Method and device for representing volumetric content by layer.

Abstract

The invention relates to a method and a device for rendering, in an image plane, a volumetric image decomposed into a plurality of 3D Gaussians. The method comprises the following steps for at least one sample (P) to be reconstructed in the image plane (IP): - identification (E21, E22) of a plurality of 3D Gaussians along a radius (R, R1, R2) launched from said sample, a said 3D Gaussian being delimited by two bounds (Bk+1) (Bk); - identification (E23) of a plurality of layers (Bk-Bk+1, Lk), a said layer being delimited by two of said adjacent bounds and comprising at least one identified 3D Gaussian, said contributing 3D Gaussian.- projection (E24) of said contributing 3D Gaussians into respective contributing 2D Gaussians in the image plane; - accumulation (E25) between said contributing 2D Gaussians of said layer comprising at least two contributing 2D Gaussians, - blending (E27) between at least two of said layers of the plurality Figure for the abbreviation: Fig. 5.

Description

Title of the invention: Method and device for representing volumetric content by layer. Previous technique

[0001] The invention relates to the general field of volumetric images. It relates more particularly to the representation and rendering of such content.

[0002] A volumetric image corresponds to three-dimensional content or a three-dimensional scene. Such an image can be part of a sequence of images or a video. Several approaches exist for representing volumetric content, notably those based on an explicit or implicit representation of the 3D scene in question.

[0003] An explicit approach represents a 3D scene using geometric primitives (volumes, surface elements, voxels, etc.) and then uses image synthesis techniques to render an image of this scene associated with a viewpoint. An implicit approach uses an implicit representation of the scene. Examples include approaches that define a 3D map of distances to objects in the scene (e.g., Signed Distance Function), or a 3D map of the density and colorimetry properties of the different 3D positions in the scene.

[0004] Recently, significant improvements have been observed in 3D scene construction and representation techniques, particularly through implicit representation based on neural networks and ray tracing. However, these approaches remain too complex in practice.

[0005] A technique for representing a 3D scene as a set of 3D Gaussian primitives, known as 3DGS, was proposed in the article "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (Kerbl et al., ACM Transactions on Graphics, July 2023). Compared to the approaches mentioned previously, this method is significantly less complex, while offering better reconstruction quality for volumetric content. It enables real-time rendering on current terminals. This is in particular linked to the rendering technique associated with a set of 3D Gaussian primitives as it has been defined in previous work on the flattening of Gaussians (in English 'splatting'), notably in the work of Zwicker (Surface Splatting Zwicker et al., in Proceedings of the 28th annual conference on Computer graphies and interactive techniques: August 12-17, 2001).

[0006] Although very effective, the rendering technique introduced by 3DGS is not without its flaws. Among other things, one can note an undesirable induced transparency, linked to the factor density of a Gaussian, a distortion of the rendering depending on the viewpoints, and a low intrinsic reconstruction quality.

[0007] There is therefore a need for a solution that effectively corrects current volumetric rendering techniques. Description of the invention

[0008] The invention relates to a method for rendering, in an image plane, a volumetric image decomposed into a plurality of 3D Gaussians, comprising the following steps for at least one sample to be reconstructed in the image plane: - identification of a plurality of 3D Gaussians along a ray launched from said sample, a said 3D Gaussian being delimited by two bounds; - identification of a plurality of layers, a said layer being delimited by two of the said adjacent boundaries and comprising at least one identified 3D Gaussian, called 3D Gaussian. - projection of said contributing 3D Gaussians into respective contributing 2D Gaussians in the image plane; - accumulation between said contributing 2D Gaussians of said layer comprising at least two contributing 2D Gaussians, - blending between at least two of said layers of plurality.

[0009] A "volumetric image" is understood to be a 3D scene that can be observed from different angles and viewpoints. The volumetric representation is the set of data associated with such an image: for example, a set of 3D samples (voxels), or a set of 3D Gaussians and associated density information.

[0010] The term "rendering" refers to the operation of representing all the volume data on a flat surface, called the image plane, corresponding to a viewpoint of the scene.

[0011] By "sample to be reconstructed" is meant a value of the image plane that one seeks to reconstruct. The sample can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). In general, a sample is represented by a vector of one or more components, or data.

[0012] The term "3D Gaussian" means a reconstruction kernel in space. Volume data is thus interpreted as a field of 3D reconstruction kernels, each kernel typically being associated with a point, or voxel, in space. Each 3D reconstruction kernel can be likened to a "fingerprint" function distributing the contribution of each voxel across nearby voxels. A 3D Gaussian is usually defined by its position in space, its density, its covariance, and its color coefficients (or harmonics). Such data can be obtained, or calculated, from latent codes, particularly during the training, construction, or learning of 3D Gaussians. Any classical state-of-the-art method can be used. A 3D Gaussian is bounded by two limits, or real values, on the radius axis.

[0013] By "projection of 3D Gaussians into 2D Gaussians" is meant the projection, or splatting, operation that produces 2D, or flattened, Gaussians in the image plane. A 2D Gaussian is represented by parameters: its position, its covariance matrix, its color coefficients (for example, (R, G, B)), and its density coefficient. Opacity is defined as a function of density. Alternatively, the notion of opacity can be replaced by that of transparency, as opacity and transparency are complementary. An object is completely opaque if its opacity value is at its maximum, that is, 1. Conversely, it is invisible if this value is 0.Each pixel in the image plane can be associated, during rendering, with color coefficients (which are those of the 2D Gaussian), and a density coefficient defined as the product of the density coefficient of the 2D Gaussian and the value of the 2D Gaussian at that point. Thus, the density coefficient of a pixel varies and tends to zero as one moves away from the center of the 2D Gaussian.

[0014] By "ray," we mean, in accordance with the state of the art, a virtual half-line whose origin is located on the camera (or the observer's eye) and which intersects the image plane at the position of the sample (pixel) that we seek to reconstruct. Volumetric rendering thus simulates a physical optical process by projecting or casting rays along which the volumetric density of the scene can be calculated. This is also referred to as ray integration.

[0015] By "layer" is meant an interval delimited by two bounds on the radius. A layer comprises at least one portion of a contributing Gaussian. A layer may contain one or more portions of contributing Gaussians. A portion of a Gaussian may correspond to the complete Gaussian if its bounds correspond to the limits of the layer, otherwise to the portion congruent to the interval. In other words, a Gaussian is considered to contribute to one or more layers between its input and output bounds.

[0016] By "accumulation" is meant a summation of the data associated with a sample of the image plane, originating from the different portions of Gaussians contributing to a layer. The 2D Gaussians resulting from the projection of the 3D Gaussians can overlap after projection (splatting) in the image plane. For example, averaging the parameters of several Gaussians allows the calculation of a resulting average color value, after possible weighting. The summation The densities allow us to obtain a resulting density value. From this resulting density, a resulting opacity can be calculated for the layer.

[0017] By "blending" we mean the combination of the contributions of the different layers associated with a sample of the image plane, according to their opacity factor. Blending performed on the parameters of several layers makes it possible to take into account the effects of transparency.

[0018] The 3D Gaussians composing the scene are thus associated with constituent layers of the scene. If portions of Gaussians overlap, the layer thus formed by their combination may correspond, in particular, to a local surface element of the scene. The process performs an accumulation step to reconstruct this layer. Once the accumulation is complete for this layer, it can be blended with the other layers. The process thus reconstructs, layer by layer, the contribution of the layers that may correspond to local surfaces, and then performs a blending operation to take into account the composition of the contributions of these different layers according to their transparency factor.

[0019] Advantageously, instead of using a recursive reconstruction formula based solely on the blending principle, as proposed in the prior art, the process interweaves accumulation and blending approaches, which notably corrects the viewpoint-dependent rendering distortion encountered in prior art methods such as 3DGS. As will be explained later with reference to Figures 2 and 3, 3DGS considers an order for the 3D Gaussians to be composed. However, this order can depend on the viewpoint, particularly if there is an overlap of the 3D Gaussians. This leads to rendering distortion when switching from one viewpoint to another. Accumulation processing within a layer greatly mitigates this phenomenon.

[0020] Moreover, in the case of a 3D overlap, 3DGS combines the contributions of each Gaussian using the blending formula, whereas in this scenario, accumulation is more appropriate. The accumulation method according to the invention thus provides the expected reconstruction in this context.

[0021] Generally speaking, the steps of a rendering process should not be interpreted as being linked to a notion of temporal succession. In other words, the steps can be carried out in a different order than that indicated in the independent claim, or even in parallel.

[0022] According to embodiments of the rendering process:

[0023] - the parameters of said 2D Gaussians include a color and a density, And : - the accumulation step involves calculating an average of the color of said at least two contributing 2D Gaussians to obtain a resulting color, and a combination of the density of said at least two contributing 2D Gaussians to obtain a resulting density and a resulting opacity. - the blending step uses said average color and said resulting opacity from the accumulation step.

[0024] Advantageously, such a method makes it possible to obtain, for a sample (pixel), a resulting color and opacity by accumulating several portions of Gaussians within a layer. The resulting color is an average color of the contributing 2D Gaussians. The resulting opacity of a layer is a function of the resulting density calculated from the densities of the contributing 2D Gaussians. Subsequently, these resulting values ​​are used in the blending step as if they had been obtained from a single Gaussian.

[0025] - the density of a said contributing 2D Gaussian is calculated according to a law distribution of the respective 3D Gaussian along the radius; advantageously according to this mode, the rendering is closer to the underlying model which would be the integration of the 3D Gaussians along the radius.

[0026] - the density of a said contributing 2D Gaussian is calculated as a function of The interval defined by the boundaries of said layer; advantageously, according to this method, the densities of a Gaussian are distributed among the different layers to which they contribute, taking into account the thickness of these layers. This allows us to consider only the portion of a contributing Gaussian within the layer. For example, the initial density of the 2D Gaussian can be corrected by a ratio calculated between the thickness of the portion of the 3D Gaussian within the layer and the total thickness of the 3D Gaussian.

[0027] - the resulting color is further weighted by said resulting density; Advantageously, this allows us to calculate the average color value on this layer, thus offering a continuous variation of the color spatially.

[0028] - the resulting opacity depends on a product of the complements of the densities of said at minus two contributing 2D Gaussians; advantageously, in this mode, the resulting opacity corresponds to the equivalent opacity following the passage of the ray through the considered Gaussians. Thus, objects located behind this layer are seen with the same transparency factor in this mode as in the alpha-blending composition mode.

[0029] - the resulting opacity depends on said resulting density; advantageously according to In this mode, the resulting opacity thus defined can correspond to the resulting density associated with the layer, calculated from the densities of the different Gaussians constituting it. This is notably the case when the opacity is assimilated by simplification. to the density. It can also correspond to any function of this resulting density.

[0030] - the resulting opacity is a function of the exponential of the resulting density; advantageously according to this variant, opacity is expressed, when not simplified, as a function of the exponential of density.

[0031] - the position of the bounds of a 3D Gaussian is defined by a parallelepiped encompassing said 3D Gaussians. Advantageously, according to this variant, the bounds are defined very simply by the intersection of the radius with the encompassing parallelepiped. A 3D Gaussian is not strictly speaking a 3D shape, but such a 3D shape can be associated with a level curve of the 3D Gaussian. By "parallelepiped encompassing said 3D Gaussians" we therefore mean, for the sake of simplicity, "parallelepiped encompassing a level curve of said 3D Gaussian".

[0032] Correspondingly, the invention also relates to a rendering device.

[0033] The characteristics and advantages of the rendering process apply in the same way to the rendering device according to the invention and vice versa.

[0034] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or rendering device according to the invention. This program includes instructions adapted to the implementation of the corresponding method. This program may use any programming language and be in the form of source code, object code, or code intermediate between source and object code, such as in a partially compiled form, or in any other desirable form.

[0035] The invention also relates to a computer-readable information or recording medium containing the aforementioned computer program instructions. The information or recording medium may be any entity or device capable of storing programs. For example, the medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard drive, a DNA sequence, or flash memory. Furthermore, the information or recording medium may be a transmissible medium such as an electrical or optical signal, which may be transmitted via an electrical or optical cable, by radio link, by wireless optical link, or by other means.

[0036] The program according to the invention can in particular be downloaded onto an Internet-type network.

[0037] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a process according to the invention. Brief description of the drawings

[0038] Other features and advantages of the present invention will become apparent from the description given below, with reference to the attached drawings which illustrate examples of embodiment without any limiting character.

[0039] [Fig-1] Fig. 1 schematically represents a volumetric rendering device according to one embodiment of the invention.

[0040] [Fig.2] Fig.2 schematically represents the volumetric rendering of a scene according to the state of the art.

[0041] [Fig.3] Fig.3 schematically represents the volumetric rendering of a scene according to two different points of view depending on the state of the art.

[0042] [Fig.4] Fig.4 represents a graphic result of the volumetric rendering of a scene from two different points of view according to the state of the art in accordance with [Fig.3].

[0043] [Fig.5] [Fig.5] Schematically represents the definition of the bounds of the Gaussian, according to one embodiment of the invention.

[0044] [Fig.6] [Fig.6] is a flowchart representing one embodiment of the process volumetric rendering according to the invention.

[0045] [Fig.7] [Fig.7] schematically represents the definition of intervals of 3D Gaussian localization along a radius according to two implementation methods. Description of the implementation methods

[0046] Figure 1 schematically represents the hardware architecture of a volumetric rendering device (RENDER) according to one embodiment of the invention. The rendering device is connected via an optional human-machine interface (HMI) to a module (BDS) comprising the representation of the 3D scene (3DS) to be rendered. This module can take the form of a server, a database, etc. It includes the 3D scene representations.

[0047] The rendering device typically comprises memory (MEM) associated with a processor (PROC) of a processing unit, controlled by a computer program (PGR) stored in read-only memory (for example, ROM or a hard drive). At initialization, the computer program's code instructions are, for example, loaded into RAM before being executed by the processor. The memory can be of the ROM (Read Only Memory) or RAM (Random Access Memory) type, or even Flash.

[0048] In accordance with [Fig. 1], the device also comprises:

[0049] - an optional GEN module for creating 3D Gaussians representing the image Volumetric. The 3D scene can indeed be provided in another format, for example, as a set of voxels or as a set of latent codes. In this case, the GEN module creates the 3D Gaussians from this data. Any known state-of-the-art method can be used, for example, the one described in the aforementioned 3DGS document.

[0050] - an ANAL module responsible for performing ray tracing and analysis (integration) of the scene along the radius. In particular, this module is responsible for projecting the 3D Gaussians into 2D Gaussians on the image plane, defining a list of 3D Gaussians along a radius, and the bounds of each Gaussian, as well as the different layers, depending on the chosen mode of defining the bounds.

[0051] - an ACC module responsible for accumulation, within a layer defined by two consecutive markers;

[0052] - a BLEND module responsible for blending between two layers;

[0053] - an optional DISP module responsible for displaying the resulting image I, by example on a computer screen.

[0054] All modules communicate conventionally with each other via a data bus (B).

[0055] Figure 2 schematically represents the volumetric rendering of a scene according to the state of the art.

[0056] The user's point of view (whose eye is located at the position of the CAM camera) is represented by the two dotted half-lines which delimit the 3D scene according to this point of view as well as the projection onto the IP image plane.

[0057] The 3D (3DS) scene shown consists of 3D Gaussians. To perform the rendering, they are projected onto the camera's image plane. These projections are referred to as 'splat' (or 2D splat) hereafter. We are particularly interested in the three Gaussians GA, GB, and GC in the figure, which lie on the ray R projected (launched) from the CAM camera. They are projected as three splats GA', GB', and GC. The ellipsoids corresponding to the projected Gaussians are artificially positioned above or below the image plane to distinguish them. However, in the image plane, these projected ellipsoids are at the same level.

[0058] Obtaining 3D Gaussians can be achieved, as proposed in 3DGS, by classical optimization techniques which allow, from an initial point cloud, control of the number and parameters of the Gaussians for optimal rendering of the scene.

[0059] The rendering principle as proposed by 3DGS is to work in the image plane, using a 3D Gaussian scanning rendering technique. A painter's algorithm can be used to scan the Gaussians. The principle of this algorithm is to draw in order of depth, from the furthest The closest layer is the different Gaussian colors. Thus, the final color of the pixel is rendered step by step using an a-blending algorithm, in which each layer contributes a coefficient of 'a', representing the opacity of the Gaussian. Gaussian colors can also be rendered from front to back. For example, the rendering of pixel P shown in the figure can be done by considering the order GA, GB, GC, or GA, GC, GB. Each time a layer is added, it contributes to the pixel's rendering with a factor of 'a' and attenuates the background by a factor of (1-a). The final color in P, when the Gaussians are treated in the order GA, GB and GC, can be expressed by C = ac * cc + ab * (1- ac ) * cb + aa*(l- ac ) (1- ab ) * ca Note that in this order, for an identical opacity factor, Gaussian A is more attenuated than B which is itself more attenuated than C, thus giving the strongest weighting to the foreground.

[0060] More generally, the reconstruction of the image plane into a pixel x can be expressed by the following formula: |,M'" C = ^TiV,

[0062] Where Ci is the color value associated with the i-th Gaussian (this value can be a scalar, a multicomponent representation as for color images, for example RGB or YUV, or a multidirectional representation of color, for example via spherical harmonic coefficients, as in 3DGS); according to one embodiment, the opacity of the i-th Gaussian for the sample x, is defined as a, = G( ( x ) , and rtM represents the attenuation (transmittance) to be applied to the i-th Gaussian due to the transparencies of the objects present "above" this Gaussian (in the chosen order).

[0063] In the aforementioned 3DGS algorithm, the transparency at a pixel x depends on two parameters: the density factor oi of the Gaussian, and the attenuation G;(x) relative to the position within the Gaussian. This attenuation can be defined by the following formula, in which p;2D is the center of the Gaussian projected onto the image plane.

[0064] \ G; x | - y.exd - ———5—-—— U r L

[0065] According to the aforementioned prior art, the normalization factor used in front of the exponential can take the value 1, or alternatively a value calculated based on the 2D covariance matrix S2D. The S2D covariance matrix is ​​obtained by projecting the 3D covariance matrix associated with the 3D Gaussian, as detailed in the prior art, where it is proposed to use a low-pass filter, modeled by a simple Gaussian, in order to avoid aliasing effects due to unsatisfactory sampling of the samples in the image plane. The effect of this low-pass filter This simply translates to adding a term to the diagonal of the matrix. In the attenuation factor expression, the covariance matrix is ​​therefore modified to S2D to account for this dilation effect of 2D Gaussians. This ability to perform anti-aliasing filtering without multiple sub-pixel renderings is unique to the use of Gaussians (according to the principle stated in Zwickler's original paper). Other approaches, particularly ray-tracing, require multiple ray tracings per pixel to achieve anti-aliasing.

[0066] As shown in figures 3 and 4, this approach suffers from a difference in rendering of the scene depending on the viewpoints.

[0067] Fig. 3 schematically represents the volumetric rendering of a scene from two different viewpoints according to the state of the art (3DGS).

[0068] Figure 3 represents a flat surface (hatched surface) with a texture gradient, which we wish to render from two viewpoints, denoted LV (Left View) and RV (Right View). The algorithm presented in support of Figure 2 orders the projected 3D Gaussians in depth. Thus, in the left view, the leftmost Gaussian will be the furthest, while the opposite will be true in the right view. In other words, in the left view, the 2D Gaussians are rendered 'from left to right', while in the right view, they are rendered 'from right to left'. This variation leads to differences in rendering that can fluctuate considerably, since, as mentioned previously, in 3DGS, the rendered value depends on the depth order of the Gaussians (the nearest one taking precedence over the furthest).This fluctuation can be particularly significant when the Gaussian zones are very close in depth, since in this case the order can change very rapidly from one viewpoint to another.

[0069] Fig. 4 represents a graphic result of the volumetric rendering of a scene from two different viewpoints according to the state of the art in accordance with Fig. 3.

[0070] In [Fig.4], 3 curves are shown: - the solid line curve (ground truth) illustrates the rendering obtained by composing the Gaussians according to an "ideal" rendering; - the thin dotted curve (left to right) shows the perceived result for a left-hand view; - the wider dotted curve (right to left) shows the perceived result for a right-hand view.

[0071] It is clear that the two curves are offset, which means that the rendering will be different depending on the left or right viewpoint according to the state of the art.

[0072] Figure 5 schematically represents the volumetric rendering of a scene according to one embodiment of the invention.

[0073] The context of the 3D scene is similar to that of [Fig.2]: the user's viewpoint and the 3D scene are represented similarly, the 3D Gaussians can be obtained by classical optimization techniques, and the three Gaussians GA, GB and GC which are located on the ray R launched from the CAM camera are projected into 3 splats GA', GB' and GC'.

[0074] The rendering principle, however, differs from that of [Fig. 2]: each Gaussian is associated with an entry boundary and an exit boundary for the considered radius. These boundaries are defined by their depth, the entry boundary being shallower than the exit boundary. A Gaussian is considered to contribute to a layer between its entry and exit boundaries. The set of boundaries of all Gaussians defines a layered partition, each layer being the area delimited by two consecutive boundaries. Within each layer, the contribution of each Gaussian is defined according to its own characteristics and the characteristics of the layer, such as its thickness, i.e., the size of the interval between its two boundaries. Within each layer, the contributions of the Gaussians are accumulated, and a blending is performed between the layers.

[0075] According to one embodiment, within a layer, as will be detailed below, the resulting color is calculated as the average of the colors of the contributing Gaussians, weighted by their density contribution.

[0076] According to one embodiment, within a layer, as will be detailed below, the density contribution of the Gaussians is accumulated to obtain a resulting density.

[0077] According to one embodiment, the opacity of the layer is calculated as a function of this resulting density. For example, this function is expressed as the 1's complement of the exponential of the inverse of the resulting density. For example, the resulting density is simply bounded in an interval [0,1].

[0078] In [Fig. 5], only the endpoints of the Gaussians intersecting with the radius are shown. A layer is represented by its two endpoints. The Gaussian GB alone contributes to layers B1-B2 and B3-B4, while layer B2-B3 has two contributing Gaussians, GB and GC. Layer B4-B5 has no contributing Gaussian, and layer B5-B6 has GA as its only contributor. Four layers must therefore be considered, labeled L1 to L4 in the figure.

[0079] Once the layers have been identified, the blending steps are carried out in a conventional manner between these layers.

[0080] Thus, the final color of the pixel is rendered step by step by an a-blending type algorithm performed between the layers (and not between the Gaussian colors), and an accumulation within the layers. Each layer contributes according to its opacity coefficient a. The layers can be rendered in any order (from (front to back, or back to front, with accumulation and blending being sequential or non-sequential). For example, the rendering of pixel P shown in [Fig. 5] can be achieved by considering the order B1 to B6 or B6 to B1. Each time a layer is added, it contributes to the pixel rendering by a factor of 'a' and attenuates the background by a factor of (1 - 'a'). The final color in P, when the Gaussians are processed in the order B1 to B6, can be expressed as C = a4 * c4 + a3 * (1 - a4) * c3 + 'a', where ck represents the resulting color in layer k and ak represents the resulting opacity of layer k.

[0081] According to one embodiment, the density contribution of the Gaussian i is distributed according to a distribution law which achieves an approximation of the Gaussian along the radius.

[0082] According to another embodiment, the density contribution of the Gaussian i is distributed proportionally to the thickness of the layers to which it contributes. Its contribution to a layer k delimited by the bounds B k -B k+] is: MM - a^k] = / J where ni, / , are the indices of the entry and exit bounds of the Gaussian i, within the list of all bounds, ordered by depth, and dt is the density of the 2D Gaussian.

[0083] For example, in [Fig.5], the Gaussian GB contributes to the interval B2-B3 by a factor of (B2-B3) / (B1-B4). We then denote the sum of the density contributions, or resulting density over the set Gk of contributing Gaussians of the k-th layer: Ak - Ei€Sfc G#]

[0084] According to this embodiment, the average color on the layer B k -B k+] is = Hiegk ai^]ci

[0085] The opacity coefficient ak associated with layer k, and which will be used for blending, can be defined in different ways from the density contributions.

[0086] According to one embodiment, it is equal to the resulting density, bounded to the interval [0,1].

[0087] According to another mode, it is a function of the product of the complements of these densities.

[0088] According to a third method, it is defined as a function of the resulting density, according to: ak = 1 - exp(-Afc)

[0089] Fig. 6 is a flowchart representing an example of a volumetric rendering process that can be implemented by the rendering device of Fig. 1 in accordance with the example of Fig. 5.

[0090] During the initialization step E20, the 3D scene is constructed as a set of 3D Gaussians. Any algorithm within the grasp of a person skilled in the art can be used to generate the Gaussians, for example, the algorithm proposed in the aforementioned 3DGS method, which uses gradient descent to minimize a quality criterion for reconstructing the rendered views. The algorithm then proceeds for each pixel of the image plane according to steps E21 to E26.

[0091] During step E21, a list of Gaussians intersecting the ray associated with the pixel of the image plane is established. According to the example in [Fig. 5], 3 Gaussians GA, GB and GC are identified on the ray, and listed for example in this order (from the furthest to the nearest).

[0092] In step E22, input / output bounds are defined for each Gaussian, and then these bounds are sorted (e.g., B1, B2, B3, B4, B5, B6). The definition of the bounds associated with a Gaussian can be done in various ways, as described in support of [Fig. 7]. The resulting values ​​of the bounds are, for example, real numbers indicating a depth position on the radius axis.

[0093] During a step E23, a plurality of layers is identified from these boundaries. A layer Lk is identified by an interval Bk-Bk+1 between two successive, or adjacent, boundaries, that is to say, the two closest boundaries in the set of boundaries.

[0094] The 3D Gaussians are projected onto the image plane as many 2D Gaussians as possible during step E24. Such a projection step has already been described previously. It is also known from the prior art, notably from the aforementioned article by Zwicker. After the projection, a 2D Gaussian, or splat, is described by its position parameters, its covariance matrix, its color coefficients (for example, (R,G,B)), and its density coefficient. It should be noted that this step can be performed at any point in the algorithm, as long as the contribution of the 3D Gaussian via its projection is available before performing the calculation steps relating to a layer to which it contributes (steps E25 and E26).

[0095] Next, the pixel value in each Lk layer is evaluated during steps E25 and E26, layer by layer.

[0096] In step E25, the density contribution of each Gaussian to this layer is evaluated, and the density contributions are accumulated to calculate the resulting density Ak. As shown in support of [Fig. 5], the resulting color Ck in the layer is calculated as an average value of the colors of the contributing Gaussians weighted by their resulting density contribution. The resulting opacity coefficient ak of the layer is then evaluated as described above, either from the density contributions of each Gaussian or from the resulting density Ak.

[0097] Then during a step E26 the layer thus obtained is composed via an alpha-blending operation using the resulting opacity of the layer and that of the previous layer.

[0098] At the end of the iteration on the layers (when the E27 test step of the last layer is positive), a finalization of the rendering operations is carried out in step E28. Typically, a background color, noted C00 on [Fig.5], can be accumulated by alpha blending behind the last boundary.

[0099] [Fig.7] schematically represents the definition of the 3D Gaussian bounds along a radius according to two embodiments.

[0100] This localization step details in particular step E22 of the logic diagram in [Fig.6]. Each of the Gaussians which has previously been placed in the sorted list (El, E2, E3) for a given radius (RI, R2) is associated with a localization interval whose bounds are the bounds associated with the Gaussian.

[0101] Figure 7 presents two embodiments:

[0102] According to a first mode, shown on the right, the localization intervals between the terminals are defined by wide, unoriented boxes, by the dotted rectangles.

[0103] Such a box, called a global bounding box, can allow us to define a location interval between two bounds, for example from the position parameters p and the covariance matrix of the 3D Gaussian, S3D, as the interval on the z-axis k the confidence interval, typically a value of 2 to 3).

[0104] The position parameters p and the covariance matrix of the 3D Gaussian can be expressed by: of the radius having center pz and width X JË being a defining factor

[0105] ^x\ U /

[0106] ^XZ ' ^yz

[0107] According to a second method, shown on the left, the localization intervals are defined by oriented bounding boxes, referred to as "tight" bounding boxes, represented by the dashed quadrilaterals. For the sample associated with radius RI, there is no overlap of the intervals; the Gaussians each contribute to only one layer. For the sample associated with radius R2, the Gaussians E1 and E2 overlap, and therefore both contribute to a common layer.

[0108] Such a tight bounding box has the advantage of adjusting to the orientation of the 3D Gaussian and thus having a narrower interval. It can be calculated for example, from the position parameters p and the inverse matrix of the covariance matrix of the 3D Gaussian, A3D, as the interval on the z-axis of the radius centered at , v As + .,As. where (x,y) are the coordinates of the pixel in the plane image, and for width j TT (k being defined as before).

Claims

Demands

1. Rendering device (RENDER), in an image plane (IP), of a volumetric image (3DS) decomposed into a plurality of 3D Gaussians, characterized in that the rendering device is configured to implement, for at least one sample (P) to be reconstructed in the image plane (IP): - identification (ANAL) of a plurality of 3D Gaussians along a radius (R, RI, R2) launched from said sample, a said Gaussian being delimited by two bounds (Bk+i) (Bk); - identification (ANAL) of a plurality of layers (Bk-Bk+1, Lk), a said layer being delimited by two of said adjacent bounds and comprising at least one identified 3D Gaussian, said 3D Gaussian contributing to the layer.- projection (ANAL) of said 3D contributing Gaussians into respective 2D contributing Gaussians in the image plane; - accumulation (ACC) between said 2D contributing Gaussians of a said layer comprising at least two 2D contributing ANAL Gaussians, - blending (BLEND.) between at least two of said layers of the plurality.

Images