Method and device for representing volumetric content

Abstract

The invention relates to a method and device for rendering, in an image plane, a volumetric image decomposed into a plurality of 3D Gaussians. The method comprises the following steps for at least one sample (P) to be reconstructed in the image plane (IP): - identifying a plurality of 3D Gaussians along a ray (R, R1, R2) traced from said sample; - projecting said 3D Gaussians as a plurality of 2D Gaussians into the image plane; - depending on an overlap criterion between at least two 3D Gaussians of the plurality on said ray, on the basis of parameters of said 2D Gaussians: - an accumulating step if said at least two 3D Gaussians overlap, - else a blending step.

Description

Method and device for representing volumetric content.

[0001] The invention relates to the general field of volumetric images. It relates more particularly to the representation and rendering of such content.

[0002] A volumetric image corresponds to three-dimensional content or a three-dimensional scene. Such an image can be part of a sequence of images, or a video. Several approaches exist for representing volumetric content, notably those based on an explicit or implicit representation of the 3D scene in question.

[0003] An explicit approach represents a 3D scene using geometric primitives (volumes, surface elements, voxels, etc.) and then uses image synthesis techniques to render an image of this scene associated with a viewpoint. An implicit approach uses an implicit representation of the scene. Examples include approaches that define a 3D map of distances to objects in the scene (e.g., Signed Distance Function), or a 3D map of the opacity and colorimetry properties of different 3D positions in the scene.

[0004] Recently, we have seen significant improvements in 3D scene construction and representation techniques, particularly through implicit representation based on neural networks and ray tracing. However, these approaches remain too complex in practice.

[0005] A technique for representing a 3D scene as a set of 3D Gaussian primitives, known as 3DGS, was proposed in the article "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (Kerbl et al., ACM Transactions on Graphics, July 2023). Compared to the approaches mentioned previously, this method is significantly less complex, while offering better reconstruction quality for volumetric content. It enables real-time rendering on current devices. This is particularly related to the rendering technique associated with a set of 3D Gaussian primitives as defined in previous work on Gaussian splatting, notably in the work of Zwicker (Surface Splatting, Zwicker et al., in Proceedings of the 28th annual conference on Computer graphics and interactive techniques: August 12-17, 2001).

[0006] Although highly effective, the rendering technique introduced by 3DGS is not without its flaws. These include undesirable induced transparency, linked to the opacity factor of a Gaussian distribution, rendering distortion depending on the viewpoint, and inherently low reconstruction quality.

[0007] Therefore, there is a need for a solution that effectively corrects current volumetric rendering techniques.

[0008] The invention relates to a method for rendering a volumetric image in an image plane, the volumetric image being decomposed into a plurality of 3D Gaussians, comprising the following steps for at least one sample to be reconstructed in the image plane: - identification of a plurality of 3D Gaussians along a ray launched from said sample, and, - projection of said 3D Gaussians into a plurality of respective 2D Gaussians in the image plane; - accumulation of at least two parameters of at least two 2D Gaussians only if the at least two respective 3D Gaussians overlap.

[0009] According to one embodiment, the rendering process includes a blending step of at least two parameters from at least two 2D Gaussians if the respective at least two 3D Gaussians do not overlap:

[0010] A "volumetric image" is defined as a 3D scene that can be observed from different angles and viewpoints. The volumetric representation is the set of data associated with such an image: for example, a set of 3D samples (voxels), or a set of 3D Gaussians and associated opacity information.

[0011] The term "rendering" refers to the operation of representing all volume data on a flat surface, called the image plane, corresponding to a viewpoint of the scene.

[0012] The term "sample to be reconstructed" refers to a value in the image plane that we seek to reconstruct. The sample could be, for example, a color pixel traditionally represented by a triplet of values, such as (R, G, B) or (Y, U, V). Generally, a sample is represented by a vector of one or more components, or data points.

[0013] A "3D Gaussian" is defined as a reconstruction kernel in space. Volume data is thus interpreted as a field of 3D reconstruction kernels, with each kernel typically associated with a point, or voxel, in space. Each 3D reconstruction kernel can be considered a "fingerprint" function, distributing the contribution of each voxel across nearby voxels. A 3D Gaussian is usually defined by its position in space, its opacity, its covariance, and its color coefficients (or harmonics). Such data can be obtained, or calculated, from latent codes, particularly during the training, construction, or learning of 3D Gaussians. Any standard, state-of-the-art method can be used.

[0014] The term "projection of 3D Gaussians into 2D Gaussians" refers to the projection, or splatting, operation that produces 2D, or flattened, Gaussians in the image plane. A 2D Gaussian is represented by parameters: its position, its covariance matrix, its color coefficients (e.g., (R, G, B)), and its opacity coefficient. Alternatively, the opacity coefficient can be replaced by the transparency coefficient, as opacity and transparency are complementary. An object is totally opaque if the opacity value is at its maximum, i.e., at the value 1. Conversely, it is invisible if this value is at 0. To each pixel of the image plane, we can associate, during rendering, color coefficients (which are those of the 2D Gaussian), and an opacity coefficient defined as the product of the opacity coefficient of the 2D Gaussian and the value of the 2D Gaussian at that point.Thus, the opacity coefficient of a pixel varies and tends to become zero as one moves away from the center of the 2D Gaussian.

[0015] By "ray," we mean, according to current best practices, a virtual half-line originating at the camera (or the observer's eye) and passing through the image plane at the position of the sample (pixel) that we wish to reconstruct. Volumetric rendering thus simulates a physical optical process by projecting or casting rays along which the volumetric density of the scene can be calculated. This is also referred to as ray integration.

[0016] By "accumulation" we mean a weighted summation of data associated with a sample of the image plane, derived from different Gaussians. The 2D Gaussians resulting from the projection of the 3D Gaussians can overlap after projection (splatting) in the image plane. The weighted summation performed on the parameters of several Gaussians allows us to calculate a resulting average value of color and opacity of the superimposed Gaussians in 3D space, according to an overlap criterion.

[0017] "Blending" refers to the combination of the contributions of different Gaussians associated with a sample of the image plane, according to their opacity factor. Blending performed on the parameters of several Gaussians allows for the consideration of transparency effects.

[0018] The "overlap criterion" refers to a criterion that must be met for at least two Gaussian lines to be considered overlapping. This criterion can take various forms, relating, for example, to the number of Gaussian lines considered, their shape, the calculated localization interval, a percentage of overlap, etc. The overlap criterion virtually defines a layer, or set of superimposed 3D Gaussian lines, during integration along the radius. One or more layers can thus be defined, each containing one or more Gaussian lines.

[0019] The 3D Gaussians composing the scene are thus associated, when they overlap, with constituent layers of the scene. If Gaussians overlap, the layer thus formed by them can correspond to a local surface element of the scene. The process performs an accumulation step to reconstruct this layer. Once the accumulation is complete for this layer, it can be blended with the other layers. The process thus reconstructs, layer by layer, the contribution of the layers that can correspond to local surfaces, and then performs a blending operation to take into account the composition of the contributions of these different layers according to their transparency factor.

[0020] Advantageously, instead of using a recursive reconstruction formula based solely on the blending principle, as proposed in the state of the art, the process interweaves accumulation and blending approaches. This allows, in particular, for correcting the viewpoint-dependent rendering distortion encountered in state-of-the-art methods such as 3DGS. As will be explained later with reference to Figures 2 and 3, 3DGS considers an order for the 3D Gaussians to be composed. However, this order can depend on the viewpoint, especially if there is an overlap of the 3D Gaussians. This leads to rendering distortion when switching from one viewpoint to another. Accumulation processing within a layer greatly mitigates this phenomenon.

[0021] Furthermore, in the case of a 3D overlap, 3DGS combines the contributions of each Gaussian using the blending formula, whereas in this scenario, accumulation is more appropriate. The accumulation process according to the invention thus enables the expected reconstruction in this context.

[0022] Generally, the steps in a rendering process should not be interpreted as being linked to a notion of temporal succession. In other words, the steps can be carried out in a different order than that indicated in the independent claim, or even in parallel.

[0023] According to different implementation methods of the rendering process:

[0024] - said parameters of the 2D Gaussians include a color and an opacity, and: - the accumulation step includes the calculation of a weighted sum of the color of said at least two 2D Gaussians to obtain a resulting color, and a combination of the opacity of said at least two 2D Gaussians to obtain a resulting opacity; - the blending step uses said resulting color and said resulting opacity from the accumulation step.

[0025] Advantageously, such a process allows us to obtain, for a sample (pixel), a resulting color and opacity for several overlapping Gaussians within a layer. Subsequently, these resulting values ​​are used in the blending step as if they had been obtained from a single Gaussian.

[0026] – The resulting opacity depends on a sum of the opacities of at least two 2D Gaussians; advantageously, according to this method, the resulting opacity thus defined corresponds to the spatial density associated with the overlap zone of the Gaussians. This is notably the case when opacity is simplified to density (whereas opacity, when not simplified, is expressed as a function of the exponential of the density).

[0027] The resulting opacity depends on the product of the complements of the opacities of at least two overlapping 3D Gaussians; advantageously, in this mode, the resulting opacity corresponds to the equivalent opacity resulting from the passage of the ray through the considered Gaussians. Thus, objects located behind this layer are seen with the same transparency factor in this mode as in the alpha-blending composition mode.

[0028] The overlap criterion depends on a location interval for at least two 3D Gaussians along the radius. Advantageously, according to this method, an interval can be simply defined to determine whether two Gaussians overlap. For example, a single interval can be defined on the depth axis, and the two Gaussians overlap when they both lie within the interval. Alternatively, an interval can be defined per Gaussian, and the Gaussians overlap when the two intervals intersect. Advantageously, according to this intersection-based method, estimation errors on these position intervals along the radius can be compensated for.

[0029] - The said overlap criterion depends on at least two parallelepipeds respectively encompassing said at least two 3D Gaussians, and said Gaussians overlap if said parallelepipeds have an intersection. Advantageously, according to this variant, overlap tests between ellipses require fewer calculations to estimate the localization intervals along said radius, since one analyzes the intersection of a line with a rectangle instead of analyzing the intersection of a line with an ellipse.

[0030] - said overlap criterion depends on a maximum number of 3D Gaussians identified along said radius; advantageously according to this variant, it is possible to control the number of Gaussians within a layer for reasons of storage and computation optimization; indeed, particularly with regard to the Gaussian parameter estimation algorithm, having several Gaussians per layer can have an impact on the memory space required.

[0031] - said overlap criterion depends on the resulting opacity of the accumulation step; advantageously according to this mode, we can in particular stop the accumulation within the Gaussians as soon as we know that, due to the level of opacity already reached, the contribution of the following Gaussians will be zero, because these Gaussians will be masked by the Gaussians already encountered so far.

[0032] - the accumulation ends with a normalization step relating to the weighting factors.

[0033] - said at least two overlapping 3D Gaussians along the radius define a layer, and: - the accumulation step is carried out inside the layer; - the blending step is carried out between two layers.

[0034] Correspondingly, the invention also aims at a rendering device.

[0035] The characteristics and advantages of the rendering process apply in the same way to the rendering device according to the invention and vice versa.

[0036] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or rendering device according to the invention. This program includes instructions adapted to the implementation of the corresponding method. This program may use any programming language and be in the form of source code, object code, or code intermediate between source and object code, such as in a partially compiled form, or in any other desirable form.

[0037] The invention also relates to a computer-readable information or recording medium containing the aforementioned computer program instructions. The information or recording medium can be any entity or device capable of storing programs. For example, the medium may include a storage means, such as a ROM (e.g., a CD-ROM or a microelectronic circuit ROM), or a magnetic recording means, such as a floppy disk or hard drive, a DNA sequence, or flash memory. Furthermore, the information or recording medium may be a transmissible medium, such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, radio link, wireless optical link, or other means.

[0038] The program according to the invention can in particular be downloaded onto an Internet-type network.

[0039] Alternatively, each information or recording medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a process according to the invention.

[0040] Other features and advantages of the present invention will become apparent from the description given below, with reference to the attached drawings which illustrate examples of embodiment without any limiting character.

[0041] Lare represents schematically a volumetric rendering device according to an embodiment of the invention.

[0042] The schematic representation of the volumetric rendering of a scene according to the state of the art.

[0043] The diagram schematically represents the volumetric rendering of a scene from two different points of view according to the state of the art.

[0044] Larepresents a graphic result of the volumetric rendering of a scene from two different points of view according to the state of the art in accordance with the.

[0045] Lare represents schematically the volumetric rendering of a scene according to an embodiment of the invention.

[0046] laest un logicgamie de regimen un e ...

[0047] laest un logicgamie represent un second embodiment of the volumetric rendering process according to the invention, which can be implemented by the rendering device of the.

[0048] lare represents schematically the definition of localization intervals of 3D Gaussians along a radius according to two modes of realization.

[0049] Lare schematically represents a step in determining the overlap of 3D Gaussians according to two embodiment modes.

[0050] The diagram schematically represents the hardware architecture of a volumetric rendering device (Render) according to an embodiment of the invention. The rendering device is connected via an optional human-machine interface (I / O) to a database module (BDS) containing the representation of the 3D scene (3DS) to be rendered. This module can take the form of a server, a database, etc. It includes the 3D scene representations.

[0051] The rendering device typically comprises memory (MEM) associated with a processor (PROC) of a processing unit, controlled by a computer program (PGR) stored in read-only memory (for example, ROM or a hard drive). At initialization, the computer program's code instructions are loaded into RAM before being executed by the processor. Memory can be of the ROM (Read Only Memory), RAM (Random Access Memory), or Flash type.

[0052] In accordance with the law, the system also includes:

[0053] – an optional GEN module for creating 3D Gaussians representing the volumetric image. The 3D scene can indeed be provided in another format, for example, as a set of voxels or as a set of latent codes. In this case, the GEN module creates the 3D Gaussians from this data. Any known state-of-the-art method can be used, for example, the one described in the aforementioned 3DGS document.

[0054] – an ANAL module responsible for performing ray tracing and scene analysis (integration) along the ray. Specifically, this module projects 3D Gaussians onto the image plane as 2D Gaussians, defines a list of 3D Gaussians along a ray, and determines if two Gaussians overlap, based on a chosen overlap criterion. It can also be used to group several Gaussians into a single layer.

[0055] – an ACC module responsible for accumulation, inside a layer;

[0056] – a BLEND module responsible for blending between two layers;

[0057] – an optional DISP module responsible for displaying the resulting image I, for example on a computer screen.

[0058] All modules communicate conventionally with each other via a data bus (B).

[0059] The schematic representation of the volumetric rendering of a scene according to the state of the art.

[0060] The user's viewpoint (whose eye is located at the position of the CAM camera) is represented by the two dotted half-lines which delimit the 3D scene according to this viewpoint as well as the projection onto the IP image plane.

[0061] The 3D (3DS) scene shown is composed of 3D Gaussians. To perform the rendering, they are projected onto the camera's image plane. These projections are referred to as 'splat' (or 2D splat) hereafter. We are particularly interested in the three Gaussians GA, GB, and GC in the figure, which lie on the projected (or launched) radius R from the CAM camera. They are projected as three splats: GA', GB', and GC'. The ellipsoids corresponding to the projected Gaussians are artificially positioned above or below the image plane to distinguish them. However, within the image plane, these projected ellipsoids are at the same level.

[0062] Obtaining 3D Gaussians can be done, as proposed in 3DGS, by classic optimization techniques which allow, from an initial point cloud, to control the number and parameters of the Gaussians for optimal rendering of the scene.

[0063] The rendering principle proposed by 3DGS is to work within the image plane, utilizing a 3D Gaussian scanning rendering technique. A painter's algorithm can be used to scan the Gaussians. This algorithm draws the different Gaussians in order of depth, from the furthest to the nearest. The final color of the pixel is thus rendered step by step using an α-blending algorithm, in which each layer contributes a coefficient α, representing the opacity of the Gaussian. Gaussians can also be rendered from front to back. For example, the rendering of pixel P, shown in the figure, can be done using the order GA, GB, GC, or GA, GC, GB. Each time a layer is added, it contributes to the pixel's rendering with a factor α and attenuates the background by a factor of (1-α).The final color in P, when the Gaussians are treated in the order GA, GB and GC, can be expressed by C = α. c * c c + α b * (1- α c ) * c b + α a * (1- α c ) (1- α b ) * c a. It should be noted that in this order, for an identical opacity factor, Gaussian A is more attenuated than B, which is itself more attenuated than C, thus giving the strongest weighting to the foreground.

[0064] More generally, the reconstruction of the image plane into a single pixel x can be expressed by the following formula:

[0065]

[0066] Where Ci is the color value associated with the Gaussian i (this value can be a scalar, a multi-component representation as for color images, for example RGB or YUV, or a multidirectional representation of color, for example via spherical harmonic coefficients, as in 3DGS); represents the opacity level of the i-th Gaussian for the sample x, and represents the attenuation (transmittance) to be applied to the i-th Gaussian due to the transparencies of the objects present "above" this Gaussian (in the chosen order).

[0067] In the aforementioned 3DGS algorithm, the transparency at a pixel x depends on two parameters: the opacity oi of the Gaussian, and the attenuation G i (x) relative to the position within the Gaussian. This attenuation can be defined by the following formula, in which µ i 2Dis the center of the Gaussian curve projected onto the image plane.

[0068]

[0069] According to the aforementioned state of the art, the normalization factor used in front of the exponential can take the value 1, or alternatively a value calculated based on the 2D covariance matrix Σ 2D . The covariance matrix Σ 2D is obtained by projecting the 3D covariance matrix associated with the 3D Gaussian, as detailed in the state of the art, where it is proposed to use a low-pass filter, modeled by a simple Gaussian, to avoid aliasing effects due to unsatisfactory sampling of samples in the image plane. The effect of this low-pass filter is simply expressed by adding a term to the diagonal of the matrix. In the expression for the attenuation factor, the covariance matrix is ​​therefore modified to Σ 2D'To account for the dilation effect of 2D Gaussians, the ability to perform anti-aliasing without multiple sub-pixel renderings is unique to the use of Gaussians (according to the principle outlined in Zwickler's original paper). Other approaches, particularly ray-tracing, require multiple ray tracings per pixel to achieve anti-aliasing.

[0070] As shown in figures 3 and 4, this approach suffers from a difference in rendering of the scene depending on the viewpoints.

[0071] The schematic representation of the volumetric rendering of a scene from two different viewpoints according to the state of the art (3DGS).

[0072] Lare represents a flat surface (hatched surface) with a texture gradient, which we want to render from two viewpoints, denoted LV (Left View) and RV (Right View). The algorithm presented in support of la orders the projected 3D Gaussians in depth. Thus, in the left view, the leftmost Gaussian will be the furthest, while the opposite is true in the right view. In other words, in the left view, the 2D Gaussians are rendered 'from left to right', while in the right view, they are rendered 'from right to left'. This variation leads to differences in rendering that can fluctuate considerably, since, as mentioned previously, in 3DGS, the rendered value depends on the depth order of the Gaussians (the closest one taking precedence over the furthest).This fluctuation can be particularly important when the Gaussians are very close in depth, since in this case the order can vary very rapidly from one viewpoint to another.

[0073] Larepresents a graphic result of the volumetric rendering of a scene from two different points of view according to the state of the art in accordance with the.

[0074] On the graph, 3 curves are represented: - the solid line curve (ground truth) illustrates the result obtained by composing the Gaussians according to an "ideal" rendering; - the thin dotted line curve (left to right) shows the result perceived for a left view; - the wider dotted line curve (right to left) shows the result perceived for a right view.

[0075] It is clear that the two curves are offset, which means that the rendering will be different depending on the left or right viewpoint according to the state of the art.

[0076] The diagram schematically represents the volumetric rendering of a scene according to an embodiment of the invention.

[0077] The context of the 3D scene is similar to that of the: the user's viewpoint and the 3D scene are represented similarly, the 3D Gaussians can be obtained by classical optimization techniques, and the three Gaussians GA, GB and GC which lie on the ray R launched from the CAM camera are projected into 3 splats GA', GB' and GC'.

[0078] The rendering principle differs from that of the previous method: when two layers overlap, according to a given criterion, an accumulation, not a blending, is performed between the Gaussian curves. The overlap criterion can take various forms; examples will be detailed in Figures 8 and 9. A set of overlapping Gaussian curves defines a layer. A layer can correspond to a physical surface of the scene, but is not limited to that.

[0079] In this case, the two Gaussians B and C overlap for the sample located at position P. Their contributions are therefore added together, and not combined ("blended"), resulting, for example, in a C-type contribution at P. acc = β b * c b + β c * c c, Or βbet βcsont des facteurs de pondération respectifs associés aux deux Gaussiennes. Selon ce mode de réalisation, la couleur résultante est une somme pondérée des couleurs des Gaussiennes. Une étape de normalisation peut être nécessaire. On peut considérer le résultat de l'accumulation comme une Gaussienne intermédiaire notée GBC sur la figure. De manière plus générale, une couche d'accumulation correspond à l'accumulation de toutes les Gaussiennes de la couche, et dans certains cas, le résultat peut être considéré comme une Gaussienne intermédiaire possédant ses propres paramètres d'opacité et de couleur résultantes.

[0080] Gaussian A, on the other hand, which is not overlapping along the radius, can be considered as a layer on its own.

[0081] Once the layers have been identified, the blending steps are carried out in a classic way between these layers, following the example of the GA layer and the accumulated GBC layer.

[0082] Thus, the final color of the pixel is rendered step by step by an α-blending algorithm performed between layers (and not between Gaussian colors), and an accumulation within the layers. Each layer contributes according to its opacity coefficient α. The layers can also be rendered in any order (from front to back, or from back to front, with accumulation and blending being sequential or not). For example, the rendering of pixel P on the image can be done by considering the order GA, GB+GC, or GB+GC, GA. Each time a layer is added, it contributes to the pixel rendering via a factor α and attenuates the background by a factor of (1-α). The final color of P, when the Gaussian colors are processed in the order GB+GC, GA, can be expressed as C = α a * c a + α b+c * (1- α a ) * c b+c = α a * c a + [α b+c * (1- α a ) * (βb * c b + β c * c c )],where (β b* cb+ βc* cc) représente la couleur résultante de la couche GBC et αb+creprésente l'opacité résultante de la couche GBC.

[0083] According to one embodiment, the resulting opacity α b+cdépend de la somme des opacités des Gaussiennes, c'est-à-dire que dans cet exemple, αb+c =αb +αc. Cettesomme peut être pondérée.

[0084] According to another embodiment, the resulting opacity α b+cdépend du produit des compléments des opacités des Gaussiennes, c'est-à-dire que dans cet exemple, αb+c= 1- (1-αb) * (1-αc)

[0085] More generally, using the notations associated with the 3DGS state-of-the-art description:

[0086] The resulting color value for the layer can be defined as a weighted average of the colors based on the opacity values ​​and attenuation factors of the Gaussians, namely:

[0087]

[0088] The resulting opacity coefficient Δ associated with the layer can be calculated in various ways, including, but not limited to (naturally this value may require normalization to remain within the interval [0,1]):

[0089] - a summation of opacities:

[0090]

[0091] - a weighted average of opacities

[0092]

[0093] - a supplement to the product of the supplements of the opacities

[0094]

[0095] A correct initialization of the algorithm can be achieved simply by taking into account the default background color C∞. In other words, the background is considered as a first virtual layer of color C∞.

[0096] This approach resolves the difference in scene rendering depending on the viewpoints (since the nearest Gaussians can be grouped into layers).

[0097] This is a flowchart representing an example of a volumetric rendering process that can be implemented by the rendering device according to the example of the.

[0098] During the initialization step E20, the 3D scene is constructed as a set of 3D Gaussians. Any algorithm within the grasp of a person skilled in the art can be used to generate the Gaussians, for example, the algorithm proposed in the aforementioned 3DGS method, which uses gradient descent to minimize a quality criterion for reconstructing the rendered views. The algorithm then proceeds for each pixel of the image plane according to steps E21 to E28.

[0099] During step E21, the Gaussians are identified. A list of Gaussians intersecting the radius associated with the pixel in the image plane is established, and then this list is sorted (for example, GA, GB, GC). The locations of these Gaussians on the radius are established using any method. Examples of such methods will be detailed with reference to Figures 8 and 9. For example, the list is sorted according to the depth order of the Gaussians scanned along the radius, from the furthest to the nearest. A reconstruction color for the current pixel is also initialized during this step using the default background value for that given pixel. For example, this value could be a fixed value for all pixels, or it could be an image of the scene at infinity viewed in that direction, as used in the "sphere-map" or "cube-map" technique commonly used in image synthesis.We also initialize a first grouping layer as an empty layer to proceed with subsequent groupings.

[0100] For each Gaussian, in step E22, the 3D Gaussian is first projected onto a 2D Gaussian in the image plane. Such a projection step has been described previously. It is also known from prior art, notably from the aforementioned article by Zwicker. After the projection, the 2D Gaussian, or splat, is described by its position parameters, its covariance matrix, its color coefficients (e.g., (R, G, B)), and its opacity coefficient.

[0101] In step E23, a test is performed to determine whether the current Gaussian overlaps the previous Gaussian, according to a given overlap criterion. Such overlap criteria can take various forms and will be detailed in Figures 8 and 9. In particular, but not limited to, they may depend on:

[0102] - of a localization interval associated with the Gaussian; for example, a localization interval is defined for the previous Gaussian and for the current Gaussian, and if the intervals are intersecting then the current Gaussian is associated with the same layer as the previous Gaussian;

[0103] - of an overlap rate between location intervals; for example, Gaussian regions are considered to overlap if their intersection exceeds a certain percentage of a location interval;

[0104] – of an intersection of the boxes (for example parallelepipeds) drawn around the Gaussians;

[0105] - a premature layer construction closure test; for example, when the number of grouped Gaussians exceeds a set threshold, or when the resulting opacity of the layer being constructed exceeds a defined threshold, or when the thickness of the current layer exceeds a defined threshold (such a threshold can be absolute or dependent on the distance to the camera). In this case, the current Gaussian is not considered to be overlapping; it is associated with a subsequent layer.

[0106] If the test in step E23 is positive, the partial calculations necessary for the accumulation operation within that layer are updated in step E24. The accumulation results in a color and opacity. As explained in the supporting documentation, the color of the Gaussian layer is added to that of the previous layer by a weighted sum, and the opacity by a summation or a product. The necessary steps for initializing the accumulation buffers are assumed to be performed in steps E22 or E26 when initializing a new layer.

[0107] If the test in step E23 is negative, the accumulation operation is finalized in an optional step E25, by a possible renormalization of the weighted sum by the weight sum used.

[0108] Then, in step E26, the layer thus obtained is composed using an alpha-blending operation that combines the resulting opacity of the layer with that of the previous layer or Gaussian. In one embodiment, the resulting opacity depends on the sum of the opacities of the Gaussians. According to un autre mode de réalisation, l'opacité résultante dépend du produit des compléments des opacités des Gaussiennes.

[0109] A new empty layer is then initialized for future layer groupings.

[0110] At the end of the iteration on the Gaussians, in step E27, a finalization of the rendering operations is carried out in step E28. Typically if a layer was under construction, we then proceed as in E25 and E26 to finalize the accumulation and update the reconstruction color by alpha-blending.

[0111] Laest is a logic diagram representing a second embodiment of the volumetric rendering process according to the invention, which can be implemented by the rendering device in accordance with the example of the.

[0112] Steps E10, E11, E12 are similar to steps E20, E21, E22 of the.

[0113] In step E13, the layers consisting of the 3D Gaussians are defined. To do this, a current layer is initialized, then the list of Gaussians is traversed and an overlap test is performed for each current Gaussian. If the test is positive, the current Gaussian is associated with the current layer. If the test is negative, a new current layer is initialized. At the end of step E13, we thus have J layers Cj, with J <I, où I est le nombre total de Gaussiennes.

[0114] Next, the layers are scanned in a predefined order, and for each layer:

[0115] An accumulation step is performed in step E14. This step is similar to step E24 but is performed on all Gaussians in the layer: the colors of the Gaussians are accumulated according to a weighted sum to obtain a resulting color for the layer, and the opacities are summed or multiplied to obtain a resulting opacity for the layer. This step is not implemented if the layer contains only one Gaussian. It optionally ends with a normalization step similar to that of step E25.

[0116] A blending step is performed in step E15, similar to step E26.

[0117] At the end of the iteration on the layers, in step E16, a finalization of the rendering operations is carried out in step E17, similar to step E28.

[0118] lare represents schematically the definition of localization intervals of 3D Gaussians along a radius according to two modes of realization.

[0119] This localization step details in particular steps E23 and E13 of the logic diagrams in figures 6 and 7. Each of the Gaussians which has previously been placed in the sorted list (E1, E2, E3) for a given radius (R1, R2) is associated with a localization interval which makes it possible to determine whether it overlaps another Gaussian or a layer of Gaussians.

[0120] Present two modes of embodiment:

[0121] According to the first method, shown on the right, the location intervals are defined by large, unoriented boxes, represented by the dashed rectangles. For the points associated with radii R1 and R2, the Gaussians E1 and E2 shown overlap and are grouped into a first layer. The Gaussian E3 remains separate and forms a second layer.

[0122] Such a box, called a global bounding box, can be used to define a localization interval, for example from the position parameters µ and the covariance matrix of the 3D Gaussian, Σ 3D , as the interval on the z-axis of the radius having center µz and width (k being a factor defining the confidence interval, typically a value between 2 and 3).

[0123] The position parameters µ and the covariance matrix of the 3D Gaussian can be expressed by:

[0124]

[0125]

[0126] According to a second method, shown on the left, the localization intervals are defined by oriented bounding boxes, referred to as "tight" bounding boxes, represented by the dashed quadrilaterals. For the sample associated with radius R1, there is no overlap of the intervals, so a layer is defined for each Gaussian. For the sample point associated with radius R2, the Gaussians E1 and E2 overlap and are grouped into a first layer. The Gaussian E3 remains separate and forms a second layer.

[0127] Such a tight bounding box has the advantage of fitting the orientation of the 3D Gaussian and thus having a narrower interval. It can be calculated, for example, from the position parameters µ and the inverse matrix of the covariance matrix of the 3D Gaussian, Λ 3D , as the interval on the z-axis of the radius having its center + , where (x,y) are the pixel coordinates in the image plane and for width (k being defined as before).

[0128] Lare schematically represents a step in determining the overlap of 3D Gaussians according to two embodiment modes.

[0129] The first mode, shown at the top of the figure, corresponds to step E23 of the flowchart, in which a test is performed to determine whether the current Gaussian overlaps the location zone of the previous Gaussian. In this case, a fixed-size overlap zone is used. When the first Gaussian E1 is encountered in the list, it defines a new layer. An overlap zone Z1 (corresponding to the location interval of E1) is associated with it, which is used to test whether subsequent Gaussians overlap this layer. The Gaussian E2, whose location interval overlaps with Z1, is grouped with E1 in the first layer. Subsequently, the Gaussian E3 constitutes the second e layer (with the associated Z2 overlap zone), the Gaussian E4 constitutes the 3 e layer (with associated Z3 overlap zone)

[0130] The second mode, shown at the bottom of the figure, corresponds to step E13 of the flowchart, in which a test is performed to determine whether the current Gaussian overlaps the current layer's location area. An increasing construction of the overlap zone is considered. When the first Gaussian E1 is extracted from the list, it initiates a new layer by considering the overlap zone Z1.1 (which corresponds to the interval associated with E1). Since the Gaussian E2 overlaps the overlap zone Z1.1, it is grouped into this first layer, and the overlap zone is updated to Z1.2, which corresponds to the union of the intervals of Z1.1 and E2. Similarly, since the Gaussian E3 overlaps the overlap zone Z1.2, it is grouped into this first layer, and the overlap zone is updated to Z1.3, which corresponds to the union of the intervals of Z1.2 and E3. The Gaussian E4 does not overlap Z1.3, the first layer is finalized, and we initiate a 2. e layer with the overlap zone Z2.1. We continue in the same way for the other Gaussians to come.

Claims

A rendering method, in an image plane, of a volumetric image decomposed into a plurality of 3D Gaussians, comprising the following steps for at least one sample (P) to be reconstructed in the image plane (IP): - identification (E21) of a plurality of 3D Gaussians along a ray (R, R1, R2) launched from said sample, - projection (E22) of said 3D Gaussians into a plurality of respective 2D Gaussians in the image plane; - accumulation (E24) of at least two parameters of at least two 2D Gaussians only if the at least two respective 3D Gaussians overlap. Rendering method according to claim 1, comprising a blending step (E26) of at least two parameters of at least two 2D Gaussians if the respective at least two 3D Gaussians do not overlap. A rendering method according to claim 1 or 2, wherein said parameters of said at least two 2D Gaussians comprise a color (Ci) and an opacity (Oi), and: - the accumulation step comprises calculating a weighted sum of the color of said at least two 2D Gaussians to obtain an average color, and a combination of the opacity of said at least two 2D Gaussians to obtain a resulting opacity; - the blending step uses said average color and said resulting opacity from the accumulation step. Rendering method according to claim 3, wherein the resulting opacity depends on the sum of the opacities of said at least two 2D Gaussians. Rendering method according to claim 3, wherein the resulting opacity depends on a product of the complements of the opacities of said at least two 2D Gaussians. Rendering method according to any one of the preceding claims, characterized in that the overlap of said two respective 3D Gaussians depends on a localization interval of said at least two 3D Gaussians along said radius. A method for rendering a volumetric image according to one of the preceding claims, characterized in that the overlap of said two respective 3D Gaussians depends on at least two parallelepipeds respectively encompassing said at least two 3D Gaussians, and said Gaussians overlap if said parallelepipeds have an intersection. A method for rendering a volumetric image according to one of the preceding claims, characterized in that the overlap of said two respective 3D Gaussians depends on a maximum number of 3D Gaussians identified along said radius. A method for rendering a volumetric image according to one of the preceding claims, characterized in that the overlap of said two respective 3D Gaussians depends on the resulting opacity of the accumulation step. Rendering device (RENDER), in an image plane (IP), of a volumetric image (3DS) decomposed into a plurality of 3D Gaussians, characterized in that the rendering device is configured to implement, for at least one sample (P) to be reconstructed in the image plane (IP): - identification (ANAL) of a plurality of 3D Gaussians along a ray (R, R1, R2) launched from said point, - projection (ANAL) of said 3D Gaussians into a plurality of respective 2D Gaussians in the image plane; - accumulation (ACC) of at least two parameters of at least two 2D Gaussians only if the at least two respective 3D Gaussians overlap. Computer program comprising instructions for carrying out the steps of an encoding or decoding process according to claim 1 when said program is executed by a computer.

Images