Method and device for representing volumetric content.

Abstract

The invention relates to a method and device for rendering, in an image plane, a volumetric image decomposed into a plurality of 3D Gaussians. The method comprises the following steps for at least one sample (P) to be reconstructed in the image plane (IP): - identification of a plurality of 3D Gaussians along a radius (R, R1, R2) launched from said sample, - projection of said 3D Gaussians into a plurality of 2D Gaussians in the image plane; - according to an overlap criterion between at least two 3D Gaussians of the plurality, on said radius, based on parameters of said 2D Gaussians: - an accumulation step if said at least two 3D Gaussians overlap, - a blending step otherwise. Figure for the abstract: Fig. 5.

Description

Title of the invention: Method and device for representing volumetric content. Previous technique

[0001] The invention relates to the general field of volumetric images. It relates more particularly to the representation and rendering of such content.

[0002] A volumetric image corresponds to three-dimensional content or a three-dimensional scene. Such an image can be part of a sequence of images or a video. Several approaches exist for representing volumetric content, notably those based on an explicit or implicit representation of the 3D scene in question.

[0003] An explicit approach represents a 3D scene using geometric primitives (volumes, surface elements, voxels, etc.) and then uses image synthesis techniques to render an image of this scene associated with a viewpoint. An implicit approach uses an implicit representation of the scene. Examples include approaches that define a 3D map of distances to objects in the scene (e.g., Signed Distance Function), or a 3D map of the opacity and colorimetry properties of the different 3D positions in the scene.

[0004] Recently, significant improvements have been observed in 3D scene construction and representation techniques, particularly through implicit representation based on neural networks and ray tracing. However, these approaches remain too complex in practice.

[0005] A technique for representing a 3D scene as a set of 3D Gaussian primitives, known as 3DGS, was proposed in the article "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (Kerbl et al., ACM Transactions on Graphics, July 2023). Compared to the approaches mentioned previously, this method is significantly less complex, while offering better reconstruction quality for volumetric content. It enables real-time rendering on current terminals. This is in particular linked to the rendering technique associated with a set of 3D Gaussian primitives as it has been defined in previous work on the flattening of Gaussians (in English 'splatting'), notably in the work of Zwicker (Surface Splatting Zwicker et al., in Proceedings of the 28th annual conference on Computer graphies and interactive techniques: August 12-17, 2001).

[0006] Although very effective, the rendering technique introduced by 3DGS is not without its flaws. Among other things, one can note an undesirable induced transparency, linked to the factor opacity of a Gaussian, a distortion of the rendering depending on the viewpoints, and a low intrinsic reconstruction quality.

[0007] There is therefore a need for a solution that effectively corrects current volumetric rendering techniques. Description of the invention

[0008] The invention relates to a method for rendering a volumetric image in an image plane, the volumetric image being decomposed into a plurality of 3D Gaussians, comprising the following steps for at least one sample to be reconstructed in the image plane: - identification of a plurality of 3D Gaussians along a ray launched from said sample, and, - projection of said 3D Gaussians into a plurality of respective 2D Gaussians in the image plane; - based on an overlap criterion between at least two 3D Gaussians of the plurality, on said radius, from parameters of at least two respective 2D Gaussians: - an accumulation step if the said at least two 3D Gaussians overlap; - a blending step otherwise.

[0009] A "volumetric image" is understood to be a 3D scene that can be observed from different angles and viewpoints. The volumetric representation is the set of data associated with such an image: for example, a set of 3D samples (voxels), or a set of 3D Gaussians and associated opacity information.

[0010] The term "rendering" refers to the operation of representing all the volume data on a flat surface, called the image plane, corresponding to a viewpoint of the scene.

[0011] By "sample to be reconstructed" is meant a value of the image plane that one seeks to reconstruct. The sample can be, for example, a color pixel traditionally represented by a triplet of values, for example (R,G,B) or (Y,U,V). In general, a sample is represented by a vector of one or more components, or data.

[0012] The term "3D Gaussian" means a reconstruction kernel in space. Volume data is thus interpreted as a field of 3D reconstruction kernels, each kernel typically being associated with a point, or voxel, in space. Each 3D reconstruction kernel can be likened to a "fingerprint" function, distributing the contribution of each voxel across nearby voxels. A 3D Gaussian is usually defined by its position in space, its opacity, its covariance, and its color coefficients (or harmonics). Such data can be obtained, or calculated, from latent codes, particularly during the training, construction, or learning of 3D Gaussians. Any classical state-of-the-art method can be used.

[0013] By "projection of 3D Gaussians into 2D Gaussians" is meant the projection, or splatting, operation that produces 2D, or flattened, Gaussians in the image plane. A 2D Gaussian is represented by parameters: its position, its covariance matrix, its color coefficients (for example, (R, G, B)), and its opacity coefficient. Alternatively, the opacity coefficient can be replaced by the transparency coefficient, as opacity and transparency are complementary. An object is totally opaque if the opacity value is at its maximum, i.e., at the value 1. Conversely, it is invisible if this value is at 0. To each pixel of the image plane, we can associate, during rendering, color coefficients (which are those of the 2D Gaussian), and an opacity coefficient defined as the product of the opacity coefficient of the 2D Gaussian and the value of the 2D Gaussian at that point.Thus, the opacity coefficient of a pixel varies and tends to become zero as one moves away from the center of the 2D Gaussian.

[0014] By "ray," we mean, in accordance with the state of the art, a virtual half-line whose origin is located on the camera (or the observer's eye) and which intersects the image plane at the position of the sample (pixel) that we seek to reconstruct. Volumetric rendering thus simulates a physical optical process by projecting or casting rays along which the volumetric density of the scene can be calculated. This is also referred to as ray integration.

[0015] By "accumulation" is meant a weighted summation of the data associated with a sample of the image plane, originating from the different Gaussians. The 2D Gaussians resulting from the projection of the 3D Gaussians can overlap after projection (splatting) in the image plane. The weighted summation performed on the parameters of several Gaussians makes it possible to calculate a resulting average value of color and opacity resulting from the superimposed Gaussians in 3D space, according to an overlap criterion.

[0016] By "blending" we mean the combination of the contributions of the different Gaussians associated with a sample of the image plane, according to their opacity factor. Blending performed on the parameters of several Gaussians makes it possible to take into account the effects of transparency.

[0017] By "overlap criterion," we mean a criterion that must be met for at least two Gaussian curves to be considered as overlapping. This criterion can take various forms relating, for example, to the number of Gaussian curves considered, their shape, the localization interval calculated for them, a percentage of overlap, etc. The overlap criterion is defined in a way A virtual layer, or set of superimposed 3D Gaussians, is created during integration along the radius. One or more layers can be defined in this way, each containing one or more Gaussians.

[0018] The 3D Gaussians composing the scene are thus associated, when they overlap, with constituent layers of the scene. If Gaussians overlap, the layer thus formed by them may correspond, in particular, to a local surface element of the scene. The process performs an accumulation step to reconstruct this layer. Once the accumulation is complete for this layer, it can be blended with the other layers. The process thus reconstructs, layer by layer, the contribution of the layers that may correspond to local surfaces, and then performs a blending operation to take into account the composition of the contributions of these different layers according to their transparency factor.

[0019] Advantageously, instead of using a recursive reconstruction formula based solely on the blending principle, as proposed in the prior art, the process interweaves accumulation and blending approaches, which notably corrects the viewpoint-dependent rendering distortion encountered in prior art methods such as 3DGS. As will be explained later with reference to Figures 2 and 3, 3DGS considers an order for the 3D Gaussians to be composed. However, this order can depend on the viewpoint, particularly if there is an overlap of the 3D Gaussians. This leads to rendering distortion when switching from one viewpoint to another. Accumulation processing within a layer greatly mitigates this phenomenon.

[0020] Moreover, in the case of a 3D overlap, 3DGS combines the contributions of each Gaussian using the blending formula, whereas in this scenario, accumulation is more appropriate. The accumulation method according to the invention thus provides the expected reconstruction in this context.

[0021] Generally speaking, the steps of a rendering process should not be interpreted as being linked to a notion of temporal succession. In other words, the steps can be carried out in a different order than that indicated in the independent claim, or even in parallel.

[0022] According to embodiments of the rendering process:

[0023] - said parameters of the 2D Gaussians include a color and an opacity, And : - the accumulation step involves calculating a weighted sum of the color of said at least two 2D Gaussians to obtain a resulting color, and a combination of the opacity of said at least two 2D Gaussians to obtain a resulting opacity; - the blending step uses said resulting color and said resulting opacity from the accumulation step.

[0024] Advantageously, such a method makes it possible to obtain, for a sample (pixel), a resulting color and opacity for several overlapping Gaussians within a layer. Subsequently, these resulting values ​​are proposed in the blending step as if they had been obtained from a single Gaussian.

[0025] - the resulting opacity depends on a sum of the opacities of said at least two 2D Gaussians; advantageously, according to this method, the resulting opacity thus defined corresponds to the spatial density associated with the overlap zone of the Gaussians. This is notably the case when the opacity is simplified to the density (whereas the opacity, when not simplified, is expressed as a function of the exponential of the density).

[0026] - the resulting opacity depends on a product of the complements of the opacities of said at least two overlapping 3D Gaussians; advantageously, in this mode, the resulting opacity corresponds to the equivalent opacity after the ray passes through the considered Gaussians. Thus, objects located behind this layer are seen with the same transparency factor in this mode as in the alpha-blending composition mode.

[0027] - the overlap criterion depends on a location interval of said at minus two 3D Gaussians along said radius; advantageously, according to this method, an interval can be defined simply to determine that two Gaussians overlap. For example, a single interval can be defined on the depth axis, and the two Gaussians overlap when they are both within the interval. For example, an interval can be defined by Gaussian, and the Gaussians overlap when the two intervals intersect. Advantageously, according to this method, based on the intersection, estimation errors on these position intervals along said radius can be compensated for.

[0028] - said overlapping criterion depends on at least two encompassing parallelepipeds respectively, said at least two 3D Gaussians, and said Gaussians overlap if said parallelepipeds have an intersection. Advantageously, according to this variant, the overlap tests between ellipses require less computation to estimate the localization intervals along said radius, since one analyzes the intersection of a line with a rectangle instead of analyzing the intersection of a line with an ellipse.

[0029] - said overlap criterion depends on a maximum number of Gaussians 3D identified along said radius; advantageously, according to this variant, it is possible to control the number of Gaussians within a layer for reasons of storage and computational optimization; indeed, particularly with regard to the algorithm for estimating the parameters of Gaussians, the fact of having several Gaussians per layer can have an impact on the memory space required.

[0030] - said overlap criterion depends on the resulting opacity of the step of accumulation; advantageously according to this mode, one can in particular stop the accumulation within the Gaussians as soon as one knows that, due to the level of opacity already reached, the contribution of the following Gaussians will be zero, because these Gaussians will be masked by the Gaussians already encountered up to that point.

[0031] - the accumulation ends with a normalization step relating to the weights of weighting.

[0032] - said at least two overlapping 3D Gaussians along the radius define one layer, and: - The accumulation step is carried out within the layer; - The blending step is carried out between two layers.

[0033] Correspondingly, the invention also relates to a rendering device.

[0034] The characteristics and advantages of the rendering process apply in the same way to the rendering device according to the invention and vice versa.

[0035] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or rendering device according to the invention. This program includes instructions adapted to the implementation of the corresponding method. This program may use any programming language and may be in the form of source code, object code, or code intermediate between source and object code, such as in a partially compiled form, or in any other desirable form.

[0036] The invention also relates to a computer-readable information or recording medium containing the aforementioned computer program instructions. The information or recording medium may be any entity or device capable of storing programs. For example, the medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard drive, a DNA sequence, or flash memory. Furthermore, the information or recording medium may be a transmissible medium such as an electrical or optical signal, which may be transmitted via an electrical or optical cable, by radio link, by wireless optical link, or by other means.

[0037] The program according to the invention can in particular be downloaded onto an Internet-type network.

[0038] Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a process according to the invention. Brief description of the drawings

[0039] Other features and advantages of the present invention will become apparent from the description given below, with reference to the attached drawings which illustrate examples of embodiment without any limiting character.

[0040] [Fig-1] Fig. 1 schematically represents a volumetric rendering device according to one embodiment of the invention.

[0041] [Fig.2] Fig.2 schematically represents the volumetric rendering of a scene according to the state of the art.

[0042] [Fig.3] Fig.3 schematically represents the volumetric rendering of a scene from two different points of view according to the state of the art.

[0043] [Fig.4] Fig.4 represents a graphic result of the volumetric rendering of a scene from two different points of view according to the state of the art in accordance with [Fig.3].

[0044] [Fig.5] Fig.5 schematically represents the volumetric rendering of a scene according to one embodiment of the invention.

[0045] [Fig.6] [Fig.6] is a logic diagram representing a first embodiment of the volumetric rendering process according to the invention, which can be implemented by the rendering device of [Fig.1].

[0046] [Fig.7] [Fig.7] is a flowchart representing a second embodiment of the volumetric rendering method according to the invention, which can be implemented by the rendering device of [Fig.1].

[0047] [Fig.8] [Fig.8] schematically represents the definition of localization intervals of 3D Gaussians along a radius according to two embodiments.

[0048] [Fig.9] Fig.9 schematically represents a step in determining 3D Gaussian overlap according to two embodiments. Description of the implementation methods

[0049] Figure 1 schematically represents the hardware architecture of a volumetric rendering device (RENDER) according to one embodiment of the invention. The rendering device is connected via an optional human-machine interface (HMI) to a module (BDS) comprising the representation of the 3D scene (3DS) to be rendered. This module can take the form of a server, a database, etc. It includes the 3D scene representations.

[0050] The rendering device classically comprises memories (MEM) associated with a processor (PROC) of a processing unit, controlled by a computer program (PGR) stored in read-only memory (for example, ROM or disk). (hard). During initialization, the computer program's code instructions are loaded into RAM before being executed by the processor. Memory can be of the ROM (Read Only Memory), RAM (Random Access Memory), or Flash type.

[0051] In accordance with [Fig. 1], the device also comprises:

[0052] - an optional GEN module for creating 3D Gaussians representing the image Volumetric. The 3D scene can indeed be provided in another format, for example, as a set of voxels or as a set of latent codes. In this case, the GEN module creates the 3D Gaussians from this data. Any known state-of-the-art method can be used, for example, the one described in the aforementioned 3DGS document.

[0053] - an ANAL module responsible for performing ray tracing and analysis (integration) of the scene along the radius. In particular, this module is responsible for projecting 3D Gaussians into 2D Gaussians on the image plane, defining a list of 3D Gaussians along a radius, and determining if two Gaussians overlap, based on a chosen overlap criterion. It can also be used to group several Gaussians into a single layer.

[0054] - an ACC module responsible for accumulation, inside a layer;

[0055] - a BLEND module responsible for blending between two layers;

[0056] - an optional DISP module responsible for displaying the resulting image I, by example on a computer screen.

[0057] All modules communicate conventionally with each other via a data bus (B).

[0058] Figure 2 schematically represents the volumetric rendering of a scene according to the state of the art.

[0059] The user's point of view (whose eye is located at the position of the CAM camera) is represented by the two dotted half-lines which delimit the 3D scene according to this point of view as well as the projection onto the IP image plane.

[0060] The 3D (3DS) scene shown consists of 3D Gaussians. To perform the rendering, they are projected onto the camera's image plane. These projections are referred to as 'splat' (or 2D splat) hereafter. We are particularly interested in the three Gaussians GA, GB, and GC in the figure, which lie on the ray R projected (launched) from the CAM camera. They are projected as three splats GA', GB', and GC. The ellipsoids corresponding to the projected Gaussians are artificially positioned above or below the image plane to distinguish them. However, in the image plane, these projected ellipsoids are at the same level.

[0061] Obtaining 3D Gaussians can be achieved, as proposed in 3DGS, by classical optimization techniques which allow, starting from a point cloud Initially, to control the number and parameters of the Gaussians for optimal rendering of the scene.

[0062] The rendering principle as proposed by 3DGS is to work in the image plane, using a 3D Gaussian scanning rendering technique. A painter's algorithm can be used to scan the Gaussians. The principle of this algorithm is to draw the different Gaussians in order of depth, from the furthest to the nearest. Thus, the final color of the pixel is rendered step by step by an a-blending type algorithm, in which each layer contributes by a factor of a, representing the opacity of the Gaussian. The Gaussians can also be rendered from front to back. For example, the rendering of pixel P shown in the figure can be done by considering the order GA, GB, GC, or GA, GC, GB. Each time a layer is added, it contributes to the rendering of the pixel by a factor of a and attenuates the background by a factor of (1-a).The final color in P, when the Gaussians are treated in the order GA, GB and GC, can be expressed by C = ac * cc + ab * (1- ac ) * cb + aa*(l- ac ) (1- ab ) * ca. Note that in this order, for an identical opacity factor, Gaussian A is more attenuated than B, which is itself more attenuated than C, thus giving the strongest weighting to the foreground.

[0063] More generally, the reconstruction of the image plane into a pixel x can be expressed by the following formula: C = ^TiVl

[0065] Where Ci is the color value associated with Gaussian i (this value can be a scalar, a multicomponent representation as for color images, for example RGB or YUV, or a multidirectional representation of color, for example via spherical harmonic coefficients, as in 3DGS); ai = 67 (r ) *o( represents the opacity level of the i-th Gaussian for the sample '■elr= represents the attenuation (transmittance) to be applied to the nth Gaussian due to the transparencies of the objects present "above" this Gaussian (in the chosen order).

[0066] In the aforementioned 3DGS algorithm, the transparency at a pixel x depends on two parameters: the opacity oi of the Gaussian, and the attenuation G;(x) relative to the position within the Gaussian. This attenuation can be defined by the following formula, in which p;2D is the center of the Gaussian projected onto the image plane.

[0067] / \ GJx = y.exd - -— ! -------—

[0068] According to the aforementioned prior art, the normalization factor used in front of the exponential can take the value 1, or alternatively a value calculated based on the 2D covariance matrix S2D. The S2D covariance matrix is ​​obtained by projecting the 3D covariance matrix associated with the 3D Gaussian, as detailed in the prior art, where it is proposed to use a low-pass filter, modeled by a simple Gaussian, to avoid aliasing effects due to unsatisfactory sampling of the samples in the image plane. The effect of this low-pass filter is simply expressed by adding a term to the diagonal of the matrix. In the expression for the attenuation factor, the covariance matrix is ​​therefore modified to S2D to take into account this dilation effect of the 2D Gaussians.This ability to perform anti-aliasing without multiple sub-pixel renderings is unique to the use of Gaussians (according to the principle stated in Zwickler's original paper). Other approaches, particularly ray-tracing-based approaches, require multiple ray tracings per pixel to achieve anti-aliasing.

[0069] As shown in Figures 3 and 4, this approach suffers from a difference in rendering of the scene depending on the viewpoints.

[0070] Fig. 3 schematically represents the volumetric rendering of a scene from two different viewpoints according to the state of the art (3DGS).

[0071] Figure 3 represents a flat surface (hatched surface) with a texture gradient, which we wish to render from two viewpoints, denoted LV (Left View) and RV (Right View). The algorithm presented in support of Figure 2 orders the projected 3D Gaussians in depth. Thus, in the left view, the leftmost Gaussian will be the furthest, while the opposite will be true in the right view. In other words, in the left view, the 2D Gaussians are rendered 'from left to right', while in the right view, they are rendered 'from right to left'. This variation leads to differences in rendering that can fluctuate considerably, since, as mentioned previously, in 3DGS, the rendered value depends on the depth order of the Gaussians (the nearest one taking precedence over the furthest).This fluctuation can be particularly significant when the Gaussian zones are very close in depth, since in this case the order can change very rapidly from one viewpoint to another.

[0072] Fig. 4 represents a graphic result of the volumetric rendering of a scene from two different points of view according to the state of the art in accordance with Fig. 3.

[0073] In [Fig.4], 3 curves are shown: - the solid line curve (ground truth) illustrates the rendering obtained by composing the Gaussians according to an "ideal" rendering; - the thin dotted curve (left to right) shows the perceived result for a left-hand view; - the wider dotted curve (right to left) shows the perceived result for a right-hand view.

[0074] It is clear that the two curves are offset, which means that the rendering will be different depending on the left or right viewpoint according to the state of the art.

[0075] Figure 5 schematically represents the volumetric rendering of a scene according to one embodiment of the invention.

[0076] The context of the 3D scene is similar to that of [Fig.2]: the user's viewpoint and the 3D scene are represented similarly, the 3D Gaussians can be obtained by classical optimization techniques, and the three Gaussians GA, GB and GC which are located on the ray R launched from the CAM camera are projected into 3 splats GA', GB' and GC'.

[0077] The rendering principle, however, differs from that of [Fig. 2]: when two layers overlap, according to a given criterion, an accumulation, and not a blending, is performed between the Gaussians. The overlap criterion can take different forms; examples will be detailed in support of Figures 8 and 9. A set of overlapping Gaussians defines a layer. A layer can correspond to a physical surface of the scene, but is not limited to this.

[0078] In [Fig. 5], the two Gaussians B and C overlap for the sample located at position P. Their contributions are therefore added together, and not combined ("blended"), resulting, for example, in a contribution at P of the type Cacc = [3b * cb + [3C * cCj] where [3b and [3c] are respective weighting factors associated with the two Gaussians. According to this embodiment, the resulting color is a weighted sum of the colors of the Gaussians. A normalization step may be necessary. The result of the accumulation can be considered as an intermediate Gaussian, denoted GBC in the figure. More generally, an accumulation layer corresponds to the accumulation of all the Gaussians in the layer, and in some cases, the result can be considered as an intermediate Gaussian possessing its own resulting opacity and color parameters.

[0079] The Gaussian A, on the other hand, which is not overlapping along the radius, can be considered as a layer by itself.

[0080] Once the layers have been identified, the blending steps are carried out in a conventional manner between these layers, according to the example of the GA layer and the accumulated GBC layer.

[0081] Thus, the final color of the pixel is rendered step by step by an a-blending type algorithm performed between the layers (and not between the Gaussian layers), and an accumulation within the layers. Each of the layers contributes to the extent of

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Its opacity coefficient is a. Layers can also be rendered in any order (from front to back, or from back to front, with accumulation and blending being sequential or non-sequential). For example, the rendering of pixel P shown in [Fig. 5] can be done using the order GA, GB+GC, or GB+GC, GA. Each time a layer is added, it contributes to the pixel rendering by a factor of a and attenuates the background by a factor of (1-a). The final color in P, when the Gaussians are treated in the order GB+GC, GA, can be expressed by C = aa * ca + ab+c * (1- aa) * cb+c = aa * ca + [ab+c * (1- aa ) * (|3b * cb + [3C* cc)], where (|3b * cb + [3C* cc) represents the resulting color of the GBC layer and ab+c represents the resulting opacity of the GBC layer. According to one embodiment, the resulting opacity ab+c depends on the sum of the opacities of the Gaussians, that is to say that in this example, ab+c=ab + ac. This sum can be weighted. According to another embodiment, the resulting opacity ab+c depends on the product of the complements of the opacities of the Gaussians, that is to say that in this example, ab+c = 1-(l-ab) * (l-ac) More generally, using the notations associated with the [Fig.2] descriptive representation of the 3DGS state of the art: The resulting color value for the layer can be defined as a weighted average of the colors based on the opacity values ​​and attenuation factors of the Gaussians, namely: Z^l(x).Oi.Cl The resulting opacity coefficient O associated with the layer can be calculated in various ways, including, but not limited to (naturally this value may require normalization to remain within the interval [0,1]): - a summation of opacities: - a weighted average of opacities 0 ~ - a complement of the product of the complements of the opacities o=i_n.( 1-^.(1-).0,) A correct initialization of the algorithm can be achieved simply by taking into account the default background color Coo. In other words, the background is considered as a first virtual layer of Coo color.

[0095] This approach resolves the difference in rendering of the scene depending on the viewpoints (since the nearest Gaussians can be grouped into layers).

[0096] Fig. 6 is a flowchart representing an example of a volumetric rendering process that can be implemented by the rendering device of Fig. 1 in accordance with the example of Fig. 5.

[0097] During the initialization step E20, the 3D scene is constructed as a set of 3D Gaussians. Any algorithm within the grasp of a person skilled in the art can be used to generate the Gaussians, for example, the algorithm proposed in the aforementioned 3DGS method, which uses gradient descent to minimize a quality criterion for reconstructing the rendered views. The algorithm then proceeds for each pixel of the image plane according to steps E21 to E28.

[0098] During step E21, the Gaussians are identified. A list of Gaussians intersecting the ray associated with the pixel in the image plane is established, and then this list is sorted (for example, GA, GB, GC). The locations of these Gaussians on the ray are established using any method. Examples of such methods will be detailed with reference to Figures 8 and 9. For example, the list is sorted according to the depth order of the Gaussians scanned along the ray, from the furthest to the nearest. A reconstruction color for the current pixel is also initialized during this step using the default background value for that given pixel. For example, this value could be a fixed value for all pixels, or it could be an image of the scene at infinity viewed in that direction, as used in the so-called "sphere-map" or "cube-map" technique commonly used in image synthesis.We also initialize a first grouping layer as an empty layer to proceed with subsequent groupings.

[0099] For each Gaussian, in step E22, the 3D Gaussian is first projected onto a 2D Gaussian in the image plane. Such a projection step has already been described previously. It is also known from the prior art, notably from the aforementioned article by Zwicker. After the projection, the 2D Gaussian, or splat, is described by its position parameters, its covariance matrix, its color coefficients (for example, (R,G,B)), and its opacity coefficient.

[0100] In step E23, a test is performed to determine whether the current Gaussian overlaps the previous Gaussian, according to a given overlap criterion. Such overlap criteria can take various forms and will be detailed in support of Figures 8 and 9. In particular, and without limitation, they may depend on:

[0101] - of a localization interval associated with the Gaussian; for example, an interval location is defined for the previous Gaussian and for the current Gaussian, and if the intervals are intersecting then the current Gaussian is associated with the same layer as the previous Gaussian;

[0102] - of an overlap rate between the location intervals; for example, the Gaussian curves are considered to overlap if their intersection exceeds a certain percentage of a location interval;

[0103] - of an intersection of the boxes (for example parallelepipeds) drawn around the Gaussian;

[0104] - of a premature layer construction closure test; for example, when the The number of grouped Gaussians exceeds a set threshold, or the resulting opacity of the layer being built exceeds a defined threshold, or the thickness of the current layer exceeds a defined threshold (this threshold can be absolute or dependent on the distance to the camera). In this case, the current Gaussian is not considered to be overlapping; it is associated with a subsequent layer.

[0105] If the test in step E23 is positive, the partial calculations necessary for the accumulation operation within that layer are updated in step E24. The accumulation results in a color and opacity. As explained in support of [Fig. 5], the color of the Gaussian is accumulated to that of the previous one by a weighted sum, and the opacity by a summation or a product. The necessary steps for initializing the accumulation buffers are assumed to be performed in steps E22 or E26 during the initialization of a new layer.

[0106] If the test in step E23 is negative, the accumulation operation is finalized in an optional step E25, by a possible renormalization of the weighted sum by the weight sum used.

[0107] Then, in step E26, the layer thus obtained is composed using an alpha-blending operation that combines the resulting opacity of the layer with that of the previous layer or Gaussian. In one embodiment, the resulting opacity depends on the sum of the opacities of the Gaussians. In another embodiment, the resulting opacity depends on the product of the complements of the opacities of the Gaussians.

[0108] A new empty layer is then initialized for future layer groupings.

[0109] Following the iteration on the Gaussians, in step E27, a finalization of the rendering operations is carried out in step E28. Typically, if a layer was under construction, the accumulation and the update of the reconstruction color by alpha-blending are then carried out as in E25 and E26.

[0110] [Fig.7] is a flowchart representing a second embodiment of the volumetric rendering process according to the invention, which can be implemented by the rendering device of [Fig.1] in accordance with the example of [Fig.5].

[0111] Steps E10, Eli, E12 are similar to steps E20, E21, E22 of [Fig.6].

[0112] In step E13, the layers consisting of the 3D Gaussians are defined. To this end, a current layer is initialized, then the list of Gaussians is traversed and an overlap test is performed for each current Gaussian. If the test is positive, the current Gaussian is associated with the current layer. If the test is negative, a new current layer is initialized. At the end of step E13, there are thus J layers Cj, with J < 1, where I is the total number of Gaussians.

[0113] Next, the layers are traversed in a predefined order, and for each layer:

[0114] An accumulation step is performed at step El4. This step is similar In step E24, this process is performed on all Gaussians in the layer: the colors of the Gaussians are summed weighted to obtain a resulting color for the layer, and the opacities are summed or multiplied to obtain a resulting opacity for the layer. This step is not implemented if the layer contains only one Gaussian. It optionally concludes with a normalization step similar to that of step E25.

[0115] A blending step is carried out in step E15, similar to step E26.

[0116] At the end of the iteration on the layers, in step E16, a finalization of the rendering operations is carried out in step E17, similar to step E28.

[0117] [Fig.8] schematically represents the definition of localization intervals of 3D Gaussians along a radius according to two embodiments.

[0118] This localization step details in particular steps E23 and E13 of the logic diagrams in figures 6 and 7. Each of the Gaussians which has previously been placed in the sorted list (El, E2, E3) for a given radius (RI, R2) is associated with a localization interval which makes it possible to determine whether it overlaps another Gaussian or a layer of Gaussians.

[0119] Figure 8 presents two embodiments:

[0120] According to a first method, shown on the right, the location intervals are defined by large, unoriented boxes, by the dashed rectangles. For the points associated with the radii RI and R2, the Gaussians E1 and E2 shown overlap and are grouped into a first layer. The Gaussian E3 remains separate and forms a second layer.

[0121] Such a box, called a global bounding box, can allow a localization interval to be defined, for example from the position parameters p and the covariance matrix of the 3D Gaussian, S3D, as the interval on the z-axis of the radius having center pz and width x (k being a factor defining the confidence interval, typically a value between 2 and 3).

[0122] The positional parameters p and the covariance matrix of the 3D Gaussian can be expressed by:

[0123] ^y

[0124] E. Eyx \ ^Z* Ej.y ^zy E xz \ ^yz ^zz /

[0125] According to a second method, shown on the left, the localization intervals are defined by oriented bounding boxes, referred to as "tight" bounding boxes, represented by the dashed quadrilaterals. For the sample associated with radius RI, there is no overlap of the intervals, so a layer is defined for each Gaussian. For the sample point associated with radius R2, the Gaussians E1 and E2 overlap; they are grouped into a first layer. The Gaussian E3 remains separate and forms a second layer.

[0126] Such a tight bounding box has the advantage of conforming to the orientation of the 3D Gaussian and thus having a narrower interval. It can be calculated, for example, from the position parameters p and the inverse matrix of the covariance matrix of the 3D Gaussian, A3D, as the interval on the z-axis of the radius centered at , . + A>". where (x,y) are the coordinates of the pixel in the image plane and for width & x A being defined as before).

[0127] Fig. 9 schematically represents a step in determining the overlap of 3D Gaussians according to two embodiments.

[0128] The first mode, shown at the top of the figure, corresponds to step E23 of the flowchart in [Fig. 6], in which a test is performed to determine whether the current Gaussian overlaps the localization zone of the previous Gaussian. In this case, a fixed-size overlap zone construction is considered. When the first Gaussian E1 is encountered in the list, it defines a new layer. An overlap zone Z1 (corresponding to the location interval of E1) is associated with it, which is used to test whether subsequent Gaussians overlap this layer. The Gaussian E2, whose location interval overlaps with Z1, is grouped with E1 in the first layer. Subsequently, the Gaussian E3 constitutes the second layer (with the associated overlap zone Z2), and the Gaussian E4 constitutes the third layer (with the associated overlap zone Z3).

[0129] The second mode, shown at the bottom of the figure, corresponds to step E13 of the flowchart in [Fig.7], in which a test is performed to determine whether the current Gaussian overlaps the localization area of ​​the current layer. Consider an increasing construction of the overlap zone. When the first Gaussian E1 is extracted from the list, it initiates a new layer by considering the overlap zone Z1.1 (which corresponds to the interval associated with E1). Since the Gaussian E2 overlaps the overlap zone Z1.1, it is grouped into this first layer, and the overlap zone is updated to Z1.2, which corresponds to the union of the intervals of Z1.1 and E2. Since the Gaussian E3 overlaps the overlap zone Z1.2, it is grouped into this first layer, and the overlap zone is updated to Z1.3, which corresponds to the union of the intervals of Z1.2 and E3. Since the Gaussian E4 does not overlap Z1.3, the first layer is finalized, and a second layer is initiated with the overlap zone Z2.1. This process continues for the other Gaussians to come.

Claims

Demands

1. A method for rendering, in an image plane, a volumetric image decomposed into a plurality of 3D Gaussians, comprising the following steps for at least one sample (P) to be reconstructed in the image plane (IP): - identification (E21) of a plurality of 3D Gaussians along a radius (R, RI, R2) launched from said sample, - projection (E22) of said 3D Gaussians into a plurality of respective 2D Gaussians in the image plane; - according to an overlap criterion (E23) between at least two 3D Gaussians of the plurality, on said radius, from parameters of at least two respective 2D Gaussians: - an accumulation step (E24) if said at least two 3D Gaussians overlap, - a blending step (E26) otherwise.

2. A rendering method according to claim 1, wherein said parameters of said at least two 2D Gaussians comprise a color (Ci) and an opacity (Oi), and: - the accumulation step comprises calculating a weighted sum of the color of said at least two 2D Gaussians to obtain an average color, and a combination of the opacity of said at least two 2D Gaussians to obtain a resulting opacity; - the blending step uses said average color and said resulting opacity from the accumulation step.

Images