Methods and devices for compressing and reconstructing at least one multidimensional scene composed of a plurality of elementary objects and corresponding computer program.

Abstract

Title: Methods and devices for compressing and reconstructing at least one multidimensional scene composed of a plurality of elementary objects and corresponding computer program. The invention relates to a method for compressing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing, for at least one of said elementary objects, referred to as the current object: for at least one descriptive piece of information of said current object, referred to as the current descriptive information: quantization (21) of said current descriptive information over an integer number of bytes, optimization (22) of the quantization of said current descriptive information. Figure for the abstract: Figure 2

Description

Title of the invention: Methods and devices for compressing and reconstructing at least one multidimensional scene composed of a plurality of elementary objects and corresponding computer program.

[0001] 1. Scope of the invention

[0002] The field of the invention is that of the encoding and decoding of digital signals. More specifically, the invention relates to the compression of data representing at least one multidimensional scene, for example for the purpose of their transmission and / or storage, and the reconstruction of such scenes.

[0003] In particular, the invention proposes a solution enabling real-time decompression and / or rendering of at least one multidimensional scene on equipment with limited resources in terms of RAM (“Random Access Memory”), CPU (“Central Processing Unit”) and / or GPU (“Graphics Processing Unit”), for example on a mobile terminal such as a “smartphone”.

[0004] The proposed solution has applications in all areas where having multiple points of view can be of interest, such as training, sport, entertainment, maintenance, etc.

[0005] 2. Prior art

[0006] Numerous solutions have been proposed for reconstructing a 3D scene (also called a 3D model) from photos taken from different viewpoints. When this reconstruction is obtained from video content, by taking the same time instant from each video clip, a temporal sequence of 3D scenes is obtained, also called a 3D video or volumetric video, which can be viewed from any point of view.

[0007] In order to represent volumetric content (3D scene or 3D video), several approaches exist. We can thus distinguish two main approaches, one based on an explicit representation and the other on an implicit representation of a 3D scene at a given moment.

[0008] An explicit approach represents a 3D scene using geometric primitives (volumes, surface elements, voxels, ...) and subsequently uses image synthesis techniques to render an image of this scene associated with a viewpoint.

[0009] An implicit approach does not explicitly represent the 3D scene. For example, one could cite approaches consisting of defining a 3D map of distances to objects of the scene (for example "Signed Distance Function", or signed distance function), or also a 3D map of the opacity and colorimetry properties of the different 3D positions of the scene.

[0010] Recently, Kerbl et al. presented a technique for representing a 3D scene as a set of 3D Gaussian primitives, which can be projected onto the camera's image plane. These projections are called "splat" or "Gaussian splat." Such a technique, called "3D Gaussian Splatting" or 3DGS, is notably presented in the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering," SIGGRAPH 2023. In simplified terms, a point cloud is generated from the input images, which correspond to different viewpoints. Each point is then converted into a Gaussian that can be split, cloned, deleted, etc. The resulting 3D splat can be considered a spherical object possessing spatial, geometric, and colorimetric properties. The properties of a 3D splat can be described by a set of parameters or descriptive information, an example of which is shown in the table below:

[0011] [Tables 1] Position of the splat center (x, y, z) Color: Red, green, blue Density or transparency (alpha) Spherical harmonics (srO, sgO, sbO, srl, sbl, sgi, ...) Scale factor (sx, sy, sz) Rotation factor (qx, qy, qz, angle)

[0012] Combinations of these properties can produce a wide variety of visual representations. For example, it is considered that a few tens of thousands of splats can represent a 3D scene in a photorealistic way.

[0013] Reconstructing a 3D video at a specific point in time, referred to here as a "frame," using a "3D Gaussian Splatting" technique offers good visual quality but relies on processing very large amounts of data. For example, even if only the central subject of a video is extracted, frames of approximately 30 MB (30,000,000 bytes) in size are obtained from several video clips, using the same point in time from each video. Reconstructing a sequence of frames from videos, for example, reconstructing a series of 30 3D scenes per second, requires a data rate of approximately 7 Gbps (gigabits per second).

[0014] It is necessary to compress such a mass of data in order to consider a transfer to a playback equipment, in particular if one considers a wireless transmission technique (for example of the Wifi®, 5G or other type).

[0015] It is also necessary to decompress and restore such a large amount of data quickly.

[0016] Current techniques used for point cloud compression include quantization and the use of coding alphabets, or "codebooks," which involve partitioning the space of possible values ​​and using the index of this code in place of the actual values ​​in a sequence of bits of arbitrary size. During decoding, the bitstream is manipulated bit by bit to reconstruct the "codebooks," which, via the index, will allow the initial values ​​to be reconstructed and the display to be enabled on the GPU.

[0017] Fig. 1 illustrates an example of a classic processing chain implemented for the decompression / restoration of an image.

[0018] In a first step 11, the compressed data is read. This step may activate RAM. In a step 12, the compressed data is stored in a CPU buffer. This step may activate RAM. In a third step 13, the codebooks are decoded and the data is decompressed. This step may activate the CPU. In a fourth step 14, the decompressed data is transformed into a format suitable for display by the GPU. This step may activate the CPU. In a fifth step, the transformed data is stored in a CPU buffer. This step may activate RAM. Then the transformed data is processed by the GPU for rendering. This step may activate the GPU.

[0019] Thus, each image, once read, is decompressed and then transformed into a format suitable for display by the GPU of the playback equipment. This process entails a significant amount of memory transfer at each stage, as well as numerous processing operations by the CPU and then the GPU.

[0020] To manage the transport and display of a video, this process can be repeated on the order of 30 times per second.

[0021] However, with "3D Gaussian Splatting" type techniques, it is not currently possible to perform all these operations in real time on a mobile terminal without specific dedicated hardware acceleration, as exists for certain types of videos.

[0022] Thus, only still images, or very short sequences, can currently be displayed in real time on a mobile terminal. To do this, after an initial decoding / preparation phase, which is not real-time, the display can be performed in real time, either on an image prepared for the GPU, or on a sequence of prepared images. The size of this sequence obviously depends on the amount of RAM available on the mobile device, but remains limited to a few seconds at most.

[0023] However, when we want to process a video of arbitrary size, in streaming mode, we seek to perform all these operations in real time, for example in less than 33 milliseconds per image to reach a frequency of 30 frames per second.

[0024] Therefore, although "3D Gaussian Splatting" type techniques make it possible to reconstruct very realistic 3D scenes from several points of view, current compression techniques do not allow for the streaming of a time series of 3D scenes on mobile terminals, due to the size of the data and the complexity of decoding and rendering.

[0025] There is therefore a need for a solution enabling efficient compression of data representative of at least one 3D scene, for example in order to be able to transfer them via a wireless network (streaming) and / or display them in real time (for example at a frequency of around 30 frames per second), in particular on a mobile terminal such as a smartphone.

[0026] 3. Description of the invention

[0027] The invention proposes a solution in the form of a compression method for at least one multi-dimensional scene composed of a plurality of multi-dimensional elementary objects, implementing, for at least one of said elementary objects, called the current object:

[0028] for at least one descriptive piece of information said current object, referred to as current descriptive information: • quantification of said descriptive current information over an integer number of bytes, • optimization of the quantization of said current descriptive information including, for at least one bit of said current quantified descriptive information, called current bit: • obtaining a representative reference image of said scene, • modifying a value of said current bit, relative to an initial value, delivering a modified current quantified descriptive information, • obtaining a degraded image representative of said scene, from said current modified quantified descriptive information, • measurement of distortion between said degraded and reference images, • if said distortion between said degraded and reference images is less than a threshold, storage of said modified value of said current bit, otherwise reset to said initial value of said current bit.

[0029] The invention thus proposes a new solution for compressing volumetric content, for example (a 3D scene, or a 3D video composed of a succession of 3D scenes), consisting of a plurality of elementary objects. For example, such elementary objects are non-Gaussian splats, or Gaussian splats, or polygons, edges and vertices of a 3D mesh...

[0030] In particular, the proposed solution is based on an optimization of the quantification of descriptive information of at least one object, by supervisedly reducing the quantification space of descriptive information.

[0031] According to at least one embodiment, such a reduction in the data quantization space allows for fast entropic decompression. Furthermore, the modified quantized descriptive information retains a structure that can be efficiently processed by the GPU since it is quantized over an integer number of bytes.

[0032] According to at least one embodiment, the proposed technique makes it possible to compress at least one multidimensional scene sufficiently to ensure streaming of the scene or time sequences over a wireless network, such as a Wifi® or 5G network or any other network, while guaranteeing real-time processing and display compatible with the capabilities of a mobile terminal such as a smartphone.

[0033] It offers a compromise between compression performance and speed of execution.

[0034] In a particular embodiment, said at least one elementary object is a Gaussian splat and the proposed solution makes it possible to compress a 3D scene composed of 3D splats efficiently in terms of compression gain, decompression time and / or 3D rendering speed by the GPU.

[0035] In a particular embodiment, said optimization of the quantification of said current descriptive information is iterated over the bits of said current quantified descriptive information as long as said distortion is below said threshold.

[0036] Thus, for at least one descriptive piece of information for at least one elementary object, one or more bits of the quantized descriptive information may be modified. This modification is performed in a supervised manner, by checking for each modified "candidate" bit whether the resulting degraded image is sufficiently close to the reference image. If so, the "candidate" modification is accepted. Otherwise, the "candidate" modification is rejected. The aim is to maintain a high-quality rendering of a multidimensional scene while reducing the data quantization space.

[0037] In particular, said optimization of the quantization of said current descriptive information is implemented by starting with the least significant bit, or LSB. The process begins by modifying the value of the least significant bit and checking whether the resulting degraded image is sufficiently close to the reference image. If so, the "candidate" modification If the modification is accepted, the process moves to the next bit (i.e., the second least significant bit) in a subsequent iteration. Otherwise, the "candidate" modification is rejected, and the iterative process terminates. This is because modifying the least significant bit, or LSB, has little impact on the resulting degraded image.

[0038] In particular, the least significant bit(s) are forced to a value of 0, as long as the resulting degraded image is sufficiently close to the reference image. In this way, the aim is to reduce the number of possible values, which helps to improve entropic compression, and consequently entropic decompression.

[0039] In a particular embodiment, the threshold is a peak-to-noise signal-to-noise ratio threshold. Such a peak-to-noise signal-to-noise ratio, or PSNR (Peak Signal to Noise Ratio), makes it possible to measure a difference between two images simply and efficiently.

[0040] For example, such a PSNR is between 20dB and 80dB, and in particular between 30dB and 50dB.

[0041] Other techniques are conceivable for measuring the distortion between two images, such as the SSIM (“structural similarity index measure”).

[0042] In a particular embodiment, said at least one descriptive piece of said current object belongs to the group comprising: • a position of the center of said current object, • a color associated with the current object, • a density associated with said current object, • a scale factor associated with said current object, • a rotation factor associated with said current object.

[0043] If the current object is a 3D splat, in one embodiment it is considered unnecessary to quantify and transmit descriptive information such as spherical harmonics. Indeed, such spherical harmonics can be omitted for processing by the GPU. In this way, the amount of data transmitted is further reduced.

[0044] In a particular embodiment, for a current descriptive information such as center position or scale factor, the quantification of said current descriptive information is implemented using two bytes. For example, two bytes are used for the quantification of the center position in x, two bytes are used for the quantification of the center position in y, two bytes are used for the quantification of the center position in z, two bytes are used for the quantification of the scale factor in x, two bytes are used for the quantification of the scale factor in y, and two bytes are used for the quantification of the scale factor in z. Twelve bytes can thus be provided for the quantification of descriptive information such as center position or scale factor.

[0045] In a particular embodiment, for a current descriptive information such as color, density, or rotation factor, the quantification of said current descriptive information is implemented on one byte. For example, one byte is used for the quantification of the color red, one byte is used for the quantification of the color green, one byte is used for the quantification of the color blue, one byte is used for the quantification of the density, one byte is used for the quantification of the rotation factor in the x-axis, one byte is used for the quantification of the rotation factor in the y-axis, one byte is used for the quantification of the rotation factor in the z-axis, and one byte is used for the quantification of the rotation angle. Eight bytes can thus be provided for the quantification of descriptive information such as color, density, or rotation factor.

[0046] According to these examples, a fixed-size group of bytes (for example, 20 bytes) is obtained after quantifying the various descriptive information of an elementary object. The amount of data to be encoded and transmitted is thus reduced compared to the prior art.

[0047] In a particular embodiment, the reference image is obtained from the current quantized descriptive information before modifying the value of at least one bit. In this way, the same reference image is used during the different iterations on the bits of the current quantized descriptive information. The number of bits modified according to this first approach is lower, but the final rendering is of better quality.

[0048] Alternatively, the reference image is obtained from the modified current quantized descriptive information, taking into account the modified value of at least one bit preceding said current bit, except for the first iteration. The number of bits modified according to this second approach is higher, but the final rendering quality is slightly degraded.

[0049] In one embodiment, the compression process includes an entropy coding step of said at least one optimized quantified descriptive information, and for example of all the optimized quantified descriptive information associated with said current object.

[0050] For example, such an entropic coding belongs to the group comprising: an LZFSE type coding (“Lempel-Ziv Finite State Entropy”), an LZ4 type coding (“Lempel-Ziv 4”), LZMA-type encoding ("Lempel-Ziv Markov chain algorithm"), ZLIB-type encoding, an LZBITMAP type encoding (“Lempel-Ziv Bitmap”).

[0051] In one embodiment, the compression process includes a step of storing and / or transmitting said at least one quantified descriptive information Optimized. The optimized quantified descriptive information(s) of at least one elementary object, obtained as output from the compression process, can be stored (in a file, memory, etc.) and / or transmitted. They can also be encoded (for example, by an entropy coder) before transmission and / or storage.

[0052] In particular, they can be transmitted via streaming in a wireless network.

[0053] In another embodiment, the invention relates to a compression device corresponding.

[0054] Such a compression device is particularly suitable for implementing the compression process described above. It may, of course, include the various features relating to the compression process according to the invention, which may be combined or considered individually. Thus, the features and advantages of the compression device are the same as those of the process described above. Consequently, they are not described in further detail.

[0055] Such a compression device is for example integrated into a computer, a tablet, a mobile terminal such as a "smartphone", a headset or virtual or augmented reality glasses, etc.

[0056] It can be a hardware entity or a software entity, which can be distributed over one or more network functions or hosted by one or more hardware devices.

[0057] The invention also relates to a method for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing: • obtaining at least one optimized quantized descriptive information of at least one elementary object (e.g., reading from a memory, receiving a signal, etc.), said optimized quantized descriptive information being obtained by modifying the value of at least one bit of the quantized descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantized descriptive information, and a reference image representative of said scene is less than a threshold, • reconstruction of said at least one elementary object from said at least one optimized quantified descriptive information.

[0058] For example, the reconstruction implements a decompression of said at least one optimized quantified descriptive information (e.g. entropic).

[0059] Such a reconstruction method is particularly suitable for receiving or reading a signal generated using the compression method described above. It may, of course, include the various characteristics relating to the compression method according to the invention. Thus, the characteristics and advantages of the reconstruction process are the same as those of the compression process described previously.

[0060] In a particular embodiment, the method comprises displaying said at least one multidimensional scene on a mobile terminal. Such a mobile terminal includes a GPU for processing the decompressed data.

[0061] In particular, such a display can be done in real time, for example at a frequency of about 30 frames per second.

[0062] In another embodiment, the invention relates to a corresponding reconstruction device.

[0063] Such a reconstruction device is particularly suited to implementing the reconstruction process described above. It may, of course, include the various features relating to the reconstruction process according to the invention, which may be combined or taken individually. Thus, the features and advantages of the reconstruction device are the same as those of the process described above. Consequently, they are not described in further detail.

[0064] Such a reconstruction device is for example integrated into a mobile terminal such as a smartphone, tablet, PDA, virtual or augmented reality headset or glasses, etc.

[0065] It can be a hardware entity or a software entity, which can be distributed over one or more network functions or hosted by one or more hardware devices.

[0066] The invention further relates to one or more computer programs comprising instructions for implementing a process as described above when this or these programs are executed by at least one processor.

[0067] The invention also relates to a computer-readable information carrier, comprising instructions for a computer program as mentioned above.

[0068] 4. List of figures

[0069] Other features and advantages of the invention will become more apparent upon reading the following description of a particular embodiment, given by way of simple illustrative and non-limiting example, and the accompanying drawings, among which: - Figure [1] illustrates an example of a classic processing chain implemented for the decompression / restoration of an image, - Figure [Fig. 2] illustrates the main steps implemented by a compression process according to a particular embodiment of the invention, - Figure 3 illustrates the main steps implemented by a reconstruction process according to a particular embodiment of the invention, - Figure 4 presents an algorithm for reducing the quantization space according to an example of an implementation of the invention, - Figure 5 illustrates a processing chain for the decompression / restoration of a 3D scene according to an example of an implementation of the invention, - Figure 6 illustrates an example of a parallel processing chain for decompressing / restoring a 3D scene according to an example of an implementation of the invention. - Figure 7 shows the simplified structure of a compression device according to one embodiment of the invention, - Fig. 8 presents the simplified structure of a reconstruction device according to one embodiment of the invention.

[0070] 5. Description of an embodiment

[0071] 5.1 General Principle

[0072] The general principle of the invention is based on optimizing the quantification of descriptive information, also called properties, of at least one multidimensional elementary object composing a multidimensional scene. Such optimization relies in particular on reducing the value space for quantifying the descriptive information of the elementary object. This reduction of the value space makes it possible to improve the compression, or encoding, particularly entropic compression, of the descriptive information of the elementary object.

[0073] Furthermore, the quantification of descriptive information is performed on an integer number of bytes (for example, one byte, two bytes, or more). This makes it possible to limit the processing time for rendering the scene at a given moment, while maintaining a format directly usable by a graphics processing unit.

[0074] The multidimensional scene is subsequently considered to be a 3D scene and the elementary object to be a 3D object. Other dimensions are conceivable, notably 2D or 4D.

[0075] Figure 2 illustrates the main steps of a compression process according to one embodiment of the invention.

[0076] Such a method takes as input at least one 3D scene composed of a plurality of elementary 3D objects. As described in relation to the prior art, such a 3D scene can be obtained from photographs captured from different viewpoints, or from videos taken from different viewpoints at the same time instant. Alternatively, such a 3D scene can be obtained from images generated by a graphics processing unit.

[0077] Consider a 3D scene at a time instant of index t, where t is an integer such that 1 < t < T and T > 1. Such a scene, denoted S_t, is composed of a plurality of elementary 3D objects. For example, the scene S_t is composed of I elementary 3D objects O_i, where 1 ≤ i ≤ 1, and i is an integer that can be several hundred thousand.

[0078] Consider one of said elementary 3D objects O_i, called the current object. Such an object O_i can be represented or described by at least one descriptive piece of information or property P_p, with l <p 1.

[0079] We consider one of said descriptive information P_p, called current descriptive information, associated with the elementary object O_i of the scene S_t.

[0080] In a first step 21, the current descriptive information P_p is quantized over an integer number of bytes, delivering a current quantized descriptive information P_p_q.

[0081] It should be noted that if we consider a quantization on one byte, or 8 bits, the space of possible values ​​comprises 256 values. If we consider a quantization on two bytes, or 16 bits, the space of possible values ​​comprises 65536 values.

[0082] Therefore, if we consider a 1m scene quantized using 256 values, we obtain a precision on the order of 0.004m, or 4mm. Such precision is not necessarily required if we seek to reconstruct objects in this scene larger than 10cm, for example. We can therefore reduce the quantization space.

[0083] If, for example, the least significant bit of the current quantified descriptive information P_p_q is forced to 0, the value space is reduced to 128 values ​​instead of 256. If we consider a scene of 1m that we seek to quantify on 128 values, we obtain a precision of the order of 0.008m, or 8mm.

[0084] If, for example, the 2 least significant bits of the current quantized descriptive information P_p_q are forced to 0, the value space is reduced to 64 values ​​instead of 256. If we consider a scene of 1m which we seek to quantify on 64 values, we obtain a precision of the order of 0.016m, or 1.6cm.

[0085] The quantification space can thus be reduced according to the composition of the scene.

[0086] Optionally, a first rough estimate of the number of bits of the current quantified descriptive information that can be modified can be implemented.

[0087] Taking into account the above observation, during a second step 22, the current quantified descriptive information P_p_q is optimized, delivering an optimized current quantified descriptive information P_p_q*.

[0088] To do this, for at least one bit B_b of said current quantified descriptive information P_p_q, with 1 b < B and B the number of bits used to quantify the current descriptive information P_p (corresponding to an integer number of bytes), called current bit, the optimization step 22 implements at least one iteration of the following steps.

[0089] During a step 221, a reference image I_REF representative of the scene S_t is obtained. Such a reference image can be obtained from the current quantized descriptive information P_p_q obtained at the output of the quantization step. 21, i.e. before modification of the value of at least one bit. Alternatively, the reference image I_REF is obtained from the current modified quantized descriptive information, obtained at a previous iteration on the bits of the current quantized descriptive information (for example at the directly preceding iteration or another previous iteration).

[0090] In a subsequent step 222, the value of the current bit B_b is modified relative to an initial value. For example, if the initial value of the current bit is equal to 1, the modified value of the current bit is equal to 0. This yields a modified current quantified descriptive information P_p_q_m.

[0091] In a subsequent step 223, a degraded image I_M0D representative of the scene S_t is obtained from the current modified quantified descriptive information P_p_q_m.

[0092] In a subsequent step 224, a distortion is measured between the degraded image I_M0D and the reference image I_REF.

[0093] If the distortion between the degraded image I_M0D and the reference image I_REF is less than a threshold, the modified value of the current bit B_b is stored during a step 225. We can then optionally move to the next bit B_b+1 of the current quantified descriptive information P_p_q and iterate steps 221 to 224.

[0094] If the distortion between the degraded image I_M0D and the reference image I_REF is greater than or equal to a threshold, the current bit B_b is reset to its initial value during a step 226. The iterative process implemented on the bits of the current quantified descriptive information P_p_q can then be stopped.

[0095] In other words, the iterative process on the bits of the current quantized descriptive information is implemented as long as the distortion between the degraded image I_M0D and the reference image I_REF is less than a threshold, for example, a value of PSNR or SSIM. For example, such an iterative process is implemented by starting with the least significant bit, and then considering the subsequent bits in ascending order of their significance.

[0096] The output of step 22 is obtained the current optimized quantified descriptive information P_p_q*.

[0097] We can then possibly move on to the next descriptive information P_p+1 and iterate steps 21 and 22.

[0098] For example, steps 21 and 22 are iterated for all the descriptive information of the current object O_i. Alternatively, steps 21 and 22 are iterated for only part of the descriptive information of the current object O_i.

[0099] The quantization of the current object O_i at least is optimized as described above for the 3D scene S_t. For example, all elementary objects of the 3D scene S_t are quantized and their quantization optimized, as described above. Alternatively, only a subset of the elementary 3D objects is quantized, and the quantization is optimized, as described above. Specifically, if several elementary 3D objects have identical descriptive information / properties, it is possible to process the plurality of elementary 3D objects simultaneously, rather than each object individually.

[0100] Compression or coding, for example entropic, of the optimized quantified descriptive information(s) can then be implemented.

[0101] At the end of these various steps, a signal is obtained that represents at least one 3D scene composed of a plurality of elementary 3D objects, intended to be transmitted and / or stored. Such a signal carries at least one optimized quantified descriptive information for at least one of said elementary 3D objects.

[0102] The main steps of a reconstruction process according to an embodiment of the invention are now described in relation to [Fig.3].

[0103] In a first step 31, such a method reads or receives a signal carrying at least one optimized quantified descriptive information of at least one elementary 3D object, said optimized quantified descriptive information being obtained by modifying the value of at least one bit of the quantified descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantified descriptive information, and a reference image representative of said scene is less than a threshold.

[0104] For example, the reconstruction process includes obtaining a first optimized quantified descriptive information P_l_q* and a p-th optimized quantified descriptive information P_p_q* of a current elementary 3D object.

[0105] Such a signal is for example generated by implementing the compression process described in relation to [Fig.2].

[0106] During a second step 32, at least one elementary 3D object is reconstructed from said at least one optimized quantified descriptive information.

[0107] For example, if the optimized quantified descriptive information has been coded before transmission and / or storage, the reconstruction step 32 implements a decompression or decoding, for example entropic, of the coded optimized quantified descriptive information.

[0108] As with the compression process, the steps of the reconstruction process can be implemented for at least one descriptive information associated with an elementary 3D object (for example, for all descriptive information associated with the object), for at least one elementary 3D object of a 3D scene (for example, for all objects of the 3D scene), for the reconstruction of at least one 3D scene composed of the elementary 3D objects.

[0109] In particular, such a reconstruction process can be implemented by equipment with limited resources in terms of RAM, CPU and / or GPU, for example a mobile terminal such as a "smartphone".

[0110] Such a process thus includes a step of rendering, or displaying, a 3D scene, or a succession of 3D scenes thus reconstructed.

[0111] It is noted that certain steps of the compression process or of the reconstruction process according to an embodiment, such as the step of obtaining a reference image, can be carried out in a different order than that indicated in the claims, or even in parallel.

[0112] 5.2 Implementation example

[0113] An example of an implementation of the invention for the compression and reconstruction of at least one 3D scene composed of a plurality of elementary 3D objects of the Gaussian 3D splat type is presented below. Of course, other elementary 3D objects can be considered, such as non-Gaussian splats, or polygons, edges and vertices of a mesh, etc.

[0114] In the implementation example described below, the aim is to reduce the value space of the quantized descriptive information, or properties, of the splats by controlling the modification of the value of one or more least significant bits of at least one quantized descriptive information of the splat, for example by forcing it to 0, while taking into account a PSNR-type distortion measure. The aim is thus to optimize the efficiency of a global entropy encoding.

[0115] In particular, the descriptive information of a splat is quantized to an integer number of bytes, resulting in a data structure that can be efficiently used by a graphics processing unit. Thus, the data compressed according to this implementation example reflects the format used by the GPU, minimizing processing and contributing to the real-time decompression / rendering of at least one 3D scene.

[0116] The compression algorithm according to this implementation example is illustrated in [Fig.4] and described below.

[0117] 5.2.1 Quantification

[0118] The main steps for implementing a quantification of at least one descriptive information of at least one splat composing at least one 3D scene according to a particular embodiment are described below.

[0119] We thus consider a set of splats 41, as illustrated in [Fig.4].

[0120] Among this set of splats 41, we consider a first splat, called the current splat, described by at least one descriptive piece of information, hereafter called a property.

[0121] We consider a first property p = P _ 142 of the current splat, called the current property.

[0122] The value of the current property p 42 is quantized and stored in one or more bytes during a step 43. To do this, the minimum (nùn) and maximum possible values ​​of the property in question are calculated, for example. The quantized value 0 in the space of size sPace is obtained by applying the following equation:

[0123] V max-min A

[0124] For example, the space of size space corresponds to 256 values ​​if we consider a quantization of a property on one byte, 65536 values ​​if we consider a quantization on two bytes, etc.

[0125] As indicated in Table 1 cited in prior art, a splat can be described by different properties.

[0126] For a property of the center position or scale factor type, quantization can be implemented on two bytes, that is, on a value from an sPace containing 65536 values. Indeed, the required precision can be significant because precise values ​​are encoded within an interval of several meters.

[0127] For example, two bytes are used for the quantification of the center position in x, two bytes are used for the quantification of the center position in y, two bytes are used for the quantification of the center position in z, two bytes are used for the quantification of the scale factor in x, two bytes are used for the quantification of the scale factor in y, and two bytes are used for the quantification of the scale factor in z. Twelve bytes can thus be provided for the quantification of center position or scale factor type properties.

[0128] For a property of the color, density, or rotation factor (quaternion) type, quantization can be implemented on a single byte, that is, on a value in an sSpace containing 256 values. Indeed, the required precision is less important for these properties, and quantization on a single byte is sufficient.

[0129] For example, one byte is used for the quantification of the color red, one byte is used for the quantification of the color green, one byte is used for the quantification of the color blue, one byte is used for the quantification of the density, one byte is used for the quantification of the rotation factor in x, one byte is used for the quantification of the rotation factor in y, one byte is used for the quantification of the rotation factor in z, and one byte is used for the quantification of the rotation angle. Eight bytes can thus be allocated for the quantification of properties such as color, density, or rotation factor.

[0130] Finally, it is possible to suppress spherical harmonics to avoid having to quantize, transmit, and / or store these properties. Indeed, the non-transmission of spherical harmonics has little impact on the final rendering. Moreover, during reconstruction From the 3D scene from the different images, it is possible to estimate the color resulting from the projection of spherical harmonics along a given angle, typically the direction of the gaze in front of the 3D scene, i.e. x, jfz = ( 0,0, -1 ).

[0131] After quantifying at least one property from those listed in Table 1, we obtain a fixed-size group of bytes.

[0132] If we consider all the properties in Table 1, except for the spherical harmonics, we obtain a fixed-size group of bytes (for example, 20 bytes) after quantizing the different properties of a splat, allowing us to describe a complete splat. The amount of data to be encoded and transmitted is thus reduced compared to the prior art, according to which each property is quantized on a 16-bit floating-point number ("Floatlô").

[0133] In particular, a conventional table listing the quantization information, i.e., the minimum (min) and maximum values ​​of the properties concerned, or the minimum value and size of the properties concerned, can be transmitted and / or stored. For example, each value is a 16-bit floating-point number. The size of such a table is negligible compared to that of the splats.

[0134] Such a quantification step 43 can be implemented for at least one descriptive property of a splat, for example for each property of the splat, and for at least one splat of a 3D scene at a time instant, for example for all the splats 41 composing the 3D scene.

[0135] 5.2.2 Optimization of quantification

[0136] In order to improve the efficiency of the entropic compression of the quantified properties, the invention proposes to optimize the quantification by performing a controlled reduction of the spa™ space

[0137] In particular, according to the implementation example described here, the value distribution space is reduced by forcing the least significant bits of a quantized property to 0, as long as the distortion between a reference image and a degraded image resulting from setting the least significant bits to zero remains acceptable, i.e., below a PSNR-type threshold. This contributes to improving the efficiency of global entropy encoding over the entire dataset.

[0138] Indeed, if we consider quantization on one byte, or 8 bits, the space of possible values ​​comprises 256 values, as indicated above. If the least significant bit is forced to 0, the space of values ​​is reduced to 128 values. If two of the least significant bits are forced to 0, the space of values ​​is reduced to 64 values. The quantization space can thus be reduced according to the composition of the scene.

[0139] In particular, a first rough estimate of the number of bits that can be forced to zero can be implemented beforehand.

[0140] It is also noted that the higher the number of bits equal to 0, the lower the number of possible values, and the more efficient the entropy coding.

[0141] Thus, returning to Figure 4, the following steps are applied for at least the current quantized property p obtained at the output of the quantization step 43.

[0142] In the example considered, the current property p is quantized over an integer number of bytes, each comprising 8 bits.

[0143] We consider a current bit 6, corresponding to the first bit of least weight for the first iteration.

[0144] In a step 44, a reference image I_REF is calculated from the current property p quantified.

[0145] During a step 45, the current bit b (corresponding to the first least significant bit of the current quantized property for the first iteration) is forced to zero, and then during a step 46 a new image I_M0D obtained from the current modified quantized property is calculated.

[0146] According to a first example, the reference image I_REF is obtained from the current quantized property p, before any zeroing of the least significant bits. In this case, the same reference image can be used throughout the entire quantization optimization process of a current property, and possibly even of a subsequent property. If the same reference image is used in all iterations of quantization optimization of a current property, the number of bits forced to zero is lower, but the final rendering is of better quality because there is less deviation. According to a second example, the reference image is obtained from the current quantized property p, taking into account the bits forced to 0 during a previous iteration on the bits of the current quantized property. According to this variant, the computational errors related to space reduction are accumulated.This variant allows for improved compression, since the number of bits actually forced to zero taking into account the reference image according to the second example is generally greater than the number of bits actually forced to zero taking into account the reference image according to the first example, while maintaining an acceptable perceived quality according to the PSNR value.

[0147] The reference image I_REF and the degraded image I_M0D are then compared in step 47, which makes it possible to measure the degradation in quality between the two images I_REF and I_M0D. For example, the distortion between the images I_REF and I_M0D is measured.

[0148] If the resulting distortion is less than (471) a PSNR-type threshold, for example, then we move on to the next bit of the current quantized property (corresponding to the second least significant bit for the second iteration) and repeat steps 45, 46, and 47. Step 44 can also be repeated if the reference image for the bit the following is calculated from the current quantized property with the modified value of the previous bit (i.e. obtained in a previous iteration).

[0149] On the other hand, if the distortion obtained is greater than or equal to (472) the threshold, then the current bit is reset to its initial value "1" during a step 48 and the iterative process on the bits of the current quantized property is stopped.

[0150] In the example illustrated in [Fig.4], this treatment is applied to the different properties of the splat.

[0151] Thus, during a step 49, it is checked whether the current property p is the last property associated with the current splat.

[0152] If the current property p is not the last property associated with the current splat, then we move on to the next property p - p + 1, and we iterate, for the new quantified current property, the steps described above.

[0153] If the current property p is the last property associated with the current splat, then in a step 491 the modified quantified properties associated with the current splat are recorded and it is possible to move on to the next splat in the set of splats 41.

[0154] These steps can be repeated for each of the tens of thousands of splats composing a 3D scene.

[0155] The result is a group of bytes with minimal entropy, while maintaining a good level of quality.

[0156] The table below illustrates an example of a 20-byte group resulting from setting the least significant bits of the quantized values ​​of the different properties of a splat to zero, with: • The first line illustrates the two bytes used for quantifying the x-position of the center, • The second line illustrates the two bytes used for quantifying the center's y-position, • The third line illustrates the two bytes used for quantifying the center's z-position, • The fourth line illustrates a first byte used for quantizing the color red and a second byte used for quantizing the color green. • The fifth line illustrates a first byte used for quantifying the blue color and a second byte used for quantifying the density. • The sixth line illustrates the two bytes used for quantifying the x-scaling factor, • The seventh line illustrates the two bytes used for quantifying the y-scaling factor, The eighth line illustrates the two bytes used for quantifying the z-scaling factor, The ninth line illustrates a first byte used for quantifying the rotation factor in x and a second byte used for quantifying the rotation factor in y. The tenth line illustrates a first byte used for quantifying the rotation factor in z and a second byte used for quantifying the rotation angle.

[0157]

[0158] This gives a compact and almost complete representation of the splat. [Tables2] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

[0159] The implementation example illustrated in [Fig.4] thus makes it possible to optimize the number of bits forced to zero while guaranteeing a minimum quality for the reconstruction of the 3D scene.

[0160] As previously stated, the various optimized quantified properties can be stored and / or transmitted.

[0161] In particular, the optimized quantized property or properties associated with at least one splat of a 3D scene can be encoded or compressed by an entropy coder before transmission. Thus, once the reduced data of all the splats has been obtained and stored in a buffer, for example, global entropy compression can be applied to them.

[0162] It is notably possible to encode / compress the data corresponding to the optimized quantized properties byte by byte, or property by property. However, such processing requires more memory manipulation. Thus, according to a particular embodiment, the data corresponding to all the optimized quantized properties associated with a splat are encoded block by block (for example per block of 20 bytes). Alternatively, the data corresponding to the set of optimized quantized properties associated with several splats can be encoded together.

[0163] In particular, a compression technique is chosen that allows for rapid decompression, enabling, for example, playback on a terminal with limited resources in terms of RAM, CPU and / or GPU.

[0164] 5.2.3 Restitution

[0165] As illustrated in [Fig.5], different steps are implemented for the reconstruction of at least one 3D scene composed of a plurality of elementary 3D objects, such as splats.

[0166] The compressed / encoded optimized quantized properties of at least one splat are obtained by a reconstruction device, for example a mobile terminal, and stored in a buffer during step 51. This step may activate RAM. This data is then decompressed during step 52 and may be stored in a GPU buffer. This step 52 may activate a CPU. The decompressed data is then processed by the GPU during step 53 to reconstruct said at least one splat. This step may activate the GPU.

[0167] In particular, due to quantization over an integer number of bytes and entropic compression applied to this integer number of bytes, decompression makes it possible to obtain decompressed data in a format that can be used directly and efficiently by the GPU, thereby reducing the processing time on the reconstruction device side. In a particular embodiment, real-time decompression can be implemented while maintaining a structure directly usable by GPUs.

[0168] The table below presents the compression and execution time performance for different examples of compression / decompression techniques that can be implemented according to the invention.

[0169] [Tables3] Technical Specifications Initial Data Size (bytes) Compressed Data Size (bytes) Compression Ratio Compression Time (s) Decompression Time (s) Frequency per Second (decompression) LZFSE 23469929 12721901 1.84 2.68 0.0262 38 LZ4 23469929 17649272 1.33 0.05 0.0087 114 LZMA 23469929 9550484 2.46 7.00 0.4502 2 ZLIB 23469929 12575977 1.87 0.62 0.0784 12 LZBITM AP 23469929 14726168 1.59 0.19 0.0124 80

[0170] It is observed that the LZMA method gives the best results in terms of compression efficiency but that the data decompression times are very long.

[0171] Conversely, the LZ4 method is very effective during decompression but less effective in terms of compression.

[0172] The LZFSE method offers a good compromise between compression efficiency and decompression time. Indeed, its compression ratio is close to 2, while allowing for real-time decompression.

[0173] Thus, the proposed solution allows, according to at least one embodiment, to decompress and display a 3D scene in real time, for example on a mobile terminal.

[0174] 5.3 Advantages

[0175] Such compression and / or reconstruction processes according to at least one embodiment of the invention offer certain advantages, for example among those listed below: • the efficiency of the compression process can reach a compression factor or ratio equal to 20, • The losses in quality related to compression can be limited by controlling the level of distortion / degradation, • Compression can be optimized for each scene, • Decompression can be implemented in real time on a mobile terminal, • Decompression can produce a structure directly usable by GPUs. • Decompression can be performed in parallel with rendering; a scene n can be displayed while a scene n+1 is being decompressed.

[0176] Such parallel processing is illustrated for example in [Fig.6].

[0177] Thus, the optimized compressed / coded quantized properties of at least one splat of a first scene are received by a reconstruction device at time t=Os and stored in a buffer during a step 611.

[0178] The optimized compressed / encoded quantized properties of at least one splat of a second scene are received by the reconstruction device at time t=0.03s and stored in a buffer during a step 612. In parallel, the representative data of the first scene are decompressed during a step 621, and can be stored in a GPU buffer.

[0179] The optimized compressed / encoded quantized properties of at least one splat of a third scene are received by the reconstruction device at time t=0.06s and stored in a buffer during a step 613. In parallel, the representative data of the second scene are decompressed during a step 622, and can be stored in a GPU buffer, and the decompressed data of the first scene are processed by the GPU during a step 631 to reconstruct the first scene.

[0180] The optimized compressed / encoded quantized properties of at least one splat of a fourth scene are received by the reconstruction device at time t=0.1 s and stored in a buffer during a step 614. In parallel, the representative data of the third scene are decompressed during a step 623, and can be stored in a GPU buffer, and the decompressed data of the second scene are processed by the GPU during a step 632 to reconstruct the second scene, and so on.

[0181] In other words, the proposed compression and reconstruction processes allow, according to at least one embodiment, for sufficiently efficient reading of data from a storage volume or network to allow for decompression and display in real time, for example on a mobile terminal.

[0182] It is therefore possible to create volumetric videos and transmit them via streaming, for example with bitrates around 100 Mbs.

[0183] 5.4 Compression and reconstruction devices

[0184] Finally, in relation to figures 7 and 8, the simplified structures of a compression device and a reconstruction device according to at least one embodiment of the invention are presented.

[0185] As illustrated in [Fig.7], a compression device comprises at least one memory 71, at least one processing unit 72, equipped for example with a programmable computing machine or a dedicated computing machine, for example a processor P, and controlled by the computer program 73, implementing the steps of the compression process according to at least one embodiment of the invention.

[0186] At initialization, the code instructions of program 73 are for example loaded into a RAM memory before being executed by the processor of the processing unit 72.

[0187] The processor of the processing unit 72 implements steps of the compression process described above on at least one multidimensional scene, according to the instructions of the computer program 73, to generate optimized quantified descriptive information.

[0188] As illustrated in [Fig. 8], a reconstruction device comprises at least one memory 81, at least one processing unit 82, equipped for example with a machine programmable computing or a dedicated computing machine, for example a processor P, and controlled by the computer program 83, implementing the steps of the reconstruction process according to at least one embodiment of the invention.

[0189] At initialization, the code instructions of program 83 are for example loaded into a RAM memory before being executed by the processor of the processing unit 82.

[0190] The processor of the processing unit 82 implements steps of the reconstruction process described above on the optimized quantified descriptive information, according to the instructions of the computer program 83, to reconstruct at least one multidimensional scene.

Claims

Demands

1. A compression method for at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing, for at least one of said elementary objects, referred to as the current object: for at least one descriptive piece of information of said current object, referred to as the current descriptive information: • quantization (21) of said current descriptive information over an integer number of bytes, • optimization (22) of the quantization of said current descriptive information comprising, for at least one bit of said current quantized descriptive information, referred to as the current bit: • obtaining (221) a reference image representative of said scene, • modification (222) of a value of said current bit, relative to an initial value, delivering a modified current quantized descriptive information, • obtaining (223) a degraded image representative of said scene,from said current modified quantified descriptive information, • measurement (224) of a distortion between said degraded and reference images, • if said distortion between said degraded and reference images is less than a threshold, storage (225) of said modified value of said current bit, otherwise reset to said initial value of said current bit.

2. Method according to claim 1, characterized in that said optimization of the quantization of said current descriptive information is iterated over the bits of said current quantified descriptive information as long as said distortion is below said threshold.

3. A method according to any one of claims 1 and 2, characterized in that said optimization of the quantification of said Current descriptive information is implemented starting with the least significant bit.

4. A method according to any one of claims 1 to 3, characterized in that said threshold is a peak signal-to-noise ratio threshold.

5. A method according to any one of claims 1 to 4, characterized in that said at least one elementary object is a Gaussian splat.

6. A method according to any one of claims 1 to 5, characterized in that said at least one descriptive piece of information of said current object belongs to the group comprising: • a position of the center of said current object, • a color associated with said current object, • a density associated with said current object, • a scale factor associated with said current object, • a rotation factor associated with said current object.

7. Method according to claim 6, characterized in that for a current descriptive information of the type center position or scale factor, said quantification of said current descriptive information is implemented on two bytes.

8. Method according to claim 6, characterized in that for a current descriptive information of the type colour, density or rotation factor, said quantification of said current descriptive information is implemented on one byte.

9. Method according to claim 2, characterized in that said reference image is obtained from said current modified quantified descriptive information, taking into account the modified value of at least one bit preceding said current bit, except for the first iteration.

10. A method according to any one of claims 1 to 9, characterized in that it comprises an entropic coding step of all the optimized quantified descriptive information associated with said current object.

11. A method for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing: • obtaining (31) at least one optimized quantified descriptive information of at least one elementary object, said optimized quantified descriptive information being obtained

12.

13. by modifying the value of at least one bit of the quantified descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantified descriptive information, and a reference image representative of said scene is less than a threshold, • reconstruction (32) of said at least one elementary object from said at least one optimized quantified descriptive information. Reconstruction method comprising displaying said at least one scene on a mobile terminal. A compression device for at least one multidimensional scene composed of a plurality of multidimensional elementary objects, comprising at least one processing unit configured for, for at least one of said elementary objects, referred to as the current object: for at least one descriptive piece of information for said current object, referred to as the current descriptive information: • quantify said current descriptive information over an integer number of bytes, • optimize the quantization of said current descriptive information, including, for at least one bit of said current quantified descriptive information, called the current bit: • obtain a reference image representative of said scene, • modify a value of said current bit, relative to an initial value, providing a modified current quantified descriptive information, • obtain a representative degraded image of said scene, from said current modified quantified descriptive information, • measure the distortion between said degraded and reference images, • if said distortion between said degraded and reference images is less than a threshold, store the modified value of said current bit, otherwise reset said current bit to said initial value.

14. A device for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, comprising at least one processing unit configured to: • obtain at least one optimized quantized descriptive information of at least one elementary object, said optimized quantized descriptive information being obtained by modifying the value of at least one bit of the quantized descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantized descriptive information, and a reference image representative of said scene is less than a threshold, • reconstruct said at least one elementary object from said at least one optimized quantized descriptive information.

15. Computer program comprising instructions for carrying out a method according to any one of claims 1 to 12 when such program is executed by a processor.

Images