Methods and devices for compressing and reconstructing at least one multidimensional scene composed of a plurality of elementary objects and corresponding computer program

Abstract

The invention relates to a method for compressing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, which implements the following, for at least one of said elementary objects, called the current object: for at least one piece of descriptive information relating to said current object, called the current descriptive information: quantising (21) the current descriptive information over an integer number of bytes, and optimising (22) the quantisation of said current descriptive information.

Description

Methods and devices for compressing and reconstructing at least one multidimensional scene composed of a plurality of elementary objects and corresponding computer program.

[0001] 1. Scope of the invention

[0002] The field of the invention is that of the encoding and decoding of digital signals. More specifically, the invention relates to the compression of data representing at least one multidimensional scene, for example for the purpose of their transmission and / or storage, and the reconstruction of such scenes.

[0003] In particular, the invention proposes a solution enabling real-time decompression and / or rendering of at least one multidimensional scene on equipment with limited resources in terms of RAM (Random Access Memory), CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit), for example on a mobile terminal such as a smartphone.

[0004] The proposed solution has applications in all areas where having multiple points of view can be of interest, such as training, sports, entertainment, maintenance, etc.

[0005] 2. Prior art

[0006] Numerous solutions have been proposed for reconstructing a 3D scene (also called a 3D model) from photos taken from different viewpoints. When this reconstruction is obtained from video content, by taking the same time moment from each video clip, a temporal sequence of 3D scenes is obtained, also called 3D video or volumetric video, which can be viewed from any point of view.

[0007] There are several approaches to representing volumetric content (3D scenes or 3D videos). We can distinguish two main approaches: one based on an explicit representation and the other on an implicit representation of a 3D scene at a given moment.

[0008] An explicit approach represents a 3D scene using geometric primitives (volumes, surface elements, voxels, etc.) and then uses image synthesis techniques to render an image of this scene associated with a viewpoint.

[0009] An implicit approach does not explicitly represent the 3D scene. Examples include approaches that define a 3D map of distances to objects in the scene (e.g., "Signed Distance Function"), or a 3D map of the opacity and colorimetry properties of different 3D positions in the scene.

[0010] Recently, Kerbl et al. presented a technique for representing a 3D scene as a set of 3D Gaussian primitives, which can be projected onto the camera's image plane. These projections are called "splat" or "Gaussian splat." This technique, known as "3D Gaussian Splatting" or 3DGS, is notably presented in the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering," SIGGRAPH 2023. In simplified terms, a point cloud is generated from input images corresponding to different viewpoints. Each point is then converted into a Gaussian that can be split, cloned, deleted, etc. The resulting 3D splat can be considered a spherical object possessing spatial, geometric, and colorimetric properties. The properties of a 3D splat can be described by a set of parameters or descriptive information, an example of which is shown in the table below:

[0011] Position of the splat center (x, y, z) Color: Red, green, blue Density or transparency: alpha Spherical harmonics: sr0, sg0, sb0, sr1, sb1, sg1, ... Scale factor: x, sy, sz Rotation factor: qx, qy, qz, angle

[0012] Combinations of these properties can produce a wide variety of visual representations. For example, it is estimated that a few tens of thousands of splats can represent a 3D scene in a photorealistic way.

[0013] Reconstructing a 3D video at a specific point in time, referred to here as a "frame," using a technique like "3D Gaussian Splatting" offers good visual quality but relies on processing very large amounts of data. For example, even if only the central subject of a video is extracted, frames on the order of 30 MB (30,000,000 bytes) are obtained from multiple video clips, using the same point in time from each video. Reconstructing a sequence of frames from videos, for example, reconstructing a series of 30 3D scenes per second, requires a bandwidth of around 7 Gbps (gigabits per second).

[0014] It is necessary to compress such a large amount of data in order to consider a transfer to a playback device, especially if one considers a wireless transmission technique (for example, Wifi®, 5G or other).

[0015] It is also necessary to quickly decompress and restore such a large amount of data.

[0016] Current point cloud compression techniques include quantization and the use of coding alphabets, or "codebooks," which partition the space of possible values ​​and use the index of this code in place of the actual values ​​within a sequence of bits of arbitrary size. During decoding, the bitstream is manipulated bit by bit to reconstruct the codebooks, which, via the index, allow the initial values ​​to be reconstructed and displayed on the GPU.

[0017] Laillustre an example of a classic processing chain implemented for the decompression / restoration of an image.

[0018] In the first step (11), the compressed data is read. This step may activate RAM. In the second step (12), the compressed data is stored in a CPU buffer. This step may activate RAM. In the third step (13), the codebooks are decoded and the data is decompressed. This step may activate the CPU. In the fourth step (14), the decompressed data is transformed into a format suitable for display by the GPU. This step may activate the CPU. In the fifth step, the transformed data is stored in a CPU buffer. This step may activate RAM. Finally, the transformed data is processed by the GPU for display. This step may activate the GPU.

[0019] Thus, each image, once read, is decompressed and then transformed into a format suitable for display by the GPU of the playback equipment. This process results in a significant amount of memory transfer at each stage, as well as numerous processing operations by the CPU and then the GPU.

[0020] To manage the transport and display of a video, this process can be repeated on the order of 30 times per second.

[0021] However, with "3D Gaussian Splatting" type techniques, it is currently not possible to perform all these operations in real time on a mobile terminal without dedicated specific hardware acceleration, as exists for certain types of videos.

[0022] Currently, only still images or very short sequences can be displayed in real time on a mobile device. To achieve this, after an initial decoding / preparation phase, which is not real-time, the display can be performed in real time, either on an image prepared for the GPU or on a sequence of prepared images. The size of this sequence obviously depends on the amount of RAM available on the mobile device, but remains limited to a few seconds at most.

[0023] However, when we want to process a video of arbitrary size, in streaming mode, we seek to perform all these operations in real time, for example in less than 33 milliseconds per image to reach a frequency of 30 frames per second.

[0024] Therefore, although "3D Gaussian Splatting" type techniques allow for the reconstruction of very realistic 3D scenes from multiple viewpoints, current compression techniques do not allow for the streaming of a time series of 3D scenes on mobile devices, due to the size of the data and the complexity of decoding and rendering.

[0025] There is therefore a need for a solution that allows for the efficient compression of data representing at least one 3D scene, for example in order to be able to transfer them via a wireless network (streaming) and / or display them in real time (for example at a frequency of around 30 frames per second), particularly on a mobile terminal such as a smartphone.

[0026] 3. Description of the invention

[0027] The invention proposes a solution in the form of a compression method for at least one multi-dimensional scene composed of a plurality of multi-dimensional elementary objects, implementing, for at least one of said elementary objects, called the current object:

[0028] for at least one descriptive information of said current object, called current descriptive information: quantification of said current descriptive information over an integer number of bytes, optimization of the quantization of said current descriptive information including, for at least one bit of said current quantized descriptive information, called current bit: obtaining a reference image representative of said scene, modification of a value of said current bit, relative to an initial value, delivering a current modified quantized descriptive information, obtaining a degraded image representative of said scene, from said current modified quantized descriptive information, measurement of a distortion between said degraded and reference images, if said distortion between said degraded and reference images is less than a threshold, storage of said modified value of said current bit, otherwise reset to said initial value of said current bit.

[0029] The invention thus proposes a new solution for compressing volumetric content, for example (a 3D scene, or a 3D video composed of a succession of 3D scenes), consisting of a plurality of elementary objects. For example, such elementary objects are non-Gaussian splats, or Gaussian splats, or polygons, edges and vertices of a 3D mesh…

[0030] In particular, the proposed solution is based on optimizing the quantification of descriptive information for at least one object, by supervisedly reducing the quantification space for descriptive information.

[0031] In at least one embodiment, such a reduction in the data quantization space enables fast entropic decompression. Furthermore, the modified quantized descriptive information retains a structure that can be efficiently processed by the GPU since it is quantized over an integer number of bytes.

[0032] According to at least one embodiment, the proposed technique makes it possible to compress at least one multidimensional scene sufficiently to ensure streaming of the scene or time sequences over a wireless network, such as a Wifi® or 5G network or any other network, while guaranteeing real-time processing and display compatible with the capabilities of a mobile terminal such as a smartphone.

[0033] It offers a compromise between compression performance and speed of execution.

[0034] In a particular embodiment, said at least one elementary object is a Gaussian splat and the proposed solution makes it possible to compress a 3D scene composed of 3D splats efficiently in terms of compression gain, decompression time and / or 3D rendering speed by the GPU.

[0035] In a particular embodiment, said optimization of the quantization of said current descriptive information is iterated over the bits of said current quantified descriptive information as long as said distortion is below said threshold.

[0036] Thus, for at least one descriptive piece of information for at least one elementary object, one or more bits of the quantized descriptive information can be modified. This modification is performed in a supervised manner, by checking for each modified "candidate" bit whether the resulting degraded image is sufficiently close to the reference image. If so, the "candidate" modification is accepted. Otherwise, the "candidate" modification is rejected. The aim is to maintain a high-quality rendering of a multidimensional scene while reducing the data quantization space.

[0037] In one particular embodiment, the optimization of the quantization of the current descriptive information is implemented by starting with a least significant bit with a non-zero value. The process begins by modifying the value of this least significant bit and checking whether the resulting degraded image is sufficiently close to the reference image. If so, the "candidate" modification is accepted, and the process moves to the next bit (i.e., the second least significant bit with a non-zero value) in a subsequent iteration. Otherwise, the "candidate" modification is rejected, and the iterative process terminates. This is because modifying a least significant bit has less impact on the resulting degraded image. The process can then proceed to optimize the quantization of another descriptive piece of information, quantized to at least one byte.

[0038] In particular, the least significant bit(s) are forced to a value of 0, as long as the resulting degraded image is sufficiently close to the reference image. This aims to reduce the number of possible values, which helps improve entropic compression and, consequently, entropic decompression.

[0039] In another embodiment, the optimization of the quantization of the current descriptive information is implemented by starting with any non-zero bit of the quantized current descriptive information, called the current bit. The value of this current bit is then modified, and it is checked whether the resulting degraded image is sufficiently close to the reference image.

[0040] If so, the "candidate" modification is accepted, and all bits with a lower weight than the current bit are forced to zero. Indeed, if any bit of the current quantized descriptive information can be forced to zero without generating too much distortion, it means that all bits with a lower weight than the current bit can also be forced to zero. We can then move on to the next non-zero bit (i.e., a bit with a higher weight than the current bit) during a subsequent iteration.

[0041] Otherwise, the "candidate" modification is rejected. We can then revert to a bit with a zero value and a weight less than the current bit in a subsequent iteration.

[0042] For example, we use a bisection method to determine which bits can be forced to zero.

[0043] In one particular embodiment, the threshold is a peak-to-noise ratio threshold. Such a peak-to-noise ratio, or PSNR (Peak Signal to Noise Ratio), allows for a simple and efficient measurement of the difference between two images.

[0044] For example, such a PSNR is between 20dB and 80dB, and in particular between 30dB and 50dB.

[0045] Other techniques are possible to measure the distortion between two images, such as the SSIM (“structural similarity index measure”).

[0046] In a particular embodiment, said at least one descriptive information of said current object belongs to the group comprising: a position of the center of said current object, a color associated with said current object, a density associated with said current object, a scale factor associated with said current object, a rotation factor associated with said current object.

[0047] If the current object is a 3D splat, one embodiment considers it unnecessary to quantify and transmit descriptive information such as spherical harmonics. Indeed, such spherical harmonics can be omitted for processing by the GPU. In this way, the amount of data transmitted is further reduced.

[0048] In a particular embodiment, for a common descriptive piece of information such as center position or scale factor, the quantification of said descriptive information is implemented using two bytes. For example, two bytes are used for the quantification of the center position in x, two bytes are used for the quantification of the center position in y, two bytes are used for the quantification of the center position in z, two bytes are used for the quantification of the scale factor in x, two bytes are used for the quantification of the scale factor in y, and two bytes are used for the quantification of the scale factor in z. Twelve bytes can thus be allocated for the quantification of descriptive information such as center position or scale factor.

[0049] In one particular embodiment, for a common descriptive information such as color, density, or rotation factor, the quantification of said descriptive information is implemented using one byte. For example, one byte is used for the quantification of the color red, one byte for the quantification of the color green, one byte for the quantification of the color blue, one byte for the quantification of the density, one byte for the quantification of the rotation factor in the x-axis, one byte for the quantification of the rotation factor in the y-axis, one byte for the quantification of the rotation factor in the z-axis, and one byte for the quantification of the rotation angle. Eight bytes can thus be allocated for the quantification of descriptive information such as color, density, or rotation factor.

[0050] These examples demonstrate that quantifying the various descriptive details of an elementary object results in a fixed-size group of bytes (e.g., 20 bytes). This reduces the amount of data to be encoded and transmitted compared to previous methods.

[0051] In one particular embodiment, the reference image is obtained from the current quantized descriptive information before any modification of the value of at least one bit. In this way, the same reference image is used during the different iterations on the bits of the current quantized descriptive information. The number of bits modified using this approach is lower, but the final rendering is of higher quality.

[0052] Alternatively, the reference image is obtained from the current, modified, quantized descriptive information, taking into account the modified value of at least one bit preceding the current bit, except for the first iteration. The number of bits modified using this second approach is higher, but the final rendering quality is slightly degraded.

[0053] In one embodiment, the compression process includes an entropy coding step of said at least one optimized quantified descriptive information, and for example of all the optimized quantified descriptive information associated with said current object.

[0054] For example, such an entropy coding belongs to the group including: an LZFSE type coding (“Lempel-Ziv Finite State Entropy”), an LZ4 type coding (“Lempel-Ziv 4”), an LZMA type coding (“Lempel-Ziv Markov chain algorithm”), a ZLIB type coding, an LZBITMAP type coding (“Lempel-Ziv Bitmap”).

[0055] In one embodiment, the compression process includes a step of storing and / or transmitting said at least one optimized quantified descriptive information. The optimized quantified descriptive information(s) of at least one elementary object, obtained as output from the compression process, may in particular be stored (in a file, memory, etc.) and / or transmitted. They may, in particular, be encoded (for example, by an entropy coder) before transmission and / or storage.

[0056] In particular, they can be streamed over a wireless network.

[0057] In another embodiment, the invention relates to a corresponding compression device.

[0058] Such a compression device is particularly well-suited for implementing the compression process described above. It may, of course, incorporate the various features of the compression process according to the invention, which may be combined or considered individually. Thus, the characteristics and advantages of the compression device are the same as those of the process described above. Consequently, they are not described in further detail.

[0059] Such a compression device is for example integrated into a computer, a tablet, a mobile terminal such as a "smartphone", a headset or virtual or augmented reality glasses, etc.

[0060] It can be a hardware entity or a software entity, which can be distributed across one or more network functions or hosted by one or more hardware devices.

[0061] The invention also relates to a method for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing: obtaining at least one optimized quantified descriptive information of at least one elementary object (for example, reading from a memory, receiving a signal, etc.), said optimized quantified descriptive information being obtained by modifying the value of at least one bit of the quantified descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantified descriptive information, and a reference image representative of said scene is less than a threshold, reconstruction of said at least one elementary object from said at least one optimized quantified descriptive information.

[0062] For example, the reconstruction implements a decompression of said at least one optimized quantified descriptive information (e.g., entropic).

[0063] Such a reconstruction method is particularly well-suited for receiving or reading a signal generated using the compression method described above. It may, of course, incorporate the various features of the compression method according to the invention. Thus, the characteristics and advantages of the reconstruction method are the same as those of the compression method described previously.

[0064] In one particular embodiment, the method includes displaying said at least one multidimensional scene on a mobile terminal. Such a mobile terminal includes a GPU for processing the decompressed data.

[0065] In particular, such a display can be done in real time, for example at a frequency of about 30 frames per second.

[0066] In another embodiment, the invention relates to a corresponding reconstruction device.

[0067] Such a reconstruction device is particularly well-suited to implementing the reconstruction process described above. It may, of course, incorporate the various features of the reconstruction process according to the invention, which may be combined or considered individually. Thus, the features and advantages of the reconstruction device are the same as those of the process described above. Consequently, they are not described in further detail.

[0068] Such a reconstruction device is for example integrated into a mobile terminal such as a smartphone, tablet, PDA, virtual or augmented reality headset or glasses, etc.

[0069] It can be a hardware entity or a software entity, which can be distributed across one or more network functions or hosted by one or more hardware devices.

[0070] The invention further relates to one or more computer programs comprising instructions for implementing a process as described above when this or these programs are executed by at least one processor.

[0071] The invention also relates to a computer-readable information carrier, comprising instructions for a computer program as mentioned above.

[0072] 4. List of figures

[0073] Other features and advantages of the invention will become more apparent upon reading the following description of a particular embodiment, given by way of simple illustrative and non-limiting example, and the accompanying drawings, among which: an example of a conventional processing chain implemented for the decompression / restoration of an image; the main steps implemented by a compression process according to a particular embodiment of the invention; the main steps implemented by a reconstruction process according to a particular embodiment of the invention; an algorithm for reducing the quantization space according to an example of an implementation of the invention; and a processing chain for the decompression / restoration of a 3D scene according to an example of an implementation of the invention.Figure 1 illustrates an example of a parallel processing chain for the decompression / restoration of a 3D scene according to an example of an implementation of the invention; Figure 2 presents the simplified structure of a compression device according to an embodiment of the invention; Figure 3 presents the simplified structure of a reconstruction device according to an embodiment of the invention.

[0074] 5. Description of an embodiment

[0075] 5.1 General Principle

[0076] The general principle of the invention is based on optimizing the quantification of descriptive information, also called properties, of at least one multidimensional elementary object composing a multidimensional scene. This optimization relies in particular on reducing the value space for quantifying the descriptive information of the elementary object. This reduction of the value space improves the compression, or encoding, particularly entropic compression, of the descriptive information of the elementary object.

[0077] Furthermore, the quantification of descriptive information is performed on an integer number of bytes (for example, one byte, two bytes, or more). This makes it possible to limit the processing time for rendering the scene at a given moment, while maintaining a format directly usable by a graphics processing unit.

[0078] We will subsequently consider the multidimensional scene to be a 3D scene and the elementary object to be a 3D object. Other dimensions are conceivable, notably 2D or 4D.

[0079] Laillustrates the main steps of a compression process according to an embodiment of the invention.

[0080] Such a process takes as input at least one 3D scene composed of a plurality of elementary 3D objects. As described in relation to prior art, such a 3D scene can be obtained from photographs captured from different viewpoints, or from videos taken from different viewpoints at the same time. Alternatively, such a 3D scene can be obtained from images generated by a graphics processing unit.

[0081] We consider a 3D scene at a time instant with index t, where t is an integer such that 1 And Such a scene, denoted S_t, is composed of a plurality of elementary 3D objects. For example, the scene S_t is composed of elementary 3D objects O_i, with 1 , And an integer that can be worth several hundred thousand.

[0082] We consider one of these elementary 3D objects O_i, called the current object. Such an object O_i can be represented or described by at least one descriptive piece of information or property P_p, with 1 And .

[0083] We consider one of the said descriptive information P_p, called current descriptive information, associated with the elementary object O_i of the scene S_t.

[0084] In a first step 21, the current descriptive information P_p is quantized over an integer number of bytes, delivering a current quantized descriptive information P_p_q.

[0085] Note that if we consider a quantization of one byte, or 8 bits, the space of possible values ​​comprises 256 values. If we consider a quantization of two bytes, or 16 bits, the space of possible values ​​comprises 65536 values.

[0086] Therefore, if we consider a 1m scene quantized using 256 values, we obtain a precision on the order of 0.004m, or 4mm. Such precision is not necessarily required if we are trying to reconstruct objects in this scene larger than 10cm, for example. We can therefore reduce the quantization space.

[0087] If, for example, the least significant bit of the current quantized descriptive information P_p_q is forced to 0, the value space is reduced to 128 values ​​instead of 256. If we consider a scene of 1m that we seek to quantize on 128 values, we obtain a precision of the order of 0.008m, or 8mm.

[0088] If, for example, the 2 least significant bits of the current quantized descriptive information P_p_q are forced to 0, the value space is reduced to 64 values ​​instead of 256. If we consider a scene of 1m that we seek to quantize on 64 values, we obtain a precision of the order of 0.016m, or 1.6cm.

[0089] This allows us to reduce the quantification space according to the composition of the scene.

[0090] Optionally, a first rough estimate of the number of bits of current quantified descriptive information that can be modified can be implemented.

[0091] Taking into account the above observations, during a second step 22, the current quantified descriptive information P_p_q is optimized, delivering an optimized current quantified descriptive information P_p_q*.

[0092] To do this, for at least one bit B_b of said current quantified descriptive information P_p_q, with And the number of bits used to quantify the current descriptive information P_p (corresponding to an integer number of bytes), called the current bit, optimization step 22 implements at least one iteration of the following steps.

[0093] During step 221, a reference image I_REF representative of the scene S_t is obtained. Such a reference image can be obtained from the current quantized descriptive information P_p_q obtained at the output of quantization step 21, i.e., before any modification of the value of at least one bit. Alternatively, the reference image I_REF is obtained from the modified current quantized descriptive information, obtained at a previous iteration on the bits of the current quantized descriptive information (for example, at the directly preceding iteration or another preceding iteration).

[0094] In a subsequent step 222, the value of the current bit B_b is modified from an initial value. For example, if the initial value of the current bit is 1, the modified value of the current bit is 0. This results in a modified current quantified descriptive information P_p_q_m.

[0095] In a subsequent step 223, a degraded image I_MOD representative of the scene S_t is obtained from the current modified quantified descriptive information P_p_q_m.

[0096] In a subsequent step 224, a distortion is measured between the degraded image I_MOD and the reference image I_REF.

[0097] If the distortion between the degraded image I_MOD and the reference image I_REF is less than a threshold, the modified value of the current bit B_b is stored during a step 225. We can then possibly move to the next bit B_b+1 of the current quantified descriptive information P_p_q and iterate steps 221 to 224.

[0098] If the distortion between the degraded image I_MOD and the reference image I_REF is greater than or equal to a threshold, the current bit B_b is reset to its initial value during a step 226. The iterative process implemented on the bits of the current quantified descriptive information P_p_q can then be stopped.

[0099] In other words, the iterative process on the bits of the current quantized descriptive information is implemented as long as the distortion between the degraded image I_MOD and the reference image I_REF is below a threshold, for example, a PSNR or SSIM value. For example, such an iterative process is implemented by starting with the least significant bit, then considering the subsequent bits in ascending order of their significance.

[0100] The output of step 22 is the current optimized quantified descriptive information P_p_q*.

[0101] We can then possibly move on to the next descriptive information P_p+1 and iterate steps 21 and 22.

[0102] For example, steps 21 and 22 are iterated for all the descriptive information of the current object O_i. Alternatively, steps 21 and 22 are iterated for only part of the descriptive information of the current object O_i.

[0103] The quantization of at least the current object O_i is optimized as described above for the 3D scene S_t. For example, all elementary objects in the 3D scene S_t are quantized, and their quantization is optimized as described above. Alternatively, only a subset of the elementary 3D objects are quantized, and their quantization is optimized as described above. Notably, if several elementary 3D objects have identical descriptive information / properties, it is possible to process the plurality of elementary 3D objects simultaneously, rather than each object individually.

[0104] Compression or coding, for example entropic, of the optimized quantified descriptive information can then be implemented.

[0105] At the end of these various steps, a signal is obtained that represents at least one 3D scene composed of a plurality of elementary 3D objects, intended for transmission and / or storage. Such a signal carries at least one optimized quantified descriptive information for at least one of said elementary 3D objects.

[0106] We now describe, in relation to the, the main steps of a reconstruction process according to an embodiment of the invention.

[0107] In a first step 31, such a process reads or receives a signal carrying at least one optimized quantified descriptive information of at least one elementary 3D object, said optimized quantified descriptive information being obtained by modifying the value of at least one bit of the quantified descriptive information, when the distortion between a degraded image representative of said scene, obtained from the modified quantified descriptive information, and a reference image representative of said scene is less than a threshold.

[0108] For example, the reconstruction process includes obtaining a first optimized quantified descriptive information P_1_q* and a p-th optimized quantified descriptive information P_p_q* of a current elementary 3D object.

[0109] Such a signal is generated, for example, by implementing the compression process described in relation to the.

[0110] In a second step 32, at least one elementary 3D object is reconstructed from said at least one optimized quantified descriptive information.

[0111] For example, if the optimized quantified descriptive information has been coded before transmission and / or storage, the reconstruction step 32 implements a decompression or decoding, for example entropic, of the coded optimized quantified descriptive information.

[0112] As with the compression process, the steps of the reconstruction process can be implemented for at least one descriptive information associated with an elementary 3D object (for example, for all descriptive information associated with the object), for at least one elementary 3D object of a 3D scene (for example, for all objects of the 3D scene), for the reconstruction of at least one 3D scene composed of the elementary 3D objects.

[0113] In particular, such a reconstruction process can be implemented by equipment with limited resources in terms of RAM, CPU and / or GPU, for example a mobile terminal such as a "smartphone".

[0114] Such a process thus includes a step of rendering, or displaying, a 3D scene, or a succession of 3D scenes thus reconstructed.

[0115] It is noted that certain steps of the compression process or the reconstruction process according to an embodiment, such as the step of obtaining a reference image, can be carried out in a different order than that indicated in the claims, or even in parallel.

[0116] 5.2 Example of implementation

[0117] An example of an implementation of the invention for compressing and reconstructing at least one 3D scene composed of a plurality of elementary 3D objects of the Gaussian 3D splat type is presented below. Of course, other elementary 3D objects can be considered, such as non-Gaussian splats, or polygons, edges and vertices of a mesh, etc.

[0118] In the implementation example described below, the goal is to reduce the value space of the quantized descriptive information, or properties, of the splats by controlling the modification of the value of one or more least significant bits of at least one quantized descriptive information of the splat, for example by forcing it to 0, while taking into account a PSNR-type distortion measure. The aim is thus to optimize the efficiency of a global entropy encoding.

[0119] In particular, the descriptive information of a splat is quantized to an integer number of bytes, resulting in a data structure that can be efficiently used by a graphics processing unit. Thus, the data compressed according to this implementation example reflects the format used by the GPU, minimizing processing and contributing to the real-time decompression / rendering of at least one 3D scene.

[0120] The compression algorithm according to this implementation example is illustrated and described below.

[0121] 5.2.1 Quantification

[0122] The main steps for implementing a quantification of at least one descriptive information of at least one splat composing at least one 3D scene according to a particular embodiment are described below.

[0123] We thus consider a set of splats 41, as illustrated in.

[0124] Among this set of splats 41, we consider a first splat, called the current splat, described by at least one descriptive piece of information, hereafter called a property.

[0125] We consider a first property 42 of the current splat, called current property.

[0126] The value of the current property 42 is quantized and stored on one or more bytes during a step 43. To do this, for example, the minimum value ( ) and the maximum value ( ) possible of the property in question. We obtain the quantified value in the size space by applying the following equation:

[0127]

[0128] For example, the size space corresponds to 256 values ​​if we consider a quantization of a property on one byte, 65536 values ​​if we consider a quantization on two bytes, etc.

[0129] As indicated in Table 1 cited in prior art, a splat can be described by different properties.

[0130] For a property such as center position or scale factor, quantization can be implemented on two bytes, that is, on a value of one space comprising 65536 values. Indeed, the necessary precision can be significant because precise values ​​are encoded within an interval of several meters.

[0131] For example, two bytes are used for quantifying the center position in x, two bytes are used for quantifying the center position in y, two bytes are used for quantifying the center position in z, two bytes are used for quantifying the scale factor in x, two bytes are used for quantifying the scale factor in y, and two bytes are used for quantifying the scale factor in z. Twelve bytes can thus be allocated for quantifying properties such as center position or scale factor.

[0132] For a property such as color, density, or rotation factor (quaternion), quantization can be implemented on a byte, that is, on a value in a space comprising 256 values. Indeed, the required precision is less important for these properties and quantization on one byte is sufficient.

[0133] For example, one byte is used to quantify the color red, one byte to quantify the color green, one byte to quantify the color blue, one byte to quantify the density, one byte to quantify the rotation factor in the x-axis, one byte to quantify the rotation factor in the y-axis, one byte to quantify the rotation factor in the z-axis, and one byte to quantify the rotation angle. Eight bytes can thus be allocated for quantifying properties such as color, density, or rotation factor.

[0134] Finally, it is possible to suppress spherical harmonics to avoid having to quantify, transmit, and / or store these properties. Indeed, the non-transmission of spherical harmonics has little impact on the final rendering. Moreover, during the reconstruction of the 3D scene from the different images, it is possible to estimate the color resulting from the projection of spherical harmonics at a given angle, typically the direction of the viewer's gaze in front of the 3D scene. .

[0135] After quantifying at least one property from those listed in Table 1, we obtain a fixed-size group of bytes.

[0136] If we consider all the properties in Table 1, except for the spherical harmonics, we obtain a fixed-size group of bytes (for example, 20 bytes) after quantizing the different properties of a splat, allowing us to describe a complete splat. The amount of data to be encoded and transmitted is thus reduced compared to the previous method, in which each property is quantized to a 16-bit floating-point number ("Float16").

[0137] In particular, a classic table listing quantification information, i.e., minimum values ​​( ) and maximum ( The relevant properties, or their minimum value and size, can be transmitted and / or stored. For example, each value is a 16-bit floating-point number. The size of such a table is negligible compared to that of the splats.

[0138] Such a quantification step 43 can be implemented for at least one descriptive property of a splat, for example for each property of the splat, and for at least one splat of a 3D scene at a time instant, for example for all the splats 41 composing the 3D scene.

[0139] 5.2.2 Optimization of quantification

[0140] In order to improve the efficiency of the entropic compression of the quantized properties, the invention proposes to optimize the quantization by performing a controlled reduction of the space .

[0141] In particular, according to the implementation example described here, the value distribution space is reduced by forcing the least significant bits of a quantized property to 0, as long as the distortion between a reference image and a degraded image resulting from zeroing the least significant bits remains acceptable, i.e., below a PSNR-type threshold. This contributes to improving the efficiency of global entropy encoding across the entire dataset.

[0142] Indeed, if we consider quantization on one byte, or 8 bits, the space of possible values ​​comprises 256 values, as indicated above. If the least significant bit is forced to 0, the space of values ​​is reduced to 128 values. If two of the least significant bits are forced to 0, the space of values ​​is reduced to 64 values. The quantization space can thus be reduced according to the composition of the scene.

[0143] In particular, a first rough estimate of the number of bits that can be forced to zero can be implemented beforehand.

[0144] It is also noted that the higher the number of bits equal to 0, the smaller the number of possible values, and the more efficient the entropy coding.

[0145] Thus, returning to Figure 4, we apply the following steps for at least the current property quantified obtained at the output of the quantification step 43.

[0146] In the example considered, the current property is quantized over an integer number of bytes, each comprising 8 bits.

[0147] We consider a current bit , corresponding to the first least significant bit for the first iteration.

[0148] In step 44, a reference image I_REF is calculated, obtained from the current property. quantified.

[0149] During step 45, the current bit is forced (corresponding to the first least significant bit of the current quantized property for the first iteration) to zero, then during a step 46 a new I_MOD image obtained from the current modified quantized property is calculated.

[0150] According to a first example, the reference image I_REF is obtained from the current property quantized, before any zeroing of the least significant bits. In this case, the same reference image can be used throughout the entire quantization optimization process for a current property, and possibly even for a subsequent property. If the same reference image is used in all iterations of quantization optimization for a current property, fewer bits are forced to zero, but the final rendering is of higher quality because there is less deviation. According to a second example, the reference image is obtained from the current property quantized by taking into account the bits forced to 0 during a previous iteration on the bits of the current quantized property. According to this variant, calculation errors related to space reduction are accumulated. This variant improves compression, since the number of bits actually forced to zero considering the reference image according to the second example is generally greater than the number of bits actually forced to zero considering the reference image according to the first example, while maintaining acceptable perceived quality depending on the PSNR value.

[0151] The reference image I_REF and the degraded image I_MOD are then compared in step 47, which allows the quality degradation between the two images I_REF and I_MOD to be measured. For example, the distortion between the images I_REF and I_MOD is measured.

[0152] If the distortion obtained is less than (471) a PSNR type threshold for example, then we move on to the next bit of the current quantized property (corresponding to the second least significant bit for the second iteration) and we repeat steps 45, 46 and 47. Step 44 can also be repeated if the reference image for the next bit is calculated from the current quantized property with the modified value of the previous bit (i.e. obtained in a previous iteration).

[0153] Conversely, if the distortion obtained is greater than or equal to (472) the threshold, then the current bit is reset to its initial value "1" during a step 48 and the iterative process on the bits of the current quantized property is stopped.

[0154] In the example illustrated, this treatment is applied to the different properties of the splat.

[0155] Thus, during step 49, we check if the current property is the last property associated with the current splat.

[0156] If the current property is not the last property associated with the current splat, so we move 492 to the next property and we iterate, for the new quantified current property, the steps described previously.

[0157] If the current property is the last property associated with the current splat, so during a step 491 we record the modified quantified properties associated with the current splat and we can move on to the next splat in the set of splats 41.

[0158] These steps can be repeated for each of the tens of thousands of splats that make up a 3D scene.

[0159] The result is a group of bytes with minimal entropy, while maintaining a good level of quality.

[0160] The table below illustrates an example of a 20-byte group resulting from setting the least significant bits of the quantized values ​​of the different properties of a splat to zero, with: the first row illustrating the two bytes used for the quantization of the center's x position, the second row illustrating the two bytes used for the quantization of the center's y position, the third row illustrating the two bytes used for the quantization of the center's z position, the fourth row illustrating a first byte used for the quantization of the red color and a second byte used for the quantization of the green color, the fifth row illustrating a first byte used for the quantization of the blue color and a second byte used for the quantization of the density, the sixth row illustrating the two bytes used for the quantization of the x scale factor.The seventh line illustrates the two bytes used for quantifying the scale factor in y; the eighth line illustrates the two bytes used for quantifying the scale factor in z; the ninth line illustrates a first byte used for quantifying the rotation factor in x and a second byte used for quantifying the rotation factor in y; the tenth line illustrates a first byte used for quantifying the rotation factor in z and a second byte used for quantifying the rotation angle.

[0161] This results in a compact and almost complete representation of the splat.

[0162] 000000000000000000000000

[0163] The illustrated implementation example thus makes it possible to optimize the number of bits forced to zero while guaranteeing a minimum quality for the reconstruction of the 3D scene.

[0164] As mentioned previously, the various optimized quantified properties can be stored and / or transmitted.

[0165] In particular, the optimized quantized property or properties associated with at least one splat of a 3D scene can be encoded or compressed by an entropy coder before transmission. Thus, once the reduced data of all the splats has been obtained and stored in a buffer, for example, global entropy compression can be applied to them.

[0166] It is possible to encode / compress the data corresponding to the optimized quantized properties byte by byte, or property by property. However, such processing requires more memory manipulation. Thus, according to a particular embodiment, the data corresponding to all the optimized quantized properties associated with a splat are encoded block by block (for example, in 20-byte blocks). Alternatively, the data corresponding to all the optimized quantized properties associated with several splats can be encoded together.

[0167] In particular, a compression technique is chosen that allows for fast decompression, enabling, for example, playback on a terminal with limited resources in terms of RAM, CPU and / or GPU.

[0168] 5.2.3 Restitution

[0169] As illustrated in, different steps are implemented for the reconstruction of at least one 3D scene composed of a plurality of elementary 3D objects, such as splats.

[0170] The compressed / encoded, optimized quantized properties of at least one splat are obtained by a reconstruction device, for example, a mobile terminal, and stored in a buffer during step 51. This step may activate RAM. This data is then decompressed during step 52 and may be stored in a GPU buffer. This step 52 may activate a CPU. The decompressed data is then processed by the GPU during step 53 to reconstruct said at least one splat. This step may activate the GPU.

[0171] In particular, due to the quantization over an integer number of bytes and the entropic compression applied to that integer number of bytes, decompression yields decompressed data in a format directly and efficiently usable by the GPU, thereby reducing processing time on the reconstruction device side. In a specific embodiment, real-time decompression can be implemented while maintaining a structure directly usable by GPUs.

[0172] The table below presents the compression and execution time performance for different examples of compression / decompression techniques that can be implemented according to the invention.

[0173] Technique Initial data size (bytes) Compressed data size (bytes) Compression ratio Compression time (s) Decompression time (s) Frequency per second (decompression) LZFSE23469929127219011.842.680.026238LZ423469929176492721.330.050.0087114LZMA2346992995504842.467.000.45022ZLIB23469929125759771.870.620.078412LZBITMAP23469929147261681.590.190.012480

[0174] It is observed that the LZMA method gives the best results in terms of compression efficiency but that data decompression times are very long.

[0175] Conversely, the LZ4 method is very effective during decompression but less effective in terms of compression.

[0176] The LZFSE method offers a good compromise between compression efficiency and decompression time. Indeed, its compression ratio is close to 2, while allowing for real-time decompression.

[0177] Thus, the proposed solution allows, according to at least one embodiment, to decompress and display a 3D scene in real time, for example on a mobile terminal.

[0178] 5.3 Advantages

[0179] Such compression and / or reconstruction processes according to at least one embodiment of the invention have certain advantages, for example among those mentioned below: the efficiency of the compression process can reach a compression factor or ratio of 20, the losses in terms of quality related to compression can be limited by controlling the level of distortion / degradation, the compression can be optimal for each scene, the decompression can be implemented in real time on a mobile terminal, the decompression can produce a structure directly usable by GPUs, the decompression can be carried out in parallel with the rendering; a scene n can be displayed while a scene n+1 is being decompressed.

[0180] Such parallel processing is illustrated, for example, in.

[0181] Thus, the optimized compressed / coded quantized properties of at least one splat from a first scene are received by a reconstruction device at time t=0s and stored in a buffer during a step 611.

[0182] The optimized compressed / encoded quantized properties of at least one splat from a second scene are received by the reconstruction device at time t=0.03s and stored in a buffer during a step 612. In parallel, the representative data of the first scene are decompressed during a step 621, and can be stored in a GPU buffer.

[0183] The optimized compressed / encoded quantized properties of at least one splat from a third scene are received by the reconstruction device at time t=0.06s and stored in a buffer during a step 613. In parallel, the representative data of the second scene are decompressed during a step 622, and can be stored in a GPU buffer, and the decompressed data of the first scene are processed by the GPU during a step 631 to reconstruct the first scene.

[0184] The optimized compressed / encoded quantized properties of at least one splat from a fourth scene are received by the reconstruction device at time t=0.1s and stored in a buffer during a step 614. In parallel, the representative data of the third scene are decompressed during a step 623, and can be stored in a GPU buffer, and the decompressed data of the second scene are processed by the GPU during a step 632 to reconstruct the second scene, and so on.

[0185] In other words, the proposed compression and reconstruction processes allow, according to at least one embodiment, for sufficiently efficient reading of data from a storage volume or network to allow for decompression and real-time display, for example on a mobile terminal.

[0186] It is therefore possible to create volumetric videos and transmit them via streaming, for example with bitrates around 100 Mbs.

[0187] 5.4 Compression and Reconstruction Devices

[0188] Finally, in relation to figures 7 and 8, simplified structures of a compression device and a reconstruction device are presented according to at least one embodiment of the invention.

[0189] As illustrated in, a compression device includes at least one memory 71, at least one processing unit 72, equipped for example with a programmable computing machine or a dedicated computing machine, for example a processor P, and controlled by the computer program 73, implementing the steps of the compression process according to at least one embodiment of the invention.

[0190] At initialization, the code instructions of program 73 are, for example, loaded into RAM memory before being executed by the processor of the processing unit 72.

[0191] The processor of the processing unit 72 implements steps of the compression process described above on at least one multidimensional scene, according to the instructions of the computer program 73, to generate optimized quantified descriptive information.

[0192] As illustrated in, a reconstruction device includes at least one memory 81, at least one processing unit 82, equipped for example with a programmable computing machine or a dedicated computing machine, for example a processor P, and controlled by the computer program 83, implementing the steps of the reconstruction process according to at least one embodiment of the invention.

[0193] At initialization, the code instructions of program 83 are, for example, loaded into RAM memory before being executed by the processor of the processing unit 82.

[0194] The processor of the processing unit 82 implements steps of the reconstruction process described above on the optimized quantified descriptive information, according to the instructions of the computer program 83, to reconstruct at least one multidimensional scene.

Claims

A compression method for at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing, for at least one of said elementary objects, referred to as the current object: for at least one descriptive piece of information of said current object, referred to as the current descriptive information: quantization (21) of said current descriptive information over an integer number of bytes, optimization (22) of the quantization of said current descriptive information comprising, for at least one non-zero bit of said current quantized descriptive information, referred to as the current bit: obtaining (221) a reference image representative of said scene, modification (222) of a value of said current bit, relative to an initial value, delivering a modified current quantized descriptive information, obtaining (223) a degraded image representative of said scene, from said modified current quantized descriptive information.measurement (224) of a distortion between said degraded and reference images; if said distortion between said degraded and reference images is less than a threshold, storage (225) of said modified value of said current bit, otherwise reset to said initial value of said current bit. Method according to claim 1, characterized in that said optimization of the quantization of said current descriptive information is iterated over the non-zero value bits of said current quantized descriptive information as long as said distortion is below said threshold. A method according to any one of claims 1 and 2, characterized in that said optimization of the quantification of said current descriptive information is implemented starting with the least significant bit. A method according to any one of claims 1 to 3, characterized in that said threshold is a peak signal-to-noise ratio threshold. A method according to any one of claims 1 to 4, characterized in that said at least one elementary object is a Gaussian splat. A method according to any one of claims 1 to 5, characterized in that said at least one descriptive piece of information of said current object belongs to the group comprising: a position of the center of said current object, a color associated with said current object, a density associated with said current object, a scale factor associated with said current object, a rotation factor associated with said current object. Method according to claim 6, characterized in that for a current descriptive information of the type center position or scale factor, said quantification of said current descriptive information is implemented on two bytes. Method according to claim 6, characterized in that for a current descriptive information of the type colour, density or rotation factor, said quantification of said current descriptive information is implemented on one byte. Method according to claim 2, characterized in that said reference image is obtained from said current modified quantified descriptive information, taking into account the modified value of at least one bit preceding said current bit, except for the first iteration. A method according to any one of claims 1 to 9, characterized in that it comprises an entropic coding step of all the optimized quantified descriptive information associated with said current object. Method for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, implementing: obtaining (31) at least one optimized quantified descriptive information of at least one elementary object according to claim 1, reconstructing (32) said at least one elementary object from said at least one optimized quantified descriptive information. Reconstruction method comprising displaying said at least one scene on a mobile terminal. A compression device for at least one multidimensional scene composed of a plurality of multidimensional elementary objects, comprising at least one processing unit configured to, for at least one of said elementary objects, referred to as the current object: for at least one descriptive piece of information of said current object, referred to as the current descriptive piece of information: quantify said current descriptive piece of information over an integer number of bytes, optimize the quantization of said current descriptive piece of information, comprising, for at least one non-zero bit of said current quantized descriptive piece of information, referred to as the current bit: obtain a reference image representative of said scene, modify a value of said current bit, relative to an initial value, delivering a modified current quantized descriptive piece of information, obtain a degraded image representative of said scene, from said modified current quantized descriptive piece of information.Measure the distortion between the degraded and reference images; if the distortion between the degraded and reference images is less than a threshold, store the modified value of the current bit; otherwise, reset the current bit to its initial value. Device for reconstructing at least one multidimensional scene composed of a plurality of multidimensional elementary objects, comprising at least one processing unit configured to: obtain at least one optimized quantified descriptive information of at least one elementary object according to claim 1, reconstruct said at least one elementary object from said at least one optimized quantified descriptive information. Computer program comprising instructions for carrying out a method according to any one of claims 1 to 12 when this program is executed by a processor.

Images