Training a model for representing a three-dimensional scene
By integrating colorimetric and dispersion criteria with noise and distribution models, the method optimizes 3D scene representation models for reduced storage and faster training, addressing the inefficiencies of existing methods.
Patent Information
- Application Number
- FR2023010966
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-10-12
- Publication Date
- 2025-04-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing 3D scene representation methods, such as neural radiance fields and latent variable-based models, face challenges in balancing training efficiency, quality, and storage space, with explicit/hybrid representations being time-efficient but memory-intensive, and implicit methods being slow and less compressible.
A training process that incorporates a quality criterion combining colorimetric quality and dispersion of latent variables, using noise and mathematical distribution models like Laplace or Cauchy to optimize compressibility, reducing the model's storage footprint.
The method improves the compressibility of 3D scene representations by minimizing latent variable dispersion while maintaining quality, leading to more efficient storage and training times.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Training a model for representing a three-dimensional scene Field of invention
[0001] This invention relates to the field of generating 3D three-dimensional scenes from 2D views.
[0002] More specifically, the invention relates to minimizing the size of data files representative of 3D scenes generated from 2D views. Prior art
[0003] Differentiable volume rendering solutions and neural radiance fields (Neural Radiance Fields or NeRFs) are solutions that make it possible in particular to construct / reconstruct / represent 3D scenes from 2D viewpoints (for example photos or videos taken from a limited number of viewpoints) by generating, after a 3D scene learning or training process, all the viewpoints intermediate to those provided a priori.
[0004] Some representations, called implicit, allow the density and color values to be calculated directly for all the voxels in the scene, and this for all possible viewpoints within it. These representations have the advantage of being attached to architectures that are relatively simple to implement and do not require much storage space. However, they have the disadvantage of being very long to train and offer limited results compared to explicit or hybrid solutions.
[0005] Other representations, called explicit or hybrid, use latent variables (sometimes called latent codes or features in English). These latent variables are used to designate a number or vector of numbers, associated with an area of 3D space. Several examples of latent codes are known to those skilled in the art, such as the "trainable features" of instant-ngp, the "spherical harmonies" of plenoxels, etc.
[0006] For example, we know the publication Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis.".
[0007] The set of latent variables makes it possible to provide, via a calculation model, a scene model representing a real scene as photographed according to several positions (extrinsics) and characteristics (intrinsics) of a virtual camera. There are several scene models known to those skilled in the art such as, for example, the models: instant-ngp, plenoxels, gaussian splatting, etc.
[0008] The method for training a model for representing a three-dimensional scene is an iterative process consisting of deducing, for each training iteration, a “scene model” from the “scene model of the previous iteration”, by evaluating the quality of the model calculated from said previous “scene model”.
[0009] The quality of the model, generally designated by the term loss in English (or perte in French) is calculated by a mathematical function making it possible to represent at least one expected quality of the scene model. Examples of quality of the model are known to those skilled in the art, in particular colorimetric loss or total variation loss (perte de variation totale in French) as defined in plenoxels.
[0010] Here, the colorimetric quality (RGB loss in English) designates a mathematical function calculating the fidelity of the representation of a 3D scene as a function of the ground truth / reality (ground truth in English) resulting from the known images of the scene and their equivalent representations produced by the “scene model”. Examples of colorimetric quality are known to those skilled in the art, for example MSE (“Mean Squared Error” in English), SSIM (“Structural Similarity” in English).
[0011] In the context of MSE, for each pixel, the square of the difference between the generated color and the image color from the ground truth / reality is calculated and then the average is calculated over all of these elementary errors for all the pixels of the current iteration.
[0012] In the case of structural similarity, the comparison is made between the image reconstructed by the current iteration and the image from the ground truth / reality on blocks of NxN pixels (rather than pixel by pixel). Then for each block, we calculate the average of the values, the variance, and the covariance between the two images.
[0013] Such a structural similarity is expressed as follows, for two blocks, x and y of the same size NxN:
[0014] ("HAS
[0015] With : - qx equal to the average of x, - qy equal to the average of y, - Oj the covariance of x, - the covariance of y - covxy the covariance of x and y, - Ci, c2 and c3 three variables intended to stabilize the division when the denominator is very weak;
[0016] If explicit or hybrid representations have the advantage of requiring time much shorter training time and better quality rendering compared to implicit representations, they have the disadvantage of being very voluminous in terms of storage / memory due to the very large variety of latent variables associated with them.
[0017] Some approaches can partially circumvent this problem, for example by using hashtables, such as the INGP solution (Instant Neural Graphics Primitives). However, these representations remain very large and problematic in terms of storage compared to implicit representations.
[0018] Object and summary of the invention
[0019] One of the objectives of the invention is therefore to resolve at least one of these drawbacks.
[0020] To this end, an object of the present invention relates to a method for training a model for representing a three-dimensional scene by latent variables, implementing a quality criterion for the representation of said scene, characterized in that said model is also trained according to a criterion for dispersion of all or part of said latent variables.
[0021] This method offers the advantage of improving the compressibility of the representation model, thus making it possible, after compression, to significantly reduce the weight of the file relating to the representation model, which constitutes a known drawback of explicit or hybrid representation models.
[0022] According to a particular embodiment, the representation model of a three-dimensional scene is trained according to a global quality criterion composed of a colorimetric quality criterion LÆGg and a dispersion criterion Vfdipsersim^ so that Lglobal — ^rgB + ^'dipsersim » °ù is a Lagrange multiplier dependent on Lrgb.
[0023] This embodiment taking into account both the colorimetric quality and dispersion criteria advantageously makes it possible to train the model so that the dispersion is minimal thanks to the dispersion criterion while maintaining / guaranteeing a quality of the representation / reconstruction of the scene above a minimal threshold, via the colorimetric quality criterion.
[0024] According to a particular embodiment, the quality criterion of the representation of the scene implements, upstream of a step of calculating the quality of the representation of the scene, an application of noise to the latent variables previously processed or not.
[0025] Adding noise to the latent variables, whether previously processed or not, advantageously improves the compressibility of the model by making the robust model to quantization which will take place after training the model, which helps to reduce the file size of the scene model.
[0026] According to a particular embodiment, the dispersion criterion of all or part of said latent variables is approximated as follows:
[0027] - associate the dispersion of the latent variables with a distribution model ma thematic, said association comprising the following:
[0028] — identify a parameterization of the latent variable distribution model op timizing the dispersion criterion,
[0029] — for any latent variable among said latent variables, calculate the dispersion based on said mathematical distribution model.
[0030] Calculating the dispersion quality of all or part of the latent variables advantageously makes it possible to optimize the improvement of the compressibility of the representation model of the three-dimensional scene. By approximating the latent variables resulting from learning using a mathematical distribution model rather than using a neural network, this embodiment makes it possible to improve the overall learning time of the model.
[0031] According to a particular embodiment, noise is applied to the latent variables, upstream of the calculation of the dispersion.
[0032] This embodiment advantageously makes it possible to further optimize the improvement of the compressibility of the representation model of the three-dimensional scene for two reasons. Firstly, this makes it possible to regularize the value of the learned latent variables before evaluating the quality of the model, in particular at the start of the training of the model when the majority of the latent variables almost all have the same value and thus generally all tend towards zero. Secondly, this noise makes it possible to pre-model the quantization noise which may result from the quantization which will be applied at the end of the training of the model.
[0033] According to a particular embodiment, the mathematical distribution model with which the dispersion of the latent variables is associated is based on a Laplace distribution.
[0034] This embodiment is particularly advantageous when implementing a model intended to represent scenes where there are several objects represented, in particular real scenes (as opposed to digitally / artificially generated scenes) where any distortion within the calculated latent variables has a significant effect on the overall quality of the scene model.
[0035] According to one embodiment, the dispersion of the latent variables is associated with a Cauchy distribution model and the dispersion criterion according to which the representation model is trained is fixed once, at the end of an iteration of the training of the model, the quality of representation of the scene exceeds a threshold. determined.
[0036] This embodiment has the advantage of limiting the dispersion of the approximated latent variables when associating the dispersion with a Cauchy distribution.
[0037] According to one embodiment, at the end of the training of the model, the latent variables are quantified with an arbitrary step (for example less than 1, equal to 1 or greater than 1), and compressed.
[0038] This embodiment advantageously makes it possible to achieve effective compression of the representation model of the reconstructed three-dimensional scene, said compression benefiting from the increased / improved compressibility of the model.
[0039] The various embodiments or characteristics mentioned above can be added independently or in combination with each other, to the method for training a model for representing a three-dimensional scene as defined above.
[0040] The invention also relates to a terminal for training a model for representing a three-dimensional scene by latent variables, implementing a quality criterion for the representation of said scene, characterized in that it is configured to also train said model as a function of a dispersion criterion for all or part of said latent variables.
[0041] Such a terminal is particularly suitable for implementing the method of training a model for representing a three-dimensional scene according to any one of the embodiments described previously.
[0042] The invention also relates to a computer program comprising program code instructions for implementing the method of training a model for representing a three-dimensional scene, when it is executed on a computer.
[0043] Such instructions may be stored permanently in a non-transitory memory medium of a device implementing the method for training a model representing a three-dimensional scene.
[0044] This program may use any programming language, and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.
[0045] The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions of a computer program for implementing the method of training a model for representing a three-dimensional scene as mentioned above.
[0046] The recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM (“Read Only Memory”), for example a CD ROM (“ Compact Disc Read-Only Memory (CD-ROM) in English), a synthetic DNA (deoxyribonucleic acid) or a microelectronic circuit ROM, or a magnetic recording medium, for example a mobile media, a hard disk or an SSD (Solid State Drive) in English.
[0047] On the other hand, the recording medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means, so that the computer program it contains is remotely executable. The program according to the invention may in particular be downloaded over a network, for example an Internet-type network.
[0048] Alternatively, the recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the aforementioned method of training a model for representing a three-dimensional scene.
[0049] According to an exemplary embodiment, the present technique is implemented by means of software and / or hardware components. With this in mind, in this document, the term “device” or “module” can correspond to a software component, a hardware component or a set of hardware and software components. Brief description of the drawings
[0050] Other characteristics and advantages will appear on reading particular embodiments of the invention, given as illustrative and non-limiting examples, and the appended drawings, among which:
[0051] [Fig. 1] represents an example of architecture in which the method of training a model representing a three-dimensional scene is implemented.
[0052] [Fig.2] represents the steps of a method for training a model representing a three-dimensional scene, in one embodiment of the invention.
[0053] [Fig. 3] represents the steps of a method for training a model representing a three-dimensional scene, in another embodiment of the invention.
[0054] [Fig.4] represents the steps of a method for training a model representing a three-dimensional scene, in another embodiment of the invention.
[0055] [Fig.5] represents the steps of a method for training a model representing a three-dimensional scene, in another embodiment of the invention.
[0056] [Fig.6] represents the steps of a method for training a representation model representation of a three-dimensional scene, in another embodiment of the invention.
[0057] [Fig.7] represents an example of a training terminal of a representation model representation of a three-dimensional scene, in one embodiment of the invention. Detailed description
[0058] Description of an embodiment of a model for representing a three-dimensional scene
[0059] With reference to [Fig. 1], an example of a scene representation model MS is described comprising: — a set of latent variables F representing a three-dimensional scene, — a three-dimensional scene reconstruction module MR, — an LC module for calculating the overall quality Lç / ^ / of the model.
[0060] According to other embodiments, the MR reconstruction module also comprises a device for interpolating the latent variables (not shown).
[0061] According to other embodiments, the MR reconstruction module may designate one or more multi-layer perceptrons (not shown) or any other alternative device, such as for example a convolutional, recurrent or hybrid neural network, etc.
[0062] [Fig. 1] also describes an iterative loop I corresponding to the training loop of a representation model of a three-dimensional scene consisting of deducing, for each training iteration, a “scene model” from the “scene model of the previous iteration” generated by the MR module, by evaluating the quality of the model calculated by the LC module from said previous “scene model”.
[0063] According to certain embodiments, the quality of the model depends on the colorimetric quality, designated as ^rgb, obtained by the following formula: 100641 he ew-cf-) ii *
[0065] Where C (r ) denotes the color of a pixel estimated by the model and C( r ) the known color of the corresponding pixel r.
[0066] Description of an embodiment of a method for training a model representing a three-dimensional scene
[0067] Figure [Fig.2] represents the steps implemented by a method for training a model representing a three-dimensional scene, in a particular embodiment of the invention, said representation model being that represented in figure [Fig.l].
[0068] The training method comprises a step E201 in which said calculation module LC measures the overall quality Lg / o / ,a / of the representation model, taking into account takes into account the colorimetric quality L^GB and the dispersion quality I^dispersian of the latent variables.
[0069] According to certain embodiments, the overall quality of the model is calculated according to the formula Lgiobai — LRCB + ^ï^dispersion ct °ù is a Lagrange multiplier which depends on L^GS
[0070] According to certain embodiments, the I-dispersion criterion is evaluated with an arbitrary step (for example less than 1, equal to 1 or greater than 1).
[0071] Description of another embodiment of a method for training a model representing a three-dimensional scene
[0072] Figure [Fig.3] represents the steps implemented by the method for training a model for representing a three-dimensional scene, in another particular embodiment of the invention, said representation model being that represented in figure [Fig.l].
[0073] The method comprises a step E301 of calculating the overall quality of the model integrating the dispersion quality of the model, in a similar manner to step E201 illustrated in the figure [Fig.2].
[0074] The method also includes a step E302, during which noise is applied to the latent variables, prior to a scene reconstruction step.
[0075] According to the embodiments, this noise can be applied directly to the latent variables or after prior processing thereof, for example after linear or non-linear interpolation.
[0076] According to the embodiments, the noise applied may be uniform noise, Gaussian noise or any other type of noise known to those skilled in the art.
[0077] According to the embodiments, this noise is used to anticipate the noise linked to the quantization which will be applied to the model once trained. The noise then corresponds to the following equation; 4 ] where A is the quantization interval.
[0078] Description of another embodiment of a method for training a model representing a three-dimensional scene
[0079] Figure [Fig.4] represents the steps implemented by the method for training a model for representing a three-dimensional scene, in another particular embodiment of the invention, said representation model being that represented in figure [Fig.l].
[0080] The method comprises a step E400 of calculating the quality of dispersion of the scene representation model, by associating the dispersion of the latent variables with a mathematical distribution model.
[0081] According to the embodiments, the dispersion of the latent variables can be associated with a Laplace, Cauchy or any other distribution function. contribution known to those skilled in the art.
[0082] This step E400 comprises two sub-steps E400-1 and E400-2.
[0083] Step E400-1 consists of identifying a parameterization of the latent variable distribution model optimizing the dispersion criterion.
[0084] According to the embodiments, the parameterization of the distribution model consists of identifying the overall mean and the standard deviation b of the retained distribution function.
[0085] Step E400-2 consists of calculating the dispersion according to said mathematical distribution model for each latent variable among said latent variables.
[0086] According to certain embodiments, the dispersion quality ,Krsien is calculated by the following equation:
[0087]
[0088] Where:
[0089] - N& is the number of latent variables of the scene,
[0090] - O the set of latent variables,
[0091] - 0 a latent variable,
[0092] - P(0) is a cumulative density function of the distribution function used to approximate the dispersion of latent variables.
[0093] According to certain embodiments, if the dispersion of the latent variables is approximated by an association with a Laplace distribution function, the overall quality of the model is calculated according to the following equation:
[0094] ^global ~ ^RGB 0 ^-'dispersion)
[0095] where 2 = LRGe X
[0096] So that ^giobai results from a harmonious compromise between the dispersion ^dispersion and the LKGS colorimetric quality.
[0097] The method continues with a step E401 similar to step E201 illustrated in the figure [Fig.2].
[0098] According to certain embodiments, the dispersion of the latent variables can be approximated by associating this dispersion with a Cauchy distribution function. To this end, during step E401, the overall quality criterion is calculated first as a function of a fixed dispersion criterion and then as a function of a variable dispersion criterion as soon as the colorimetric quality criterion of the model exceeds a predetermined quality threshold at the end of an iteration of the training of the model.
[0099] According to certain embodiments, the calculation of the representation of the scene on the basis of said latent variables is not optimized to operate on discrete values, but to work with variables whose precision is less than an arbitrary step (a minimum deviation), potentially less than 1.
[0100] Description of another embodiment of a method for training a model representing a three-dimensional scene
[0101] Figure [Fig.5] represents the steps implemented by the method for training a model for representing a three-dimensional scene, in another particular embodiment of the invention, said representation model being that represented in figure [Fig.l].
[0102] The method comprises a step E500 similar to step E400 illustrated in figure [Fig.4] but further comprises a preliminary sub-step E500-0 consisting of applying noise to the latent variables, upstream of the evaluation of the dispersion.
[0103] Depending on the embodiments, the noise applied may be uniform or Gaussian noise or any other type of noise known to those skilled in the art.
[0104] The method continues with sub-steps E500-1 and E500-2 similar to sub-steps E400-1 and E400-2 illustrated in figure [Fig.4] and a step E501 similar to step E201 illustrated in figure [Fig.2].
[0105] According to an embodiment where the dispersion of the latent variables is associated with a Cauchy distribution function, during step E501, the global quality criterion ^global is calculated firstly as a function of a fixed dispersion criterion then as a function of a variable dispersion criterion as soon as the colorimetric quality criterion of the model exceeds a predetermined quality threshold at the end of an iteration of the training of the model.
[0106] Description of another embodiment of a method for training a model representing a three-dimensional scene
[0107] Figure [Fig.6] represents the steps implemented by the method for training a model for representing a three-dimensional scene, in another particular embodiment of the invention, said representation model being that represented in [Fig.1], with the difference that it additionally comprises a lossless data compression MC module.
[0108] According to the embodiments, the compression module MC implements compression in ZIP, ACE, TAR format or any other lossless compression format known to those skilled in the art.
[0109] The method comprises a step E601 similar to step E201 illustrated in figure [Fig.2] as well as a step E602 in which the compression module receives the latent variables F and produces a compressed file containing said latent variables.
[0110] According to the embodiments, this step includes a quantization sub-step preceding the compression.
[0111] According to the embodiments, if the distribution of the latent variables is associated with a Laplace distribution, this quantization sub-step implements a quantization function Q of the type: eOLMHHl
[0112] Whereas if the distribution of the latent variables is associated with a Cauchy distribution, the quantification sub-step implements a quantification function Q of the type:
[0113] g(e) = A[£ + l]
[0114] According to the embodiments, the latent variables are quantified with an arbitrarily determined step (for example less than 1, equal to 1 or greater than 1).
[0115] Description of an embodiment of a terminal for training a model representing a three-dimensional scene
[0116] Figure [Fig.7] represents the simplified structure of a terminal T configured to train a model for representing a three-dimensional scene such as that illustrated in [Fig.l].
[0117] According to a particular embodiment of the invention, the steps of the method for training a model for representing a three-dimensional scene of the present invention are implemented by instructions of a computer program PG. For this, the terminal T has the conventional architecture of a computer and notably comprises a memory MEM, a processing unit UTR, equipped for example with a processor PROC, and controlled by the computer program PG stored in memory MEM. The computer program PG comprises instructions for implementing the steps of the method for training a model for representing a three-dimensional scene, in particular the calculation of an overall quality criterion of the model taking into account the colorimetric quality and the quality of dispersion of the latent variables.
[0118] According to other embodiments, based on a so-called cloud computing architecture, the processor PROC and the memory MEM can be associated with another terminal, different from the terminal T.
Claims
Claims
1. Method for training a model for representing a three-dimensional scene using latent variables, implementing a quality criterion for the representation of said scene, characterized in that said model is also trained according to a dispersion criterion for all or part of said latent variables.
2. Method for training a model for representing a three-dimensional scene according to claim 1, characterized in that said model is trained according to a global quality criterion Lg / p / œ / composed of a colorimetric quality criterion L«gs and a dispersion criterion ^dispersions so that 4- '^^'dispersions or is a Lagrange multiplier dependent on L^g.
3. Method for training a model for representing a three-dimensional scene according to any one of claims 1 to 2, characterized in that the quality criterion for the representation of said scene implements, upstream of a step of calculating the quality of the representation of the scene, an application of noise to the latent variables, whether previously processed or not.
4. Method for training a model for representing a three-dimensional scene according to any one of claims 1 to 3, characterized in that the dispersion criterion of all or part of said latent variables is approximated as follows: - associating the dispersion of the latent variables with a mathematical distribution model, said association comprising the following: - identifying a parameterization of the distribution model of the latent variables optimizing the dispersion criterion, - for any latent variable among said latent variables, calculating the dispersion as a function of said mathematical distribution model.
5. Method for training a model for representing a three-dimensional scene according to claim 4, characterized in that said method comprises an application of noise to the latent variables, upstream of the calculation of the dispersion.
6. Method for training a model for representing a three-dimensional scene according to claim 4 or 5, characterized in that the mathematical distribution model with which the dispersion of the latent variables is associated is based on a Laplace distribution.
7. Method for training a model for representing a three-dimensional scene dimensional according to claim 4 or 5, characterized in that: - the mathematical distribution model with which the dispersion of the latent variables is associated is based on a Cauchy distribution, - said dispersion criterion according to which the representation model is trained is fixed once, at the end of an iteration of the training of the model, the quality of representation of the scene exceeds a determined threshold.
8. Method for training a model for representing a three-dimensional scene according to any one of claims 1 to 7, characterized in that at the end of the training of the model, the latent variables are quantified with an arbitrary step and compressed.
9. Terminal for training a model for representing a three-dimensional scene using latent variables, implementing a quality criterion for the representation of said scene, characterized in that it is configured to also train said model according to a dispersion criterion for all or part of said latent variables.
10. Computer program comprising program code instructions for implementing the method of training a model for representing a three-dimensional scene according to any one of claims 1 to 8, when executed on a computer.
11. Information medium readable by a computer, and comprising instructions of a computer program for implementing the method of training a model for representing a three-dimensional scene according to any one of claims 1 to 8.