Dynamic 3D imaging method

Abstract

A dynamic three-dimensional imaging method is disclosed that allows the generation of a three-dimensional numerical model representing an observed three-dimensional scene. The generated model optimally combines (404) data from two intermediate three-dimensional numerical models, which are obtained respectively by a stereoscopic three-dimensional reconstruction calculation method (402) and by a learned three-dimensional reconstruction calculation method (403). In addition, each new model generated helps improve the overall performance of the learned three-dimensional reconstruction calculation method.

Description

Technical Field

[0001] The present invention relates generally to three-dimensional imaging and more particularly to a three-dimensional imaging method that allows for dynamic reconstruction of a three-dimensional numerical model from multiple two-dimensional digital images of a given three-dimensional scene. Background Art

[0002] Currently, there are various methods that can generate a three-dimensional numerical model to represent a three-dimensional scene. Among these methods, the so-called stereo reconstruction methods and the so-called learning reconstruction methods are particularly attractive for industrial applications targeting the public, because they use a two-dimensional digital image stream from a passive image sensor to reconstruct a three-dimensional numerical model of the scene.

[0003] Figure 1 is a step diagram showing in a simplified manner the steps common to the two methods described above, which allow the generation of a three-dimensional numerical model from a two-dimensional digital image. These methods are performed in an iterative manner, wherein each iteration is able to generate a three-dimensional numerical model that properly enriches and completes the three-dimensional model established during the previous iteration.

[0004] After a step 101 of receiving one or more two-dimensional digital images representing a given three-dimensional scene, steps 102 and 103 are constituted respectively by a so-called position estimation phase and a so-called three-dimensional information estimation phase. On the one hand, the position estimation phase consists in determining, at a given moment t, the position of the image sensor used to acquire the two-dimensional digital image relative to the three-dimensional numerical model already established at this moment t. In a manner known per se, this determination is performed by optimizing criteria taking into account both the two-dimensional image and the three-dimensional model (for example, by two-dimensional or three-dimensional excision or by optimizing photometric criteria). On the other hand, the three-dimensional information estimation phase consists in calculating the three-dimensional numerical model by means of calculation formulas specific to each of the two methods from the two-dimensional image received at the same moment t.

[0005] Finally, the three-dimensional information obtained at the end of the three-dimensional reconstruction calculation is added to the three-dimensional numerical model already established, taking into account the actual position of the image sensor, during step 104. This information can appropriately change certain features of the model already established in order to improve its accuracy.

[0006] Beyond these aspects that both methods share, each method has its own advantages and disadvantages.

[0007] On the one hand, the stereoscopic 3D reconstruction calculation method does not require any information before its application and can be applied to a wide variety of 3D scenes. However, its performance depends on the specific type of scene observed. Specifically, when the area of ​​the observed scene has no texture or there are reflections or occlusions, the reconstruction performance of the method is significantly reduced. In addition, an important limitation of this method is that at least two separate 2D images are required to generate a 3D model that presents any 3D scene.

[0008] On the other hand, the three-dimensional reconstruction calculation method by learning relies on specific algorithms (e.g., algorithms based on deep neural networks) to generate three-dimensional numerical models from single or multiple two-dimensional digital images. These algorithms can be trained, that is, they can improve their performance due to the learning process, during which the algorithm faces a large number of different two-dimensional image groups associated with a determined three-dimensional model. Advantageously, the method can therefore be applied indiscriminately to any type of scene. Its performance depends essentially only on the quality of the learning performed previously and the data used during the learning period. However, targets or scenes that are not integrated into the learning process will only be reconstructed incompletely, if at all. In addition, in general, the accuracy of the three-dimensional model generated by this method is currently lower than the accuracy of the model generated by the stereoscopic three-dimensional reconstruction calculation method.

[0009] Document WO2018 / 039269A1 discloses a device using multiple image sensors on augmented reality glasses. The device uses the acquired two-dimensional images to determine the position of the device in its environment via a learning computational method and performs a dynamic three-dimensional reconstruction of the observed scene. However, the accuracy of the reconstructed three-dimensional model is not an issue in this application. It should also be noted that the device uses only one of the two methods described above. Summary of the invention

[0010] The object of the present invention is to obviate or at least mitigate all or some of the above-mentioned disadvantages of the prior art.

[0011] To this end, a first aspect of the present invention provides a dynamic three-dimensional imaging method, which includes the following steps performed by a computing unit:

[0012] - receiving at least two two-dimensional digital images representing an observed three-dimensional scene, the at least two two-dimensional digital images being acquired by at least one determined image sensor from at least two corresponding positions in space;

[0013] - generating a first intermediate three-dimensional numerical model associated with the observed three-dimensional scene from the two-dimensional digital image based on a stereoscopic three-dimensional reconstruction calculation method;

[0014] - generating a second intermediate three-dimensional numerical model associated with the observed three-dimensional scene from the two-dimensional digital image based on a three-dimensional reconstruction calculation method through learning;

[0015] - generating a final three-dimensional numerical model by combining the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model, the generating comprising: for a determined portion of the final three-dimensional numerical model, selecting a corresponding portion of the determined portion of the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model based on a comparison of at least one characteristic parameter of the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model for the portion, maximizing a rendering quality criterion of the observed three-dimensional scene, the characteristic parameter representing the rendering quality of the observed three-dimensional scene;

[0016] - storing in a memory digital information comprising at least one received two-dimensional digital image, a final three-dimensional numerical model associated with the two-dimensional digital image, and characteristic parameters of the final three-dimensional numerical model representative of the rendering quality of the observed three-dimensional scene; and

[0017] - Using the stored digital information as training data to train and retrain the computing unit to perform the three-dimensional reconstruction computing method by learning.

[0018] Thanks to the invention, it is possible to optimally combine two different three-dimensional reconstruction calculation methods in order to generate a final three-dimensional numerical model that is as realistic as possible. This combination goes far beyond a simple choice between two methods of generating a 3D model that may be known to be equivalent to each other. It allows real synergy, especially in the case of embodiments in which the generated second intermediate three-dimensional numerical model can be used as a priori for generating a first intermediate three-dimensional numerical model by a stereoscopic three-dimensional reconstruction calculation method, even if this embodiment remains a non-mandatory option. In fact, in some embodiments, not only the stereo method feeds learning, but the dynamic learning method also provides a priori for executing the stereo method.

[0019] Furthermore, the method is self-enriching, so that each newly generated model can be trained or retrained in the framework of unsupervised learning by using at least in part the results obtained by the stereo method. The stereo method is able to generate three-dimensional models even for 3D scenes that were not included in the initial learning. Therefore, these newly generated models can constitute a new subset that can be added to the initial learning data for relearning. This will be richer than the initial learning because the new three-dimensional models have not yet been considered. This can improve the overall performance of the method in a self-enrichment process, in which the learning or relearning guides the stereo effect rather than being essential to the stereo effect, and the stereo effect provides new data for relearning when it is used.

[0020] The embodiments, alone or in combination, are further configured as follows:

[0021] The method further comprises acquiring, by means of at least one determined image sensor, preferably a passive video sensor of the CMOS or CCD type, a two-dimensional digital image representing said observed three-dimensional scene.

[0022] - The method further comprises displaying the final three-dimensional numerical model via a display device, such as a two-dimensional scanning or LED display screen, virtual or augmented reality glasses, or a three-dimensional display screen.

[0023] - generating the final three-dimensional numerical model by combining the first and second intermediate three-dimensional numerical models comprises:

[0024] If only the first intermediate three-dimensional numerical model or the second intermediate three-dimensional numerical model has a value for a three-dimensional pixel modelling a given part of the representation of the observed three-dimensional scene, this value is assigned to the final three-dimensional numerical model;

[0025] If both the first intermediate three-dimensional numerical model or the second intermediate three-dimensional numerical model have a value for a three-dimensional pixel modelling a given part of the representation of the observed three-dimensional scene, the value assigned to the final three-dimensional numerical model is selected from the two corresponding values ​​based on a comparison of at least one characteristic parameter of the three-dimensional numerical models, the characteristic parameter being associated with the value of the three-dimensional pixel; and

[0026] If no three-dimensional numerical model has a value for a three-dimensional pixel modeling a given portion of the representation of the observed three-dimensional scene, no value is assigned to the three-dimensional pixel or the value assigned to the three-dimensional pixel is determined according to a smoothing calculation method.

[0027] - the generation of the second intermediate three-dimensional numerical model is performed before the generation of the first intermediate three-dimensional numerical model, and the generated second intermediate three-dimensional numerical model is used as a priori for generating the first intermediate three-dimensional numerical model by a stereoscopic three-dimensional reconstruction calculation method;

[0028] - A characteristic parameter of the three-dimensional numerical model is the uncertainty associated with each value of the three-dimensional numerical model.

[0029] - the learning data used by the computing unit to train and retrain the learned three-dimensional reconstruction computing method includes:

[0030] Data from shared or unshared databases, including two-dimensional digital images and associated three-dimensional numerical models; and / or

[0031] Data comprising a two-dimensional digital image and an associated three-dimensional numerical model, which is generated by another computing unit executing the method; and / or

[0032] Data comprising a two-dimensional digital image and an associated three-dimensional numerical model are generated by a computing unit executing another method.

[0033] - Each intermediate 3D numerical model is a depth map or a 3D point cloud.

[0034] - During one or more steps of generating an intermediate three-dimensional numerical model, a second remote computing unit cooperates with said computing unit to generate said one or more intermediate three-dimensional numerical models.

[0035] - the received two-dimensional digital image is a color digital image, and wherein the generated final three-dimensional numerical model comprises a texture determined based on the color of said two-dimensional digital image.

[0036] - During one or more steps of generating an intermediate three-dimensional numerical model, the three-dimensional reconstruction calculations performed use a foveal imaging computational model to obtain an intermediate three-dimensional numerical model that is relatively more accurate in the portion of the model projected onto the center of the image and relatively less accurate in the portion of the model projected onto the edge of the image.

[0037] In a second aspect, the invention also relates to a computing unit comprising hardware and software means for performing all the steps of the method according to the previous aspect.

[0038] A final aspect of the invention relates to a three-dimensional imaging device comprising at least one digital image sensor, a memory and a computing unit according to the preceding aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features and advantages of the present invention will become more apparent upon reading the following description, which is purely illustrative and should be read in conjunction with the accompanying drawings, in which:

[0040] Figure 1 The diagram shows in a simplified manner the steps common to a plurality of known three-dimensional reconstruction calculation methods for generating a three-dimensional numerical model of a scene.

[0041] Figure 2 is a schematic diagram of a system that can perform one embodiment of the method according to the present invention.

[0042] Figure 3 is a functional diagram of an apparatus for performing the method according to the present invention.

[0043] Figure 4 is a step diagram which shows in a simplified manner the steps of an embodiment of the method according to the invention.

[0044] Figure 5It is a schematic diagram of the iterative process of the stereoscopic three-dimensional reconstruction calculation method.

[0045] Figure 6 Schematic diagram of the method for 3D reconstruction calculation through learning.

[0046] Figure 7 is a step diagram which shows in a simplified manner the steps of a further exemplary embodiment of the method according to the invention. DETAILED DESCRIPTION

[0047] In the following description of embodiments and the accompanying drawings, the same or similar elements are provided with the same reference numerals in the drawings.

[0048] Combine the following Figure 2 , Figure 3 and Figure 4 An embodiment of a dynamic three-dimensional imaging method according to the present invention is described.

[0049] In the context of embodiments of the present invention, the term "dynamic" means that the method may be continuously performed by a computing unit, for example in an iterative manner. In particular, each new acquisition of an image or a two-dimensional digital image representing a given three-dimensional scene may result in the execution of method steps. Each execution of the method (i.e., each iteration) results in the generation of a three-dimensional numerical model representing the three-dimensional scene. In other words, the three-dimensional numerical model generated during a given method iteration appropriately constitutes an update of the three-dimensional numerical model generated during a previous method iteration (e.g., during a previous iteration).

[0050] In addition, those skilled in the art will understand that the expression "three-dimensional scene" refers as broadly as possible to any element or combination of elements from the real world that can be observed through one or more two-dimensional digital images acquired by one or more digital image sensors. These elements include, but are not limited to, one or more (in large or small objects) pictures or animals, scenery, places, buildings seen from the outside, rooms or spaces in buildings, etc.

[0051] During step 401, computing unit 203 receives data corresponding to two-dimensional digital images acquired by image sensors 202a, 202b and 202c. All these two-dimensional digital images represent the same observed three-dimensional scene, in this case three-dimensional scene 201, from corresponding positions in space (i.e., different from each other). In other words, each sensor acquires a two-dimensional digital image representing three-dimensional scene 201 from a specific viewing angle that is different from the viewing angles of other two-dimensional digital images of the scene that can be obtained.

[0052] exist Figure 2 and Figure 3In the example shown, three image sensors 202a, 202b, 202c form an acquisition unit 202 that repeatedly acquires three-in-one digital images representing a three-dimensional scene 201 from different viewpoints (i.e., from different perspectives). However, a person skilled in the art will appreciate that the number of image sensors used is not limited to three, but only has to be greater than or equal to one. In practice, the various three-dimensional reconstruction calculation methods performed in the remainder of the method may require only one two-dimensional image in order to operate, or use multiple two-dimensional images of the same scene, the multiple two-dimensional images being acquired by a single image sensor from different perspectives at different times, or by multiple sensors at different orientations toward the scene.

[0053] In general, the image sensors used to acquire two-dimensional digital images can be any digital sensor traditionally used for such acquisition. For example, they can be passive video sensors of the CMOS (Complementary Metal Oxide Semiconductor) type or CCD (Charge Coupled Device) type. In addition, these sensors can be integrated into image acquisition devices accessible to the public, such as video cameras, smartphones or even augmented reality glasses. Finally, in certain embodiments of the present invention, these sensors acquire digital images in color, which makes it possible to integrate textures in the three-dimensional numerical model generated by executing the method. In this way, the final rendering acquired is more realistic than the observed three-dimensional scene.

[0054] The computing unit 203 comprises hardware and software means which allow the computing unit to perform the steps of the method. Typically, the computing unit comprises a mainboard, a processor and at least one memory. Furthermore, in a particular embodiment, the image sensor and the computing unit may be integrated into the same device or apparatus. For example, the device may be a smartphone which acquires a two-dimensional digital image which is directly processed by its computing unit. Advantageously, the transmission of data related to the image is thus simple and secure.

[0055] In a method variant not shown, some steps, in particular the steps described below, can be performed in whole or in part by a second remote computing unit cooperating with the computing unit 203. For example, the computing unit 203 can exchange data (in particular image data) with a remote computer. Advantageously, this can increase the available computing power and thus the execution speed of the method steps. In addition, such data can be transmitted according to a protocol suitable for such communication (for example the HTTP protocol).

[0056] During step 402, the calculation unit 203 generates a first intermediate three-dimensional numerical model from the two-dimensional digital image it has received. This first intermediate three-dimensional numerical model is a 3D rendering of the observed three-dimensional scene. In an example of a non-limiting embodiment, this intermediate three-dimensional numerical model can be a depth map or a three-dimensional point cloud. In all cases, it is generated by a so-called stereo three-dimensional reconstruction calculation method. In particular, the calculation unit 203 integrates a subunit called stereo reconstruction subunit 203a, which is particularly dedicated to the reconstruction of a three-dimensional numerical model according to this calculation method.

[0057] As mentioned above, this method is known per se to the person skilled in the art. It is based in particular on observing the same three-dimensional scene from at least two different angles in order to be able to reconstruct a three-dimensional model of the scene. An example of using this method is described in the article "StereoScan: Dense 3D Reconstruction in Real-time" by Geiger et al. in 2011. In addition, there are many algorithms that apply this method that are free to use and grouped in libraries, such as the library that can be found at http: / / www.cvlibs.net / software / libviso / .

[0058] This computational method is able to reconstruct a three-dimensional numerical model from a two-dimensional digital image without requiring a priori knowledge. In other words, a three-dimensional model can be obtained based solely on the two-dimensional image, without requiring any data previously known and used by the computational method.

[0059] However, the proper use of a priori, that is to say the use of data used as a starting point for the stereo reconstruction calculation, allows a gain in calculation time and improves the accuracy of the method. In fact, this calculation method proceeds iteratively by gradually reducing the size of the calculated region of interest (i.e., volume), in which the three-dimensional surfaces of the various elements constituting the observed three-dimensional scene are reconstructed. The use of a priori therefore makes it possible to target the region of interest more accurately and more quickly.

[0060] Figure 5 It is a schematic description of the iterative process of the stereoscopic three-dimensional reconstruction calculation method.

[0061] In the example shown, two-dimensional image sensors 501 and 502 acquire two-dimensional digital images representing the same three-dimensional surface 508 (which is itself included in a larger observed three-dimensional scene).

[0062] In order to best estimate the shape of the surface 508, the calculation is performed in a search volume 503 that delimits the calculated region of interest. A first estimate 505 of the shape (i.e., the first iteration of the calculation) is performed in this volume. In particular, within this volume 503, the calculation method is applied to three-dimensional pixels 504a (i.e., a volumetric sub-region of the three-dimensional space of the integrated scene). For each of these three-dimensional pixels 504a, a photometric quantitative index is calculated, for example an index of the NCC (normalized cross correlation) type. A second estimate 506 is performed only in three-dimensional pixels whose photometric quantitative index is greater than a determined value. In particular, the selected three-dimensional pixel 504a is subdivided into a plurality of smaller three-dimensional pixels 504b, in each of which a new estimate is performed.

[0063] The final estimate 507 occurs after a determined number of iterations that is sufficient to achieve the desired level of accuracy in the shape estimate of the three-dimensional surface. Specifically, the accuracy achieved comes directly from the size of the three-dimensional pixels 504c used during the final estimate. A three-dimensional numerical model representing the three-dimensional surface and other elements is then generated from the final estimate.

[0064] Reference now Figure 4 , step 403 comprises generating, by means of the computing unit 203, a second intermediate three-dimensional numerical model from the two-dimensional digital image that has been received. This second intermediate three-dimensional numerical model also represents the observed three-dimensional scene. Furthermore, in the case of a three-dimensional numerical model generated by a learning computing method, this model may only be a depth map. However, the latter can easily be subsequently converted into a three-dimensional point cloud according to methods known to those skilled in the art.

[0065] However, unlike the first intermediate three-dimensional numerical model described above, this model is generated by means of a three-dimensional reconstruction calculation method through learning. Specifically, the calculation unit 203 integrates a subunit called a learning reconstruction subunit 203b, which is specifically dedicated to the reconstruction of the three-dimensional numerical model according to the calculation method.

[0066] Again, as mentioned above, this method is known to those skilled in the art, so that a detailed description thereof would be beyond the scope of this specification. It will be appreciated by those skilled in the art that the execution of the method is particularly based on prior learning (so-called training). During this learning period, the execution of the method uses a large database in which many two-dimensional digital images have been associated with three-dimensional numerical models. Examples of the use of this method are described in, for example, the article "Deeperdepth prediction with fully convolutional residual networks" by Laina et al. in 2016. In addition, there are many algorithms that apply this method, which are free to use and grouped in a repository, which can be found, for example, at https: / / github.com / iro-cp / FCRN-DepthPrediction.

[0067] Those skilled in the art will appreciate that the order of steps 402 and 403 is immaterial and is not limited to the order in which steps 402 and 403 are described herein. Figure 4 The step diagram describes the example.

[0068] Figure 6 is a schematic description of an example of a training process for a method of computing a three-dimensional reconstruction by learning.

[0069] exist Figure 6 In the example shown, the computational method is based on a deep learning neural network or DNN (deep neural network) 602 to reconstruct a three-dimensional numerical model 604 from a single two-dimensional digital image 601. The network can learn the weights to be assigned to each learning layer from a training set (i.e., from a database) that couples each determined two-dimensional digital image with a given three-dimensional numerical model, in order to predict the three-dimensional numerical model as accurately as possible from the associated two-dimensional digital image. Once the neural network is trained, it can be used predictively. Therefore, advantageously, the method only requires one image of the observed scene to generate a three-dimensional numerical model of the scene of interest.

[0070] The process of learning such a DNN is known to those skilled in the art. Figure 6 An example of this process is shown in a simplified manner. The first stage of learning executes three convolutional layers 603a, 603b, 603c in succession. Each time these convolutional layers are executed after the first time, the size of the convolutional matrices (also called kernels) associated with them respectively is reduced compared to the previously executed layer. The second stage consists of executing two layers of fully connected neural networks 603d and 603e. The final layer 603e is fully connected to the final depth map.

[0071] return Figure 4, in step 404, the computing unit 203 accesses the decision unit 203c to generate a final three-dimensional numerical model by combining the first and second intermediate three-dimensional numerical models. Specifically, the computing unit 203 is adapted to select each part of the final three-dimensional numerical model to be generated in the first intermediate three-dimensional numerical model or the second intermediate two-dimensional numerical model based on the comparison of at least one characteristic parameter of the first and second intermediate three-dimensional numerical models according to the quality standard of the presentation of the observed three-dimensional scene. The characteristic parameter represents the quality of the presentation of the observed three-dimensional scene. In other words, the computing unit selects the corresponding parts of the first and second intermediate three-dimensional numerical models (generated by the stereo calculation method and the learning calculation method, respectively), and the comparison of the characteristic parameters of the first and second intermediate three-dimensional numerical models maximizes the standard of the presentation quality of the observed three-dimensional scene for the parts.

[0072] The quality criterion of the representation of the observed scene enables each time (that is to say for each part of the final three-dimensional numerical model to be generated) to select the intermediate three-dimensional numerical model that most accurately represents the observed three-dimensional scene. Of the two intermediate three-dimensional numerical models, this is the model whose corresponding part maximizes this quality criterion. In some embodiments, this result can be obtained by comparing a characteristic parameter common to both models, based on which a part of one of the intermediate models and a part of the other of the intermediate models are selected, respectively, according to the comparison result, these parts corresponding to the considered parts of the final model to be formed, i.e. the parts that model the same part of the observed scene.

[0073] For example, in a particular embodiment of the method, the characteristic parameters of the intermediate three-dimensional numerical models may be the uncertainty associated with each value that has been determined for each three-dimensional numerical model, i.e. determined during method steps 402 and 403. More precisely, the decision unit is adapted to compare the known uncertainties between the two models for each three-dimensional pixel of the intermediate model respectively associated with the same determined point in the space of the observed three-dimensional scene.

[0074] In this embodiment, it can be set as follows:

[0075] - if only the first or the second intermediate three-dimensional numerical model has a value for a given three-dimensional pixel of the observed three-dimensional scene, this value is assigned to the final three-dimensional numerical model;

[0076] - if either the first or the second intermediate three-dimensional numerical model both have a value for a given three-dimensional pixel of the observed three-dimensional scene, the value assigned to the final three-dimensional numerical model is selected from the two corresponding values ​​based on a comparison of the respective uncertainties of the two three-dimensional numerical models associated with the value of the three-dimensional pixel; and finally

[0077] If no intermediate three-dimensional numerical model has a value for a given three-dimensional pixel of the observed three-dimensional scene, no value is assigned to this three-dimensional pixel, or in a variant the value assigned to this three-dimensional pixel is determined according to a so-called smoothing calculation method. One such method consists in interpolating element values ​​based on the values ​​of neighboring three-dimensional pixels in the observed three-dimensional scene.

[0078] In summary, this method makes it possible to optimize the accuracy of the final three-dimensional numerical model obtained by combining the respective qualities of two different processes. The model is thus as faithful as possible to the actual observed three-dimensional scene.

[0079] Those skilled in the art will appreciate that the present invention can be extended to more than two intermediate three-dimensional numerical models, and to more than two two-dimensional images of the observed three-dimensional scene. For example, a plurality of intermediate three-dimensional numerical models of any type considered above can be generated from corresponding two-dimensional digital image p-tuples, where p is an integer greater than or equal to 1. Furthermore, types of intermediate three-dimensional numerical models other than the two types described above are also conceivable, based on the specific characteristics of each application.

[0080] Furthermore, in certain embodiments of the method, the final three-dimensional numerical model can be returned to the user via a display device. For example, the model can be projected onto a two-dimensional scanning or LED display screen, onto virtual or augmented reality glasses, or even onto a three-dimensional display screen.

[0081] Finally, in order to reduce the duration of the calculations performed during the step of generating the three-dimensional numerical model, the calculations inherent to the calculation method can be performed according to a calculation model of the foveal imaging type. The term "foveal" refers, by analogy with human vision, to a calculation model that seeks to obtain results that are more accurate in the center of the image or calculation and less accurate at the edges. Thus, the calculation power is reduced at the edges and results in resource savings, thereby reducing the overall calculation time.

[0082] In addition to the advantages already described, further advantages according to the present invention result from the following factor: The combination of two different calculation methods also enables the two methods to cooperate with each other to improve their respective performances.

[0083] For example, firstly, it is possible to store in a memory the digital information obtained at the end of an iteration of the method in order to use it to improve the performance of the method during subsequent iterations, i.e. to generate another final three-dimensional numerical model of the same observed scene or another observed scene, for example considered from another observation point. More precisely, the calculation unit 203 can transmit to the memory 205 a set of data comprising the two-dimensional digital image it has received, the final three-dimensional numerical model associated with said two-dimensional digital image and one or more parameter characteristics of this model. In this way, the digital information stored in this way can be reused by the calculation unit as learning data, which allows it to train or retrain the calculation unit used for the three-dimensional reconstruction calculation method through learning, if necessary. Therefore, advantageously, in addition to the database initially used to train the calculation unit, each new iteration of the method completes and enriches the learning of the calculation unit used to continuously perform the method, and therefore completes and enriches its performance level.

[0084] Furthermore, in a particular embodiment of the method, the data described above and / or other additional data can help improve the learning of the computing unit during a phase referred to as the relearning phase. For example, it can be data from a car manufacturer providing a set of digital images presenting a vehicle, which are associated with a three-dimensional numerical model of the vehicle predefined based on the images. It can also be data generated by another comparable computing unit (e.g., by another device or by another user) executing the same method, or even data generated by the execution of another method (e.g., laser acquisition of a three-dimensional scene) that is unrelated to the present invention but still leads to the generation of a three-dimensional numerical model from a two-dimensional digital image. In addition, the data can also come from a learning data sharing platform that is capable of synthesizing such data from multiple sources. In all cases, the three-dimensional reconstruction computing method can be regularly retrained to improve its performance level. With particular reference to the self-enrichment of the method. In fact, advantageously, each iteration of the method can improve its performance.

[0085] Secondly, as already described above, the stereoscopic 3D reconstruction calculation method can benefit from the use of priors in its calculation process. Therefore, as long as the 3D reconstruction calculation method through learning is trained before any specific implementation (iteration) of the method, the latter can provide a 3D numerical model that can be used as a priori for the stereoscopic 3D reconstruction calculation method when appropriate, thanks to the use of a specific database called the learning database.

[0086] Figure 7 is a diagram of the steps of a specific embodiment of a method for using this capability.

[0087] In this embodiment, reference has been made to Figure 4Each of the steps 401 to 404 described is duplicated. However, the order in which these steps are unfolded is specific. Specifically, step 403 for generating the second intermediate three-dimensional numerical model is performed before step 402 for generating the first intermediate three-dimensional numerical model. In this way, the generated second intermediate three-dimensional numerical model is used as a priori by the stereoscopic three-dimensional reconstruction calculation method to generate its three-dimensional numerical model more accurately and faster. For example, referring to Figure 5 The use of this prior makes it possible to start directly at an estimation step 506 and any other subsequent estimation steps that are more accurate than the estimation step 505 in the iterative process of the stereoscopic three-dimensional reconstruction calculation method shown in . In addition, step 404 of the method remains unchanged, and the final three-dimensional numerical model remains the combination of the two intermediate models, and in all cases, for each part of the final model, the combination of the two intermediate models uses the most accurate (i.e., the most faithful to the observed three-dimensional scene) intermediate model.

[0088] The present invention has been described and illustrated in possible embodiments in the detailed description and the accompanying drawings. However, the present invention is not limited to the embodiments presented. By reading the description and the accompanying drawings, those skilled in the art will be able to deduce and implement other variations and embodiments.

[0089] In the claims, the terms "comprising" and "having" do not exclude other elements or other steps. A single processor or a plurality of other units may be used to perform the invention. The various features presented and / or claimed can be advantageously combined. Their appearance in the description or in different independent claims does not exclude this possibility. The reference signs shall not be understood as limiting the scope of the invention.

Claims

1. A dynamic three-dimensional imaging method, comprising the following steps performed by a computing unit: receiving at least two two-dimensional digital images representing an observed three-dimensional scene, the at least two two-dimensional digital images being acquired by at least one determined image sensor from at least two corresponding positions in space; generating a first intermediate three-dimensional numerical model associated with the observed three-dimensional scene from the two-dimensional digital image based on a stereoscopic three-dimensional reconstruction calculation method; generating a second intermediate three-dimensional numerical model associated with the observed three-dimensional scene from the two-dimensional digital image based on a three-dimensional reconstruction calculation method through learning; Generating a final three-dimensional numerical model by combining the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model, the generating comprising: for a determined portion of the final three-dimensional numerical model, selecting a corresponding portion of the determined portion of the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model based on a comparison of at least one characteristic parameter of the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model for the portion, maximizing a rendering quality criterion of the observed three-dimensional scene, the characteristic parameter representing a rendering quality of the observed three-dimensional scene; storing digital information in a memory, the digital information comprising at least one received two-dimensional digital image, a final three-dimensional numerical model associated with the two-dimensional digital image, and characteristic parameters of the final three-dimensional numerical model representing a rendering quality of the observed three-dimensional scene; and The stored digital information is used as learning data to train and retrain the computing unit to perform the three-dimensional reconstruction computing method through learning.

2. The method according to claim 1 further comprises acquiring a two-dimensional digital image representing the observed three-dimensional scene through at least one determined image sensor.

3. The method according to claim 2, wherein: The at least one determined image sensor is a passive video sensor of the CMOS or CCD type.

4. The method according to one of claims 1 to 3, further comprising displaying the final three-dimensional numerical model through a display device, wherein the display device is a two-dimensional scanning or LED display screen, virtual or augmented reality glasses, or a three-dimensional display screen.

5. The method according to claim 1, wherein: Generating the final three-dimensional numerical model by combining the first intermediate three-dimensional numerical model and the second intermediate three-dimensional numerical model comprises: If only the first intermediate three-dimensional numerical model or the second intermediate three-dimensional numerical model has a value for a three-dimensional pixel modelling a given part of the representation of the observed three-dimensional scene, this value is assigned to the final three-dimensional numerical model; If both the first intermediate three-dimensional numerical model or the second intermediate three-dimensional numerical model have a value for a three-dimensional pixel modelling a given part of the representation of the observed three-dimensional scene, the value assigned to the final three-dimensional numerical model is selected from the two corresponding values ​​based on a comparison of at least one characteristic parameter of the three-dimensional numerical models, the characteristic parameter being associated with the value of the three-dimensional pixel; and If no three-dimensional numerical model has a value for a three-dimensional pixel modeling a given portion of the representation of the observed three-dimensional scene, no value is assigned to the three-dimensional pixel or the value assigned to the three-dimensional pixel is determined according to a smoothing calculation method.

6. The method according to claim 1, wherein: The generation of the second intermediate three-dimensional numerical model is performed before the generation of the first intermediate three-dimensional numerical model, and wherein the generated second intermediate three-dimensional numerical model is used as a priori for generating the first intermediate three-dimensional numerical model by a stereoscopic three-dimensional reconstruction calculation method.

7. The method according to claim 1, wherein: The characteristic parameter of the three-dimensional numerical model is the uncertainty associated with each value of the three-dimensional numerical model.

8. The method according to claim 1, wherein: The learning data used by the computing unit to train and retrain the learned three-dimensional reconstruction computing method includes: Data from shared or unshared databases, including two-dimensional digital images and associated three-dimensional numerical models; and / or Data comprising a two-dimensional digital image and an associated three-dimensional numerical model, the data being generated by another computing unit executing the method; and / or The data comprises a two-dimensional digital image and an associated three-dimensional numerical model, the data being generated by a computing unit executing another method.

9. The method according to claim 1, wherein: Each intermediate 3D numerical model is either a depth map or a 3D point cloud.

10. The method according to claim 1, wherein: During one or more steps of generating the intermediate three-dimensional numerical models, a second remote computing unit cooperates with the computing unit to generate the one or more intermediate three-dimensional numerical models.

11. The method according to claim 1, wherein: The received two-dimensional digital image is a color digital image, and wherein the generated final three-dimensional numerical model includes a texture determined based on the color of the two-dimensional digital image.

12. The method according to claim 1, wherein: During one or more steps of generating an intermediate three-dimensional numerical model, the three-dimensional reconstruction calculations performed use a foveal imaging computational model to obtain an intermediate three-dimensional numerical model that is relatively more accurate in the portion of the model projected onto the center of the image and relatively less accurate in the portion of the model projected onto the edge of the image.

13. A computing unit, comprising: Motherboard; processor; as well as at least one memory; The at least one memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to enable the processor to perform the method according to one of claims 1 to 12.

14. A three-dimensional imaging device comprising at least one digital image sensor, a memory and a computing unit according to claim 13.

Images