METHOD FOR PREDICTING A THREE-DIMENSIONAL (3D) REPRESENTATION AND CORRESPONDING DEVICE, SYSTEM AND COMPUTER PROGRAM

Description

Domain of invention

[0001] The general field of the invention is the processing of three-dimensional (3D) scenes. The invention can be applied in particular to the tracking of a three-dimensional target by a robot or to the determination of the pose parameters of a camera of an augmented reality system. Prior art

[0002] The invention falls within the framework of the prediction of three-dimensional representations.

[0003] The article Image-based localization using LSTMs for structured feature correlation, in IEEE International Conference on Computer Vision 5ICCV, Oct 2017, by F. Walch, C. Hazirbas, L. Leal-Taixe, T. Sattler, S. Hilsenbeck, and D. Cremers describes a solution based on the use of deep learning to determine the location of a three-dimensional structure. One of the advantages of this solution is that it does not require the storage of reference images. However, it does not allow the localization of 3D structures that are in motion.

[0004] We know from the paper Duong Nam-Duong et al. “Accurate Sparse Feature Regression Forest Learning for Real-time Camera Relocalization” 2018 International conference on 3D vision (3DV), IEEE 2018-09-05, pages 643-652, XP033420103 and from the paper Tommaso Cavallari “On-the-fly Adaptation of Regression Forests for Online Camera Relocalization” ARXIV.ORG, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, 2017-02-09, XP0800747283 solutions based on a regression tree forest to estimate the pose parameters of a camera used for capturing a 2D image and disclosing the preamble of claim 1.

[0005] The invention aims, among other things, to remedy the drawback of the aforementioned prior art. Exposition of the invention

[0006] According to a first aspect of the invention, there is provided a method for predicting a three-dimensional (3D) representation in a three-dimensional scene, using one or more two-dimensional (2D) representations (preferably invariant), obtained from at least one image generated by a camera. According to a general characteristic of this first aspect of the invention, the method comprises obtaining a forest of regression trees, and for said at least one image: an extraction of a 2D representation associated with one or more positions within the image, a prediction of a 3D representation, corresponding to the extracted 2D representation, using the regression tree forest which defines includes a set of possible associations between at least one 2D representation and one 3D representation, each possible association resulting from a predictive model, an evaluation of one or more of the possible associations defined by the regression tree forest, according to a predetermined confidence criterion, and an update of the regression tree forest comprising a deactivation of one or more possible associations, depending on the evaluation of said possible associations.

[0007] In other words, the invention proposes a method for dynamically evolving a forest of regression trees. This evolution can, for example, take place when one or more objects in the scene considered are in motion. An update of the regression model, in this case the forest of regression trees, makes it possible to improve the discriminating power of the latter and therefore the quality of the prediction made. In addition, adapting the predictive model that constitutes the forest of regression trees implies better robustness in the face of external disturbances.

[0008] According to the invention, during the evaluation step, the prediction can be evaluated for at least one possible association, as being correct or incorrect according to the predetermined confidence criterion.

[0009] Determining a confidence criterion makes it possible to qualify a prediction as probably correct or incorrect, and therefore to act accordingly with regard to the different stages dependent on this prediction.

[0010] According to the invention, the predictive model of a possible association is defined by a distribution characterizing the at least one 2D representation associated with the 3D representation for said possible association, and by at least one parameter from among the set of the following parameters: a first parameter representative of a status of the predictive model of the possible association; a second parameter representative of a number of consecutive predictions evaluated as being incorrect for the possible association; a third parameter representative of the extracted 2D representations, associated with the 3D representation by said possible association.

[0011] By proposing new parameters to define the predictive model, it is possible to control the evolution of the regression tree forest. Thus, the parameter representing a "status" (for example a Boolean) makes it possible to control on the one hand the activity and the influence of a possible association on the final prediction, and on the other hand the application of the update process of the other parameters of this same predictive model. The second parameter acts as a histogram of the predictions considered correct. It plays the role of an alarm, allowing the update of the regression tree forest to be initiated. Finally, the third parameter makes it possible to update the 3D representation during the evolution of the regression model.

[0012] According to one implementation mode, the status of the predictive model of a possible association is 'active' when said association is activated or 'passive' when said association is deactivated.

[0013] Choosing a binary parameter that represents the status of a predictive model allows for effective control of the update process. This helps avoid irrelevant updates.

[0014] According to one embodiment, said update of the regression tree forest comprises for at least one predictive model of a possible association whose status is 'active' an update of the value of the second parameter for said possible association, based on the result of the evaluation, and if the value of said second parameter is greater than a predetermined threshold, a deactivation of said possible association, otherwise maintaining the 'active' status of the predictive model of said possible association.

[0015] The predictive model of a possible association uses the second parameter to evaluate the quality of the possible association for several successive 2D representations. The value of this second parameter reflects the sensitivity and stability of the regression model and makes it possible to anticipate its failure and adapt it automatically in time.

[0016] According to one implementation mode, the update of the regression tree forest includes for at least one predictive model of a possible association, an update of the value of the third parameter depending on the extracted 2D representation, and for at least one predictive model of a possible association having the 'passive' status: if said updated third parameter meets a predetermined condition, a new determination of the distribution characterizing the 3D representation relating to said possible association and a new determination of the 3D representation of said possible association, and ∘ if the covariance of the newly determined distribution is less than a predetermined threshold, a validation of said update of the value of the third parameter, of the newly determined distribution and of said newly determined 3D representation, and an activation of the predictive model of said possible association, ∘ otherwise a reinitialization of the value of the third parameter and a maintenance of the 'passive' status of the predictive model of said possible association.

[0017] In other words, the predictive model of a possible association is able to readjust its own parameters during the various successive predictions. To be able to definitively validate the relevance of the updates, an evaluation of the newly determined values ​​is triggered, based on a covariance of the newly determined distribution (and more precisely in practice on the trace of the covariance matrix). This control mechanism makes the update step robust to changes that should not have led to an update of the regression model.

[0018] According to one embodiment, the method may further comprise an update of the pose parameters of said camera, based on the possible associations evaluated as being correct or incorrect.

[0019] The process can be implemented by a computer.

[0020] According to a second aspect of the invention, there is provided a device for predicting a three-dimensional (3D) representation in a three-dimensional scene, using one or more two-dimensional (2D) representations, obtained from at least one image generated by a camera, said device comprising a computing machine dedicated to or configured to: obtaining a regression tree forest, and for said at least one image: extracting a 2D representation associated with one or more positions within the image, predicting a 3D representation, corresponding to the extracted 2D representation, using the regression tree forest which defines includes a set of possible associations between at least one 2D representation and one 3D representation, each possible association resulting from a predictive model, said predictive model of a possible association being defined by a distribution characterizing the at least one 2D representation associated with the 3D representation for said possible association, and by the following parameters: a first parameter representative of a status of the predictive model of the possible association; a second parameter representative of a number of consecutive predictions evaluated as being incorrect for the possible association;a third parameter representative of the extracted 2D representations, associated with the 3D representation by said possible association; evaluating one or more of the possible associations defined by the regression tree forest, according to a predetermined confidence criterion, where the prediction can be evaluated for at least one possible association, as being correct or incorrect according to the predetermined confidence criterion and updating the regression tree forest including a deactivation of one or more possible associations, depending on the evaluation of said possible associations. ;

[0021] According to one embodiment, said computing machine may be dedicated to or configured for at least one possible association, evaluating the prediction as being correct or incorrect according to the predetermined confidence criterion.

[0022] According to one embodiment, the device may further comprise a module dedicated to or configured to update pose parameters relating to said camera, based on the possible associations evaluated as being correct or incorrect.

[0023] According to a third aspect of the invention, an augmented reality system is proposed comprising: a camera capable of acquiring an image of a three-dimensional scene, a module capable of delivering an output image from an input image acquired by said camera, representing said three-dimensional scene and at least one real or virtual object, from: ∘ a 3D representation representing said at least one object in the scene, and ∘ camera pose parameters, a display module capable of rendering the output image.

[0024] According to a general characteristic of this third aspect of the invention, the system comprises a device according to the second aspect of the invention, making it possible to update the pose parameters of said camera.

[0025] According to a fourth aspect of the invention, there is provided a computer program comprising instructions for implementing a method according to any one of the embodiments of the first aspect of the invention, when said program is executed by a processor.

[0026] According to a fifth aspect of the invention, there is provided a computer-readable recording medium on which is recorded a computer program comprising program code instructions for carrying out the steps of the method according to any one of the embodiments of the first aspect of the invention. List of figures

[0027] Other advantages and characteristics of the invention will appear more clearly on reading the following description of a particular embodiment of the invention, given as a simple illustrative and non-limiting example, and the appended drawings, among which: there Figure 1 describes a mode of implementation of a method for predicting a 3D representation according to the invention; the Figure 2 represents in more detail an example of the step of predicting a 3D representation from 2D representations extracted from an image in RGB (Right Green Blue) format; the Figure 3 describes a mode of implementing a learning step so as to obtain an initial regression tree forest for a prediction of a 3D representation according to the invention; Figure 4 schematically represents an example of a regression tree forest used by the method for predicting a 3D representation according to the invention; the Figure 5 illustrates more particularly a mode of implementation of the step of updating the regression tree forest according to the invention; Figure 6 illustrates very schematically a method of triangulation of a point in three dimensions; and the Figure 7 illustrates an example of an embodiment of a device according to the invention. Description of modes of mise en œuvre particuliers de l'invention

[0028] There Figure 1 illustrates an example of an implementation mode of a method for predicting a 3D representation according to the invention.

[0029] According to this implementation mode, images 10 for example in RGB format are generated by a camera. These images can be extracted from a video sequence and be temporally independent. Conversely, these images can form a series of successive, temporally dependent images.

[0030] These images represent a three-dimensional (3D) scene captured by the camera including one or more three-dimensional objects. A prediction step 20 makes it possible, from these images 10, to predict a 3D representation of one of these objects. A 3D representation designates, for example, a single 3D point, a 3D point cloud or the three-dimensional location of this point or of the points of the point cloud.

[0031] Preferably, the images of the image set 10 are processed one by one.

[0032] The prediction is performed using an initial regression tree forest 20, matching a 3D representation with one or more two-dimensional (2D) representations. Recall that regression tree forests consist of running a large number (≈400 for example) of randomly constructed regression trees in parallel, before averaging them.

[0033] In other words, the regression tree forest defines several possible associations between one or more 2D representations and a 3D representation. A 2D representation can be, for example, a single 2D point, a 2D line, a 2D pattern, or the position of one of these elements. An example of obtaining the initial regression tree forest is detailed in the Figure 3 below.

[0034] As illustrated in more detail on the Figure 2 , the prediction 20 comprises a step 21 of extracting one or more 2D representations 22 associated with one or more positions within the RGB image considered. In the remainder of the description, it is considered that a 2D representation is defined by a characteristic fi represented by a vector fi ( v 1 , v 2,..., vd ) as defined below. For example, it comes from d = 64.

[0035] More precisely, a feature vector defining a 2D representation can be extracted from a patch, i.e. a portion of the RGB image considered, arranged or centered around a point of interest. In the remainder of the description, unless otherwise indicated, it will be considered for simplification purposes that a single patch is obtained per image, and that a feature vector is extracted from this same patch. This feature vector will allow the prediction of a 3D representation. The case of several patches per image is easily deduced from the description.

[0036] Features describe a variability and orientation of the intensities of the image pixels (or components of the pixels) in the vicinity of the point of interest. An extraction function φ allows to obtain a feature vector fi for a point of interest KP i ,is defined according to the following equation: f i = ϕ KP i .

[0037] According to a first variant, the function f can perform invariant feature extraction from the point of interest KP i of the thumbnail, for example using a SURF-type technique described in the document by Bay et al., entitled "Speeded-up Robust Features (SURF)", published in the journal Computer Vision and Image Understanding, issue 110, pages 346-359, in 2008. A feature is said to be invariant when it is associated with a point of interest that is invariant to transformations of the image, such as rotations, transformations or even changes in scale.

[0038] According to a second variant, the function fcan perform feature extraction using a supervised automatic prediction system. The supervised automatic prediction system can implement a layered neural network (CNN) as described in the article by: A. Kendall, M. Grime, and R. Cipolla: Posenet: A convolutional network for real-time 6-DOF camera relocalization, published in Proceedings of the IEEE International Conference on Computer Vision, pages 2938-2946, 2015 .

[0039] A step 23 of predicting the 3D representation from the extracted 2D representations 22 is carried out as explained below with reference to the Figure 5 .

[0040] Reference is again made to the Figure 1 . At the end of the prediction step 20, a predicted 3D representation 40 is delivered.

[0041] This is processed a posteriori in order to evaluate the accuracy of the prediction in a step 50. Results 60 are then used: on the one hand for step 23 of predicting a 3D representation from the extracted 2D representations, and on the other hand in the example of the Figure 1 , for an estimation step 70 of the camera pose parameters and more precisely, for updating the pose parameters.

[0042] More generally, the results 60 based on the prediction 40 of the 3D representation can be used to estimate parameters related to the calibration of the camera. On the Figure 1 , they are used for estimating camera pose parameters. A well-known principle of pose estimation consists of following in the scene, a marker whose position and 3D geometry are known a priori, in order to calibrate the camera (position, orientation among others).

[0043] In the context of augmented reality, a virtual object can be embedded in the three-dimensional scene using an augmented reality system based on the camera pose parameters.

[0044] In an industrial context, the parameters estimated using the results 60 and in particular the camera pose parameters can enable target tracking: when the camera allows automatic navigation (in the context of autonomous vehicles for example), when the camera is embedded for piloting a robot, or when the camera is an intelligent surveillance camera, with for example automatic tracking of an object in the filmed scene.

[0045] Reference is now made to the Figure 3 .This figure represents a flowchart that illustrates an example of obtaining the initial regression tree forest 30. Preferably, this is obtained by implementing a prior learning phase 31 from a learning database 32. An example of a prior learning phase is given in the article by N.-D. Duong, A. Kacete, C. Soladié, P.-Y. Richard, and J. Royan, Accurate sparse feature regression forest learning for real-time camera relocalization, in 2018 International Conference on 3D Vision (3DV), pages 643-652, IEEE, 2018.

[0046] The training database 32 comprises a group of images in RGB format. A set of characteristics are extracted for example using the SURF method described above with reference to the Figure 2The learning phase aims to classify each feature and therefore the 2D representation they define, using a regression tree forest. Each leaf of each tree defines a possible association between one or more 2D representations and a 3D representation.

[0047] The training phase of the regression tree forest requires labeling the data in the training base with a 3D representation, for example defined by its coordinates P i w in a three-dimensional frame of reference P i w = X i w Y i w Z i w , i representing the index of the 3D representation considered, over the set of possible 3D representations, and w representing a first reference frame used ("world referential" in English). In other words, labeling involves associating one or more 2D representations extracted from the images in the database with a 3D representation defined here by its coordinates.

[0048] One possible method consists of performing a triangulation between two RGB images representing a scene at at least two different viewpoints, in order to achieve a coupling between a point of interest found in each of the images, and a three-dimensional point in the first reference frame. Finally, according to another method, it is possible to use for the database 32, RGB-D images from the correctly calibrated camera. An RGB-D image makes it possible to associate with each pixel of the image, a depth value obtained from the camera pose parameters. In this last method, from the position of the point of interest pi = ( ui , vi ) detected in an RGB-D image and the depth value D i associated with it, an associated 3D representation P i c = X i c Y i c Z i c in a second frame of reference linked to the camera, is calculated. This calculation is carried out using a model of the camera, based on a standard model of a pinhole camera as defined below: P i c = D i K − 1 p i 1 .

[0049] Where K is a matrix defining the intrinsic parameters of the camera, and the exponent c denotes the second frame of reference, linked to the camera. The camera pose is defined by T = [ R | t ] including a rotation matrix R and a translation vector t for each image, which are assumed to be known beforehand. The 3D representation P i w = X i w Y i w Z i w defined in the first reference frame, corresponding to the point of interest pi is obtained using the following equation: P i w 1 = R t 0 1 P i c 1 .

[0050] There Figure 4illustrates a tree of a regression tree forest. A so-called initial regression tree forest 30 is obtained by learning during step 31.

[0051] A regression tree forest is formed by a set of trees, an ABR example of which is shown in this Figure 4 . Each tree is formed of nodes NP i , each node being associated with a split function. The first node of the tree is called the root node NR. Each terminal node NT forms a leaf which represents a possible association between one or more 2D representations and a 3D representation. This possible association results from a predictive model based on the different classifications made by each split function which led to the terminal node NT.

[0052] More precisely, an NP i , NR node is characterized by the parameters of its cutoff function. At the beginning of the training phase 31, each cutoff function is said to be weak, that is, for a parent node NP i , the cutoff function distributes the data into a left-hand child node NFG i or a right-hand child node NFD i with a result slightly better than a random distribution. The weak cutoff function for node NP i is parameterized by θ i = { ref i , τ i} , Or ref i is a reference feature vector (or 2D representation), and the is a predetermined threshold.

[0053] During the training phase 31, a set of feature vectors is extracted, then for each tree in the regression tree forest, a subset Yes of feature vectors is randomly chosen from the extracted set.

[0054] So, for each tree, a feature vector fj ( v 1 , v 2,..., vd ) (j being an index of the characteristic varying between 1 and an integer K) of the corresponding subset Yes , is processed from the root node NR to a terminal node NT (leaf), with a classification carried out at each node NP i by the cut function associated with the node considered: h f j θ i = 0 , si d ref i f j < τ i , classification dans le noeud gauche 1 , si d ref i f j ≥ τ i , classification dans le noeud droit Or d ref i f j = ref i − f j 2 2 is a Euclidean distance function i i = {refi , the | refi ∈ S i}.

[0055] The cut-off function h associated with each node NP i of the tree considered, is therefore applied to each of the characteristics fj of this subset Yes so as to classify it in a right node NFD i S i D or left NFG i S i L . The cutoff function h, then called the weak prediction function (or predictor), is then trained to minimize the spatial variance of the feature vectors classified in the NT or leaf nodes. This principle is described for example in the article by J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon, Scene coordinate regression forests for camera relocalization in rgb-d images, in Proceedings of the IEEE conference of Computer Vision and Pattern Recognition, pages 2930-2937, 2013. More precisely, the parameters of the weak predictor are optimized by minimizing an entropic energy, based on an information gain Q calculated on the result of the separation carried out by the weak predictor which can be defined as follows: Q S i θ i = ∑ d ∈ L , R S i d θ i S i V S i d θ i with V S = 1 S ∑ m ∈ S P i w − P i w ¯ 2 2 Or Q corresponds to the information gain which represents an entropic measure allowing the purity of each node to be evaluated, that is to say the degree of homogeneity of the characteristics associated with it at each classification carried out by the cut-off function, And denote the nodes respectively to the left and to the right of a node NP i , respectively associated with characteristics fj ; m represents the coordinates of the characteristic vector considered in the first frame and m represents an average of these coordinates.

[0056] Note that in the case of an unbalanced tree (i.e., with an unbalanced distribution of 2D representations in the terminal nodes), deep learning may be required with a higher learning time. However, to limit this learning time without compromising on the prediction accuracy, the parameters of the weak cutoff functions can be balanced from the higher levels of the tree. Thus, the cutoff function of a higher-level NPi node learns to balance the feature vectors between its right and left node by choosing a reference feature ref i and a threshold the adequate, such as: θ i = ref i τ i S i L = S i R .

[0057] The initial regression tree forest is acquired when a maximum depth value is obtained for each tree or the number of data classified in a node does not fall below a previously predefined minimum value. In each terminal node NT (leaf) defining a possible association 2D representations / 3D representation, the 2D representations are characterized by a Gaussian distribution or ( P , P, Σ P ), where Σ P represents the covariance of the Gaussian distribution, where P symbolizes the coordinates P i w = X i w Y i w Z i w .

[0058] We now refer to the Figure 5 which more precisely describes step 20 of predicting a 3D representation and evaluating 50.

[0059] Step 23 comprises a sub-step 236 of predicting a 3D representation from feature vectors extracted from a thumbnail from an image of index k and from a regression tree forest of index k 235.

[0060] This forest 235 was updated during a sub-step 230, from the validated results obtained for feature vectors extracted from a thumbnail from an image of index k-1, 60a and from the regression tree forest of index k-1 used to process the feature vectors extracted from the image of index k-1. The indices correspond to an order of processing of the images and do not necessarily correspond to a temporal order.

[0061] According to the invention the predictive model leading to each terminal or leaf node l of the trees of the k-1 regression forest is characterized by a Gaussian distribution or ( P,P, Σ P ) as described above with reference to the Figure 4 , and three parameters (or at least one of these three parameters, by inferring the other parameters for example), such that: l = η s n false S data Or, scorresponds to the status of the predictive model of a leaf - preferably it can take two values: "0": passive status or "1": active status - a leaf whose predictive model has the status "passive" is either a leaf having a large variance or a leaf corresponding to an association with a 3D representation corresponding to a moving object in the captured scene - the predictions for which the 2D representations are assigned to leaves whose status of the predictive model is "passive" are not considered to be a prediction 40, only the leaves whose predictive model has the status "active" are - when at the end of the evaluation step 50, the accuracy of a prediction is called into question, the status of the predictive model of the leaf in question is changed to "passive" - ​​the "active" status can be restored later as described below; n falsecorresponds to the number of consecutive preference predictions, which were validated as incorrect during step 50 - for example, the value "0" is assigned to this parameter when a prediction is considered to be fair or correct - as explained below, this parameter allows to restore an "active" status; S data is representative of the extracted 2D representations, which are associated with the 3D representation by the sheet considered - this parameter also allows to restore an “active” status.

[0062] Firstly, a step 232 makes it possible to detect possible associations to be inactivated within the k-1 regression tree forest. For this, step 232 receives as input the results 60a from the processing of the k-1 image. These results 60a make it possible to count the predictionsincorrect ("outliers") for each possible association (or leaf) whose predictive model status is "active". If a leaf, whose predictive model status is active, has resulted in an incorrect prediction, the parameter n false is incremented by 1. On the contrary, if a sheet whose predictive model status is "active" has given rise to a correct prediction ("inlier" in English), the value of the parameter n false is reset. Finally, if it is detected that n false > max false , max false being a predetermined threshold, the status s of the predictive model of the sheet in question becomes "passive". Indeed, the possible association gave rise to max false consecutive incorrect predictions, it is therefore inactivated for the rest of the process, that is to say that any prediction made from an inactivated leaf is not considered during the evaluation step 50.

[0063] At the output of the detection step 232, we therefore obtain a modified k-1 regression tree forest 233, with leaves having the value of their parameter n false e update, and potentially the status of their inactivated predictive model.

[0064] This modified k-1 regression tree forest 233 is delivered as input to a step 234 of updating the parameters of all the leaves of the k-1 regression tree. The use of invariant characteristics allows the use of the same regression tree forest for the same three-dimensional scene, even when objects are moving within the scene. By only updating the parameters of the resulting leaves, an optimal prediction of the 3D representations can be carried out. The purpose of step 234 is precisely to update the parameters characterizing the predictive model of the leaves whose status is “passive”. During the processing of each image, statistics vectors are classified using the prediction tree forest, making it possible to collect for each leaf, a set of statistics vectors.

[0065] Thus, step 234 includes updating the contents of the parameter S datafor each leaf whose predictive model has a passive status, by adding to a stack for example, each new feature vector classified in the leaf whose predictive model has a "passive" status. If the size of the stack S data is greater than a predetermined threshold max data (| S data | > max data ), a new distribution or ( P,P, Σ P ) of the passive sheet is determined. The coordinates of the 3D representation, P , are then recalculated using a function F label such as P = F label ( S data ). The predictive model of the regression tree forest is thus redefined. Finally, if |Σ P | < max var (or if the trace of the covariance matrix is ​​lower than a predetermined threshold), max var being a predetermined threshold, the update of the Gaussian distribution and the update of Pis validated and the "active" status is reactivated. Conversely, if |Σ P | ≥ max var the feature vector stack of S data is reset and the status is kept at “passive”.

[0066] The regression tree forest 235 is then delivered to predict a 3D representation from the feature vectors (2D representations) extracted from the image of index k.

[0067] The invention therefore proposes to use a dynamic regression tree forest, where the possible associations between one or more 2D representations and a 3D representation can evolve over time. The invention is particularly suitable for estimating the pose parameters of a camera capturing a three-dimensional scene in which one or more objects are in motion. The modification of the regression tree forest generated by the movement of an object results in a better on-the-fly estimation of the pose parameters, for a better capture of the scene. This update of the camera pose parameters is described in more detail with reference to the Figure 6 below.

[0068] Note that for the first image processed, the regression tree forest used is the initial regression tree forest 30. Steps 232 and 234 do not modify the forest 30 when processing the first image, in the absence of existing results 60 a. Then this initial regression tree forest 30 is progressively updated, as images are processed and results are generated.

[0069] The prediction step 236 classifies the feature vectors 22 using the regression tree forest of index k, so as to predict a 3D representation. If we generalize to several thumbnails obtained per image, for image k there comes a set of feature vectors { fi} corresponding respectively to a set of thumbnails { I am}. Each feature vector is processed by the decision tree forest, which returns for an input { I am i , fi} a prediction of a 3D representation to a sheet l j i of the tree of index j of said forest. When all the feature vectors are processed, a set LP = l j i of leaves having given rise to a prediction is obtained. All leaves of the LP set having an “active” status are used as input to a post-processing step 501 to calculate processed predictions 502 using a post-processing function F post such as the one proposed below: F post Im i , f i , l j i s j i = 1 Or s j i is the status of the leaf (or possible association) of tree j used for the prediction.

[0070] Then a validation step 503 allows a confidence criterion to be calculated for the post-processed predictions. The validation step allows those which are correct (“inliers” in English) or incorrect (“outliers” in English) to be defined. Validation is carried out using a validation function F value , For example : F val α F post Im i l j i = 1 , α . < max val , pr é diction correcte 0 , α . ≥ max val , pr é diction incorrecte Or α F post Im i l j i is a prediction delay function associated with the leaf l j i based on F post and the thumbnail image I am The post-processing steps 501 and validation steps 503 form the evaluation step 50.

[0071] After validation, 60 results are obtained which include n inliers predictions considered correct and n outlierspredictions considered to be incorrect. A confidence criterion is determined and compared to a predetermined threshold, so that the update carried out during step 234 is validated when the confidence criterion is greater than the predetermined threshold such that: n inliers n inliers + n outliers > max conf , Or max conf is a threshold that allows validation of the update step 234 for the regression tree forest of index k, which will be used for processing the image k+1.

[0072] We now refer to the Figure 6which illustrates the step of estimating the pose parameters of the camera 70. This step takes into account only the predictions associated with leaves whose predictive model has the “active” status. According to one embodiment, the results corresponding to three points of interest in an input image k may be required to perform the estimation. According to another embodiment using an algorithm known to those skilled in the art, called PnP (for “Perspective-n-Points” in English), makes it possible to overcome this constraint. Another embodiment proposes to combine the PnP algorithm and a method called RANSAC (for “RANdom Sample Consensus” in English) to delete the predictions defined as incorrect (“outliers”) and keep only the predictions defined as correct (“inliers”).

[0073] During the validation step 503 prior to the estimation step 70, the RANSAC method then makes it possible to generate a set of hypothetical poses = { } and applying the PnP algorithm on a randomly chosen subset of the 3D representation predictions. Then the most satisfactory correct predictions are obtained by maximizing this number of correct predictions corresponding to each hypothetical pose based on a re-projection error: ∀ T i ∈ T max ∑ p j ∈ p ρ α ij ρ α ij = 1 , si α ij < τ 0 , sinon Or a ij = p j − KT i − 1 P ^ j w 2 And t is the maximum reprojection error threshold that defines a correct prediction and where the function r (.) is a metric used to assess the accuracy or otherwise of a prediction. This function is another example of a validation function F value of step 503, in the case where the results 60 are used for an estimation of the pose parameters.

[0074] The point of interest transformed by the pose of index i, having an index j , is considered to give rise to a correct prediction for the hypothetical poses { } if r ( a j )=1.

[0075] Final pose parameters T of the camera are obtained by applying in step 70 the PnP algorithm on all correct predictions (“inliers”) in order to minimize the re-projection error function E(t): E T = ∑ i ∈ J p i − KT − 1 P ι w ^ 2 Or corresponds to the set of indices of correct predictions, E(T) being the re-projection error corresponding to an energy that we seek to minimize. In other words, by minimizing the function E , we obtain the pose T optimal.

[0076] The camera pose parameters can be updated based on the proportion of correct predictions: n inliers n inliers + n outliers . Only when this proportion is sufficiently large, an update of the estimated parameters is carried out.

[0077] Estimating the camera pose parameters involves recalculating them from the results of the detection 232 and update 234 steps. In this case, the data stored in the stack S data of each passive leaf includes the position pi associated with the 2D point of interest and the camera pose T . When the stack size exceeds a predetermined value ( max data ), the value of the 3D representation associated with the passive leaf is re-evaluated using a triangulation algorithm from two points of interest stored in the stacks p 1 And p 2 represented on the Figure 6 These points of interest are stored with their respective poses. T 1 and T 2 . Such a triangulation method is for example described in the article R. Hartley and A. Zisserman, Multiple view geometry in computer vision, Robotica, 23(2):271-271, 2005.

[0078] The new value PW< of the 3D representation in the first frame of reference, is obtained by solving the equation system below: p 1 ∗ KT 1 − 1 P W = 0 p 2 ∗ KT 2 − 1 P W = 0

[0079] There Figure 7illustrates an embodiment of a device 100 (for a transmitter or a receiver) configured to implement an embodiment of a method according to the invention. The device 100 comprises a random access memory 130 (for example a RAM memory), a processing unit 110 equipped for example with a processor, and controlled by a computer program stored in a read-only memory 120 (for example a ROM memory or a hard disk). Upon initialization, the code instructions of the computer program are for example loaded into the random access memory 130 before being executed by the processor of the processing unit 110.

[0080] This device 100 can be integrated into an augmented reality system (not shown) which can further comprise: a camera capable of acquiring an image of a three-dimensional scene, the pose parameters of which are estimated by the device 100, a module capable of delivering an output image from an input image acquired by said camera, representing said three-dimensional scene and at least one real or virtual object, from a 3D representation representing said at least one object in the scene and pose parameters of the camera, and a display module capable of rendering the output image.

[0081] There Figure 7illustrates a particular way, among several possible ones, of producing the device100 capable of implementing at least certain steps of the method according to the invention. Indeed, these steps can be carried out indifferently on a reprogrammable computing machine (a PC computer, a DSP processor or a microcontroller) executing a program comprising a sequence of instructions, or on a dedicated computing machine (for example a set of logic gates such as an FPGA or an ASIC, or any other hardware module).

[0082] In the case where the processing means is carried out with a reprogrammable computing machine, the corresponding program (i.e. the sequence of instructions) may be stored in a removable or non-removable storage medium, this storage medium being partially or totally readable by a computer or a processor.

[0083] It goes without saying that the methods of implementation and embodiment which have been described above have been given for purely indicative purposes and are in no way limiting, and that numerous modifications can easily be made by those skilled in the art without departing from the scope of the invention.

Claims

1. A method for predicting a three-dimensional (3D) representation in a 3D scene, using one or more two-dimensional (2D) representations obtained from at least one camera-generated image, the method comprising a step for obtaining a forest of regression trees, and for said at least one image: extracting (21) a 2D representation associated with one or more positions within the image, wherein the method further comprises, for said at least one image: - predicting (236) a 3D representation, corresponding to the extracted 2D representation, using the regression tree forest which comprises a set of possible associations between at least one 2D representation and a 3D representation, each possible association resulting from a predictive model, said predictive model of a possible association being defined by a distribution characterising the at least one 2D representation associated with the 3D representation for said possible association, and by the following parameters: - a first parameter representative of a status of the predictive model of the possible association; - a second parameter representative of a number of consecutive predictions evaluated as incorrect for the possible association; - a third parameter representative of the extracted 2D representations associated with the 3D representation by said possible association. evaluating (50) one or more of the possible associations of the regression tree forest, according to a predetermined confidence criterion, in which the prediction can be evaluated for at least one possible association as being correct or incorrect according to the predetermined confidence criterion, and updating (230) the regression tree forest comprising a deactivation of one or more possible associations, depending on the evaluation of said possible associations.

2. The method according to claim 1, wherein the status of the predictive model of a possible association is 'active' when said association is enabled or 'passive' when said association is disabled.

3. The method according to claim 2, wherein said updating of the regression tree forest comprises for at least one predictive model of a possible association whose status is 'active' - updating the value of the second parameter for said possible association, based on the result of the evaluation, and - if the value of said second parameter is above a predetermined threshold, deactivating (232) said possible association, otherwise maintaining the 'active' status of the predictive model of said possible association.

4. The method according to any of claims 1 to 3, wherein updating the regression tree forest comprises for at least one predictive model of a possible association, updating (234) the value of the third parameter according to the extracted 2D representation, and for at least one predictive model of a possible association having the status 'passive': - if said third updated parameter meets a predetermined condition, a new determination of the distribution characterizing the 3D representation related to said possible association and a new determination of the 3D representation of said possible association, and ∘ if the covariance of the newly determined distribution is below a predetermined threshold, a validation of said update of the value of the third parameter, the newly determined distribution and said newly determined 3D representation, and an activation of the predictive model of said possible association, ∘ otherwise a reset of the value of the third parameter and a maintenance of the 'passive' status of the predictive model of said possible association.

5. The method according to any one of claims 1 to 4, further comprising an update of the pose parameters of said camera, according to the possible associations assessed as correct or incorrect.

6. The method according to any of the preceding claims, characterised by the fact that it is implemented by a computer.

7. A device (100) for predicting a three-dimensional (3D) representation in a three-dimensional scene, using one or more two-dimensional (2D) representations, obtained from at least one image generated by a camera, said device comprising a computing machine dedicated to or configured for: obtaining a forest of regression trees, and for said at least one image: extracting a 2D representation associated with one or more positions within the image, - predicting a 3D representation, corresponding to the extracted 2D representation, using the regression tree forest which comprises a set of possible associations between at least one 2D representation and a 3D representation, each possible association resulting from a predictive model, said predictive model of a possible association being defined by a distribution characterising the at least one 2D representation associated with the 3D representation for said possible association, and by the following parameters: - a first parameter representative of a status of the predictive model of the possible association; - a second parameter representative of a number of consecutive predictions evaluated as incorrect for the possible association; - a third parameter representative of the extracted 2D representations associated with the 3D representation by said possible association. evaluating one or more of the possible associations defined by the forest of regression trees, according to a predetermined confidence criterion, in which the prediction can be evaluated for at least one possible association as being correct or incorrect according to the predetermined confidence criterion, and updating the regression tree forest including a deactivation of one or more possible associations, depending on the evaluation of said possible associations.

8. The device according to claim 7, wherein said computing machine is dedicated to or configured for at least one possible association, evaluating the prediction as correct or incorrect according to the predetermined confidence criteria.

9. The device according to claim 8, further comprising a module dedicated to or configured to update pose parameters related to said camera, according to the possible associations assessed as correct or incorrect.

10. An augmented reality system comprising: - a camera capable of acquiring an image of a three-dimensional scene, - a module capable of delivering an output image from an input image acquired by said camera, representing said scene in three dimensions and at least one real or virtual object, from : ∘ a 3D representation of the at least one object in the scene, and ∘ camera pose parameters, - a display module capable of displaying the output image, characterised by the fact that it comprises a device according to claim 9 for updating the installation parameters of said camera.

11. A computer program comprising instructions for implementing a method according to any of claims 1 to 6, when said program is executed by a processor.

12. A computer-readable storage medium having recorded thereon a computer program comprising program code instructions for performing steps of a method according to any one of claims 1 to 6.