Video Flux Extension of Real Scenes

Abstract

To provide a method, a system, and a computer program for improving augmented reality used for augmenting a video flux of a real scene including a real object.SOLUTION: A method according to the present invention includes the steps of: capturing a video flux by a video camera; extracting one or a plurality of 2D images each indicating a real object from the video flux; providing a 3D model indicating the real object; and determining a pose of the 3D model with respect to the vide flux from candidate poses. This determination gives a reward to mutual information for at least one 2D image and each of given candidate poses, which indicates interdependency between virtual 2D rendering and the at least one 2D image. The method also includes a step of augmenting the video flux based on the pose.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of computer programs and systems, and more particularly to a method, system and program for enhancing video flux of a real scene containing real objects. [Background technology]

[0002] For the design, engineering and manufacturing of objects, numerous systems and programs are offered on the market. CAD is an acronym for Computer-Aided Design, e.g., related to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, e.g., related to software solutions for simulating the physical behavior of a future product. CAM is an acronym for Computer-Aided Manufacturing, e.g., related to software solutions for defining manufacturing processes and operations. In such Computer-Aided Design systems, the graphical user interface plays a key role regarding the efficiency of the technique. These technologies may be incorporated within Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes and leverage corporate knowledge to help develop products from concept to life, beyond long-term corporate concepts. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and a Corporate Hub that allows corporate integration and connection to both the Engineering and Manufacturing Hubs. Overall, the system provides an open object model that links products, processes, and resources to enable dynamic knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this and other contexts, augmented reality is gaining widespread importance. It encompasses solutions for augmenting a video flux of a real scene with real objects. These solutions make it possible, for example, to add information to the display of the real object by means of the video flux. These solutions require knowing the pose of a 3D model of the real object relative to the different 2D frames of the video flux, so that information associated with the 3D model can be added to the video flux to augment it. Indeed, in order to superimpose a virtual model on the object, it is necessary to know as accurately as possible where the object is located in the reference frame of the augmented reality system.

[0004] When a 3D model (e.g., a virtual 3D mesh) has the same texture as a real object in the image frames of a video flux, one existing solution is to synthesize a virtual view of the object and match keypoints (such as FERN or SIFT descriptors) between the virtual and real images. Since the keypoints on the virtual image correspond to known 3D points, some existing solutions include using the EPnP algorithm to calculate the pose. This is notably what is proposed in the paper by VLepetit, FMoreno-Noguer, and PFua, EPnP: An Accurate O(n) Solution to PnP Problem, International Journal of Computer Vision (IJCV), 2008. However, it is very common that the 3D model does not have texture (or a different texture than the real object displayed on the image), which prevents the calculation of keypoints that are robust enough to yield an accurate pose. Furthermore, real objects can be difficult to segment in the image, which also prevents any reliable descriptors over the contours of the object. In such cases, existing learning-based approaches can provide a solution by training neural networks to predict the right. However, such methods do not work well for real objects that were not seen during training.

[0005] Therefore, there is a need for improved augmented reality solutions for enhancing the video flux of a real scene with real objects. Summary of the Invention

[0006] Therefore, the present invention provides a computer-implemented method of augmented reality. The method is for augmenting a video flux of a real scene including a real object. The method includes capturing the video flux with a video camera. The method also includes extracting one or more 2D images from the video flux, each of which represents a real object. The method also includes providing a 3D model representing the real object. The method also includes determining a pose of the 3D model for the video flux among the candidate poses. The determination rewards mutual information for the at least one 2D image and each given candidate pose. The mutual information represents an interdependency between the virtual 2D rendering and the at least one 2D image. The virtual 2D rendering is a virtual 2D rendering of the 3D model with a value of a given candidate pose for the at least one 2D image. The method also includes augmenting the video flux based on the pose.

[0007] The method may include one or more of the following: The one or more 2D images include several 2D images each corresponding to a respective frame of the video flux, and the determining step rewards mutual information for each 2D image of the several 2D images. The determining step includes optimizing an objective function that includes terms, each term rewarding mutual information for a respective 2D image of a number of 2D images. The objective function is the sum of the terms. Each term is equal to the mutual information for the corresponding 2D images. The optimization includes providing N (N>1) starting poses and running a non-differential iterative optimization algorithm on the objective function N time points, each time starting from each starting pose and outputting a respective resulting pose, and the determining includes calculating a pose based on one of the N resulting poses; The optimization step includes providing N starting poses, where N>1; running a non-differentiable iterative optimization algorithm on an objective function N times, each time starting from a respective starting pose and outputting a respective resulting pose; and calculating the pose based on one of the N resulting poses. The determining step further rewards photo-consistency between projections of one or more points of the 3D model onto the multiple 2D images. The determining step performs a pruning step to perform the reward for photo consistency in a calculation and output one of N result poses, and a refinement step to give the reward for photo consistency after optimization. The determining step further rewards matches between projections of 3D edges of the 3D model and 2D edges in at least one 2D image, and the refining optionally performs the rewarding of the matches. For each image of the at least one image, the 3D edges form a silhouette of the 3D model in the respective image. The mutual information is mutual information between a first variable extracted from the virtual 2D rendering and a second variable extracted from the at least one 2D image.

[0008] The distribution over the pixel value domain is a count of pixels having values ​​that fall within one of predefined subranges.

[0009] A computer program comprising instructions for carrying out the method is further provided.

[0010] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0011] Further provided is a system that includes a processor coupled to a memory, the memory having a computer program recorded thereon. [Brief description of the drawings]

[0012] [Figure 1] 1 shows a flow chart of an example method. [Diagram 2] FIG. 1 illustrates an example of a system. [Diagram 3] This method is shown below. [Figure 4] This method is shown below. [Diagram 5] The results obtained with this method are shown below. [Figure 6] The results obtained with this method are shown below. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A computer-implemented method of augmented reality is proposed. The method is for augmenting a video flux of a real scene including a real object. The method includes capturing the video flux with a video camera. The method also includes extracting one or more 2D images from the video flux, each of which represents a real object. The method also includes providing a 3D model representing the real object. The method also includes determining a pose of the 3D model for the video flux among the candidate poses. The (pose) determination rewards mutual information for the at least one 2D image and each given candidate pose. The mutual information represents an interdependency between the virtual 2D rendering and the at least one 2D image. The virtual 2D rendering is a virtual 2D rendering of the 3D model with a value of a given candidate pose for the at least one 2D image. The method also includes augmenting the video flux based on the (determined) pose.

[0014] Such a method forms an improved solution of augmented reality for extending the video flux of a real scene with real objects.

[0015] The method determines a pose of a 3D model representing a real object relative to a video flux of a real scene including said real object, the video flux being captured by a video camera. The method thus allows, for example, to augment the video flux based on the determined pose according to any known method. The method thus provides the capability of an augmented reality solution in the specific context of video flux capture.

[0016] The method then determines the pose of the 3D model relative to the video flux in a specific way. In particular, it rewards mutual information, which represents the interdependence between a virtual 2D rendering of the 3D model relative to at least one 2D image extracted from the video flux, and the at least one 2D image itself, in which the determination of the pose by the method. Thanks to such reward, the method determines a pose that is relatively consistent with the video flux. In other words, the method determines a pose that corresponds relatively accurately to the true pose parameters of the camera, or, in terms of augmented reality, pose parameters that emulate these true pose parameters in an acceptable way. In fact, the mutual information captures the interdependence between one or more extracted images and its (their) corresponding rendering, based on the pose. By rewarding such mutual information, the method assumes that such a dependency must be verified to some extent, which turns out to be true when evaluating the accuracy of the results obtained by the method. The augmentation is therefore also relatively accurate, since it depends on the determined pose.

[0017] Moreover, the method does not need to rely on any deterministic correlation between the texture of the 3D model and the segmentation of the 2D image. Indeed, the method uses a statistical measure of similarity, namely mutual information. This is made possible by taking into account a 2D rendering of the 3D model, obtaining a format comparable to the 2D image. With such a statistical analysis, the method has enough information to be able to correlate, on the one hand, the 3D model rendered along the candidate pose, with the extracted 2D image on the other hand. Thus, the 3D model can be untextured or alternatively textured, but with a texture that does not correlate to the segmentation of the 2D image, and / or the 2D image can be unsegmentable (i.e. difficult or impossible to segment). Also, it was revealed during testing that mutual information (which has a probabilistic theoretical definition) provides more accurate results than solutions that rely only on criteria with a non-probabilistic theoretical definition, such as the L2 norm between a virtual 2D rendering and its (their) corresponding extracted 2D image. Moreover, the method does not need to rely on the prior provision of a machine learning dataset. In fact, the pose determination of the present method can be fully deterministic (i.e., it implements a deterministic algorithm, i.e., it is not a learned and therefore trained neural network).

[0018] The one or more (extracted) 2D images may include several 2D images (i.e., several 2D images), each corresponding to a respective frame of the video flux. In such a case, the decision may reward mutual information for each such 2D image (i.e., for / between each image of several 2D images), i.e., for each frame of the extracted video flux that represents a real object, a reward is given. This allows to relatively fully exploit the richness of information provided by the video flux (e.g., compared to relying on a single extracted image frame), which improves the accuracy and robustness of the method.

[0019] In particular, some 2D images may include 2D images of the real object from different viewpoints. The different viewpoints may include, for example, different viewing angles of the real object. To that end, the filming may include moving (e.g., including rotating) the camera in the scene (e.g., including around the real object) to film video frames of the real object from different viewpoints (e.g., including different angles). In such a case, the method takes advantage of the full spatial coverage of the real object allowed by video flux filming. This further improves the accuracy and robustness of the method. The extracted 2D images may include, for example, multiple images distributed over a period of less than 20 or 10 and / or more than 2 or 3 and / or longer than 1 or 5 seconds and / or less than 1 minute or 30 seconds.

[0020] A real object may be a manufactured product and / or fixed / immovable in the real world. A real object may be a product of various industries, including, without limitation, aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. Thus, a real object may be an industrial product, which may be any machine part, such as a part of a land vehicle (including, for example, automobiles and light truck equipment, racing cars, motorcycles, trucks and motor equipment, trucks and buses, trains), a part of an air vehicle (including, for example, airframe equipment, aerospace equipment, propulsion equipment, defense equipment, control equipment, space equipment), a part of a marine vehicle (including, for example, marine equipment, commercial vessels, offshore equipment, yachts and workboats), a general machine part (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment, fabricated metal products, tire manufacturing products), an electric machine or electronic component (including, for example, consumer electronics, security and / or control and / or instrumentation products, computer and communication equipment, semiconductors, medical devices and equipment), a household item (including, for example, furniture), etc. The objects may be home goods, garden goods, leisure goods, fashion goods, hard retailer products, soft retailer products, packaging (including, for example, food and beverage and tobacco, beauty and personal care, household goods packaging). A real scene may be any real-world environment in which physical instances of real objects are located.

[0021] The method is computer-implemented. This means that the steps of the method (or substantially all steps) are performed by at least one computer or any system. Thus, the steps of the method are performed by a computer, possibly fully automatically or semi-automatically. In an example, the triggering of at least some of the steps of the method may be performed via user / computer interaction. The level of user / computer interaction required depends on the level of automation expected and may be balanced against the need to implement user wishes. In an example, this level may be user-defined and / or predefined.

[0022] A typical example of a computer implementation of the method is to execute the method using a system adapted for this purpose. The system includes a processor coupled to a memory, which may be recorded with a computer program including instructions for carrying out the method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally comprise several physically distinct parts (e.g. a part for the program and possibly a part for the database).

[0023] A computer program may include instructions executable by a computer, the instructions including means for causing the device to carry out the method. The program may be recordable on any data storage medium including the memory of the system. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, for example an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the method may be executed by a programmable processor executing a program of instructions for performing the functions of the method by operating on input data and generating output. Thus, the processor is programmable and may be coupled to receive data and instructions from, and to transmit data and instructions to, the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high level procedural or object oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. Applying the program on the system will in any case provide instructions for carrying out the method.

[0024] The system may include a display screen. The processor may be coupled to the display screen and configured to display the (e.g., enhanced) video flux on the display screen. The method may further include displaying the enhanced video flux (i.e. the enhanced video flux resulting from the enhancing step performed by the method) on the display screen, for example (substantially) in real time during the capture of the video flux. Alternatively, the display may be performed separately after the method.

[0025] The system may comprise a video camera. The camera may be coupled to a processor and configured to capture video flux. The capture may be performed, for example, via user interaction. A user may carry the camera and move it in (for example, around) a scene to capture video of the real object. The user may, for example, hold the camera in his / her hand or may wear an accessory (for example, glasses) to which the camera is attached. Other steps of the method may, for example, be performed (substantially) fully automatically in real time while the video flux is being captured.

[0026] The system may be configured to determine the relative position of the camera during its movement, for example, by having a sensor mounted on the camera or fixed relative to the camera. The sensor may be coupled to the processor and configured to determine the relative position of the camera during its movement to track the position of the camera. The sensor may include a motion sensor and / or an inertial sensor.

[0027] The system may for example comprise a mobile phone or a tablet programmed for AR (Augmented Reality) and / or AR glasses. Such a system may be configured for AR, as known per se from the field of augmented reality. The method may comprise carrying such a system by a user in a scene in order to perform a video capture with a video camera of the system, as known per se from the field of augmented reality.

[0028] The method provides an augmented reality by augmenting the video flux. "Augmenting the video flux" means any operation that includes calculating a flux of graphics to be superimposed (i.e. placed on the video flux). The augmented video flux may completely include the video flux itself, to which the calculated flux of graphics is added (superimposed). This is the case, for example, when the system is a mobile phone or tablet, and the display is performed on the screen of the mobile phone or tablet while being carried by the user to film the video flux. Alternatively, the augmented video flux may be separated from the video flux and thereby consist of merely the calculated flux of graphics. This is the case, for example, when the system includes transparent AR glasses, where the user does not need to see the initially filmed video flux itself, since thanks to its transparency the user already sees the reality through the AR glasses, so that only the flux of the calculated graphics needs to be displayed on the glasses, superimposed on the reality being seen. All this is known per se from the field of augmented reality and will not be further detailed in the present disclosure.

[0029] By "flux" we mean a sequence (i.e., a time series) of images. Thus, a video flux is a set of consecutive 2D images representing a portion of a real scene, each 2D image corresponding to (i.e., being) a respective frame of the video flux. The capture may be performed consecutively, such that two consecutive frames of the video flux may be close to each other (e.g., separated by less than 1 second or 0.1 seconds).

[0030] By "extract" it is simply meant here that the method selects one or more captured video frames, each containing a view (full or partial) of the real object, and provides a 2D image corresponding to that frame to the pose determination step. Each extracted 2D image may be the selected frame containing a view of the real object, or a subpart thereof. The method may include identifying each such subpart in any manner. For example, identifying may include defining a (e.g. rectangular) bounding box around the view of the real object, e.g. drawn by a user.

[0031] A 3D model is a 3D modeled object. A modeled object is any object that is defined by data stored, for example in a database. By extension, the expression "modeled object" specifies the data itself, and by "3D modeled object" we mean any object that is modeled by data that allows its 3D representation. A 3D representation allows the part to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, can be manipulated and rotated around any of its axes or around any axis in the screen on which the representation is displayed. This in particular excludes 2D icons that are not 3D modeled. The display of 3D representations facilitates design (i.e. statistically increases the speed with which designers accomplish their tasks). This speeds up the manufacturing process in industry, since the design of a product is part of the manufacturing process.

[0032] A 3D model therefore comprises a specification of a 3D shape of a real object. The 3D shape may be defined or definable in a 3D reference frame. The specification may be under any format that allows a virtual 2D rendering(s) of the pose determination. The 3D model may for example be a 3D mesh or any other 3D modeled object that allows the calculation of a 3D mesh, for example a boundary representation or B-Rep (wherein the B-Rep may be converted into a 3D mesh by tessellation, as known per se from the field of CAD).

[0033] The 3D model may be designed in a CAD system or may result from a design made in a CAD system. By CAD system is meant any system adapted to at least design a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, the data defining the modeled object includes data enabling the representation of the modeled object. The CAD system may, for example, provide a representation of the CAD modeled object using edges or lines, in some cases having faces or surfaces. The lines, edges, or surfaces may be represented in various ways, for example, with Non-Uniform Rational B-Splines (NURBS). In particular, a CAD file includes specifications from which a geometric shape may be generated, and thus a representation may be generated. The specifications of the modeled object may be stored in a single CAD file or in multiple CAD files. A typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. And the modeled object may typically be an assembly of thousands of parts.

[0034] Providing the 3D model can be performed in any manner: the 3D model may, for example, be searched for by a user, or may be automatically recognized by the system, for example in a library of 3D models.

[0035] "Pose of a 3D model relative to the video flux" refers to the frame-by-frame pose parameters of a video camera relative to the 3D model, e.g., defined in the reference frame of the 3D model. Pose parameters are also known as "extrinsic parameters", and thus the "value" of a pose for (i.e., relative to) an extracted 2D image is the value of the pose parameters for the frame of the video flux that corresponds to the extracted 2D image.

[0036] As known per se from the field of computer vision, the pose parameters together with the (frame-by-frame) intrinsic parameters of the video camera make it possible to project a 3D model onto each frame of the video flux, and the method may include such a projection, thanks to which the video flux can be expanded based on the result of such a projection.

[0037] The 3D model in particular provides a mathematical representation of the 3D shape of the real object, thereby allowing to add to the video flux graphics (such as text, numbers and / or textures) derived from calculations performed on such mathematical representation. Alternatively or additionally, the 3D model may be associated with any type of data that can be used for augmentation, for example to generate a graphism to be added to the video flux. The associated data may for example include metadata (for example in a database) linked to the 3D model or to different parts of the 3D model. The metadata may for example include a text description or text specification linked to each part of the 3D model, if the 3D model includes different parts. The pose allows to position said graphism in the correct position of the video flux, with respect to the view in the real object. All this is known per se from the field of augmented reality and will not be further elaborated in the present disclosure.

[0038] The method accurately determines such a pose from among the candidate poses, where the pose is an unknown variable of the method and the determining step outputs a calculated value for it. Here, the candidate pose simply refers to the searchable domain provided to the determining step. Since the pose determined by the method is accurate, the projection of the 3D model in each frame is also relatively accurate in the sense that the projected 3D model closely matches the 2D view of the real object in the frame. In other words, the pose defines a virtual position of the video camera relative to the 3D model, and the method ensures that the virtual position corresponds accurately to the actual position of the video camera relative to the real object.

[0039] The intrinsic parameters are (at least partially) considered as unknown variables as well and may be determined by the method, for example, within the pose determination step (in such a case determining to perform a reward not only for each given candidate pose, but also for each given candidate frame-by-frame values ​​of the intrinsic parameters). Alternatively, the intrinsic parameters may be (at least partially) pre-determined and thus provided to the method, for example output by the camera itself (modern cameras usually provide their used intrinsic parameters). Further alternatively, the method may set constants and arbitrary values ​​for the intrinsic parameters. Although this may lead to slightly less accurate poses, this still provides a relatively accurate projection result and thus augments reality.

[0040] In all these cases, the method outputs a flux of camera parameters with respect to the video flux in a candidate flux of camera parameters, each set of camera parameters corresponding to a respective frame of the video flux (the flux consists of a set of camera parameter sets).The method can also determine, for at least one 2D image and for each given candidate flux of camera parameters, an output flux of camera parameters by rewarding mutual information representing an interdependency between a virtual 2D rendering of a 3D model having values ​​of the candidate flux of the given camera parameters for the at least one 2D image (i.e. the set of camera parameters in the flux corresponding to the image) and the at least one 2D image.

[0041] The determination can perform the mutual information reward in any manner that allows penalizing low values ​​or scores of mutual information. In other words, the determination is configured to output, for a given extracted 2D image, the pose that leads to the highest value of mutual information among all candidate poses, other things being equal (or ceteris paribus). The method performs such reward for each extracted 2D image, and the method can perform other types or rewards, so that the determined pose is generally not one that maximizes the mutual information for all 2D images, but one that provides a globally good score, thereby achieving a relatively high accuracy of the augmentation.

[0042] A particular way of implementing such rewards may follow this method, but is to run an optimization program, i.e., such that determining includes optimizing an objective function. The objective function includes terms that reward different measures. In practice such an optimization program may be implemented as a minimization program that includes cost (or penalty) terms, each of which penalizes the inverse of a measure against a reward.

[0043] The method can optimize the program using any type of optimization algorithm, for example, by running an iterative optimization algorithm at least once. The iterative optimization algorithm may start from a starting point (i.e., a starting pose) and then optimize. Each iteration can include providing a current given candidate pose (initialized with the starting pose) and outputting a new candidate pose (for the next iteration) that improves the value of the objective function (e.g., reducing it in the case of a minimization program). The output of the new candidate pose can include calculating mutual information for the current candidate pose and / or another candidate pose, and comparing the calculated mutual information values ​​for the current candidate pose and the other candidate pose. The calculation of mutual information can include calculating a virtual 2D rendering of the 3D model.

[0044] If the method extracts several images, the pose determination of the method may include optimizing an objective function that includes terms. Each term may then reward mutual information for each 2D image of the several 2D images. In other words, each extracted 2D image may have its own reward term (e.g., cost term). This provides an efficient way to implement the reward.

[0045] Specifically, the objective function may be a sum of terms. Alternatively or additionally, each term (for each of the extracted 2D images) may be equal to the mutual information of the corresponding 2D images. This allows for a robust and efficient convergence of the optimization. This has been notably proven to provide accurate results even when some of the extracted 2D images are of poor quality.

[0046] The method may, for example, include tracking the relative pose of the video camera in any known manner based on the aforementioned sensors (modern AR systems usually provide such functionality based on such sensors). In such a case, the relative pose of the 3D model between different frames is known. Thus, the pose determination may consist in initializing the value of the pose of the 3D model for the video flux based on one or more extracted 2D images, and the remaining values ​​of the pose are determined based on the initialized values ​​and the relative pose tracking. As a result, in order to know the full pose of the 3D model, the method can be limited to determining an initial pose, and thus, the extension can be performed throughout the entire video flux capture. Alternatively or additionally, if the method extracts several images and the determination rewards mutual information for each such image, the pose determination can consider the value of each given candidate pose for only one 2D image (e.g., the first image in chronological terms) as an unknown variable, and the value of the pose of the other images is defined by a variable as a function of the unknown variable and the relative pose tracking.

[0047] The video camera can capture RGB video flux or alternatively grayscale video flux. If the initially captured video flux is RGB video flux, the method can optionally include converting the video flux to grayscale video flux so that the extracted 2D image is a grayscale image. Alternatively, the method can include converting only selected video frames, or only subparts thereof, to a grayscale image, so that again the extracted 2D image is a grayscale image. This improves the efficiency of the method, since color information is ignored (such ignoring is particularly relevant in cases where the 3D model does not convey the correct color information, or where there is no texture, for example, and / or in the case of shading-based 2D rendering, where texture is not taken into account, even if there is).

[0048] Here, the method considers mutual information that represents the interdependence between a virtual 2D rendering of the 3D model with the pose values ​​and a corresponding 2D image. The term "virtual" simply refers to the fact that a 2D rendering does not have to be output by the method. The virtual 2D rendering of the 3D model can consist of any generation of a photorealistic 2D image of the 3D model from a viewpoint defined by the pose values. The virtual 2D rendering may further be based on camera-specific parameters (predetermined or determined by the method, as described above). The generation of the virtual 2D rendering can include any rasterization or shading, such as application of a Phong shader or a normal map shader.

[0049] As known per se from the fields of probability theory and information theory, the mutual information of two variables is a measure of the interdependence between the two variables. This concept is detailed in TM; Thomas, JA (1991), Elements of Information Theory (Wiley ed.), chapter 2, and in the website accessible at the following URL: https: / / en.wikipedia.org / wiki / Mutual_Information. The contents of this textbook and the contents of this website (as of the priority date of the patent application) are incorporated herein by reference, and the mutual information rewarded by the pose determination of the method may be as defined therein, i.e., may be included in the Kullback-Leibler bifurcation.

[0050] In particular, the mutual information may be between a first variable extracted from a virtual 2D rendering and a second variable extracted from at least one 2D image, the first and second variables representing a distribution over a pixel value range. The mutual information may in particular be the mutual information of a joint distribution in such a pixel value range (Kullback-Leibler divergence). In a concrete example, the distribution over the pixel value domain may be a count of pixels having values ​​belonging to one of the predefined subranges. The subranges (or "bins") may form a subdivision of the range of values ​​taken by each pixel. The subdivision may be regular. In practice, the subdivision may include a number of bins of the order of 20, for example, more than 5 or 10, and / or less than 100 or 50.

[0051] Optional features of the method will now be described with reference to the example shown in FIG.

[0052] The method of the present example includes a step S10 of capturing a video flux with a video camera. The method of the present example includes a step S20 of extracting from the video flux several 2D images, each of which represents a real object. In parallel, the method of the present example includes a step S30 of providing a 3D model representing the real object. The method of the present example then includes steps S40-S80 of determining the pose of the 3D model relative to the video flux. The determining steps S40-S80 reward mutual information for each 2D image of the several 2D images thanks to the objective function in S50, each of which includes a term that rewards mutual information for the corresponding 2D image (e.g., the sum as mentioned above). The method of the present example finally includes a step S90 of expanding the video flux based on the pose.

[0053] In addition to mutual information, the pose determination can further reward photo-matching between the projections on several 2D images of one or more points of the 3D model. Furthermore, in addition to mutual information, the pose determination can further reward matching between the projections of 3D edges of the 3D model and 2D edges in at least one 2D image. Rewarding these additional criteria improves the accuracy of the determined pose. Such criteria (photo-matching and edge matching) are used in the prior art as the only primary criteria for determining pose. By using mutual information instead as the primary statistical criterion, the pose determination of the present method converges much faster towards a relatively accurate result. However, to improve efficiency, the method of this example still uses other criteria as secondary criteria, but in the second stage (after the mutual information-based optimization is performed). This results in overall accuracy, robustness, and speed.

[0054] The photo-consistency criterion would assume that the final pose must be graphically consistent from the projection (according to the final pose) of a particular 3D point of the 3D model onto several 2D images, where the pose 2D points (i.e. all such existing 2D points depending on the per-image visibility of the corresponding 3D points) mean that they must correspond to positions in the 2D images that present the same local graphical features. Graphical consistency may be measured by applying at least one graphical descriptor, which may be rotationally invariant, to such 2D points in different images and comparing the values ​​of the graphical descriptor across the 2D images. The graphical descriptor may optionally be invariant to light intensity changes and / or differentiable. The graphical descriptor may be selected from among a predefined set of one or more graphical descriptors, including for example SIFT descriptors, FERN descriptors, and / or (simply) pixel colors. The 3D points may come from any method, for example from a randomly performed sampling of the 3D model. The method may reward photo-consistency at different stages, where at least the graphical descriptor may be different at each such stage.

[0055] The edge matching criterion will assume that the final pose must be such that a particular 3D edge of the 3D model projects (according to the final pose) onto at least one of the extracted 2D images, e.g., onto a particular 2D edge of each extracted 2D image. The 3D edges may for example form a silhouette of the 3D model in each image (according to the value of the final pose for the image). The silhouette of the 3D model is the contour of the 3D model according to the pose. The 2D edges of the images are also called "contours" and the method can determine the contour map in any known manner, e.g., using a trained neural network or by computing a chamfer map of the results of Canny edge detection. The contour map may be a contour probability map (i.e., each pixel is assigned a probability of being a contour).

[0056] The optimization may include a step S40 of providing a (predetermined) number N of starting poses, where N>1 (in other words, multiple starting points for the optimization subsequently performed in S50). The providing step S40 may be performed in any manner, and specific implementations are described below. The optimization may then include running an optimization algorithm that repeats the process of step S50 N times (e.g., the same one) on an objective function (i.e., an algorithm that optimizes the objective function), each time starting from a respective one of the starting poses, and each time outputting a respective resulting pose. In practice, N may be higher than 10 or 20 and / or lower than 100 or 50.

[0057] The method can then calculate a (final) pose based on one of the N result poses, via S60 to S80 in the illustrated example. This makes it possible to reduce the risks associated with accidentally inaccurate results provided by potential degradation of the algorithm in local extrema. In particular, for the problem at hand, the optimization algorithm may be non-differentiable (the method may for example implement the Nelder-Mead algorithm). Such non-differentiability creates the risk factors mentioned above.

[0058] The calculating steps S60 to S80 can include a step S60 of holding a (predetermined) number k of optimized poses during N executions, where k < N. In other words, the method can identify k best results from among the N result poses output by S50 (for example, k poses that minimize the objective function). In practice, k can be higher than 2 or 3, and / or lower than 20 or 10. The other result poses may be discarded at this stage, and only the selected k results are input to the pruning step S70. Alternatively, the method can skip S60, thereby inputting all of the N result poses to the pruning step S70.

[0059] Inputting only the best result poses output by S50 and above makes it possible to reduce the risk associated with accidentally inaccurate results. Such inaccurate results can occur when relying only on mutual information, since mutual information is a statistical measure and thus involves ambiguity (for example, in the case of symmetry). Step S60 likewise reduces the risk associated with accidentally inaccurate results (compared to inputting all N result poses to the pruning step S70) due to the photo consistency criterion of the pruning step S70, which can potentially lead to an accidental selection of poses that are not sufficiently satisfactory with respect to mutual information.

[0060] The steps of computing S60-S80 may comprise a step S70 of pruning the N result poses (which are then input to step S70) to output a single pose. Although the pruning step S70 is shown separately from S60, S60 may be considered equivalent to a part of pruning. In the illustrated example, the pruning step S70 results in selecting the best pose from among the k optimized poses output by S60 by rewarding the aforementioned photo-consistency. For example, the pruning step S70 may select one pose from among the k optimized poses that performs best against a photo-consistency criterion. This may involve sampling the 3D model and then evaluating the photo-consistency criterion for each of the k poses (including, for example, computing any of the aforementioned descriptors for each 2D point resulting from the projection of the sampled 3D points, such as SIFT descriptors).

[0061] Next, the calculation steps S60 to S80 may include performing a refinement step S80 of the pose output by S70, and the refined pose may be considered as the final determined pose and input to S90 to extend the video flux.

[0062] The refinement step S80 may reward photo-consistency (used again here) and / or 2D edge 3D edge matching as described above, e.g. both of these criteria. The refinement step S80 may for example minimize a global error. The global error may include a first term that penalizes the error between the projected contour of the 3D model (e.g. the virtual mesh (i.e. 2D view) of each of the 2D images) and the detected edge of the real object in said 2D images, and / or a second term that penalizes the error between the values ​​of the descriptor(s) across different 2D images, e.g. at the 2D points corresponding to the sampled 3D points. In the case of an edge matching reward based on a contour probability map as described above, the first term may penalize a low contour probability of the 2D pixel onto which the 3D edge (e.g. corresponding to a silhouette) projects. In the case of a photo-consistency reward based on graphical descriptors as described above, the second term may penalize the difference between the values ​​of the graphical descriptor on the projection of the (e.g. sampled) 3D point across the 2D images. The minimization may be gradient-based, which improves efficiency. In such cases, the graphical descriptors may be distinguishable, such as pixel colors.

[0063] FIG. 2 shows an example of the system.

[0064] The client computer in this example comprises a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer is further provided with a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated in specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a display screen 1080 and a camera 1090.

[0065] Next, the implementation of the method according to the example of FIG. 1 will be described.

[0066] These implementations of the method aim to compute the pose (and possibly intrinsic parameters) of a camera taking different views of a known 3D object. By "known" we mean that the system possesses a (e.g., texture-free) 3D model (e.g., a 3D mesh) representing the object. Furthermore, we assume here that the implementation of the method is in the context of an augmented reality system, and that the system knows the relative pose between the different views taken by the camera.

[0067] The implementation of the method relies on the definition of a pose-dependent energy based on mutual information between virtual renderings of a real image. This energy is discretized among several poses (e.g., obtained on a Thompson sphere). The implementation of the method optimizes this energy to find the best possible pose. Next, the implementation of the method sorts out incorrect poses by utilizing a photo-consistency criterion. Finally, the implementation of the method fine-tunes the pose by fitting to image edges.

[0068] Implementations of the method are able to compute pose even when segmentation is impossible or inaccurate and when the texture of the object is unknown. Furthermore, embodiments of the method combine both mutual information methods, photo consistency criteria, and silhouette / contour matching in a multi-view framework, thus removing ambiguity and obtaining more accurate pose. Implementations of the method exploit all views at once and are not biased towards any one view in the mutual information optimization. Furthermore, implementations of the method locally refine the pose using photo consistency criteria as well as matching the projected silhouette of the 3D model onto the closest contour in the image. Finally, implementations of the method are independent of any dataset / machine learning algorithm.

[0069] I1,…,I n Let be n real images, and R i Let I1 be

number

number

[0070] T i (P) is the view I given by the pose P of frame I1. i That is, if P=KR (where K is the projection matrix and R is the rigid transformation from the mesh frame to the I1 frame), then

number

[0071] An implementation of the method may include providing a portion of the image I1 in which the object is located, for example a user may draw a bounding box around it.

[0072] <Multi-view optimization> Pretreatment First, the implementation of the method involves dividing the images I1, ..., I n can be preprocessed to convert to grayscale luminance space. An implementation of the method can then equalize the histogram of each image. Each pixel color is a one-dimensional integer in the range [0,255].

[0073] Once all the images have been transformed, the implementation of the method can be used to transform the poses P1, ..., P n can be sampled.

[0074] For example, the explored pose

number

number

number

[0075] For example, if no such approximation is known, an implementation of the method samples points on a Thomson sphere around the mesh, P1,…,P n (The camera is located at a single sampled point on the sphere, looking at the center.) As well as a translation in the optical plane of the camera, the radius of the Thomson sphere may be set such that the projected bounding box M is equal to the expected position of the object in image I1.

[0076] If the intrinsic parameters (focal point and principal point) of the actual camera are known, then implementations of the method can use them for the intrinsic matrix. If they are unknown, implementations of the method initialize the principal point to be in the center of the image, and also initialize the focal point to (width+height) / 2 (where "width" and "height" are the width and height of image I1). These intrinsic parameters may be optimized along with the pose, or may be fixed.

[0077] Definition of Mutual Information Let f be a function that maps a pose P to a virtual rendering (color or grayscale) of mesh M taken at pose P. For example, an implementation of the method may use a Phong shader on f to render mesh M, or may use a normal map shader, where the color of each pixel is in the range [0,255]. 3 An implementation of this method may apply f to generate a view with the same resolution as the real image.

[0078] Given a pair of images (J1, J2), and let c1 and c2 be the numbers of channels in J1 and J2,

number

number

number

number

[0079] The implementation of this method is shown in Figure 1.

number

number

number

number

number

number

[0080] Finally, let D be the joint distribution

number

number

[0081] Such mutual information is the Kullback-Leibler divergence between the joint and marginal distributions of colors in J1 and J2. If the color distributions in J1 and J2 are independent, then

number

[0082] Note that mutual information does not characterize a linear relationship between the colors of J1 and J2 (unlike correlation), it is a value that describes how well one can infer the color of J2 knowing the color of J1, i.e., how probabilistically dependent the colors are in J1 and J2.

[0083] Mutual Information Optimization Best pose

number

[0084] Thus, implementation of the method can minimize the energy:

number

[0085] To minimize E1(P) and avoid poor local minima, the implementation of the method involves the selection of P1,...,P N We can perform N optimizations starting from . The optimization can be done with any algorithm. An efficient implementation of the method can be the Nelder-Mead one (as in the tested implementation described below).

[0086] An embodiment of the method may maintain the optimized k poses (k=5 in the tested embodiment) within N runs.

number

[0087] This is shown in Figure 3.

[0088] Pose Reaper From now on, the implementation of this method will be done without preprocessing, using the original images I1,…,I N It may work with.

[0089] The implementation of this method is

number

[0090] x1,…,x m is a sampling of m points on a mesh M, where n1,…,n m Let s(x,I) be a function that maps a 2D coordinate x on image I onto a local descriptor that is, for example, (ideally) invariant to changes in lighting and orientation. s(x,I) may be as simple as the color I(x) itself, or it may take into account a local neighborhood as is done with SIFT descriptors, or it may be extracted using a pre-trained neural network.

[0091] lastly,

number

number

number

[0092] The implementation of this method is

number

number

[0093] where σ is a penalty function such as the L2 norm, and ρ is the normal of the point n i is perpendicular to the camera's eye vector (pose PR jis a weighting function that underweights more energy, as given by

number

number

number

number

[0094] This is shown in Figure 4.

[0095] Pose Refinement Finally, the implementation of the method increases photo-consistency and edge matching between the contours of the projected mesh and edges in the image by minimizing local color-based and edge-based energy.

number

[0096] First, the implementation of the method involves: j normalize the projected mesh, e.g.

number

[0097] Let σ be, for example, the L2 norm or

number

number

[0098] The method is carried out by:

number

number

number

[0099] This minimization can be handled by any differentiable optimization where the Jacobian of the pixel colors with respect to their location in the image is given by a Sobel filter.

[0100] 5-6 show the results obtained with the tested embodiment of the method according to FIG. 1 and the formulas provided for the considered embodiment of the method.

[0101] 5 shows how a 2D frame of a video flux of a scene containing a table and a chair can be accurately augmented by the present method with the projection of its 3D model. As can be seen, the error between the graphical representation 64 of the table on the 2D frame and the 3D model 62 of the table is relatively small.

[0102] Figure 6 shows how this method can be used to accurately augment a video flux of a scene including a coffee maker 62 with textual descriptions 64 of different parts of the coffee maker associated with a 3D model of the coffee maker in a database. On Figure 6, protrusions 66 of the 3D model of the coffee maker are also shown semi-transparently superimposed on the 2D view of the coffee maker in the image.

Claims

1. 1. A computer-implemented method of augmented reality for augmenting a video flux of a real scene including a real object, comprising: capturing a video flux with a video camera; - extracting from said video flux one or more 2D images, each of which represents a real object; providing a 3D model representing the real object; determining a pose of a 3D model from candidate poses for said video flux; having In said determining step, for at least one 2D image and for each given candidate pose, a virtual 2D rendering of the 3D model with given candidate pose values ​​for said at least one 2D image; and said at least one 2D image; The mutual information that represents the dependency between the two is rewarded, said at least one 2D image comprises several 2D images each corresponding to a respective frame of said video flux; the determining step rewards mutual information for each 2D image of the number of 2D images; the determining step includes optimizing an objective function including a term; Each term rewards mutual information for each 2D image of several 2D images. method.

2. The objective function is the sum of the terms The method of claim 1.

3. Each term is equal to the mutual information for the corresponding 2D images. The method of claim 2.

4. The optimization, providing N starting pauses, where N>1; running a non-differentiable iterative optimization algorithm on said objective function N times, each time starting from a respective starting pose and outputting a respective resulting pose; Calculate the pose based on any of the N poses in the result The method of claim 1, 2 or 3.

5. The determining step further rewards photo-consistency between projections of one or more points of the 3D model onto the multiple 2D images.

5. The method according to any one of claims 1 to 4.

6. The determining step further comprises: a pruning step for performing the photo consistency reward in a computation and outputting one of the N resulting poses; The optimizing step is followed by a refinement that rewards photo-matching. The method according to claim 5.

7. The determining step further rewards matches between projections of 3D edges of the 3D model and 2D edges in the at least one 2D image, and the refinement performs the rewards of the matches. The method according to claim 6.

8. For each of the at least one 2D images, the 3D edges form a silhouette of the 3D model in the respective image. The method of claim 7.

9. The mutual information is A first variable extracted from the virtual 2D rendering; and A second variable extracted from the at least one 2D image. is the mutual information between The first variable and the second variable indicate a distribution of a range of pixel values.

9. The method according to any one of claims 1 to 8.

10. The distribution over the region of pixel values ​​is the count of pixels having values ​​that belong to one of the predefined subranges.

10. The method of claim 9.

11. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having the computer program according to claim 11 recorded thereon.

13. 12. A system comprising a processor coupled to a memory having the computer program of claim 11 stored thereon.

Images