Enhanced video throughput for real-world scenarios

Abstract

The present invention relates in particular to a computer-implemented method for augmented reality. The method comprises: capturing a video flux with a camera; extracting one or more 2D images from the video flux, each 2D image representing a real object; and providing a 3D model representing the real object. The method further comprises determining a pose of the 3D model relative to the video flux among candidate poses. The determination rewards mutual information for at least one 2D image and for each given candidate pose, which represents the mutual dependency between the virtual 2D rendering and the at least one 2D image. The method further comprises enhancing the video flux based on the pose. This forms an improved solution for augmented reality for enhancing the video flux of a real scene including real objects.

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more particularly to methods, systems and programs for enhancing video throughput of real scenes including real objects. Background Art

[0002] There are many systems and programs available on the market for the design, engineering and manufacturing of objects. CAD is an acronym for Computer-Aided Design, i.e. it refers to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, i.e. it refers to software solutions for simulating the physical behavior of future products. CAM is an acronym for Computer-Aided Manufacturing, i.e. it refers to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play an important role in the efficiency of the technology. These technologies may be embedded in a Product Lifecycle Management (PLM) system. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage corporate knowledge for product development across the extended enterprise concept, from concept to the end of the product's life. The PLM solution provided by Dassault Systèmes (under the trademarks CATIA, ENOVIA and DELMIA) provides an engineering center that organizes product engineering knowledge, a manufacturing center that manages manufacturing engineering knowledge, and an enterprise center that enables the enterprise to integrate and connect engineering and manufacturing centers. The entire system provides an open object model that connects products, processes and resources to enable dynamic, knowledge-based product creation and decision support, thereby driving optimized product definition, manufacturing preparation, production and service.

[0003] In this and other contexts, augmented reality is becoming increasingly important. Augmented reality comprises solutions for augmenting a video flux of a real scene comprising real objects. These solutions allow, for example, to add information to the display of the real object via the video flux. These solutions require knowing the pose of a 3D model of the real object relative to the different 2D frames of the video flux, so that information associated with the 3D model can be added to the video flux in order to augment it. In fact, in order to superimpose a virtual model on an object, it is necessary to know the position of the object in the reference frame of the augmented reality system as accurately as possible.

[0004] If the 3D model (e.g. a virtual 3D mesh) has the same texture as the real object in the image frames of the video flux, one existing solution is to synthesize a virtual view of the object and match the keypoints between the virtual image and the real image (e.g. FERN or SIFT descriptors). Since the keypoints on the virtual image correspond to known 3D points, some existing solutions consist in computing the pose using an EPnP algorithm. This was notably proposed by V. Lepetit, F. Moreno-Noguer and P. Fua in the paper, EPnP: An Accurate O(n) Solution to the PnP Problem, International Journal of Computer Vision (IJCV), 2008. However, it is very common that the 3D model has no texture (or a different texture than the real object shown on the image), and this prevents computing keypoints that are robust enough to produce an accurate pose. Furthermore, real objects may be difficult to segment in the image, which also prevents any reliable descriptors on the object contours. In this case, existing learning-based methods can provide a solution by training a neural network to predict the correct However, this approach does not work well on real objects that are not seen during training.

[0005] Therefore, there is a need for an improved solution for augmented reality for enhancing video throughput of real scenes including real objects. Summary of the invention

[0006] Therefore, a computer-implemented augmented reality method is provided. The method is used to enhance a video flux of a real scene including a real object. The method includes capturing the video flux with a camera. The method also includes extracting one or more 2D images from the video flux, each 2D image representing a real object. The method also includes providing a 3D model representing the real object. The method also includes determining a pose of the 3D model relative to the video flux in a candidate pose. The determination rewards mutual information for at least one 2D image and for each given candidate pose. The mutual information represents the mutual dependence between the virtual 2D rendering and the at least one 2D image. The virtual 2D rendering is a virtual 2D rendering of the 3D model using the value of a given candidate pose of the at least one 2D image. The method also includes enhancing the video flux based on the pose.

[0007] The method may include one or more of the following:

[0008] - the one or more 2D images comprises a plurality of 2D images, each 2D image corresponding to a respective frame of the video flux, the determining rewarding the mutual information for each 2D image of the plurality of 2D images;

[0009] - the determining comprises optimizing an objective function comprising terms, each term rewarding the mutual information for a respective 2D image of the plurality of 2D images;

[0010] -The objective function is the sum of the terms;

[0011] - Each term is equal to the mutual information of the corresponding 2D images;

[0012] - The optimization comprises: providing N starting poses, where N>1; and running a non-differentiable iterative optimization algorithm N times on the objective function, each time starting from a corresponding one of the starting poses and outputting a corresponding final pose; and the determination comprises: calculating a pose based on one of the N final poses.

[0013] - the determining further rewards optical consistency between projections of one or more points of the 3D model on the plurality of 2D images;

[0014] - said determining comprises: pruning in the computation, which performs said reward for photoconsistency and outputs one of N final poses; and / or after optimization, performing a refinement of the reward for said photoconsistency;

[0015] - said determining further rewards a match between a projection of a 3D edge of said 3D model and a 2D edge in at least one 2D image, said refining optionally performing said reward of said match;

[0016] - for each respective image of the at least one image, the 3D edge forms a silhouette of the 3D model in the respective image;

[0017] - the mutual information is the mutual information between: a first variable extracted from a virtual 2D rendering, and a second variable extracted from at least one 2D image, the first variable and the second variable representing a distribution over a range of pixel values; and / or

[0018] - The distribution over the pixel value range is the count of pixels having values ​​belonging to one of the predetermined sub-ranges.

[0019] A computer program comprising instructions for performing the method is also provided.

[0020] There is also provided a computer-readable storage medium having the computer program recorded thereon.

[0021] Also provided is a system comprising a processor coupled to a memory having a computer program recorded thereon. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments of the invention will now be described by way of non-limiting examples and with reference to the accompanying drawings, in which:

[0023] - Figure 1 A flow chart showing an example of the method;

[0024] - Figure 2 An example of a system is shown;

[0025] - Figure 3-4 The method is shown; and

[0026] - Figure 5-6 The results obtained using this method are shown. DETAILED DESCRIPTION

[0027] A computer-implemented augmented reality method is proposed. The method is used to enhance a video flux of a real scene including a real object. The method includes capturing the video flux with a camera. The method also includes extracting one or more 2D images from the video flux, each 2D image representing a real object. The method also includes providing a 3D model representing the real object. The method also includes determining a pose of the 3D model relative to the video flux in a candidate pose. The determination (of the pose) rewards mutual information for at least one 2D image and for each given candidate pose. The mutual information represents the mutual dependency between a virtual 2D rendering and at least one 2D image. The virtual 2D rendering is a virtual 2D rendering of the 3D model using the value of a given candidate pose of at least one 2D image. The method also includes enhancing the video flux based on the (determined) pose.

[0028] This approach forms an improved solution for augmented reality for enhancing video throughput of real scenes including real objects.

[0029] The method determines a pose of a 3D model representing a real object relative to a video flux of a real scene including said real object, the video flux being captured by a camera. Thus, the method allows to augment the video flux based on the determined pose, for example according to any known method. Thus, the method provides the functionality of an augmented reality solution in the specific context of video flux capture.

[0030] Now, the method determines the pose of the 3D model relative to the video flux in a specific way. In particular, the determination of the pose by the method rewards mutual information, which represents the mutual dependence between the virtual 2D rendering of the 3D model for at least one 2D image extracted from the video flux and the at least one 2D image itself. Due to this reward, the method determines a pose that is relatively consistent with the video flux. In other words, in terms of augmented reality, the method determines a pose that corresponds relatively accurately to the real pose parameters of the camera or to pose parameters that simulate these real pose parameters in an acceptable way. In fact, the mutual information captures the mutual dependence between one or more extracted images and their corresponding rendering based on the pose. By rewarding this mutual information, the method assumes that this dependence must be verified to a certain extent, and this is correct when evaluating the accuracy of the results obtained by the method. Therefore, the augmentation is also relatively accurate, because it relies on the determined pose.

[0031] In addition, the method does not need to rely on any deterministic correlation between the texture of the 3D model and the segmentation of the 2D image. In fact, the method uses a statistical measure of similarity, namely the mutual information. This feature is enabled by considering the 2D rendering of the 3D model to obtain a format comparable to the 2D image. Through this statistical analysis, the method has enough information to be able to associate the 3D model rendered along the candidate pose on the one hand with the extracted 2D image on the other hand. Therefore, the 3D model can be textureless, or alternatively have texture, but with a texture that is not relevant to the segmentation of one or more 2D images, and / or one or more 2D images may be non-segmentable (i.e., difficult or impossible to segment). Moreover, the mutual information (with a probability theory definition) is revealed during testing to provide more accurate results than solutions that rely solely on criteria with non-probabilistic theory definitions (e.g., the L2-norm between the virtual 2D rendering and its (their) corresponding extracted 2D image). In addition, the method does not have to rely on any prior provision of a machine learning dataset. In fact, the pose determination of the method can be completely deterministic (i.e., a deterministic algorithm is executed, i.e., it is not learned and therefore not a trained neural network).

[0032] The one or more (extracted) 2D images may include several 2D images (i.e., a plurality of 2D images), each 2D image corresponding to a respective frame of the video flux. In this case, the determination may reward mutual information for each such 2D image (i.e., for each of the several 2D images / each of the several 2D images). In other words, the reward is performed for each of the extracted plurality of frames of the video flux representing the real object. This allows relatively full exploitation of the richness of information provided by the video flux (e.g., compared to relying on a single extracted image frame). This improves the accuracy and robustness of the method.

[0033] In particular, several 2D images may include 2D images of a real object from different viewpoints. Different viewpoints may, for example, include different viewing angles of a real object. To this end, capture may include moving (e.g., including rotating) a camera in a scene (e.g., including around a real object) so as to capture video frames of a real object from different viewpoints (e.g., including different angles). In this case, the method utilizes the complete spatial coverage of the real object allowed by video flux capture. This still improves the accuracy and robustness of the method. The extracted 2D images may, for example, include multiple images distributed over a time period of less than 20 or 10 and / or greater than 2 or 3 and / or greater than 1 second or 5 seconds and / or less than 1 minute or 30 seconds.

[0034] Real objects can be manufactured products of the real world and / or fixed / immovable property. Real objects can be products in a variety of unlimited industrial fields, including: aerospace, architecture, construction, consumer products, high-tech equipment, industrial equipment, transportation, marine and / or offshore oil and gas production or transportation. Therefore, a real object can be an industrial product, any mechanical part, such as a part of a land vehicle (including automobile and light truck equipment, racing cars, motorcycles, trucks and motor vehicle equipment, trucks and buses, trains), a part of an aircraft (including fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, aerospace equipment), a part of a naval vehicle (including naval equipment, commercial ships, offshore equipment, yachts and workboats, marine equipment), a general mechanical part (including industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment products, metal products, tire products), electromechanical or electronic parts (including consumer electronics, safety and / or control and / or instrument products, computing and communication equipment, semiconductors, medical equipment and equipment), consumer products (including furniture, home and garden products, leisure products, fashion products, products of hard goods retailers, products of soft goods retailers), packaging (including food and beverage and tobacco, beauty and personal care, household product packaging). A real scene can be any real environment where a physical instance of a real object is located.

[0035] The method is computer-implemented. This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, and may be fully or semi-automatically performed. In an example, the triggering of at least some steps of the method may be performed by user-computer interaction. The required level of user-computer interaction may depend on the level of automation foreseen, balanced with the need to implement the user's wishes. In an example, the level may be user-defined and / or pre-defined.

[0036] A typical example of a computer implementation of a method is to perform the method using a system suitable for this purpose. The system may include a processor coupled to a memory having a computer program recorded thereon, the computer program including instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for such storage, and may include several physically distinct parts (e.g., one for the program and possibly one for the database).

[0037] The computer program may include instructions executable by a computer, the instructions including means for causing the above-mentioned system to perform the method. The program may be recorded on any data storage medium, including the memory of the system. The program may be implemented, for example, in digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, such as a product, tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by executing an instruction program by a programmable processor to perform the functions of the method by operating on input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to a data storage system, at least one input device, and at least one output device. If desired, the application may be implemented in a high-level procedural or object-oriented programming language or in an assembly or machine language. In any case, the language may be a compiled language or an interpreted language. The program may be a complete installer or updater. In any case, the application of the program on the system results in the execution of the instructions of the method.

[0038] The system may include a display screen. The processor may be coupled to the display screen and configured to display the (e.g., enhanced) video flux on the display screen. The method may further include displaying the enhanced video flux (i.e., the enhanced video flux produced by the enhancement step performed by the method) on the display screen, e.g. in (substantially) real time during the capture of the video flux. Alternatively, the display may be performed separately after the method.

[0039] The system may include a camera. The camera may be coupled to a processor and configured to capture video flux. Capturing may be performed, for example, via user interaction. A user may carry the camera and move the camera in a scene (e.g., around) to capture video of a real object. The user may, for example, hold the camera in their hands, or alternatively wear an accessory (e.g., glasses) on which the camera is mounted. Other steps of the method may be performed fully automatically, for example, in (substantially) real time while capturing video flux.

[0040] The system can be configured to determine the relative position of the camera during its motion, for example by including a sensor mounted on the camera or fixed relative to the camera. The sensor can be coupled to the processor and configured to determine the relative position during the camera motion so as to track the position of the camera. The sensor can include motion and / or inertial sensors.

[0041] The system may for example comprise a mobile phone or tablet computer programmed for AR (augmented reality) and / or AR glasses. Such a system may be configured for AR as is known per se from the field of augmented reality. The method may comprise carrying such a system around the scene by the user in the manner discussed above to perform the capture of video with a camera of the system, which is per se known from the field of augmented reality.

[0042] The method provides augmented reality by augmenting video flux. "Augmenting video flux" refers to any operation that includes calculating a graphics flux to be superimposed (i.e. overlaid) on the video flux. The augmented video flux may completely include the video flux itself, to which the calculated graphics flux is added (superimposed). This is the case, for example, if the system is a mobile phone or tablet, and the display is performed on the screen of the mobile phone or tablet while being carried by the user to capture the video flux. Alternatively, the augmented video flux may be deprived from the video flux, thereby only including the calculated graphics flux. This is the case, for example, if the system includes transparent AR glasses, where the user does not need to see the originally captured video flux itself, because the user already sees the reality through the AR glasses due to its transparency, and therefore only the calculated graphics flux needs to be displayed on the glasses superimposed on the seen reality. This is known per se from the field of augmented reality and is not described in further detail in the present disclosure.

[0043] "Flux" herein refers to a succession (i.e., a temporal sequence) of images. Thus, a video flux is a set of consecutive 2D images representing a portion of a real scene, each 2D image corresponding to (i.e., being) a respective frame of the video flux. Capturing may be performed continuously, such that two consecutive frames of the video flux may be close to each other (e.g., less than 1 second or 0.1 second apart).

[0044] "Extracting" here simply means that the method selects one or more captured video frames, each frame containing a (full or partial) view of the real object, and provides 2D images corresponding to said frame (or multiple) to the gesture determination step. Each extracted 2D image may be a selected frame or a sub-portion thereof containing a view of the real object. The method may include identifying each such sub-portion in any manner. For example, the identification may include defining a (e.g. rectangular) bounding box around said view of the real object, for example drawn by a user.

[0045] A 3D model is a 3D modeled object. A modeled object is any object defined by data stored, for example, in a database. By extension, the expression "modeled object" denotes the data itself. A "3D modeled object" is any object modeled by data that allows its 3D representation. The 3D representation allows viewing the part from all angles. For example, when represented in 3D, 3D modeled objects can be manipulated and rotated about any of their axes or about any axis in the screen on which the representation is displayed. In particular, this does not include 2D icons that are not modeled in 3D. The display of a 3D representation facilitates design (i.e., increases the speed at which a designer statistically completes a task). Since the design of a product is part of the manufacturing process, this can speed up the industrial manufacturing process.

[0046] Thus, the 3D model comprises a specification of the 3D shape of the real object. The 3D shape may be defined or definable in a 3D reference frame. The specification may be in any format that allows for a virtual 2D rendering of a pose determination. The 3D model may be, for example, a 3D mesh or any other 3D modeled object that allows for the calculation of a 3D mesh, such as a boundary representation or a B-Rep (wherein a B-Rep may be converted to a 3D mesh by tessellation, such as is well known per se in the CAD field).

[0047] The 3D model may have been designed using a CAD system, or may have been derived from a design using a CAD system. By CAD system, it is meant any system, such as CATIA, that is at least suitable for designing a modeled object based on a graphical representation of the modeled object. In this case, the data defining the modeled object includes data that allows the modeled object to be represented. The CAD system may, for example, provide a representation of a CAD modeled object using edges or lines (in some cases with faces or surfaces). Lines, edges or surfaces may be represented in various ways, such as non-uniform rational B-splines (NURBS). Specifically, a CAD file contains specifications from which geometric figures can be generated, so that a representation can be generated. The specifications for a modeled object can be stored in a single CAD file or in multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of 1MB per part. A modeled object can typically be an assembly of thousands of parts.

[0048] The provision of the 3D model may be performed in any manner. The 3D model may, for example, be retrieved by a user or automatically identified by the system, for example in a 3D model library.

[0049] By "pose of the 3D model relative to the video flux", it is meant herein frame-by-frame pose parameters of the camera relative to the 3D model, e.g. defined in a reference frame of the 3D model. The pose parameters are also referred to as "extrinsic parameters". Thus, a "value" of a pose for (i.e. relative to) an extracted 2D image is the value of the pose parameter for a frame of the video flux corresponding to said extracted 2D image.

[0050] As known per se from the field of computer vision, and the (frame-by-frame) intrinsic parameters of the video camera, the pose parameters allow for projecting a 3D model in each frame of the video flux, and the method may comprise the projection. Thus, the video flux may be enhanced based on the result of such projection.

[0051] The 3D model in particular provides a mathematical representation of the 3D shape of the real object, thereby allowing graphics (e.g. text, numbers and / or textures) generated based on calculations based on such a mathematical representation to be added to the video flux. Alternatively or additionally, the 3D model can be associated with any type of data that can be used for augmentation, such as for generating graphics to be added to the video flux. The associated data can, for example, include metadata (e.g. metadata in a database) linked to the 3D model or to different parts of the 3D model. In the case where the 3D model includes different parts, the metadata can, for example, include a text description or a text specification linked to each part of the 3D model. The gesture allows positioning the graphics at the correct position of the video flux relative to the view of the real object therein. This is known per se from the field of augmented reality and is not described in further detail in the present disclosure.

[0052] The method accurately determines such a pose among the candidate poses. This means that the pose is an unknown variable of the method and the determination step outputs a calculated value for it. Here, the candidate pose refers only to the explorable domain provided to the determination step. Since the pose determined by the method is accurate, the projection of the 3D model in each frame is also relatively accurate in the sense that the projected 3D model matches well with the 2D view of the real object in the frame. In other words, the pose defines the virtual position of the camera relative to the 3D model, and thanks to the method, the virtual position accurately corresponds to the real position of the camera relative to the real object.

[0053] The intrinsic parameters may (at least partially) similarly be considered as unknown variables and may be determined by the method, e.g. within the pose determination step (in which case the determination performs a reward not only for each given candidate pose, but also for each given candidate frame-by-frame value of the intrinsic parameters). Alternatively, the intrinsic parameters may (at least partially) be predetermined and thus provided to the method, e.g. output by the camera itself (modern cameras typically provide the intrinsic parameters for their use). Alternatively, the method may set constant and arbitrary values ​​for the intrinsic parameters. This may result in poses with slightly less precision, but still provide relatively accurate projection results, thereby enhancing realism.

[0054] In all these cases, the method outputs a flux of camera parameters in a candidate flux of camera parameters relative to the video flux, each set of camera parameters corresponding to a respective frame of the video flux (the flux comprising a sequence of camera parameter sets), and the method may determine the output flux of camera parameters by rewarding, for at least one 2D image and for each given candidate flux of camera parameters, a representation of mutual information representing a mutual dependency between a virtual 2D rendering of a 3D model and the at least one 2D image using the values ​​of the given candidate flux of camera parameters for the at least one 2D image (i.e., the set of camera parameters in the flux corresponding to the image).

[0055] The determination may perform a reward for the mutual information in any manner that allows for penalizing low values ​​or scores of the mutual information. In other words, the determination is configured to output, other things being equal (or under other conditions), for a given extracted 2D image the pose that results in a maximum value of the mutual information among all candidate poses. The method performs such a reward for each extracted 2D image, and the method may perform other types or rewards such that the determined pose is typically not the one that maximizes the mutual information for all 2D images, but rather is the pose that provides a good score overall, thereby achieving a higher increase in accuracy.

[0056] One particular way that the method can follow to achieve such a reward is to run an optimization procedure, that is, to make the determination process include optimizing an objective function. The objective function includes terms that reward different metrics. In practice, such an optimization procedure can be performed as a minimization procedure including cost (or penalty) terms, each of which penalizes the inverse of a metric for reward.

[0057] The method can utilize any type of optimization algorithm to optimize the program, for example by performing at least one run of an iterative optimization algorithm. The iterative optimization algorithm can start from a starting point (i.e., a starting posture that is then to be optimized). Each iteration may include providing a currently given candidate posture (initialized with the starting posture), and outputting a new candidate posture (for the next iteration) that improves the value of the objective function (e.g., reduces the value of the objective function in the case of a minimization program). The output of the new candidate posture may include calculating the mutual information of the current candidate posture and / or another candidate posture, and comparing the calculated mutual information values ​​of the current candidate posture and another candidate posture. The calculation of the mutual information may include calculating a virtual 2D rendering of the 3D model.

[0058] In the case where the method extracts several images, the pose determination of the method may include optimizing an objective function including a term. Each term may then reward the mutual information of a corresponding 2D image in the several 2D images. In other words, each extracted 2D image may have its own reward term (e.g., cost term). This provides an efficient way to implement the reward.

[0059] Specifically, the objective function may be the sum of the terms. Alternatively or additionally, each term (relative to the extracted 2D images) may be equal to the mutual information for the respective 2D images. This allows for a robust and efficient convergence of the optimization. It has been shown that accurate results can be provided even if the partially extracted 2D images are of poor quality.

[0060] The method may comprise tracking the relative pose of the camera in any known manner, for example based on the previously mentioned sensors (modern AR systems are typically based on such sensors to provide such functionality). In this case, the relative pose of the 3D model between different frames is known. Thus, pose determination may comprise initializing the value of the pose of the 3D model relative to the video flux based on the extracted one or more 2D images, determining the remaining value of the pose based on the initialization value and the relative pose tracking. As a result, the method may be limited to determining the initial pose so as to know the complete pose of the 3D model and thus be able to perform augmentation throughout the entire process of video flux capture. Alternatively or additionally, in the case where the method extracts multiple images and determines the mutual information reward for each such image, pose determination may consider the value of each given candidate pose of only one 2D image (e.g. the first one, in chronological order) as an unknown variable, the values ​​of the poses of the other images being defined by variables as a function of the unknown variables and the relative pose tracking.

[0061] The camera may capture an RGB video flux, or it may capture a grayscale video flux. In the case where the initially acquired video flux is an RGB video flux, the method may optionally include converting the video flux into a grayscale video flux, so that the extracted 2D image is a grayscale image. Alternatively, the method may include converting only selected video frames or only sub-parts of them into grayscale images, so that the extracted 2D image is here again a grayscale image. The efficiency of the method is improved due to the fact that color information is ignored (this neglect is useful in cases where the 3D model does not convey correct color information or even has no color information at all (for example, if it has no texture) and / or in the case of 2D rendering based on shading, so that no texture is taken into account even if it exists).

[0062] Now, the method considers mutual information representing the interdependence between a virtual 2D rendering of a 3D model using pose values ​​and the corresponding 2D image. The term "virtual" simply refers to the fact that the method does not require the output of a 2D rendering. The virtual 2D rendering of the 3D model may include any generation of a realistic 2D image of the 3D model from the point of view defined by the pose values. The virtual 2D rendering may be further based on camera intrinsic parameters (which, as described above, are predetermined or determined by the method). The generation of the virtual 2D rendering may include any rasterization and / or shading, such as the application of a Phong shader or a normal map shader.

[0063] As is known from the fields of probability theory and information theory itself, the mutual information of two variables is a measure of the mutual dependence between the two variables. This concept is described in detail in the textbook by Cover, TM and Thomas, JA (1991) "The Elements of Information Theory" (Wiley ed.), Chapter 2, and can also be found on the website accessible at the following URL: https: / / en.wikipedia.org / wiki / Mutual_information. The contents of this textbook and the contents of this website (at the priority date of this patent application) are incorporated herein by reference, and the mutual information rewarded by the posture determination of this method can be as defined therein, i.e., including the Kullback-Leibler divergence.

[0064] In particular, the mutual information may be the mutual information between a first variable extracted from the virtual 2D rendering and a second variable extracted from at least one 2D image, and the first variable and the second variable may represent a distribution over a range of pixel values. The mutual information may in particular be the mutual information of a joint distribution over such a range of pixel values ​​(Kullback-Leibler divergence). In a specific example, the distribution over the pixel value domain may be a count of pixels having values ​​belonging to one of the predetermined sub-ranges. The sub-ranges (or "bins") may form a subdivision of the range of values ​​taken by each pixel. The subdivision may be conventional. In practice, the subdivision may include a plurality of bins that are higher than 5 or 10, and / or lower than 100 or 50, for example about 20.

[0065] Reference now Figure 1 Examples are shown to discuss optional features of the method.

[0066] The method of the example includes capturing S10 a video flux with a camera. Then, the method of the example includes extracting S20 a plurality of 2D images from the video flux, each 2D image representing a real object. In parallel, the method of the example includes providing S30 a 3D model representing the real object. Then, the method of the example includes determining S40-S80 a pose of the 3D model relative to the video flux. Since the objective function at S50 includes a term that rewards the mutual information of each 2D image (e.g., the aforementioned sum), it is determined S40-S80 that the mutual information of each 2D image in the plurality of 2D images is rewarded. The method of the example finally includes increasing S90 the video flux based on the pose.

[0067] In addition to mutual information, posture determination can also reward photo-consistency between the projections of one or more points of the 3D model on multiple 2D images. In addition to mutual information, posture determination can also reward the match between the projection of the 3D edge of the 3D model and the 2D edge in at least one 2D image. Rewarding these further criteria can improve the accuracy of the determined posture. In the prior art, such criteria (photo-consistency and edge matching) are used as the only and main criteria for determining posture. By using mutual information as the main statistical criterion, the posture determination of this method converges faster, thereby obtaining relatively accurate results. However, the example method still uses other criteria as secondary criteria to improve efficiency, but in the second stage (after running the mutual information based optimization). This generally achieves accuracy, robustness and high speed.

[0068] The photo consistency criterion may be equivalent to assuming that the final pose must be such that the 2D points resulting from the projection of a particular 3D point of the 3D model onto multiple 2D images (according to the final pose) (all these existing 2D points depending on the visibility of the corresponding 3D point for each image) must be graphically consistent, meaning that they must correspond to 2D image positions presenting the same local graphical features. Graphic consistency may be measured by applying at least one rotationally invariant graphical descriptor to such 2D points of different images and comparing the values ​​of the graphical descriptors across the 2D images. The graphical descriptors may also optionally be invariant and / or distinguishable to brightness variations. The graphical descriptors may be selected from a predetermined set of one or more graphical descriptors, for example including SIFT descriptors, FERN descriptors and / or (simply) pixel colors. The 3D points may originate from a sampling of the 3D model performed in any manner (e.g. randomly). The method may reward photo consistency at different stages, and at least the graphical descriptors may be different at each such stage.

[0069] The edge matching criterion may be equivalent to assuming that the final pose must be such that a specific 3D edge of the 3D model (according to the final pose) is projected onto a specific 2D edge of at least one extracted 2D image (e.g. each extracted 2D image). The 3D edges may be, for example, those edges that form the silhouette of the 3D model in each image (according to the value of the final pose of the image). The silhouette of the 3D model is the outline of the 3D model according to the pose. The 2D edges of the image are also referred to as "contours", and the method may determine a contour map in any known manner, such as using a trained neural network, or by calculating a chamfer map of a Canny edge detection result. The contour map may be a contour probability map (i.e., each pixel is assigned a probability of being a contour).

[0070] The optimization may include providing S40 a (predetermined) number N of starting poses, where N>1 (in other words, multiple starting points for optimization to be run later at S50). Providing S40 may be performed in any manner, and a specific implementation will be discussed later. The optimization may then include running S50 a (e.g., the same) iterative optimization algorithm (i.e., an algorithm that optimizes the objective function) N times on the objective function, each time starting from a corresponding one of the starting poses, and each time outputting a corresponding final pose. In practice, N may be higher than 10 or 20 and / or lower than 100 or 50.

[0071] Then, in the example shown, the method can compute the (final) pose based on one of the N final poses via S60 - S80. This can reduce the risk associated with unexpected inaccurate results due to a potential descent of the algorithm into local extrema. In particular, due to the problem at hand, the optimization algorithm may be non - differentiable (the method can run, for example, the Nelder - Mead algorithm). This non - differentiability gives rise to the above - mentioned risk.

[0072] Computing S60 - S80 can include maintaining a (predetermined) number k of optimized poses in S60a for N runs, where k < N. In other words, the method can identify k best results (e.g., the k poses that minimize the objective function) among the N final poses output by S50. In practice, k can be greater than 2 or 3 and / or less than 20 or 10. The other final poses can be discarded at this stage, and only the selected k results are input to the pruning S70. Alternatively, the method can skip S60 and thus input all N final poses to the pruning S70.

[0073] Inputting not just the best final pose output by S50 can reduce the risk associated with unexpected inaccurate results. Such inaccurate results can occur if only mutual information is relied upon, because mutual information is a statistical measure and thus can be ambiguous (e.g., in symmetric cases). Step S60 similarly reduces the risk associated with unexpected inaccurate results (compared to inputting all N final poses to the pruning S70) because the light consistency criterion of the pruning S70 may lead to an unexpected selection of a pose with insufficient satisfaction regarding mutual information.

[0074] Computing S60 - S80 can include pruning S70 of the N final poses to output a single pose (which is then input to step S70). Pruning S70 is shown as separate from S60, but S60 can equivalently be considered part of the pruning. In the example shown, pruning S70 consists of selecting the best pose among the k optimized poses output by S60 by rewarding the previously described light consistency. For example, pruning S70 can select the one pose among the k optimized poses that performs best with respect to the light consistency criterion. This can include sampling the 3D model and then evaluating the light consistency criterion for each of the k poses (e.g., including computing any of the previously mentioned descriptors for each 2D point generated by the projection of the sampled 3D points, e.g., the SIFT descriptor).

[0075] Then, computing S60 - S80 can include refining S80 the pose output by S70, and the refined pose can be considered the finalized pose and is input to S90 to enhance the video flux.

[0076] Refinement S80 may reward the light consistency described earlier (thus used again) and / or 2D edge 3D edge matching, such as these two criteria. Refinement S80 may, for example, minimize the global error. The global error may include a first term that penalizes the error between the projected contour of the 3D model (e.g., a virtual grid on each 2D image, i.e., a 2D view) and the detected edge of the real object in the 2D image, and / or a second term that penalizes the error between the descriptor values ​​across different 2D images at a 2D point corresponding to, for example, a sampled 3D point. In the case of an edge matching reward based on a contour probability map as described above, the first term may penalize the low degree of the contour probability of a 2D pixel on which a 3D edge (e.g., corresponding to a silhouette) is projected. In the case of a light consistency reward based on a graphic descriptor as described above, the second term may penalize the difference between the values ​​of the graphic descriptor on the projection of the (e.g., sampled) 3D point on the 2D image. Minimization may be based on gradients. This improves efficiency. In this case, the graphic descriptor may be distinguishable, such as pixel color.

[0077] Figure 2 An example of a system is shown.

[0078] The client computer of this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer is also provided with a graphics processing unit (GPU) 1110, which is associated with a video random access memory 1100 connected to the bus. Video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device, such as a hard disk drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; CD-ROM disks 1040. Any of the above may be supplemented or combined by specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a display screen 1080 and a camera 1090.

[0079] Now discuss the Figure 1 An example method implementation is provided.

[0080] These embodiments of the method aim to compute the pose (and possibly intrinsic parameters) of a camera taking different views of a known 3D object. By "known" it is meant that the system has a (e.g., textureless) 3D model (e.g., 3D mesh) representing the object. Furthermore, it is assumed here that the embodiments of the method are in the context of an augmented reality system, where the system knows the relative poses between the different views taken by the camera.

[0081] Embodiments of the method rely on the definition of a pose-dependent energy based on the mutual information between the virtual rendered real image. This energy is discretized in several poses (e.g., photographed on a Thomson sphere). Embodiments of the method optimize this energy to find the best possible pose. Then, embodiments of the method prune wrong poses by exploiting a light consistency criterion. Finally, embodiments of the method fine-tune the pose by fitting the image edges.

[0082] Embodiments of the method are able to compute pose even when segmentation is not possible or inaccurate and when the texture of the object is unknown. Furthermore, since implementations of the method combine mutual information methods, photo-consistency criteria, and silhouette / contour matching in a multi-view framework, they can disambiguate and obtain more accurate poses. Embodiments of the method utilize all views at once in mutual information optimization without favoring any view. Furthermore, embodiments of the method utilize photo-consistency criteria and matching the projected silhouette of the 3D model to the nearest contour in the image to locally improve the pose. Finally, embodiments of the method are independent of any dataset / machine learning algorithm.

[0083] Let I1,…,I n is n real images, R i For I1 and I i The known relative rigid transformation between (R1 = Id 3×3 ), that is, from frame I1 to frame I i Transformation of . is present in all views I1,…,I n A goal might be to compute the mesh of the objects in view I1. Posture where pose is defined as the composition of projections with rigid transformations (rotation and translation).

[0084] Let T i (P) is view I i The pose matrix of the mesh is given by the pose P of frame I1, that is, if P = KR and the projection matrix is ​​K, then R is the rigid transformation from the mesh frame to the I1 frame, then T i (P)=KR i R.

[0085] Implementations of the method may include providing the portion of the image I1 where the object is located. For example, a user may draw a bounding box around it.

[0086] Multi-view optimization

[0087] Preprocessing

[0088] First, the implementation of the method can be performed on images I1,…,I n Preprocessing is performed to convert them to grayscale brightness space. Then, the implementation of the method can equalize the histogram of each image. Each pixel color is a one-dimensional integer belonging to [0,255].

[0089] Once all images have been converted, embodiments of the method may apply any method to P1, ..., P N The poses are sampled.

[0090] For example, if the pose you are looking for is known Approximate value of As is often the case in augmented reality, users can If the image I1 is taken at an approximate known position, the implementation of this method can be The surrounding samples P1,…,P N .

[0091] For example, if this approximation is not known, an implementation of the method could sample points on a Thomson sphere around the grid and consider the camera located at one of the sampled points on the sphere and looking toward the center as P1,…,P N The radius of the Thomson sphere and the translation in the camera optical plane can be set so that the bounding box of the projection is equal to the expected position of the object in image I1.

[0092] If the intrinsic parameters (focal point and principal point) of the real camera are known, the implementation of this method can use these parameters for the intrinsic matrix. If they are unknown, the implementation of this method can initialize the principal point to the center of the image and the focal point to (width+height) / 2, where width and height are the width and height of the image I1. These intrinsic parameters can be optimized together with the pose, or can be fixed.

[0093] Mutual Information Definition

[0094] Let f be the mesh that maps pose P to a shot at pose P. For example, an implementation of this method could render a mesh using a Phong shader in f Alternatively, you can use a normal map shader. In this case, the color of each pixel is in the range [0,255]. 3 Implementations of this method can also impose f to produce views with the same resolution as the real image.

[0095] Given a pair of images (J1, J2), and c1 and c2 are the number of channels of J1 and J2, Represents the joint histogram of the multi-channel color distributions of the input images (J1, J2). That is, implementations of the method may concatenate the second image as an additional channel to the first image, and treat the pair of images as a new image with channels from both the first and second input images. The number of bins in each channel of the histogram can be between 2 and the number of possible colors. In practice, implementations of the method may use 20 bins to obtain a meaningful histogram that remains accurate. This means that implementations of the method may concatenate the interval Divide evenly into subintervals, and for each subinterval, the implementation of the method may count the number of pixels whose multi-channel color belongs to this subinterval (e.g., normalized by the total number of pixels) (where the color of the pixel in the joint image (J1, J2) belongs to ).

[0096] The implementation of this method can marginalize the distribution H on the second image to obtain only the distribution of colors in image J1 And similarly, the implementation of this method can consider only the distribution of colors in image J2 If X follows the joint distribution A random variable, then yes The distribution of yes distribution.

[0097] Finally, let D be the joint distribution The mutual information is:

[0098]

[0099] Such mutual information is the Kullback-Leibler divergence between the joint distribution and the marginal distributions in J1 and J2. If the color distributions in J1 and J2 are independent, then D(J1,J2)=0, and the more the colors in J1 and J2 are linked, the higher the mutual information. It is always non-negative and symmetric. When the color in J2 is a function of the color J1, then the mutual information is highest and equal to the entropy of J1.

[0100] Note that mutual information does not characterize a linear relationship between the colors in J1 and J2 (unlike correlation). It is a value that describes how well one can infer the colors in J2 given that one knows the colors in J1, i.e., the random correlation of the colors in J1 and J2.

[0101] Mutual Information Optimization

[0102] To calculate the best pose This method can be implemented to maximize the mutual information between the virtual rendering and the real image. In fact, if the grid If the alignment is good, the colors in the virtual rendering should strongly depend on the colors in the real image, even without a simple affine transformation to match the color spaces of the virtual and real images, e.g. an implementation of the method may not know the real texture of the mesh.

[0103] That is why the implementation of this method minimizes the energy of:

[0104]

[0105] To minimize E1(P) and avoid poor local minima, the method can be implemented from P1,…,P N Start running N optimizations. The optimizations can be done with any algorithm. In a valid example, the implementation of this method can use Nelder-Mead (such as in the test implementation discussed later).

[0106] The implementation of this method can keep k (k=5 in the test implementation) best optimized poses in N runs, expressed as

[0107] This is due to Figure 3 Shown.

[0108] Posture Pruning

[0109] From now on, the implementation of this method can be used with the original images I1,…,I n Use together without any preprocessing.

[0110] In order to To select the best pose and avoid ambiguity, the implementation of this method can use a light consistency criterion.

[0111] Let x1,…,x m For Grid The sampling of the last m points, n1,…,n mbe their corresponding normals. Let s(x,I) be a function that maps a 2D coordinate x on an image I to a local descriptor that is (ideally) invariant to changes in illumination and orientation. s(x,I) can be as simple as the color I(x) itself, or it can take into account local neighbors, as with SIFT descriptors, but can also be extracted using a pre-trained neural network.

[0112] Finally, let in I j It is visible in}. It can be simply through each posture T j (P)Medium The Z buffer is used to calculate make

[0113] This method can be implemented in The best pose is chosen as the pose that minimizes

[0114]

[0115] Where σ is a penalty function similar to the L2 norm, ρ is a weighting function, and when the normal n of the point i with the camera's eye vector (given by the pose PR j When given) is orthogonal, the energy weight is lower, is x i Average of the descriptors across all images visible:

[0116]

[0117] Indicates that the The most consistent posture in .

[0118] This is due to Figure 4 Shown.

[0119] Posture Refinement

[0120] Finally, embodiments of the method can refine the image by minimizing local color-based and edge-based energies. To increase light consistency and match the projected grid The edge between the contour and the edge in the image.

[0121] First, the implementation of this method can be each image I j Normalization, e.g. on the projected grid has zero mean and variance 1 in the domain of , so that the pixel color illumination is invariant. Other normalizations can be considered.

[0122] Let σ be the loss, penalizing two colors The difference between, for example, the L2 norm or Let J j For I j Smoothed image of edges (e.g., predictions from a neural network, chamfer maps from Canny edge detection, or chamfer maps with squared distances). is the projection on M to view T j (P)Medium A 3D point on the silhouette of .

[0123] This method can be implemented by Start refining the pose by minimizing the following energy

[0124]

[0125] in

[0126]

[0127] This minimization can be handled by any differentiable optimization, and the Jacobian of the pixel color with respect to its position in the image is given by the Sobel filter.

[0128] Figure 5-6 Shown according to Figure 1 The results obtained for tested embodiments of the method are presented and formulas are provided for the discussed embodiments of the method.

[0129] Figure 5 It shows how a 2D frame of a video flux of a scene comprising a table and chairs can be accurately enhanced by the method using the projection of its 3D model. It can be seen that the error between the graphical representation 64 of the table on the 2D frame and the 3D model 62 of the table is relatively small.

[0130] Figure 6 It is shown how this method can be used to precisely enhance the video throughput of a scene including a coffee machine 62 using text descriptions 64 of different parts of the coffee machine associated with a 3D model of the coffee machine in the database. Figure 6 In the image, a projection 66 of the 3D model of the coffee machine is also shown in a semi-transparent manner, which is superimposed on the 2D view of the coffee machine in the image.

Claims

1. A computer-implemented augmented reality method for augmenting a video stream of a real scene including a real object, the method comprising: − capturing the video flux using a camera; − extracting one or more 2D images from the video flux, each 2D image representing the real object; − providing a 3D model representing the real object; − determining among candidate poses a pose of the 3D model relative to the video flux, the determining rewarding, for at least one 2D image and each given candidate pose, mutual information representing the interdependence between: o a virtual 2D rendering of the 3D model using the value of the given candidate pose of the at least one 2D image, and o the at least one 2D image; and − enhancing the video throughput based on the posture, Wherein, the mutual information includes Kullback-Leibler divergence, where the mutual information is the Kullback-Leibler divergence between: a joint distribution of pixel value ranges extracted from the concatenation of the virtual 2D rendering and the at least one 2D image, and A first marginal distribution over a range of pixel values ​​extracted from the virtual 2D rendering, and a second marginal distribution over a range of pixel values ​​extracted from the at least one 2D image.

2. The method according to claim 1, wherein: The one or more 2D images include a plurality of 2D images, each 2D image corresponding to a respective frame of the video flux, the determining rewarding the mutual information for each 2D image of the plurality of 2D images.

3. The method according to claim 2, wherein: The determining includes optimizing an objective function including terms, each term rewarding the mutual information for a corresponding 2D image in the plurality of 2D images.

4. The method according to claim 3, wherein: The objective function is the sum of the terms.

5. The method according to claim 4, wherein: Each term is equal to the mutual information of the corresponding 2D images.

6. The method according to claim 3, 4 or 5, wherein: The optimization includes: − Provide N starting poses, where N > 1; and − running a non-differentiable iterative optimization algorithm N times on the objective function, each time starting from a corresponding one of the starting postures and outputting a corresponding final posture; and the determining comprises: − Compute a pose based on one of the N final poses.

7. The method according to any one of claims 2 to 5, wherein: The determination further rewards photo consistency between projections of one or more points of the 3D model onto the plurality of 2D images.

8. The method according to claim 6, wherein: The determination further rewards photo consistency between projections of one or more points of the 3D model onto the plurality of 2D images.

9. The method according to claim 8, wherein: The determination includes: - in computing, performing said reward for said photoconsistency and outputting a pruning of one of said N final poses; and / or - After optimization, a refinement of the reward for the photoconsistency is performed.

10. The method according to any one of claims 1 to 5, wherein: The determination also rewards a match between a projection of a 3D edge of the 3D model and a 2D edge in at least one 2D image.

11. The method according to claim 9, wherein: The determining further rewards a match between a projection of a 3D edge of the 3D model and a 2D edge in at least one 2D image, the thinning performing the reward for the match.

12. The method according to claim 10, wherein: For each respective image of the at least one image, the 3D edge forms a silhouette of the 3D model in the respective image.

13. The method according to any one of claims 1 to 5, wherein: The mutual information is the mutual information between: − a first variable extracted from said virtual 2D rendering, and − a second variable extracted from the at least one 2D image, the first variable and the second variable representing a distribution over a range of pixel values.

14. The method according to claim 13, wherein: The distribution over the pixel value domain is a count of pixels having values ​​belonging to one of the predetermined sub-ranges.

15. A computer-readable storage medium having instructions recorded thereon, wherein when the instructions are executed by a computing device, the computing device is caused to execute the method according to any one of claims 1 to 14.

16. A system comprising a processor coupled to a memory having recorded thereon the instructions of claim 15.

Images