3D reconstruction with smoothness maps

Abstract

To provide a computer-implemented method, system, and program for 3D reconstruction.SOLUTION: The method includes providing 2D images and, for each 2D image, providing camera parameters that define a perspective projection. The 2D images all represent the same real object. The real object is fixed. The method also includes providing, for each 2D image, a smooth map. The smooth map has pixel values, in which each pixel value represents a measurement of contour existence. The method also includes determining a 3D modeling object representing the real object. The determination iteratively optimizes energy. The energy rewards, for each smooth map, a projection of a silhouette vertex of the 3D modeling object having a pixel value representing a high measurement value of contour existence.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to the field of computer programs and systems, and more particularly to methods, systems and programs for 3D reconstruction. [Background technology]

[0002] For the design, engineering and manufacturing of objects, numerous systems and programs are offered on the market. CAD is an acronym for Computer-Aided Design, e.g., related to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, e.g., related to software solutions for simulating the physical behavior of a future product. CAM is an acronym for Computer-Aided Manufacturing, e.g., related to software solutions for defining manufacturing processes and operations. In such Computer-Aided Design systems, the graphical user interface plays a key role regarding the efficiency of the technique. These technologies may be incorporated within Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes and leverage corporate knowledge to help develop products from concept to life, beyond long-term corporate concepts. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and a Corporate Hub that allows connections to both the Corporate Hub and the Engineering and Manufacturing Hubs. Overall, the system provides an open object model that links products, processes, and resources to enable dynamic knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this and other contexts, 3D reconstruction is gaining widespread importance. 3D reconstruction relates to computer vision techniques that allow to determine 3D modeled objects based on data representing real objects or real scenes. Such techniques may be useful in areas such as virtual reality and augmented reality (more generally any kind of immersive experience), video games, manufacturing and 3D printing, and / or 3D modeling.

[0004] Although much research effort has been made in this field over the past few years, there is still a need for improved solutions for 3D reconstruction. Summary of the Invention

[0005] Thus, the present invention provides a computer-implemented method for 3D reconstruction. The method includes providing 2D images and, for each 2D image, providing camera parameters that define a perspective projection. All 2D images represent the same real object. The real object is fixed. The method also includes providing, for each 2D image, a smoothness map. The smoothness map has pixel values, each pixel value representing a measure of isoline presence. The method also includes determining a 3D modeled object that represents the real object. The determination iteratively optimizes an energy. The energy rewards, for each smoothness map, projections of silhouette vertices of the 3D modeled object that have pixel values ​​that represent a high measure of isoline presence.

[0006] The method may include one or more of the following. The smooth map contains local extrema corresponding to isoline probability maxima, and the energy rewards high isoline probabilities in the projection of the silhouette vertices. The smoothness map is a function of the isoline probability map. The function is a smoothing of the affine mapping. The smoothing may include applying at least one of a Gaussian blur to the isoline probability map, or convolving the isoline probability map with a kernel derived from a Laplace distribution. The smoothing includes determining an envelope of application of a family of Gaussian blurs to the isoline probability map, each Gaussian blur having a different kernel size. The energy comprises terms of the following types:

number

[0007] A computer program comprising instructions for carrying out the method is further provided.

[0008] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0009] Further provided is a system including a processor coupled to a memory and a graphical user interface, the memory having a computer program recorded thereon. [Brief description of the drawings]

[0010] [Figure 1] 1 shows an example of a graphical user interface of the system. [Diagram 2] An example of the system is shown below. [Diagram 3] This method is shown below. [Figure 4] This method is shown below. [Diagram 5] This method is shown below. [Figure 6] This method is shown below. [Figure 7] This method is shown below. [Figure 8] This method is shown below. [Figure 9] This method is shown below. [Figure 10] This method is shown below. [Figure 11] This method is shown below. [Figure 12] This method is shown below. [Figure 13] This method is shown below. [Figure 14] This method is shown below. [Figure 15] This method is shown below. [Figure 16] This method is shown below. [Figure 17] This method is shown below. [Figure 18] This method is shown below. [Figure 19] This method is shown below. [Figure 20] This method is shown below. [Figure 21] This method is shown below. [Figure 22] This method is shown below. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention provides a computer-implemented method for 3D reconstruction. The method includes providing 2D images and, for each 2D image, providing camera parameters that define a perspective projection. All 2D images represent the same real object. The real object is fixed. The method also includes providing, for each 2D image, a smoothness map. The smoothness map has pixel values, each pixel value representing a measure of isoline presence. The method also includes determining a 3D modeled object that represents the real object. The determination iteratively optimizes an energy. The energy rewards, for each smoothness map, projections of silhouette vertices of the 3D modeled object that have pixel values ​​that represent a high measure of isoline presence.

[0012] This forms an improved solution for 3D reconstruction.

[0013] This method falls within the scope of structure-motion analysis techniques. In fact, this method allows to reconstruct real objects in 3D on the basis of 2D images, which are relatively simple and inexpensive to acquire compared to other types of signals. Therefore, this method forms a relatively simple and inexpensive 3D reconstruction technique.

[0014] Furthermore, the optimization is an iterative optimization. Iterative optimization techniques are known to be practical methods for solving optimization problems. Therefore, the method is relatively efficient to perform.

[0015] Now, the method relies in particular on maps with pixel values ​​each representing a measure of isoline presence. For each contour map, a projection of silhouette vertices of the 3D modeled object with pixel values ​​representing a high measure of isoline presence, in order to optimize the energy that the 3D modeled object rewards, the method makes it possible to determine, to the extent possible, 3D modeled objects whose silhouettes are projected onto isolines in the 2D image. Thus, the method is based on the observation that the silhouettes of real objects represented by the 2D image are transformed into isolines in the 2D image. As a result, the 3D reconstructions enabled by the method are relatively accurate.

[0016] Moreover, the map provided by the method is a particularly smooth map. As a result, the optimization is able to follow paths that lead to local extrema of the map during the iterations, thereby robustly converging towards an optimal 3D modeled object. This results in a relatively high accuracy and robustness of the method. The iterative optimization may in fact be gradient-based, for example involving the calculation of the gradient of the smooth map to follow such a gradient.

[0017] Each 2D image represents physical signals forming a 2D graphical representation of a real object, for example in a grid arrangement of pixels. The grids may all be regular and / or rectangular. The 2D images may include RGB and / or grayscale images. The determination may for example be performed solely on the RGB and / or grayscale images. The 2D images may include images captured by one or more physical cameras, each having one or more respective sensors, such as photographs. Providing the 2D image may include acquiring and / or receiving from a remote system at least a part of the 2D image and / or retrieving in memory at least a (different) part of the 2D image.

[0018] All the 2D images represent the same real object. For example, the 2D images may each have been taken at different times using the same and unique camera observing the same and unique real object with different camera parameters (e.g., different camera extrinsic parameters, including from different viewpoints, and / or different camera intrinsic parameters). At least some of the 2D images may represent the real object from different viewpoints. Thus, the method combines several pieces of 2D graphic information on the real object (i.e., the 2D images) to build a 3D model of the real object. In an example, the 2D images may form different frames of the same video caption of the real object.

[0019] The camera parameters are provided by the method of determining. In other words, the camera parameters are known before the determination (i.e., the provision of the camera parameters may be performed before the determination). The camera parameters may be known, for example, before performing the method, or the method may include, for example, calculating such camera parameters (for example, approximately) separately and before the determination, according to any known algorithm. This method is therefore relatively efficient, since the determination does not need to calculate the camera parameters within the optimization. Rather, the determination may use the provided camera parameters as it to manipulate the reward (i.e., the reward by optimized energy of the projection of silhouette vertices with pixel values ​​that represent a high measure of isoline presence for each smooth map). The iterative optimization may in fact include iteratively determining silhouette vertices corresponding to a given 2D image based on the camera parameters, and projecting the silhouette vertices onto the 2D image depending on the camera parameters. The camera parameters of each 2D image may be associated / attached to the image at the time of capturing the image, according to any known technique (outside the scope of this discussion).

[0020] The camera parameters of each 2D image define the corresponding perspective projection according to which (fixed) real objects are displayed in the 2D image. This simplifies the method since the camera usually captures the 2D images according to such perspective. It also makes it possible to take into account the perspective exactly and not approximate it with, for example, a parallel projection, as this would introduce errors into the process, especially for objects that are photographed from a close distance and therefore where the perspective is more pronounced.

[0021] The real object is fixed in the 2D image. In other words, the real object is immobile across all 2D images input to the determination. This means that the real object is not moved or at least not substantially deformed (i.e., deformation and movement are negligible relative to the dimensions of the real object). As a result, the determination may be relatively simple to implement and the optimization may converge relatively quickly and robustly. The fixed real object allows for easier and faster combination of information provided by different images, since the information is more directly comparable. Therefore, the method has industrial applications that allow for 3D reconstruction of stationary physical objects, such as non-biological objects, for example industrial products.

[0022] In an example, the method may include a user-machine interaction in which the user captures 2D images with a video camera by moving the video camera around a fixed real object, for example at a free distance from the real object. The video camera may be a standard one and may capture the video frames according to a perspective projection. Thus, the method forms a simple structure analysis from motion.

[0023] Providing each 2D image with a corresponding smoothness map will now be described.

[0024] Each smooth map is made up of pixels arranged in the same grid as the 2D image, for example. Each pixel of each smooth map has a value that represents a measure of the presence of a contour in the corresponding pixel of the respective 2D image. In other words, each smooth map pixel value measures the local presence of an image contour.

[0025] By "isoline" is meant any location of a 2D image that corresponds to a separation (e.g. as sharp as possible) between two different textures of the 2D image, each texture corresponding to a different real-world material. Thus, the set of isolines (e.g. approximately) includes the projected boundary or visible silhouette of a real object, or the boundary of the visible silhouette of a (distinguishable) part of a real object. Such a concept is known per se from the field of image processing, in particular as applied to 3D reconstruction. As is known, a measure of isoline presence may be a function of the gradient of pixel values ​​of the 3D image. Many different techniques exist in the prior art for obtaining such a measure.

[0026] By "measurement" we mean any information regarding the isoline presence at a location corresponding to a pixel value. The values ​​of different pixels of a smooth map can be compared with each other, so that the isoline presence can be compared at different corresponding locations. The pixel value may correspond monotonically to the isoline presence (decreasing (or increasing) gradually, i.e., a higher (or lower) pixel value is more indicative of isoline presence than a lower (or higher) pixel value, and thus represents a relatively higher (or lower) measure of isoline presence).

[0027] By "smooth" we mean that the smooth map has a non-zero gradient at least in the vicinity (i.e., neighborhood) of the image isoline (i.e., everywhere or in such a neighborhood), that it varies monotonically towards a local optimum, and that it exhibits C0 continuity (i.e., having only point continuity, not derivative continuity, e.g., is sufficient at the optimum).

[0028] "For a smooth map, reward the projections of silhouette vertices of 3D modeled objects having pixel values ​​that represent a high measure of isoline presence" means that the optimization favors 3D modeled objects having silhouette vertices (called "projections of silhouette vertices") that project onto 2D points of the respective 2D images corresponding to pixels of the smooth map whose values ​​further indicate the presence of isolines. The optimization can perform such rewards by minimizing an objective function that includes a cost term that penalizes the absence of isolines, i.e., a low measure of isoline presence. The optimization can minimize the cost term, ceteris paribus, and thus maximize the overall measure of isoline presence by the projections of silhouette vertices.

[0029] Thanks to the measures provided as smoothness maps for each 2D image, the optimization can iteratively follow a monotonic gradient path of each of the smoothness maps so as to converge relatively quickly towards a local optimum corresponding to a local maximum of isoline existence.

[0030] Each smooth map may, for example, contain a local extremum corresponding to a local maximum of the respective isoline probability (i.e., the isoline probability is locally maximum at the local extremum of the smooth map). In such a case, the energy may simply perform the reward described above as follows: the energy may reward high isoline probabilities in the projection of the silhouette vertices. The smooth map may in particular vary from a value representing a corresponding zero isoline probability to said local extremum. Thanks to its smoothness, the gradient of the smooth map is not negative even at pixels corresponding to non-isoline probabilities, and the gradient-based iterative optimization may still proceed successfully towards the local extremum with the reward.

[0031] The smooth map may for example be a function of the isoline probability map. This means that the smooth map is calculated from the isoline probability map and inherits the information contained therein. In particular, the function maintains the positions and values ​​of the extremes (i.e. the extremes of the smooth map are the same in position and value, e.g. the same as the extremes of the isoline probability map, e.g. modulo 1-p inversion, as described below), and the function provides a non-zero gradient, a monotonic variation towards a local optimum, and C0 continuity. The method may include providing an isoline probability map for each 2D image and applying the function to each isoline probability map (to output a corresponding smooth map). This allows for a relatively accurate determination of the 3D modeled object, since the optimization depends entirely on the relatively fine information provided by the isoline probability map. The function may be a monotonic function.

[0032] As is well known, an isoline probability map is a map with pixel values ​​each representing the probability of isoline presence. Such a map can be calculated in any way. For example, it can be calculated based on a machine learning scheme implemented on a container probability map or an annotated dataset of 2D images related to the container map (wherein each pixel value represents a quadratic continuum of presence or absence). Such machine learning can be performed, for example, according to or similar to the teachings of the paper by S. Xie, Z. Tu., Holistically-nested edge detection, in Proceedings of the IEEE international conference on computer vision (pp. 1395-1403), 2015, which is incorporated herein by reference. The edge detection method described in this paper is based on learning a network to predict edges in an image by supervised training on an image dataset annotated with edges. The network extracts multi-scale predictions, and the classification loss for each pixel is a binary mask of the edge ground-truth and a cross-entropy between the prediction of the edge at each scale. The predictions at each scale are then aggregated to obtain a final prediction.

[0033] Alternatively, the method can follow the teachings to compute contour probability maps for any one of a number of other papers that disclose machine learning for predicting edges in a supervised manner (the differences being expressed mainly in terms of architectures and losses), such as the following papers (all of which are incorporated herein by reference): DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection, Gedas Bertasius, Jianbo Shi, Lorenzo Torresani, CVPR 2015. Object Contour Detection with a Fully Convolutional Encoder-Decoder Network, Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, Ming-Hsuan Yang, CVPR 2016. Pixel-Wise Deep Learning for Contour Detection, Jyh-Jing Hwang, Tyng-Luh Liu, ICLR Workshop 2015.

[0034] Further alternatively, the contour probability map may be calculated by deterministic methods, such deterministic methods are widely known and may for example involve calculating a chamfer map resulting from Canny edge detection.

[0035] The function may be (the result of) a smoothing of a monotonic (e.g., affine) mapping. The smoothing is a function as described above, i.e., the smoothing maintains the location and value of the extremes, the smoothing results in a non-zero gradient, a monotonic variation towards a local optimum, and C0 continuity. As a result of such a combination of smoothing and a monotonic (e.g., affine) mapping, the information of the initial probability map is maintained, in particular the information that allows the comparison of the presence of isolines between different pixels. The affine mapping (i.e., the function) may be a probability complementary mapping (i.e., 1 minus the probability). The smoothed map may thus form a smoothed inverse isoline probability map, with pixel values ​​each representing the probability of isoline non-existence after smoothing (i.e., the complement of the isoline probability). This improves the convergence speed. In such a case, the optimization problem is a minimization problem. Alternatively, the monotonic mapping may be an identity. In such a case, the optimization problem is a maximization problem.

[0036] Any smooth map herein may result, for example, from a scheme that includes computing an isoline probability map (e.g., as described above), applying a monotonic (e.g., affine) mapping to the isoline probability map (e.g., a probability complementary mapping, or alternatively, which may simply be an identity mapping), and smoothing the result of the mapping (e.g., in any known manner, as described below). The method may include performing the scheme on at least some (e.g., all) of the smooth map, and / or receiving and / or retrieving at least a portion of the smooth map itself.

[0037] The method may alternatively use a distance transform of the image to generate a map containing the distance to the nearest isoline at each pixel. However, such an approach requires a threshold to identify isolines with probability values ​​above the threshold and then to compute the distance transform of the image. This means that the difference between the probabilities of two isolines that are both within the threshold is lost and the isoline probabilities are not exploited as they are.

[0038] As mentioned above, the determining step includes an optimization scheme, which may be iterative and gradient-based, iteratively computing the gradient of the smooth map for each silhouette vertex projection to converge towards the 3D modeled object that achieves a global optimum over the smooth map (thereby representing a global maximum of the contour probability).

[0039] The optimization may include, at each iteration, providing a “current” 3D modeled object, determining the projections of the silhouette vertices, and modifying the 3D modeled object to reduce energy (e.g., including computing the gradient of the smoothness map at the pixels corresponding to the projection), and the modified 3D modeled object may be input to the next iteration as the “current” 3D modeled object for the next iteration until a convergence criterion is reached.

[0040] The iterative optimization may start from a 3D modeled object that forms an approximation of the optimal 3D modeled object found by the method. Such initialization of the optimization with an approximate 3D model of the real object allows for a fast and robust convergence of the optimization. This also allows for a "smoothness map" that is "locally smooth", i.e. where the projection of the initial shape is located, as long as the gradient is "good" up to the pixel of maximum isoline probability, while zero gradient zones may exist elsewhere. As a result, the smoothness requirement is only present in the vicinity of the true isoline, as long as the silhouette of the initial shape projects into these regions. Such an approximation can be obtained in any known way, for example by implementing existing 3D reconstruction techniques. Examples are given below.

[0041] The method is computer-implemented. This means that the steps of the method (or substantially all steps) are performed by at least one computer or any system. Thus, the steps of the method are performed by a computer, possibly fully automatically or semi-automatically. In an example, the triggering of at least some of the steps of the method may be performed via user / computer interaction. The level of user / computer interaction required depends on the level of automation expected and may be balanced against the need to implement the user's wishes. In an example, this level may be user-defined and / or predefined.

[0042] A typical example of a computer implementation of the method is to execute the method using a system adapted for this purpose. The system may comprise a processor coupled to a memory and a graphical user interface (GUI), in which a computer program is recorded, the computer program comprising instructions for executing the method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally comprise several physically distinct parts (e.g. a part for the program and possibly a part for the database).

[0043] The method generally operates on modeled objects. A modeled object is any object defined by data stored, for example in a database. By extension, the expression "modeled object" designates the data itself. Depending on the type of system, the modeled object may be defined by different kinds of data. A system may in fact be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these different systems, the modeled object is defined by the corresponding data. Thus, one can speak of CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, CAE data. However, these systems do not make one of the systems exclusive of the other, since a modeled object may be defined by data corresponding to any combination of these systems. Thus, a system may be both a CAD and a PLM system.

[0044] A CAD system further means any system adapted to at least design a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, the data defining the modeled object includes data enabling the representation of the modeled object. A CAD system may, for example, provide a representation of a CAD modeled object using edges or lines, in some cases with faces or surfaces. The lines, edges or surfaces may be represented in various ways, for example with Non-Uniform Rational B-Splines (NURBS). In particular, a CAD file contains specifications from which a geometric shape may be generated, and thus a representation may be generated. The specifications of the modeled object may be stored in a single CAD file or in multiple CAD files. A typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. And the modeled object may typically be an assembly of thousands of parts.

[0045] In the context of CAD, a modeled object may typically be a 3D modeled object that represents a product, such as a part or an assembly of parts, or possibly an assembly of a product. By "3D modeled object" is meant any object that is modeled by data that allows for its 3D representation. The 3D representation allows for viewing the part from all angles. For example, a 3D modeled object, when represented in 3D, can be manipulated and rotated around any of its axes or around any axis within the screen on which the representation is displayed. This notably excludes 2D icons that are not 3D modeled. The display of 3D representations facilitates design (i.e. statistically increases the speed at which designers accomplish their tasks). This speeds up the manufacturing process in industry, since the design of the product is part of the manufacturing process.

[0046] A CAM system further means any solution, hardware or software, adapted to manage the manufacturing data of a product. The manufacturing data generally includes data about the product to be manufactured, the manufacturing process and the necessary resources. CAM solutions are used to plan and optimize the entire manufacturing process of a product. For example, the CAM user can be provided with information about the feasibility of the manufacturing process, the duration of the manufacturing process or the number of resources, such as specific robots, that can be used in a particular step of the manufacturing process, thus allowing decisions regarding management or necessary investments. CAM is a follow-up process after CAD and potentially CAE processes. Such CAM solutions are offered by Dassault Systèmes under the trademark DELMIA®.

[0047] CAE system further means any solution, hardware or software, adapted to the analysis of the physical behavior of a modeled object. A well-known and widely used CAE technique is the Finite Element Method (FEM), which typically involves dividing the modeled object into elements whose physical behavior can be calculated and simulated by equations. Such CAE solutions are offered by Dassault Systèmes under the trademark SIMULIA®. Another growing CAE technique involves the modeling and analysis of complex systems that comprise multiple components from different fields of physics without CAD geometry data. CAE solutions allow the simulation and therefore the manufacturing optimization, improvement and validation of products. Such CAE solutions are offered by Dassault Systèmes under the trademark DYMOLA®.

[0048] The 3D modeled object reconstructed by the method may represent the geometry of a product to be manufactured in the real world, e.g. after completion of the virtual design using a CAD software solution or CAD system, e.g. a (mechanical) part or an assembly of parts (or equivalently an assembly of parts, since an assembly of parts can be seen as a part itself from the point of view of the method), or more generally any rigid assembly (e.g. a moving mechanism), etc. CAD software solutions enable the design of products in a variety of unlimited industrial sectors, including aerospace, architecture, construction, consumer goods, high tech devices, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. Thus, the 3D modeled object reconstructed by the method may represent an industrial product which may be any machine part, such as a part of a land vehicle (including, for example, automobiles and light truck equipment, racing cars, motorcycles, trucks and motor equipment, trucks and buses, rail vehicles), a part of an air vehicle (including, for example, airframe equipment, aerospace equipment, propulsion equipment, defense equipment, aircraft equipment, space equipment), a part of a marine vehicle (including, for example, naval equipment, commercial vessels, offshore equipment, yachts and workboats, marine equipment), a general machine part (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, fixed installations, industrial equipment, fabricated metal products, tire manufacturing products), an electric machine or electronic component (including, for example, consumer electronics, security and / or control and / or instrumentation products, computer and communication equipment, semiconductors, medical devices and equipment), a consumer product (including, for example, furniture, household goods, leisure goods, fashion goods, hard goods retail products, soft goods retailer products), and packaging (including, for example, food and beverage and tobacco, beauty and personal care, household goods packaging).

[0049] FIG. 1 shows an example of a GUI for a system, the system being a CAD system.

[0050] The GUI 2100 may be a typical CAD-like interface with standard menu bars 2110, 2120 and bottom and side toolbars 2140, 2150. Such menu bars and toolbars include a set of user-selectable icons, each icon associated with one or more operations or functions, as known in the art. Some of these icons are associated with software tools adapted to edit and / or work with the 3D modeled object 2000 displayed in the GUI 2100. The software tools may be grouped into workbenches. Each workbench includes a subset of software tools. In particular, one of the workbenches is an editing workbench suitable for editing geometric features of the modeled product 2000. During operation, the designer may, for example, pre-select a part of the object 2000 and then edit the operations (e.g., change dimensions, color, etc.) or geometric constraints by selecting the appropriate icon. For example, a typical CAD operation is modeling a stamping or folding of the 3D modeled object displayed on the screen. The GUI may, for example, display data 2500 related to the displayed product 2000. In the illustrated example, the data 2500, displayed as a "feature tree", and their 3D representation 2000, relate to a brake assembly including a brake caliper and disc. The GUI may further show various types of graphic tools 2130, 2070, 2080 for facilitating, for example, 3D orientation of objects, for triggering a simulation of the operation of the edited product, or for rendering various attributes of the displayed product 2000. A cursor 2060 may be controlled by a haptic device to allow the user to interact with the graphic tools.

[0051] FIG. 2 shows an example of a system where the system is a client computer system, eg, a user's workstation.

[0052] The client computer in this example comprises a central processing unit (CPU) 1010 connected to an internal communications bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer is further provided with a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated in specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a haptic device 1090 such as a keyboard. A cursor control device is used in the client computer to allow the user to selectively position a cursor at any desired location on the display 1080. In addition, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generators for inputting control signals to the system. Typically, the cursor control device may be a mouse, with the buttons of the mouse being used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0053] A computer program may include instructions executable by a computer, the instructions including means for causing the device to carry out the method. The program may be recordable on any data storage medium including the memory of the system. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, for example an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the method may be executed by a programmable processor executing a program of instructions for performing the functions of the method by operating on input data and generating output. Thus, the processor is programmable and may be coupled to receive data and instructions from, and to transmit data and instructions to, the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high level procedural or object oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. Applying the program on the system will in any case provide instructions for carrying out the method.

[0054] The method may, for example, be part of a process for designing a 3D modeled object, including editing the reconstructed 3D modeled object. "Designing a 3D modeled object" specifies any action or series of actions that is at least part of a process of creating a 3D modeled object.

[0055] The method may be included in a manufacturing process, which may include generating a physical product corresponding to the modeled object after performing the method (e.g., after editing). In either case, the modeled object designed by the method may represent a manufacturing object. Thus, the modeled object may be a modeled solid (i.e., a modeled object representing a solid). The manufacturing object may be a product, such as a part, or an assembly of parts. Because the method improves the design of the modeled object, the method also improves the manufacturing of the product, thus increasing the productivity of the manufacturing process.

[0056] An implementation of the method in which the determined 3D modeled object is a control mesh of a subdivision surface will now be described with reference to figures 3 to 22.

[0057] Implementation of the method provides a new solution for generating a 3D modeled object (or 3D model) that matches multiple images of a given real object. The resulting model matches the contours of the object in the multiple images and can be a control mesh for the subdivision surfaces that can be easily processed by existing CAD and / or CAM software.

[0058] The control mesh of the subdivision surface is a synthetic representation of the 3D shape and therefore requires relatively little memory space and can be relatively easily manipulated for editing. Furthermore, such a control mesh may be refined to perform specific edits, e.g., to add details and / or to perform simulations requiring specific finite element sizes. The method may include any such subsequent use of the resulting control mesh. Furthermore, the method may use the control mesh format in an efficient manner to perform optimizations. Furthermore, the method may introduce specific energy terms to control the mesh.

[0059] Subdivision surfaces have been used in 3D software for a long time. The implementation of the present method can be aimed at a quad-dominant control mesh and a subdivision surface based on the Catmull-Clark subdivision scheme. The obtained control mesh and its prescribed subdivision surface can thus be implemented according to the teachings of the paper E. Catmull, J. Clark., Recursively generated b-spline surfaces on arbitrary topological meshes, Computer-Aided Design, 10(6) (pp. 350-355), 1978, which is incorporated herein by reference.

[0060] In this discussion of the implementation of the method, the terms "fit a control mesh" and "fit a subdivision surface" are equivalent and refer to the act of automatically moving the positions of the vertices of a control mesh so that its limiting surface (or the mesh after a sufficient number of subdivision steps) fits the target. Figure 3 shows the process of successively refining a simple control mesh until a limiting surface is reached.

[0061] The implementation of the method allows to obtain a control mesh that, when sufficiently subdivided, fits multiple views of an existing object. The input data includes the raw 2D image with the corresponding camera parameters and an initial dense mesh that roughly approximates the object (in the same reference system as the camera). The latter can for example be obtained by (roughly) segmenting the image, followed by using space carving techniques. This can be implemented according to the teachings of the paper KN Kutulakos, SM Seitz, A theory of shape by space carving, International journal of computer vision, 38(3) (pp. 199-218), 2000, which is incorporated herein by reference. In this paper we have shown how a mesh can be generated from multiple images of an object by space carving. This technique performs a rough identification of the contours of the object and generates an approximate dense mesh that cannot be used as is for editing in CAD and / or CAM software. However, it provides a suitable input for the implementation of the method.

[0062] Implementation of the method forms a new pipeline for generating subdivision surfaces (i.e., parametric shapes) that fit multiple images of the same object. Implementations of the method use subdivision surfaces because they provide a simple and general model for parametric shapes, allowing easy evaluation of the surface and its geometric properties. However, alternatively, other models, e.g., parametric strict boundary representations (used in CAD software), can be optimized as well.

[0063] With reference to Fig. 4, starting from a set of images 40 of an object with corresponding camera parameters, and a dense mesh 42 that roughly approximates the shape (such as the Theory of Shape by Space Carving paper cited above), an implementation of the method may create a simplified quad-dominant control mesh that, when refined, fits the object in the images provided according to the pipeline shown in Fig. 5. The more relevant the images are, i.e., the more they cover the object from all possible viewpoints, the more accurate the resulting 3D shape will be (i.e., the more they correspond to the real object). In an embodiment, the 2D images provided in the implementation of the method may thus cover the real object from different viewpoints.

[0064] The images may then be pre-processed to detect all isolines in the images, including but not limited to isolines corresponding to objects. This can be done, for example, as discussed above following the teachings of the paper "Holistically-nested edge detection". This paper develops a new edge and boundary detection algorithm that uses a deep learning model. An implementation of the method can use this solution to generate a probability map with all isolines in a provided set of images (whether they are isolines of a target object or not).

[0065] Alternatively, implementations of the method may simplify the dense mesh to an initial control mesh. This may be done, for example, as taught in the paper J. Wenzel, M. Tarini, D. Panozzo, and O. Sorkine-Hornung, Instant field-aligned meshes, ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA), 34 (6), 2015, which is incorporated herein by reference. This paper proposes a technique for simplifying the dense mesh to a uniform quad-dominant mesh. Implementations of the method may use this technique in steps of the proposed pipeline.

[0066] Since the two are quite far apart for our purposes, implementations of the method can use a scheme to fit this initial control mesh to a dense mesh (using the point cloud corresponding to its vertices). This can be done, for example, as taught in the papers V. Estellers, FR Schmidt, and D. Cremers, Compression for smooth shape analysis, arXiv preprint arXiv:1711.10824, 2017, and V. Estellers, F. Schmidt, D. Cremers, Robust Fitting of Subdivision Surfaces for Smooth Shape Analysis, International Conference on 3D Vision (3DV) (pp. 277-285), IEEE, 2018, which are incorporated herein by reference. These papers introduce a scheme to fit a control mesh to a target point cloud. This scheme first generates an estimated control mesh that very roughly fits the shape defined by the cloud, and then applies an iterative linear optimization scheme to adapt the control mesh to the target. Existing public prototypes are available and have been tested. The results are robust and satisfy the stated objective, which is the computational analysis of the shape. However, the resulting control mesh is not uniformly dense and is therefore difficult to use for further editing. More importantly, using this scheme to reconstruct 3D shapes from multiple images requires first generating a dense point cloud from the same images. This constitutes a separate and difficult problem that the implementation of the present method can avoid.

[0067] Since the provided dense mesh is only a rough approximation, the resulting refinement of the modified control mesh still does not fit the contours of the object in the various images. Implementations of the method use an optimization scheme to best fit the contours of the object in the images. Finally, implementations of the method may also add a post-processing step to further smooth regions of the resulting shape that were not fitted because no contours were found for that particular region in any of the provided images.

[0068] Details of these various steps of the implementation of the method shown in FIG. 5 are given below.

[0069] The implementation of the method fits rigid objects to multiple views, so the more relevant images are used, the better the result describes the real object. The implementation of the method does not require and can eliminate precise identification of the correct contour line in each image, but can process the raw images automatically. The implementation of the method can exclude manually selected constraint points. The implementation of the method can take into account realistic perspective projections, which means that the implementation of the method can use photos taken at any distance from the object. Since the camera parameters are provided by current shooting devices, they do not need to be calculated. Thus, the silhouette of the object may be calculated explicitly, rather than by minimizing energy, which results in higher computational robustness and better performance. The implementation of the method can use the approximation results (e.g., from older methods) as input to obtain a shape that fits the real object better and can be further edited with existing software. Also, the result of the implementation of the method requires significantly less memory space than any dense mesh (or point cloud) obtained by existing techniques. The implementation of the method can allow for the addition of a new energy term to regularize the resulting shape. Implementation of the method may allow for a post-processing step to further smooth regions where no contours were found in any of the images.

[0070] An example of the generation of a simplified quad-dominant control mesh has been discussed herein.

[0071] The subdivision surface that an implementation of the method fits to the view can be defined by a quad-dominant control mesh. The initial form of the unfitted control mesh can be provided as input to the method or can be generated from a provided dense mesh. For example, an implementation of the method can generate a simplified quad-dominant mesh from a provided dense mesh using the method introduced by the above-cited paper "Instant field-aligned meshes," although any other method capable of generating a simplified quad-dominant mesh would be equally satisfactory. It is important to note that the limiting surface of the resulting control mesh, as shown in FIG. 6, has no reason to be exactly close to either the provided dense mesh or the actual object.

[0072] Here we consider the example of fitting a control mesh to a dense input mesh.

[0073] This implementation of the method uses the scheme proposed in the previously cited paper "Compression for smooth shape analysis" to fit the generated control mesh as closely as possible to the input dense mesh, since this is the best initial guess of the real object. The results are shown in Figure 7.

[0074] However, implementations of the method can introduce some modifications to the scheme. For example, if the object point cloud is a relatively thin object with a cavity, such as a vase as depicted in FIG. 8, and the simple control mesh is a simple convex subdivision surface that envelops the object, implementations of the method can establish the correct initial correspondence between the two to obtain the cavity in the adapted subdivision surface. Implementations of the method can in particular avoid erroneous correspondences between regions with opposite orientations.

[0075] Thus, implementations of the method can add normal comparison in the point correspondence calculation, such that two points do not correspond if the dot product of their normal vectors is below a threshold, resulting in a point on the outer shell not being attracted by a target point on the inner shell of the object.

[0076] An implementation of the method may also add an optional weight calculation function based on an initial evaluation of the various energy terms. Weighting coefficients for the energy terms may be provided or calculated during the initial evaluation of the energy terms to ensure specific ratios between the four terms. These ratios may be either system default values ​​or user provided.

[0077] Implementations of the method may also add new energy terms using spring energy terms to attract the control points to their initial positions, which may be useful to keep them as close as possible to the initial control mesh, or at least a portion of it.

[0078] Finally, the implementation of this method can also replace the generation of a non-uniform control mesh with the method introduced by the previously cited paper "Instant field-aligned meshes", which has been tested in the present context to lead to better results.

[0079] An example of contour detection is now discussed.

[0080] An implementation of the method can use the solution presented in the previously cited paper "Holistically-nested edge detection" to generate an isoline probability map for each image, as shown in Figure 9. The pixels with the highest probability values ​​in the map indicate the location of the detected isolines. The detected isolines have different probabilities. For example, boundary isolines of internal reflection artifacts have a lower probability than hard contrast isolines that correspond to the actual boundaries of objects in the image. An implementation of the method can invert the probability map to obtain a fitting energy map (i.e., to calculate a map that corresponds to 1 minus the value of the probability map). Thus, the isoline pixels correspond to local minima in the fitting energy and are more suitable for minimization methods.

[0081] An implementation of the method can then calculate points that lie on the apparent isoline of the current shape of the subdivision surface (by subdividing the control mesh). These points can be projected onto the fitting energy map using the camera parameters. The lower the fitting energy value at a projected point, the more likely the point is on the actual isoline.

[0082] Two observations can be made.

[0083] First, if the projected isoline of the current shape (i.e., isoline 102 in FIG. 10) is far from the target isoline of the object in the image, it will be more difficult to converge towards the target isoline (i.e., isoline 104 in FIG. 10). This is because the example isoline detection does not eliminate all unnecessary isolines from the fitting energy map, and some of those isolines may attract the shape during the optimal isoline fitting phase.

[0084] However, since the implementation of this method starts from an approximation of the desired shape, the initial apparent contours are generally not very far from the contours of the real object in the image (see Fig. 11) in the same reference frame in which the camera parameters were defined, i.e., the probability of converging towards the correct contours is much higher.

[0085] Second, even when starting from point projections that are near the object isoline, the local gradient of the fitting energy map can be very flat, effectively keeping the optimization converging towards the object isoline (see Fig. 12).

[0086] Since the technique of the previously cited paper "Holistically-nested edge detection" aims to generate (e.g., almost) a binary image of clearly delineated isolines as any other contour detection algorithm, locally flat gradients are essentially a feature of such techniques. Deep learning techniques such as those described in this paper are statistical in nature and therefore obtain non-binary probability values ​​that are very close to 0 or 1 (and thus still contain flat gradient regions). This is especially true for high-resolution (e.g., 4K) images such as those obtained by current imaging devices. To avoid this type of situation, implementations of the method can apply a local filter to the fitted energy map to expand the regions where the gradient is non-zero (see FIG. 13).

[0087] The smoothing filter may, for example, include at least one application of a Gaussian blur (applied to the contour probability map), which effects a transformation of the contour map from the shape of FIG. 12, characterized by flat regions of zero gradient, to the shape of FIG. 13, which allows for better convergence of the optimization.

[0088] A simple, single Gaussian blur reduces the amplitude. To avoid loss of minima in the isoline image, the method may instead consider (e.g., apply) smoothing that involves determining the envelope of the application of a family of Gaussian blurs to the isoline probability map, each Gaussian blur having a different kernel size.

[0089] For example, an implementation of the method may consider the envelope of the initial fitted energy map and all its Gaussian blurs (see FIG. 14).

[0090] The method is carried out by:

number

number

number

number

number

[0091] The smoothed fitting energy map is given by the function

number

[0092] An implementation of the method may apply an approximation of this envelope by iterating successive Gaussian blurs with increasing kernel sizes and combining them with the previous image by keeping the minimum value for each pixel. For example,

number

number

number

number

[0093] Any other filter that produces a similar map will work equally well and may therefore be applied, for example, by a method using a filter that convolves the image with a kernel taken from the Laplace distribution, resulting in something very similar (see FIG. 15):

number

[0094] FIG. 16 shows a close-up of the contour map before and after applying this type of filter.

[0095] In conclusion, our implementation can work with fitting energy maps that have non-zero gradients, which locally decrease in a monotonically manner towards a minimum, thus increasing the convergence speed. However, realization of our method may not require more than a minimum of C0 continuity.

[0096] As mentioned earlier, another approach might have been to use the distance transform of the image to generate a map containing the distance to the nearest isoline at each pixel. However, this requires a threshold to identify isolines with probability values ​​above the threshold and then to calculate the distance transform of the image. Whereas this approach would result in the difference between the probabilities of two isolines that are both within the threshold being lost, the implementation of the present invention allows the results of the isoline detection algorithm to be exploited in their entirety.

[0097] Next, an example of fitting to a contour line will be described.

[0098] An implementation of the method can fit the quad-dominant control mesh discussed above to all provided images by minimizing the fitting energy between the silhouette of the subdivision surface and the isolines in the isoline maps (one for each provided image). Rather than using the distance between the silhouette and isolines in each image, the fitting energy may use the fitting energy map discussed above calculated for each image. At each iteration, the process first calculates the silhouette for the current constraining surface, and then deforms the surface by minimizing the fitting energy for all images (see FIG. 17).

[0099] Since the camera parameters are already known, the implementation of the method may separate the calculation of the silhouette points on the 3D surface, which can be done explicitly and does not require optimization. This is in contrast to methods that consider the (simplified) camera parameters and the silhouette points on the 3D surface as unknown variables and perform a more complex optimization, whose energy function is defined in a variable space of rather high dimension, possibly leading to longer calculation times and less robust calculation results. In particular, the size of the variable space in such methods grows with the number of images provided, thus ultimately limiting the number of images that can be used. In contrast, the optimization of the implementation of the method is performed in a variable space whose dimension depends only on the number of vertices in the control mesh and is independent of the number of images provided, which makes it easier to use many images and therefore allows for a better description of the object.

[0100] The method may include, at each iteration of the optimization, providing a current control mesh, refining the current control mesh, performing a loop on each 2D image, and modifying the current control mesh to reduce the energy. The loop includes, for each 2D image, projecting the refined mesh onto the 2D image, calculating a 2D silhouette of the 3D modeled object in the 2D image, and identifying the points of the refined mesh that correspond to the 2D silhouette as silhouette vertices, as will be described in more detail below. Optionally, the topology of the refined mesh after the first iteration can be stored to be reused in the next iteration of the subdivision. This allows time savings.

[0101] Thus, the determination of the 3D silhouette vertices may be performed again at each iteration of the optimization. The topology of the mesh (connectivity between vertices, number of edges, number of polygons) does not change. The topology of the overall shape in the sense of the Euler property does not change either. Therefore, the subdivision, in the sense of determining the connectivity between the divided vertices, may be performed only once. On the other hand, as the coordinates of the vertices of the control mesh change, the method may recalculate the coordinates of the vertices of the subdivided mesh (approaching the limit surface) after each iteration. The calculation of the 3D silhouette vertices may be performed later.

[0102] A set of silhouette points can be computed for each image by using the camera parameters, computing a depth map of the current shape of the subdivision surface, and keeping only those points whose image are boundary pixels of the projection of the object (see FIG. 18: depth map and boundary pixels 180 on the left, subdivision mesh and silhouette points 182 on the right). In case several points project onto the same pixel, the ones with the same depth as the depth value at the pixel may be used, i.e. the implementation of the method can avoid using hidden points in the silhouette.

[0103] An implementation of the method can identify the location of the silhouette on the current shape of the subdivision surface for its projection in each image i. An implementation of the method can be to refine the control mesh n times a sufficient number of times to produce a mesh R n The subdivision surface can be approximated by subdividing the surface into n, where n is a parameter defined at the beginning of the process. An implementation of the method may subdivide three times, where n is a parameter defined at the beginning of the process. Thus, an implementation of the method may approximate the surface of vertices R projected onto the silhouette of the shape. n A subset of Σ i You can search for R n is sufficiently subdivided that these vertices give a good approximation of the silhouette's actual location on the subdivision surface.

[0104] Projection transformation π of image i i Using this, the implementation of the method obtains a depth map D of the scene that contains only the current shape of the subdivision surface.i The implementation of this method is that for any silhouette pixel p s ∈S i is the background pixel p bg D, adjacent to i Silhouette pixel S i Background pixels are identified as having a depth value D that is set to zero in the depth map. i (p bg ) Implementations of the method may expand a silhouette pixel by including pixels that are not directly adjacent to the background pixel, but that reside in a neighborhood (the size of which may also be defined) that includes the background pixel.

[0105] After identifying all silhouette points, the implementation of the method is n For each vertex v in, its projection π i (v) is silhouette pixel p s In this case, v is a subset of the silhouette vertices of Rn. i If several vertices are projected onto the same pixel, only the vertices with the same depth as the calculated pixel depth are projected onto the same pixel. i is held in

[0106] The implementation of the method is as follows: for each silhouette vertex v ∈ Σ i (Here E i (π i (v)) where E i is the fitted energy map computed for image i.

[0107] E i (π i When (v)) reaches a minimum value, the silhouette vertices are projected onto the isoline identified in the image.

[0108] If all fitting energy values ​​reach a minimum (globally) for all silhouette vertices and all images, then the projected silhouette of the subdivision surface is identical to the identified silhouette of the object in the image (because the implementation of the method starts from a shape that is already close to the object).

[0109] To verify how well the provided images cover the surface morphology, a visual verification was performed by coloring all vertices that appear as silhouette points in at least one image (see Figure 19: grey areas appear as silhouettes in at least one image, white areas are not part of any silhouette).

[0110] Each silhouette point can be assigned a value from its fitting energy map. The sum of the squares of these values ​​may define the main fitting energy term to be minimized. Since the silhouette points are computed on the limiting surfaces of the subdivision (or on the mesh resulting from at least some of the subdivision steps), they can be expressed as linear functions of the vertex coordinates of the control mesh, i.e., the variables of the problem to be solved.

[0111] π i The projective transformation of image i (based on the provided camera parameters). x The vertex coordinates of the control mesh.

number

number

[0112] The fitting energy term is

number

[0113] Since the 3D modeled object is a control mesh (of subdivision surfaces), certain terms can be added. In particular, energy can be rewarded for a low sum of the squares of all edge lengths in the control mesh, a low sum of the squares of the principal curvature values ​​at densely sampled points of the limiting faces of the control mesh, and / or for regularity of the faces of the control mesh. This allows to obtain industrially clean control meshes, which perform more accurately in simulations and are easier to manipulate.

[0114] This can correspond to three different regularization terms that are included in the total energy to be minimized:

[0115]

number

[0116]

number

[0117]

number

number

number

[0118] Finally, implementations of the method can also introduce an optional energy term that measures the overall variation in curvature (the sum of the squared partial derivatives of the principal curvatures can be used):

number

number

[0119] E fit is a discrete map function E i Since we use i We generate maps that approximate the gradients of E fit The Jacobian for the other energy terms can be approximated numerically.

[0120] The implementation of this method is x min A numerical optimization solver (e.g., Levenberg-Marquardt) can be used to determine

[0121] Because the implementation of the method does not have an explicit representation of the silhouette, it only needs to compute a single minimization step per iteration and then uses the resulting vertex coordinates to update the control mesh. If the energy is small enough, the process is stopped, resulting in an updated control mesh; otherwise, the process is repeated again (see FIG. 17).

[0122] As a result, based on the resulting control mesh, the silhouette of the subdivision surface is best fitted to the contours seen in the provided image (see FIG. 21).

[0123] A further advantage of implementing the method is that areas not covered by the silhouette can usually be smoothed or flattened. For example, the flat base of an object sitting or lying on a table may not be well covered by the image since such objects are usually photographed from above. Because of this, carving may generate uneven, possibly sharp, areas that may, however, be flattened by the method during the contour fitting phase (see FIG. 22).

[0124] If more information is available, e.g., planar regions, symmetry planes, etc., it can be taken into account in the optimization scheme, i.e., by constraining the positions of (some) control vertices, or by directly reducing the number of variables. In the case of an identified symmetry plane, the controller vertices on one side of the symmetry plane can be constrained to be mirror images of the identified controller vertices on the other side of the symmetry plane, thus reducing the number of controller vertices by about 50%. Control vertices in an identified planar region can be defined by two parameters on the plane instead of three spatial coordinates. The same can be done for identified spherical or cylindrical regions of the shape.

[0125] Next, an example of optional smoothing of the unfitted regions is given.

[0126] As mentioned earlier, not all regions of the resulting fitted subdivision surface are visible in silhouette and therefore not all regions are fitted, so the surface may not be locally smooth in these regions.

[0127] These examples are for a set of constraints C fix While maintaining the above, the term E edges , E curvature , E quad , E curvature variation One can try to smooth these regions by performing a post-processing step that consists in minimizing the energy constructed from

[0128] The first four terms have been discussed above. fix represents a constraint that keeps each surface point on the silhouette in its original position, that is, the position it reached before the optional smoothing. This effectively constrains the formation to stick to the original shape in these regions.

[0129] Four smoothing terms E edges , E curvature , E quad , E curvature variation All of the above use unequal weights to reduce their influence on regions that appear within the silhouette and to increase their influence in regions not covered by the silhouette. While the task of determining how to weight the different terms relative to one another is not trivial, it has been observed that by using these examples of implementation of the method, any remaining bumps in such regions have been further smoothed out.

Claims

1. 1. A computer-implemented method for 3D reconstruction, comprising: providing a number of 2D images, all representing the same fixed real object, and camera parameters defining a perspective projection for each 2D image; providing, for each 2D image, a smoothness map having pixel values ​​each representing a measure of isoline presence; determining a 3D modeled object representing a real object, said 3D modeled object being a control mesh of a subdivision surface, said control mesh being a mesh defined from an initial dense mesh that approximates said real object, and for each smoothness map iteratively optimizing an energy that rewards projections of silhouette vertices of the 3D modeled object having pixel values ​​that represent a high measure of contour presence; Including, At each iteration, providing a current first mesh that is a control mesh; providing a current second mesh obtained by subdividing the current first mesh; For each 2D image, projecting the current second mesh onto a 2D image; computing a 2D silhouette of a 3D modeled object in said 2D image; identifying points of the current second mesh that correspond to the 2D silhouette as silhouette vertices; modifying the current first mesh to reduce its energy; Also includes A computer-implemented method.

2. the smooth map includes local extrema corresponding to isoline probability maxima; The energy rewards high contour probability in the projection of the silhouette vertices. The method of claim 1.

3. The smooth map is a function of the contour probability map The method of claim 2.

4. The function is a smoothing of the affine mapping The method according to claim 3.

5. The smoothing includes applying at least one of a Gaussian blur to the contour probability map, or convolving the contour probability map with a kernel derived from a Laplace distribution. The method according to claim 4.

6. the smoothing includes determining an envelope of application of a family of Gaussian blurs to the contour probability map; Each Gaussian blur has a different kernel size The method according to claim 5.

7. The energy comprises terms of the following type: [0010] 7. The method according to any one of claims 1 to 6.

8. The energy is The low sum of the squared lengths of all edges of the control mesh, a low sum of the squares of the principal curvature values ​​at densely sampled points of the limiting surface of the control mesh; and / or Control mesh face regularity Give more rewards to The method of claim 1.

9. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having the computer program according to claim 9 recorded thereon.

11. A computer program product according to claim 9, comprising: a processor coupled to a memory; and a graphical user interface, the memory having recorded thereon a computer program product according to claim 9. system.

Images