3D reconstruction using smooth maps

Abstract

The invention relates in particular to a computer-implemented method for 3D reconstruction. The method comprises providing 2D images, and for each 2D image, providing camera parameters defining a perspective projection. The 2D images all represent the same real object. The real object is stationary. The method further comprises providing, for each 2D image, a smooth map. The smooth map has pixel values, each pixel value representing a measure of contour presence. The method further comprises determining a 3D modeled object representing the real object. The determination iteratively optimizes an energy. For each smooth map, the energy rewards a projection of silhouette vertices of the 3D modeled object having pixel values representing a high amount of contour presence. This forms an improved solution for 3D reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer programs and systems, and in particular to a method, system and program for 3D reconstruction. BACKGROUND

[0002] Many systems and programs for the design, engineering and manufacturing of objects are available on the market. CAD is the acronym for Computer Aided Design, for example it relates to software solutions for designing objects. CAE is the acronym for Computer Aided Engineering, for example it relates to software solutions for simulating the physical behavior of future products. CAM is the acronym for Computer Aided Manufacturing, for example it relates to software solutions for defining manufacturing processes and operations. In such computer aided design systems, the graphical user interface plays an important role in terms of technical efficiency. These technologies can be embedded in Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps enterprises share product data, apply common processes, and leverage business knowledge to drive extended enterprises to optimize product creation from conception to the cradle to the grave. The PLM solutions offered by Dassault Systemes (under the trademarks CATIA, ENOVIA and DELMIA) provide an engineering hub that organizes product engineering knowledge, a manufacturing hub that manages manufacturing engineering knowledge and an enterprise hub that integrates and connects the engineering and manufacturing hubs of extended enterprises. The overall system provides an open object model that connects products, processes, resources and people to drive dynamic, knowledge-based product creation and decision-making to drive optimized product definition, manufacturing preparation, production and service.

[0003] In this context and others, 3D reconstruction is becoming increasingly important. 3D reconstruction is related to computer vision techniques that can determine a 3D modeled object based on data representing a real object or a real scene. Such techniques can be useful for fields such as virtual reality and augmented reality (more generally, any kind of immersive experience), video games, manufacturing and 3D printing and / or 3D modeling.

[0004] Despite the significant research work that has been carried out in this field over the past few years, there is still a need for improved solutions for 3D reconstruction. SUMMARY

[0005] Thus, a computer-implemented method for 3D reconstruction is provided. The method comprises providing 2D images, and for each 2D image, providing camera parameters defining a perspective projection. The 2D images all represent the same real object. The real object is stationary. The method further comprises providing, for each 2D image, a smooth map. The smooth map has pixel values, each pixel value representing a measure of contour presence. The method further comprises determining a 3D modeled object representing the real object. The determination iteratively optimizes an energy. For each smooth map, the energy rewards a projection of silhouette vertices of the 3D modeled object having pixel values representing a high amount of contour presence.

[0006] The method can comprise one or more of:

[0007] - the smooth map comprises local extrema corresponding to maxima of contour probability, the energy rewarding high contour probability at the projection of the silhouette vertices.

[0008] - the smooth map is a function of a contour probability map;

[0009] - the function is a smoothing of an affine mapping;

[0010] - the smoothing comprises applying at least a Gaussian blur to the contour probability map, or convolving the contour probability map with a kernel obtained from a Laplacian distribution;

[0011] - the smoothing comprises determining an envelope of applying a family of Gaussian blurs to the contour probability map, each Gaussian blur having a different kernel size;

[0012] - the energy comprises a term of the type: fit =∑ i,j E i (π i (s ij )) 2 where, for a 2D image i: E i (·) denotes the smooth map, s ij denotes a silhouette point j, and π i (·) denotes a projection transform.

[0013] - the 3D modeled object is a control mesh of a subdivision surface;

[0014] - the method further comprises, at each iteration of the optimization:

[0015] o providing a current control mesh;

[0016] o subdividing the current control mesh into a subdivision mesh;

[0017] o for each 2D image:

[0018] ■projecting the subdivision mesh onto the 2D image;

[0019] ■computing a 2D silhouette of the 3D modeled object in the 2D image; and

[0020] ■identifying points of the subdivision mesh corresponding to the 2D silhouette as silhouette vertices; and

[0021] o modifying the current control mesh to reduce the energy; and / or

[0022] the low degree of the square sum of the lengths of all edges in the control mesh, the low degree of the square sum of the principal curvatures values at the dense sampling points of the limit surface of the control mesh, and / or the regularity of the faces of the control mesh.

[0023] There is also provided a computer program comprising instructions for carrying out the method.

[0024] There is also provided a computer readable storage medium having recorded thereon a computer program.

[0025] There is also provided a system comprising a processor coupled to a memory and a graphical user interface, the memory having recorded thereon a computer program. BRIEF DESCRIPTION OF DRAWINGS

[0026] Embodiments of the application will now be described, by way of non-limiting examples, and with reference to the accompanying drawings, in which:

[0027] Figure 1 An example of a graphical user interface of the system is shown;

[0028] Figure 2 An example of the system is shown; and

[0029] Figures 3-22 The method is shown. DETAILED DESCRIPTION

[0030] There is provided a computer-implemented method for 3D reconstruction. The method comprises providing 2D images, and for each 2D image, providing camera parameters defining a perspective projection. The 2D images all represent the same real object. The real object is stationary. The method further comprises, for each 2D image, providing a smooth map. The smooth map has pixel values, each pixel value representing a measure of contour presence. The method further comprises determining a 3D modeled object representing the real object. The determination iteratively optimizes an energy. For each smooth map, the energy rewards a projection of a silhouette vertex of the 3D modeled object having a pixel value representing a high amount of contour presence.

[0031] This forms an improved solution for 3D reconstruction.

[0032] The method is in the range of structure-from-motion analysis techniques. In fact, the method allows reconstructing a real object in 3D based on 2D images. 2D images are relatively simple and can be obtained without any cost compared to other types of signals. The method thus forms a relatively simple and costless 3D reconstruction technique.

[0033] Further, the optimization is iteratively optimized. The iteratively optimization technique is a practical method to solve optimization problems. The method is thus relatively efficiently performed.

[0034] Now, the method particularly relies on a map having pixel values, each pixel value representing a measure of the presence of a silhouette. Since the 3D modeled object optimizes an energy for each silhouette map rewarding the projection of the silhouette vertex of the 3D modeled object having a pixel value representing a high amount of the presence of the silhouette, the method allows determining the 3D modeled object maximizing the projection of the silhouette on the contour in the 2D image. Thus, the method is based on the observation that the silhouette of a real object represented by a 2D image translates into a contour in the 2D image. As a result, the 3D reconstruction allowed by the method is relatively accurate.

[0035] Further, the map provided in the method is particularly a smooth map. As a result, the optimization is enabled to follow a path in the iteration process leading to a local extremum of the map, thereby robustly converging to the optimal 3D modeled object. This leads to a relatively high accuracy and robustness of the method. The iteratively optimization can indeed be based on gradients, and for example comprises a computation of the gradient of the smooth map to follow such a gradient.

[0036] The 2D images each represent a physical signal forming a 2D graphical representation of a real object, for example arranged in a grid of pixels. The grid can be regular and / or rectangular all over. The 2D images can comprise RGB images and / or grayscale images. The determination can be performed for example based on RGB images and / or grayscale images only. The 2D images can comprise images captured by one or more physical cameras, each camera having one or more respective sensors, for example photographs. Providing the 2D images can comprise capturing at least a part of the 2D images, and / or receiving and / or retrieving at least (another) part of the 2D images from a remote system and / or in a memory.

[0037] 2D images all represent the same real object. For example, the 2D images can have been captured at different times, each image using the same and unique camera to view the same and unique real object, e.g. using different camera parameters (e.g. using different camera extrinsic parameters, including from different angles, and / or having different camera intrinsic parameters). At least a portion of the 2D images can represent the real object from different viewpoints. Thus, the method combines several pieces of 2D graphical information about the real object (i.e. the 2D images) to establish a 3D model of the real object. In an example, the 2D images can form different frames of the same video sequence of the real object.

[0038] The method provides camera parameters to the determination. In other words, the camera parameters are known prior to the determination (i.e. the provision of the camera parameters can be performed prior to the determination). The camera parameters can for example be known prior to performing the method, or the method can comprise calculating such camera parameters individually (e.g. approximately) prior to the determination, e.g. according to any known algorithm. The method is thus relatively efficient, as the determination does not need to calculate said camera parameters within the optimization. Rather, the determination can operate the reward (i.e. reward by the optimization energy having a projection of the silhouette vertices representing the amount of height present for the contour of each smooth map) using the provided camera parameters as such. The iterative optimization can indeed comprise iteratively determining the silhouette vertices corresponding to a given 2D image based on the camera parameters, and projecting the silhouette vertices onto the 2D image according to the camera parameters. According to any known technique (outside the scope of the present discussion), the camera parameters of each 2D image can have been associated / attached to the image at the time of capturing the image.

[0039] The camera parameters for each 2D image define a respective perspective projection according to which the (fixed) real object can be viewed in the 2D image. This simplifies the method, as cameras typically capture 2D images according to such angles. This also allows to properly take into account the perspective, rather than approximating the perspective, which can introduce errors in the process, especially for objects taken at close distance, whose perspective is thus more apparent, if using a parallel projection for example.

[0040] The real object is fixed in the 2D images. In other words, the real object is immovable in all 2D images input to the determination. This means that the real object does not move and deform at least substantially (i.e. the deformation and movement are negligible with respect to the size of the real object). As a result, the determination can be relatively easy to implement, and the optimization can relatively quickly and robustly converge. The fixed real object can be more easily and quickly combined information provided by different images, as these can be directly compared. Thus, the method is situated in an industrial application which allows 3D reconstruction of stationary objects, e.g. non-living objects, e.g. industrial products.

[0041] In an example, the method can comprise a user-machine interaction, in which a user captures 2D images with a camera by moving the camera around a fixed real object, e.g. keeping a free distance from the real object. The camera can be a standard camera and capture video frames according to a perspective projection. The method thus forms a simple motion structure analysis.

[0042] Each 2D image is now discussed to provide a corresponding smooth map.

[0043] Each smooth map comprises pixels in a grid arrangement, e.g. identical to the 2D image. Each pixel of each smooth map has a value, which represents a measure of the presence of a contour in the corresponding pixel of the corresponding 2D image. In other words, each smooth map pixel value measures the local presence of an image contour.

[0044] A "contour" refers to any location of a 2D image, which corresponds to a (e.g. as sharp as possible) separation between two different textures of the 2D image, each texture corresponding to a different real-world material. The set of contours thus (e.g. approximately) comprises the projected boundary of a real-world object or visible silhouette, or the boundary of a visible silhouette of an (identifiable) part of a real-world object. This concept is known per se in the field of image processing, in particular applied to 3D reconstruction. It is known that a measure of the presence of a contour can be a function of the gradient of the pixel value of a 3D image. There are many different techniques in the prior art for obtaining such a measure.

[0045] A "measure" refers to any information about the presence of a contour at a location corresponding to a pixel value. The values of different pixels of a smooth map can be compared together, so that the presence of a contour can be compared at different corresponding locations. The pixel values can monotonically correspond to the presence of a contour (gradually decreasing respectively increasing, i.e. higher respectively lower pixel values indicate the presence of a contour, and not merely lower respectively higher pixel values, so that a relatively higher respectively lower measure of the presence of a contour is indicated).

[0046] "Smooth" refers to a smooth map having a non-zero gradient at least in the vicinity (i.e. neighbourhood) of an image contour (i.e. everywhere, or alternatively, in such a vicinity), which locally varies monotonically to a local optimum, and which has C0 continuity (i.e. e.g. in an optimal state, it is sufficient to have only point continuity, without derivative continuity).

[0047] "Rewarding the 3D modeled object for the projection of silhouette vertices having pixel values of the smooth map representative of a high amount of presence of the contour", means that the optimization favors the 3D modeled object for which the projection of silhouette vertices (called "projection of silhouette vertices") on the 2D points of each 2D image corresponding to the pixels of the smooth map having values more representative of the presence of the contour. The optimization can perform this reward by minimizing an objective function comprising a cost term penalizing the absence of contour, i.e. a low amount of presence of the contour. The optimization can minimize said cost term, and thus maximize the global amount of presence of the contour by the projection of silhouette vertices, under all other conditions being equal.

[0048] Since each 2D image is provided with a measure as a smooth map, the optimization can iteratively follow a monotonic gradient path of each smooth map to converge toward a local optimum corresponding to a local maximum of presence of the contour at a relatively high speed.

[0049] Each smooth map can for example comprise a local extremum, each local extremum corresponding to a respective local maximum of the contour probability (i.e. the contour probability is locally maximal at the local extremum of the smooth map). In this case, the energy can simply perform the above-mentioned reward: the energy can reward a high contour probability at the projection of silhouette vertices. Said smooth map can in particular vary from a value representative of a zero contour probability to said local extremum. Since of its smoothness, the gradient of the smooth map is not zero even at the pixel corresponding to the zero contour probability, and the gradient-based iterative optimization can still successfully push toward the local extremum by the reward.

[0050] The smooth map can for example be a function of the contour probability map. This means that the smooth map is computed from the contour probability map, the smooth map inheriting the information contained in it. In particular, the function preserves the position and value of the extrema (i.e. the extrema of the smooth map are identical in position and value - e.g. modulus of an affine mapping, e.g. 1-p inverse, as described later - to the extrema of the contour probability map), the function leads to non-zero gradients, monotonic variations toward the local optimum, and C0 continuity. The method can comprise providing a contour probability map for each 2D image, and applying the function to each contour probability map (in order to output a corresponding smooth map). This allows to determine a relatively accurate 3D modeled object because the optimization relies entirely on the relatively fine information provided by the contour probability map. The function can be a monotonic function.

[0051] It is known that a contour probability map is a map with pixel values, each of which represents the probability of the presence of a contour. Such a map (a contour map is a map whose pixel values each represent the presence or absence of a binary contour) can be computed in any known manner, for example based on a machine learning scheme implemented on an annotated dataset of 2D images associated with the contour probability map or contour map. Such a machine learning can be carried out for example according to the teachings of the paper by S. Xie, Z. Tu, Holistically-nested edge detection, in the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pages 1395-1403, or similarly to it, which is incorporated herein by reference. The edge detection method described herein is based on the learning of a network that predicts the edges of an image by supervised training on a dataset of image data in which the edges are annotated. The network extracts multiscale predictions, and the classification loss for each pixel will compare the cross-entropy between the binary mask ground truth of the edges and the edge prediction at each scale. The predictions at the various scales are then pooled to arrive at a final prediction.

[0052] Alternatively, the method can compute the contour probability map according to the teachings of any one of a number of other papers on machine learning that predict edges in a supervised manner, the differences between which are mainly expressed in terms of architecture and loss, for example the following papers, all of which are incorporated herein by reference:

[0053] - DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection, Gedas Bertasius, Jianbo Shi, Lorenzo Torresani, CVPR 2015;

[0054] - Object Contour Detection with a Fully Convolutional Encoder-Decoder Network, Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, Ming-Hsuan Yang, CVPR 2016;

[0055] or

[0056] - Pixel-Wise Deep Learning for Contour Detection, Jyh-Jing Hwang, Tyng-Luh Liu, ICLR Workshop 2015.

[0057] Alternatively, the silhouette probability map can be computed by a deterministic method. Such deterministic methods are well known and the method may, for example, comprise computing a chamfer map of the Canny edge detection result.

[0058] The function can be a smooth (result of) a monotonic (e.g. affine) mapping. The smooth is a function as described above, i.e. the smooth preserves the location and value of the extrema, and the smooth results in non-zero gradients, monotonic variation towards the local optima and C0 continuity. As a result of this smooth in combination with the monotonic (e.g. affine) mapping, the information of the initial probability map is maintained, in particular allowing a comparison of the information of the presence of a silhouette between different pixels. The affine mapping (i.e. function) can be a probability complement mapping (i.e. 1 minus the probability). Thus, the smooth map can form a smooth inverted silhouette probability map, having pixel values each representing the probability of the absence of a silhouette after smoothing (i.e. the complement of the silhouette probability). This improves the convergence speed. In this case, the optimization problem is a minimization problem. Alternatively, the monotonic mapping can be the identity. In this case, the optimization problem is a maximization problem.

[0059] Any smooth map herein may, for example, be resulting from a scheme comprising computing a silhouette probability map (e.g. as described above), applying a monotonic (e.g. affine) mapping to the silhouette probability map (e.g. can be a probability complement mapping), or alternatively, simply the identity mapping), and smoothing the mapping result (in any known way, e.g. as described later). The method can comprise performing the scheme on at least a part (e.g. all) of said smooth maps, and / or receiving and / or acquiring at least a part of said smooth maps as such.

[0060] The method can alternatively use a distance transform of the image in order to generate a map containing the distance to the nearest silhouette at each pixel. However, this method requires a threshold to identify the silhouettes with a probability value above the threshold, and then a distance transform of the image is computed. This means that the probability difference of two silhouettes both within the threshold is lost, and the silhouette probability information is not fully exploited.

[0061] As described above, the determination process comprises an iterative optimization scheme, and can be a gradient-based optimization scheme, iteratively computing the gradient of the smooth map of each silhouette vertex projection in order to converge to a 3D modeled object that realizes an overall optimum value over the whole smooth map (thus representing a global maximum of the silhouette probability).

[0062] The optimization can comprise, at each iteration, providing a "current" 3D modeling object, determining the projection of the silhouette vertices, and modifying the 3D modeling object to reduce the energy (e.g. including computing the gradient of the smooth map at the pixels corresponding to the pixels of the projection). The modified 3D modeling object can be input to the next iteration as the "current" 3D modeling object of said next iteration until a convergence criterion is reached.

[0063] The iterative optimization can start from a 3D modeling object that forms an approximation of the best 3D modeling object sought by the method. This initialization of the optimization using an approximate 3D model of the real object allows a fast and robust convergence of the optimization. It can also be such that only a "locally smooth" "smooth map" is satisfied, i.e. as long as locally, where the projection of the initial shape lies until the pixel of maximum silhouette probability, the gradient is "good", there can be regions of zero gradient elsewhere. Thus, as long as the silhouette projection of the initial shape lies in these regions, the smoothness requirement is only located near the real silhouette. Such an approximation can be obtained in any known way, e.g. by performing existing 3D reconstruction techniques. Examples will be discussed later.

[0064] The method is computer-implemented. This means that the steps (or substantially all the steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, possibly fully automatically, possibly semi-automatically. In examples, the triggering of at least some steps of the method can be performed by user-computer interaction. The level of user-computer interaction required can depend on the level of automation foreseen and balance the need to implement the user's will. In examples, the level can be user-defined and / or pre-defined.

[0065] A typical example of computer implementation of the method is the execution of the method using a system adapted for this purpose. The system can comprise a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program comprising instructions for executing the method. The memory can also store a database. The memory is any hardware suitable for such storage, possibly comprising several physically distinct parts (e.g. one for the program and possibly one for the database).

[0066] The method generally manipulates modeling objects. A modeling object is any object defined by data stored for example in a database. By extension, the expression "modeling object" designates the data itself. Depending on the type of system, a modeling object can be defined by different kinds of data. The system can actually be any combination of a CAD system, a CAE system, a CAM system, a PDM system and / or a PLM system. In those different systems, a modeling object is defined by the corresponding data. Thus, one can speak of CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, CAE data. However, those systems are not mutually exclusive, since a modeling object can be defined by data corresponding to any combination of those systems. Thus, a system can well be both a CAD system and a PLM system.

[0067] By CAD system, it is additionally meant any system adapted at least to design a modeling object based on a graphical representation of the modeling object, such as CATIA. In this case, the data defining the modeling object comprises data allowing to represent the modeling object. The CAD system can provide a representation of the CAD modeling object for example using edges or lines, in some cases with faces or surfaces. Lines, edges or surfaces can be represented in various ways, for example Non-Uniform Rational B-Splines (NURBS). In particular, a CAD file contains specifications from which a geometry can be generated, so that a representation form can be generated. The specifications of a modeling object can be stored in a single CAD file or in several CAD files. The typical size of a file representing a modeling object in a CAD system is in the range of 1 MB per part. A modeling object can generally be an assembly of several thousands of parts.

[0068] In the context of CAD, a modeling object can generally be a 3D modeling object, for example representing a product, for example a part or an assembly of parts, or possibly a combination of products. A "3D modeling object" designates any object modeled by data allowing its 3D representation. A 3D representation allows to view a part from various angles. For example, when represented in 3D, a 3D modeling object can be manipulated and rotated around any of its axes or around any of the axes in the screen displaying the representation. In particular, this does not include 2D icons that are not 3D modeled. The display of a 3D representation facilitates the design (i.e. increases the speed at which a designer statistically completes a task). Since the design of a product is part of the manufacturing process, the manufacturing process of the industry can be accelerated.

[0069] A CAM system also means any solution, hardware software, suitable for managing product manufacturing data. Manufacturing data generally includes data related to the product to be manufactured, the manufacturing process and the resources needed. CAM solutions are used to plan and optimize the entire manufacturing process of a product. For example, it can provide the CAM user with information on the feasibility, the duration of the manufacturing process or the number of resources (such as a specific robot) that can be used at a specific step of the manufacturing process; it is thus possible to decide on the management or the investment needed. CAM is a follow-up process after the CAD process and potentially the CAE process. Such CAM solutions are provided by Dassault Systèmes under the trademark CATIA®.

[0070] A CAE system also means any solution, hardware software, suitable for analyzing the physical behavior of a modeled object. The finite element method (FEM) is a well-known and widely used CAE technique that generally includes the decomposition of a modeled object into elements that can be computed and simulated through equations for physical behavior. Such CAE solutions are provided by Dassault Systèmes under the trademark CATIA®. Another evolving CAE technique involves the modeling and analysis of complex systems composed of multiple components from different physical domains without CAD geometry data. CAE solutions can perform simulations that can be optimized, improved and validated for the product to be manufactured. Such CAE solutions are provided by Dassault Systèmes under the trademark CATIA®.

[0071] The 3D modeling object reconstructed by the method can represent the geometry of a product to be manufactured in the real world after its virtual design has been completed with, for example, a CAD software solution or a CAD system, for example a (mechanical) part or a part assembly (or equivalently an assembly of parts, as from the method point of view an assembly of parts can be seen as a part itself, or the method can be applied independently to each part of the assembly) or more generally any rigid assembly (for example a mobile mechanism). CAD software solutions allow the design of products in various unlimited industrial fields, including: aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine and / or offshore oil and gas production or transportation. Thus, the 3D modeling object reconstructed by the method can represent an industrial product, which can be any mechanical part, for example part of a land vehicle (including automotive and light truck equipment, racing car, motorcycle, truck and motor car equipment, truck and bus, train), part of an aircraft (including fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), part of a naval vehicle (including naval equipment, commercial ship, marine equipment, yacht and work boat, marine equipment), part of a general mechanical part (including industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment products, metal products, tire products), part of an electromechanical or electronic part (including consumer electronics, security and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical equipment and supplies), part of a consumer good (including furniture, home and garden products, leisure products, fashion products, products of hard goods retailers, products of soft goods retailers), part of a packaging (including food and beverage and tobacco, beauty and personal care, home products packaging).

[0072] Figure 1 An example of a GUI of a system is shown, wherein the system is a CAD system.

[0073] GUI 2100 can be a typical CAD-like interface with standard menu bars 2110, 2120 and bottom and side toolbars 2140, 2150. Such menu bars and toolbars contain a set of user-selectable icons, each of which is associated with one or more operations or functions as known in the art. Some of these icons are associated with software tools adapted to edit and / or work on the 3D modeled object 2000 displayed in the GUI 2100. The software tools can be grouped into workbenches. Each workbench contains a subset of the software tools. In particular, one of the workbenches is an editing workbench adapted to edit the geometric features of the modeled product 2000. In operation, the designer can for example pre-select a portion of the object 2000 and then initiate an operation (e.g. change dimensions, color, etc.) or edit geometric constraints by selecting the appropriate icon. For example, a typical CAD operation is to model a hole or a fold in a 3D modeled object displayed on the screen. The GUI can for example display data 2500 related to the displayed product 2000. In the example of this figure, the data 2500 displayed as a "feature tree" and its 3D representation 2000 relate to a brake assembly comprising a brake caliper and a disc. The GUI can further show various types of graphical tools 2130, 2070, 2080, for example to facilitate 3D orientation of the object, to trigger a simulation of an operation on the edited product or to render various attributes of the displayed product 2000. A cursor 2060 can be controlled by a haptic device to allow the user to interact with the graphical tools.

[0074] Figure 2 An example of a system is shown, wherein the system is a client computer system, for example a workstation of a user.

[0075] The client computer of this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, also connected to the bus are a random access memory (RAM) 1070, and a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. Video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device such as a hard disk drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; CD-ROM disks 1040. Any of the foregoing can be supplemented by, or incorporated in, an appropriately designed ASIC (application-specific integrated circuit). A network adapter 1050 manages access to a network 1060. The client computer can also include haptic devices 1090, such as a cursor control device, keyboard, etc. A cursor control device is used in the client computer to allow the user to selectively position a cursor on the display 1080 at any desired location. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices to input the control signals to the system. Typically, the cursor control device can be a mouse, the buttons of which are used to generate the signals. Alternatively or additionally, the client computer system can include a sensitive mat and / or a sensitive screen.

[0076] The computer program can include instructions executable by a computer which comprise means for causing the system described above to perform the method. The program can be recorded on any data storage medium including the memory of the system. The program can be implemented, for example, in digital electronic circuitry or computer hardware, firmware, software, or a combination thereof. The program can be implemented as a device tangibly embodied in a machine-readable storage device for execution by a programmable processor, such as a product. The method steps can be performed by a programmable processor executing a program of instructions to perform functions of the method by operating on input data and generating output. The processor can be programmable and coupled for receiving data and instructions from, and sending data and instructions to, a data storage system, at least one input device, and at least one output device. The application can be implemented with assembly or machine language, if desired. In any case, the language can be a compiled or interpreted language. The program can be a complete installation program or an update program. In any case, application of the program on the system causes instructions to be executed which perform the method.

[0077] The method can be part of a process for designing a 3D modeled object, which process for example comprises editing of a reconstructed 3D modeled object. "Designing a 3D modeled object" specifies any action or series of actions that is at least part of a process of setting out a 3D modeled object.

[0078] The method can comprise in a manufacturing process, which manufacturing process can comprise after execution of the method, producing a physical product corresponding to the modeled object, e.g. after editing. In any case, the modeled object designed by the method can represent a manufactured object. Thus, the modeled object can be a modeled entity, i.e. a modeled object representing an entity. The manufactured object can be a product, e.g. a part or a combination of parts. Since the method improves the design of the modeled object, the method also improves the manufacturing of the product, thereby improving the productivity of the manufacturing process.

[0079] Reference is now made to Figures 3-22 Implementations of the method are discussed in which the determined 3D modeled object is a control mesh of a subdivision surface.

[0080] Implementations of the method provide a new solution that can generate a 3D modeled object (or 3D model) that fits a plurality of images of a given real-world object. The generated model can be a control mesh of a subdivision surface that fits the contours of the object in the plurality of images and can be readily processed by existing CAD and / or CAM software.

[0081] A control mesh of a subdivision surface is a synthetic representation of a 3D shape, and thus it occupies relatively little storage space and can be manipulated to be relatively easy to edit. Furthermore, such a control mesh can be subdivided to perform specific edits, e.g. adding details, and / or to perform simulations that require a certain limited element size. The method can comprise any such later use of the obtained control mesh. In addition, the method can use the control mesh format in an efficient way to perform optimizations. Furthermore, the method can introduce terms in the energy that are specific to the control mesh.

[0082] Subdivision surfaces have been used in 3D software for a long time. Implementations of the method can be directed to subdivision surfaces based on a four-branch control mesh and a Catmull-Clark subdivision scheme. Thus, the obtained control mesh and its defined subdivision surface can be in accordance with the teachings of the following paper: E. Catmull, J. Clark., Recursively generated b-spline surfaces on arbitrary topological meshes, Computer Aided Design, 10(6) (pp. 350-355), 1978, which is incorporated herein by reference.

[0083] In the discussion of implementations of the method, the terms "control mesh fitting" and "subdivision surface fitting" to a given target are equivalent and refer to the operation of automatically moving the vertex positions of a control mesh so that its limit surface (or the mesh after a sufficient number of subdivision steps) fits the target. Figure 3 The process of successively subdividing a simple control mesh until a limit surface is reached is illustrated.

[0084] Implementations of the method allow to obtain a control mesh that, when sufficiently subdivided, fits to multiple views of an existing object. The input data comprises raw 2D images with corresponding camera parameters, and an initial dense mesh that roughly approximates the object (in the same reference frame as the cameras). The latter can be obtained, for example, after (roughly) segmenting the images by using a space carving technique. This can be done according to the teachings of the paper by K. N. Kutulakos, S. M. Seitz, A theory of shape by space carving, International Journal of Computer Vision, 38(3) (pp. 199-218), 2000, which is incorporated by reference. It is shown here how to generate a mesh from multiple images of an object by space carving. This technique can roughly identify the contours of the object and generate an approximate dense mesh that cannot be directly used for editing in CAD and / or CAM software. However, it provides a suitable input for implementations of the method.

[0085] Implementations of the method form a new pipeline for generating a subdivision surface (i.e. a parametric shape) that fits to multiple images of the same object. Implementations of the method use subdivision surfaces because they provide a simple and generic model for parametric shapes and allow for an easy evaluation of the surface and its geometric features. However, alternatively, other models, such as parametric exact boundary representations (used in CAD software), can also be optimized in the same way.

[0086] Reference is made to Figure 4 Starting from a set of images 40 of an object with corresponding camera parameters and a dense mesh 42 that roughly approximates the shape (obtained, for example, by the earlier cited paper "A theory of shape by space carving"), implementations of the method can create a simplified four-hinged control mesh that, when subdivided, fits to the object according to Figure 5 The pipeline shown fits the object to the provided images. The stronger the correlation of the images, i.e. the more of the object they cover from all possible angles, the more accurate the generated 3D shape (i.e. the higher its correspondence to the actual object). In the example, the 2D images provided in the implementation of the method thus cover the real object from different angles.

[0087] The images can then be pre-processed to detect all contours in the images, including but not limited to contours corresponding to objects. This can be performed, for example, as previously taught according to the teachings of the paper “Holistically-nested edge detection”. This paper develops a new edge and boundary detection algorithm using a deep learning model. Implementations of the method can use this solution to generate a probability map with all contours in the provided set of images (regardless of whether they are contours of target objects).

[0088] Separately, implementations of the method can simplify a dense mesh to an initial control mesh. This can be done, for example, following the teachings of J. Wenzel, M. Tarini, D. Panozzo, and O. Sorkine-Hornung in the paper Instant field-aligned meshes, ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA), 34(6), 2015, which is incorporated by reference herein. This paper proposes a technique to simplify a dense mesh to a uniform quad-dominant mesh. Implementations of the method can use this technique in the step of its proposed pipeline.

[0089] Since for the present purposes the two are far apart, implementation of the method can use a scheme to fit the initial control mesh back to a dense mesh (using a point cloud corresponding to its vertices). This can be performed, for example, following the teachings of V. Estellers, F. R. Schmidt, and D. Cremers, Compression for smooth shape analysis (arXiv preprint arXiv: 1711.110824, 2017), and V. Estellers, F. Schmidt, D. Cremers, Robust Fitting of Subdivision Surfaces for Smooth Shape Analysis, International Conference on 3D Vision (3DV) (pp. 277-285), IEEE, 2018, which are incorporated herein by reference. These papers introduce a scheme to fit a control mesh to a target point cloud. The scheme first generates an estimated control mesh that very roughly fits the shape defined by the cloud, and then applies an iterative linear optimization scheme to adapt the control mesh to the target. A working public prototype is available and has been tested. The results are robust and satisfy the stated goal, i.e. the computational analysis of the shape. However, the resulting control mesh is not uniformly dense, and therefore it is difficult to use for further editing. More importantly, using this scheme to reconstruct a 3D shape from multiple images would require first generating a dense point cloud from the same image. This constitutes a separate difficulty that implementation of the method can avoid.

[0090] Since the provided dense mesh is only a rough approximation, the subdivision of the resulting modified control mesh is not yet adapted to the contours of the object in the various images. Implementation of the method uses an optimization scheme to best fit the object contours in the images. Finally, implementation of the method can also add a post-processing stage to further smooth areas of the resulting shape for which no contours were found in any of the provided images, and therefore which were not fitted.

[0091] Details about these various steps of the implementation of the method shown in Figure 5 are discussed below.

[0092] The implementation of the method fits a rigid object to multiple views, so the more relevant the images used, the better the resulting description of the real object. The implementation of the method does not require and can exclude the precise identification of the correct contours in each image, but it can automatically process raw images. The implementation of the method can exclude manually selected constraint points. The implementation of the method can take into account the real perspective projection, which means that the implementation of the method can use photos taken at any distance from the object. Since the camera parameters are now provided by the current capture device, there is no need to calculate them. Thus, the silhouette of the object can be computed explicitly, rather than by minimizing an energy, which leads to higher computational robustness and better performance. The implementation of the method can use approximate results (e.g. from older methods) as input to obtain a shape that is both more adapted to the real object and can be further edited with existing software. Also, the results of the implementation of the method require much less storage space than any dense mesh (or point cloud) obtained by existing techniques. The implementation of the method can allow the addition of new energy terms to regularize the resulting shape. The implementation of the method can allow a post-processing stage that further smooths the areas where no contours were found in any image.

[0093] Now an example of the generation of a simplified four- principal control mesh is discussed.

[0094] The implementation of the method fits a rigid object to multiple views, so the more relevant the images used, the better the resulting description of the real object. The implementation of the method does not require and can exclude the precise identification of the correct contours in each image, but it can automatically process raw images. The implementation of the method can exclude manually selected constraint points. The implementation of the method can take into account the real perspective projection, which means that the implementation of the method can use photos taken at any distance from the object. Since the camera parameters are now provided by the current capture device, there is no need to calculate them. Thus, the silhouette of the object can be computed explicitly, rather than by minimizing an energy, which leads to higher computational robustness and better performance. The implementation of the method can use approximate results (e.g. from older methods) as input to obtain a shape that is both more adapted to the real object and can be further edited with existing software. Also, the results of the implementation of the method require much less storage space than any dense mesh (or point cloud) obtained by existing techniques. The implementation of the method can allow the addition of new energy terms to regularize the resulting shape. The implementation of the method can allow a post-processing stage that further smooths the areas where no contours were found in any image. Figure 6

[0095] Now an example of the fitting of the control mesh to the dense input mesh is discussed.

[0096] The implementation of the method can use the scheme presented in the earlier cited papers “Compression for smooth shape analysis” and “Robust Fitting of Subdivision Surfaces for Smooth Shape Analysis” to fit the generated control mesh to the input dense mesh as best as possible, since this is the best first estimate of the real object.​Figure 7 Results are shown.

[0097] However, the implementation of the method can introduce some changes to the described scheme. For example, if the target point cloud corresponds to a relatively thin object with cavities, such as a vase as shown in Figure 8 the simple control mesh is just a convex subdivision surface that swallows the target, the implementation of the method can establish the correct initial correspondence between the two, thus obtaining the cavity in the fitted subdivision surface. The implementation of the method can significantly avoid false correspondences between regions with opposite orientations.

[0098] Therefore, the implementation of the method can add a normal vector comparison in the point correspondence calculation: if the dot product of their normal vectors is less than a threshold, the two points do not correspond. Thus, points on the hull are not attracted by target points on the inner hull of the object.

[0099] The implementation of the method can also add an optional weight calculation function based on the initial evaluation of the various energy terms. Weight coefficients for the energy terms can be provided, or these weight coefficients can be calculated at the initial evaluation of the energy terms to ensure a certain ratio between the four terms. These ratios can be the default values of the system or user-provided ratios.

[0100] The implementation of the method can also add a new energy term using a spring energy term that attracts the control points to their initial positions. This is very useful to keep the fitted subdivision surface as close as possible to the initial control mesh or at least a portion thereof.

[0101] Finally, the implementation of the method can also replace the generation of the non-uniform control mesh with the method introduced in the earlier cited paper "Instant field-aligned meshes". In the current context, this has been tested to produce better results.

[0102] An example of contour detection is now discussed.

[0103] The implementation of the method can use the solution presented in the earlier cited paper "Holistically-nested edge detection" to generate a contour probability map for each image, as shown in Figure 9 The pixels with the highest probability in the map indicate the location of the detected contours. The detected contours have different probabilities, for example, the boundary contours of internal reflection artifacts have a lower probability than hard-contrast contours that correspond to the actual boundaries of the objects in the image. The implementation of the method can invert the probability map in order to obtain a fitting energy map (i.e., compute a map whose values correspond to 1 minus the probability map). Thus, the contour pixels correspond to local minima of the fitting energy, which are more suitable for the minimization scheme.

[0104] The implementation of the method can then compute points lying on the apparent contour of the current shape of the subdivision surface (by subdividing the control mesh). These points can be projected onto the fitting energy map using the camera parameters. The lower the fitting energy value of a projected point, the more likely it is that the point lies on the actual contour.

[0105] Two conclusions can be drawn.

[0106] First, if the projected contour of the current shape (i.e. the contour 102 in Figure 10 ) is far from the target contour of the object in the image (i.e. the contour 104 in Figure 10 ), it will be more difficult to converge towards the target contour. This is due to the fact that the example contour detection does not eliminate all unwanted contours from the fitting energy map, and some of these contours can attract the shape during the best contour fitting phase.

[0107] However, since the implementation of the method starts from an approximation of the desired shape, the initial apparent contour is usually not too far from the contour of the actual object in the image (see Figure 11 ) in the same reference frame as the one in which the camera parameters are defined, i.e. the probability of converging towards the correct contour is much higher.

[0108] Second, even if the projection starts from a point close to the target contour, the local gradient of the fitting energy map can be very flat, effectively preventing the optimization from converging to the target contour (see Figure 12 ).

[0109] Since the technique of the paper “Holistically-nested edge detection” aims at generating a (e.g. almost) contour clear binary image like any other contour detection algorithm, the locally flat gradient is essentially a normative feature of such techniques. The deep learning technique as described in the present paper is intrinsically statistical, so the non-binary probability values obtained are very close to 0 or 1 (thus still containing flat gradient areas). This is especially true for high resolution (e.g. 4K) images such as the ones obtained by current capturing devices. To avoid this, the implementation of the method can apply a local filter to the fitting energy map to widen the areas where the gradient is non-zero (see Figure 13 ).

[0110] The smoothing filter can for example comprise at least one application of a Gaussian blur (applied to the contour probability map). The Gaussian blur operation modifies the shape of the contour map from Figure 12 to the shape of Figure 13 , which can make the optimization converge better.

[0111] A simple Gaussian blur reduces the amplitude. To avoid losing minima in the profile image, the method can consider (e.g. apply) a smoothing process that includes determining an envelope of a series of Gaussian blurs applied to the profile probability map, each Gaussian blur having a different kernel size.

[0112] For example, an implementation of the method can consider the envelope of the initial fitted energy map and all Gaussian blurs thereof (see Figure 14 ).

[0113] An implementation of the method can consider the initial fitted energy map as a function defined in . An implementation of the method can define a Gaussian transform of the initial fitted energy map

[0114]

[0115] The smoothed fitted energy map can be defined as the envelope of the series of functions .

[0116] An implementation of the method can apply an approximation of the envelope by iterating successive Gaussian blurs while increasing the kernel size and combine them with previous images by keeping the minimum value for each pixel, for example, where G k is a Gaussian filter with kernel size 2k+1, is the initial smoothed map. A faster method that produces similar results but with lower gradient values or at least different gradient values that can be implemented by the method is to skip some of the intermediate filters, for example Otherwise, iteratively convolve with a Gaussian filter (as applying multiple successive Gaussian filters is equivalent to applying a single Gaussian filter with a larger kernel size), for example where and

[0117] Any other filter that produces a similar map will work equally well and can thus be applied by the method, for example a filter that convolves an image with a kernel obtained from a Laplace distribution produces very similar results (see Figure 15 ), i.e.:

[0118]

[0119] Figure 16 A close-up of the profile map before and after applying such a filter is shown.

[0120] In summary, the implementation of the method can operate on a fitting energy map with non-zero gradients that locally decrease in a monotonic way towards a minimum, thus improving the convergence speed. However, the implementation of the method can require at least only C0continuity.

[0121] As mentioned before, another approach can be to use a distance transform of the image in order to generate a map containing the distance to the closest contour for each pixel. However, this would require a threshold to identify contours with a probability value above this threshold and then compute the distance transform of the image. This means that the probability difference of two contours that are both within the threshold is lost, whereas the implementation of the method allows to exploit the results of the contour detection algorithm globally.

[0122] An example of fitting contours is now discussed.

[0123] By minimizing the fitting energy between the silhouette of the subdivision surface and the contours in the contour map (one for each provided image), the implementation of the method can adapt the four dominant control mesh discussed before to all provided images. Instead of using the distance between the silhouette and the contours in each image, the fitting energy map computed earlier for each image can be used. In each iteration, the process first computes the silhouette of the current limit surface and then deforms the surface by minimizing the fitting energy for all images (see Figure 17 ).

[0124] Since the camera parameters are known, the implementation of the method can separate the computation of the silhouette points on the 3D surface, which can be done explicitly and does not require an optimization. This is different from approaches that consider the (simplified) camera parameters and the silhouette points on the 3D surface as unknown variables and perform a more complex optimization method, whose energy function is defined in a variable space of high dimension, which leads to potentially longer computation times and possibly less reliable computation results. In particular, the dimension of the variable space in such approaches would grow with the number of provided images, thus eventually limiting the number of images that can be used. In contrast, the optimization of the implementation of the method is performed in a variable space whose dimension only depends on the number of vertices in the control mesh and is independent of the number of provided images, which makes it easier to use many images, thus better describing the object.

[0125] The method can comprise, at each iteration of the optimization, providing a current control mesh, subdividing the current control mesh into a subdivided mesh, performing a loop on each 2D image, and modifying the current control mesh to reduce the energy. For each 2D image, the loop comprises projecting the subdivided mesh onto the 2D image, computing a 2D silhouette of the 3D modeled object in the 2D image, and identifying points of the subdivided mesh corresponding to the 2D silhouette as silhouette vertices. This will be explained in more details below. Optionally, the topology of the subdivided mesh can be stored after the first iteration so as to be reused in the next iteration of the subdivision. This can save time.

[0126] Thus, the determination of the 3D silhouette vertices can be redone at each iteration of the optimization. The topology of the mesh (connections between vertices, number of edges, number of polygons) does not change. The topology of the overall shape does not change either in terms of Euler characteristic. Thus, the subdivision can only be done once in the sense of determining the connections between the subdivision vertices. On the other hand, the method can recompute the coordinates of the vertices of the subdivided mesh (approximating the limit surface) after each iteration as the coordinates of the vertices of the control mesh change. The 3D silhouette vertices can then be computed.

[0127] The set of silhouette points for each image can be computed by using the camera parameters and computing a depth map of the current shape of the subdivision surface and keeping only the points for which the image is a border pixel of the object projection (see Figure 18 : depth image pixels and border pixels 180 on the left, subdivided mesh and silhouette points 182 on the right). If several points project on the same pixel, the point whose depth is the same as the depth value at this pixel can be used, i.e. the implementation of the method can avoid using hidden points in the silhouette.

[0128] The implementation of the method can identify the position of the silhouette on the current shape of the subdivision surface for its projection in each image i. The implementation of the method can approximate the subdivision surface by dividing the control mesh a sufficient number of times n into a refinement mesh R n The implementation of the method can subdivide by three, but n is a parameter that can be defined at the beginning of the process. Thus, the implementation of the method can find a subset of vertices of R n Σ i that will project on the shape silhouette in i. As R n is sufficiently refined, these vertices can well approximate the actual position of the silhouette on the subdivision surface.

[0129] Using the projection transform π i for image i, the implementation of the method can compute a depth map D i of the scene containing only the current shape of the subdivision surface. The implementation of the method can identify the silhouette pixels S i of D ia subset of S such that any silhouette pixel p s ∈S i is adjacent to a background pixel p bg . Background pixels are identified by a depth value D i (p bg ) usually set to 0 in the depth map. The implementation of this method can extend the silhouette pixels by including pixels that are not directly adjacent to a background pixel but are located in a neighborhood (whose size can also be defined) that includes a background pixel.

[0130] After identifying all the silhouette points, the implementation of this method can verify for each vertex v of R n whether its projection π i (v) is a silhouette pixel p s , in which case v is part of a subset of the silhouette vertices Σ n of R i . If multiple vertices project on the same pixel, only the vertex with the same depth as the computed pixel depth is kept in Σ i .

[0131] The implementation of this method can now compute a fitting energy value for each silhouette vertex v ∈ Σ i using E i (π i (v)), where E i is the fitting energy map computed for image i.

[0132] If E i (π i (v)) reaches a minimum value, the silhouette vertex projects onto a contour identified in the image.

[0133] If all the fitting energy values (globally) reach a minimum value for all the silhouette vertices and for all the images, the projected silhouette of the subdivision surface is identical to the identified silhouette of the object in the images (as the implementation of this method starts with a surface shape that is close to the object).

[0134] To verify the extent to which the provided images cover the surface shape, a visual verification is performed by coloring all the vertices that appear as silhouette points in at least one or more images (see picture: the gray areas appear as silhouettes in at least one image, the white areas do not belong to any silhouette). Figure 19

[0135] Each silhouette point can be assigned a value from its fitting energy map. The sum of the squares of these values can define the main fitting energy term to be minimized. Since the silhouette points are computed on the limit surface of the subdivision (or at least on the mesh produced by multiple subdivision steps), they can be represented as a linear function of the control mesh vertex coordinates, i.e. the variables of the problem to be solved. ​

[0136] π i Projection transformation of image i (based on provided camera parameters)

[0137] x Coordinates of the vertices of the control mesh

[0138] s ij = S ij (x) For a given image i, silhouette point j (S ij is a linear function of x)

[0139] e ij = E i (π i (s ij (x))) Fitting energy of the silhouette point (using the fitting energy map E i of image i), i.e. the lower the fitting energy, the higher the probability that the point projects onto the contour

[0140] The fitting energy term can be defined by E fit =∑ i,j e ij 2 By definition, it can be minimized to fit the silhouette to the contour in all images.

[0141] Since the 3D modeled object is a control mesh (of a subdivision surface), one can add specific terms. Obviously, the energy can reward a low degree of the sum of the lengths of all edges in the control mesh, a low degree of the sum of the squared principal curvatures values at densely sampled points of the limit surface of the control mesh, and / or the regularity of the faces of the control mesh. This allows to obtain an industrial cleaner control mesh that behaves more accurately in simulation and that is easier to manipulate.

[0142] This can correspond to three different regularization terms included in the total energy to minimize:

[0143] the sum of the lengths of all edges in the control mesh. E edges is a quadratic function of x.

[0144] the sum of the squared principal curvatures values cl and c2 at densely sampled points of the limit surface. This method can particularly use the curvature values instead of the second derivatives with respect to the parameters defined on the subdivision surface, since this method is more directly linked to the shape of the surface and is independent of the parameterization. For a subdivision surface, the second derivatives can be easily evaluated using x. The principal curvatures can be computed from the second derivatives.

[0145] This term supports regular quadrilaterals in the generated control mesh. For each quadrilateral (see Figure 20 ), one can define and Possible variables are and All these values ​​can be analytically represented as functions of x.

[0146] Finally, the implementation of this method can also introduce an optional energy term to measure the global change in curvature (which can be the sum of squares of the partial derivatives of the principal curvatures):

[0147]

[0148] The optimization scheme can determine the total energy E = E fit +αE edges +βE curvature +γE quad +δE curvature variation Minimize vertex coordinates x min α, β, γ, and δ are weighting parameters that can be provided or calculated during the initial evaluation of the energy term to ensure specific ratios among these four terms. These ratios can be system defaults or user-provided ratios.

[0149] Due to E fit Using discrete graphical functions E i Therefore, this method can be implemented by applying a Sobel filter to generate an approximation of E. i The gradient of E is plotted, and these plots are used to analytically evaluate E. fit The Jacobian matrix is ​​given. The Jacobian matrices for other energy terms can be numerically approximated.

[0150] This method can be implemented using a numerical optimization solver (e.g., Levenberg-Marquardt) to determine x. min .

[0151] Since this method does not explicitly represent the silhouette, it can only compute one minimization step per iteration, then use the generated vertex coordinates to update the control mesh. If the energy is sufficiently small, the process can be stopped and an updated control mesh generated; otherwise, the process is iterated again (see [link to documentation]). Figure 17 ).

[0152] Therefore, based on the obtained control mesh, the profile of the subdivided surface is the best fit to the profile found in the provided image (see [link]). Figure 21 ).

[0153] Another benefit of the method implementation is that it generally smoothes or flattens the areas that are not covered by the silhouette. For example, the flat bottom of an object sitting or lying on a table can be difficult to cover by the image, as it is usually taken from above. Thus, the carving can result in non-flat, possibly sharp areas, but at the silhouette fitting stage, the method can flatten them (see Figure 22 ).

[0154] If more information can be used, such as planar areas, symmetry planes, etc., it can be taken into account in the optimization scheme, i.e. by restricting the position of (some) control vertices, or by directly reducing the number of variables. If a symmetry plane is identified, the control vertices on one side of the symmetry plane can be constrained to be the mirror image of the identified control vertices on the other side of the symmetry plane, thus reducing the number of control vertices by about 50%. Control vertices in a recognized planar area can be defined by two parameters on the plane instead of three spatial coordinates. The same can be done for spherical or cylindrical areas of determined shape.

[0155] An example of optional smoothing of the unfitted areas is now discussed.

[0156] As mentioned before, not all areas of the generated fitted subdivision surface are considered to be silhouettes, and thus nothing is fitted there. Therefore, the surface can be locally not smooth in these areas.

[0157] These examples can attempt to smooth these areas by performing a post-processing stage that includes minimizing an energy composed of terms E edges , E curvature , E quad , E curvature variation , while maintaining a set of constraints C fix .

[0158] The first four terms have been discussed before. C fix represents the constraint of keeping each point of the surface on the silhouette in its original position (i.e. the position reached before the optional smoothing), effectively constraining the shape to stick to its original shape in these areas.

[0159] All four smoothing terms E edges , E curvature , E quad , E curvature variation use non-uniform weights to reduce their influence on the areas that appear in the silhouette, and increase their influence on the areas that are not covered by the silhouette. While the task of determining how to weight the different terms against each other is not trivial, it has been observed that using these methods implementations, some of the remaining roughness in such areas can be further smoothed.

Claims

1. A computer-implemented method for 3D reconstruction, the method comprising: providing 2D images all representing the same real object that is stationary, and for each 2D image, providing camera parameters defining a perspective projection; providing, for each 2D image, a smooth map having pixel values each representing a measure of silhouette presence in the corresponding pixel of that 2D image; and determining a 3D modeled object representing the real object by iterative optimization for iteratively optimizing an energy that rewards, for each smooth map, a projection of silhouette vertices of the 3D modeled object having pixel values on that smooth map that represent a high amount of silhouette presence, wherein the 3D modeled object is a control mesh of a subdivision surface, and wherein the method further comprises, at each iteration of the iterative optimization: - providing a current control mesh; - subdividing the current control mesh into a subdivision mesh; - for each 2D image: ■ projecting the subdivision mesh onto the 2D image; ■ computing a 2D silhouette of the 3D modeled object in the 2D image; and ■ identifying points of the subdivision mesh corresponding to the 2D silhouette as silhouette vertices of the 3D modeled object; and - modifying the current control mesh to reduce the energy, wherein the modified current control mesh is to be input to the next iteration as the current control mesh of the next iteration.

2. The method of claim 1, wherein, The smooth map comprises local extrema corresponding to maxima of silhouette probability, the energy rewarding high silhouette probability at the projection of the silhouette vertices.

3. The method of claim 2, wherein, The smooth map for each 2D image is obtained by applying a function to a silhouette probability map for that 2D image.

4. The method of claim 3, wherein, The function is a smoothing of an affine mapping.

5. The method of claim 4, wherein, The smoothing comprises applying at least one Gaussian blur to the silhouette probability map, or convolving the silhouette probability map with a kernel obtained from a Laplacian distribution.

6. The method of claim 5, wherein, The smoothing comprises determining an envelope of applying a family of Gaussian blurs to the silhouette probability map, each Gaussian blur having a different kernel size.

7. The method of any one of claims 1 to 6, wherein, The energy comprises terms of the following type: wherein, for a 2D image i: • E i (·) indicates a smoothed plot, • s ij represents the silhouette point j, and • π i (·) denotes a projection transformation.

8. The method of any one of claims 1 to 6, wherein, The energy further rewards: - a low degree of the sum of the lengths of all edges in the control mesh, - a low degree of the sum of the squared principal curvatures values at densely sampled points of the limit surface of the control mesh, and / or - a regularity of the faces of the control mesh.

9. A computer program product comprising instructions for performing the method according to any one of claims 1-8.

10. A computer readable storage medium having recorded thereon instructions for performing the method according to any one of claims 1-8.

11. A system comprising a processor coupled to a memory and to a graphical user interface, the memory having recorded thereon instructions for performing the method according to any one of claims 1-8.

Images