A method for numerically modeling a three-dimensional surface of an object

Abstract

A method for digitally digitizing a three-dimensional surface, comprising the steps of: determining point coordinates on the three-dimensional surface to obtain a point cloud and capturing images of the three-dimensional surface; constructing a mesh of triangles on the point cloud by tetrahedrization; projecting the mesh onto the images and simultaneously determining an opacity parameter and a texture parameter for the triangles by aligning at least a portion of the images with the mesh to obtain a preliminary three-dimensional model; comparing the preliminary three-dimensional model to the images and verifying consistency between the preliminary three-dimensional model and the images; returning to the determination of the opacity and texture parameters of the triangles if the consistency is less than expected; otherwise, extracting the surface from the preliminary model to form the three-dimensional model. Figure from the abstract: Fig. 1

Description

Title of the invention: Method for numerically modeling a three-dimensional surface of an object

[0001] The present invention relates to the field of three-dimensional surface numerical modeling.

[0002] BACKGROUND OF THE INVENTION

[0003] Three-dimensional digital models are used for example in the field of virtual reality to represent objects that an observer can view randomly from different directions of observation or places in which an observer can randomly move around and wander about, giving the observer an impression of volume as close as possible to reality.

[0004] The creation of a three-dimensional digital model of an object generally begins with determining the positions of points belonging to the object's outer surface to obtain a point cloud and continues with determining a mesh connecting these points to each other. Once the mesh is complete, a surface texture is extracted from images of the object and applied to the mesh.

[0005] The positions of the points are determined for example using a digitization process by projection of a laser beam or by projection of structured light, or by a photogrammetry process.

[0006] The determination of the mesh is for example carried out by a Delaunay triangulation consisting of making a mesh from triangles whose vertices are the points whose positions have been determined (this is also called tiling or tessellation in English) or, for volume surfaces, a "tetrahedrization" consisting of making a mesh from tetrahedra having triangular faces having vertices coinciding with the points whose positions have been determined.

[0007] Once the mesh has been created, the images of the object are used to extract a texture which will be applied to each triangle of the mesh.

[0008] This method of embedding three-dimensional digital models is efficient for objects with simple shapes. However, for objects with complex shapes, the difficulty lies in determining which triangles belong to the object's surface and which do not. Triangles belonging to the object's surface are then defined as opaque and will be visible, while triangles not belonging to this surface are defined as transparent and will therefore be invisible. Determining the opacity of the triangles is performed by a region growth algorithm but is often imperfect unless intervention is required. human particularly long and tedious, so three-dimensional digital models of complex-shaped objects often have flaws.

[0009] SUBJECT OF THE INVENTION

[0010] The invention aims in particular to provide a three-dimensional digital model exhibiting relatively few defects while limiting the computing resources required for modeling. Summary of the invention

[0011] To this end, the invention provides a method for digitally digitizing a three-dimensional surface, comprising the steps of: determining point coordinates on the three-dimensional surface to obtain a point cloud and capturing images of the three-dimensional surface, constructing a mesh of triangles on the point cloud by tetrahedrization, projecting the mesh onto the images and simultaneously determining an opacity parameter and a texture parameter for the triangles by aligning at least a portion of the images with the mesh in order to obtain a preliminary three-dimensional model, comparing the preliminary three-dimensional model to the images and verifying consistency between the preliminary three-dimensional model and the images, and returning to the determination of the opacity and texture parameters of the triangles if the consistency is less than expected.extract the surface from the model draft to form the three-dimensional model; otherwise...

[0012] Projecting the triangles onto the images and determining the texture makes it possible to determine whether a triangle belongs to the visible surface of the object according to the observation direction. Thus, while images are traditionally used only to provide the texture of the model, the method of the invention also uses images to determine the opacity of the triangles. This makes it possible to quickly and easily improve the accuracy of the mesh and therefore limit the number of defects in the final model.

[0013] According to optional features, used individually or in whole or in combination: - the process includes an operation to order the triangles, according to at least one observation direction and from an observation position, for each pixel of each image and an opacity calculation for each of said pixels; - opacity is a sigmoid function varying between -oo and +oo; - tetrahedrization includes the operation of determining a bounding box around the point cloud and the operation of constructing tetrahedra on the point cloud from each angle of the bounding box; - the process includes a loop to return to the determination of the opacity and texture parameters of the triangles if the consistency is less than an expected consistency, the loop implementing a gradient backpropagation method to iteratively converge the opacity and texture parameters of the triangles towards an optimal configuration; - comparing the draft model to the images results in the calculation of a consistency score; - the consistency score is for example an error calculated between the model draft and the images by the least squares method; - when errors accumulate over a series of several images, an optimization procedure is used to adjust the parameters of the triangles in order to minimize the error by least squares; - the process includes the step of implementing an external heuristic to eliminate the triangles from the triangular mesh least likely to belong to the three-dimensional surface, the heuristic involving the conditions for obtaining the point cloud and at least one characteristic of the triangles.

[0014] Other features and advantages of the invention will become apparent from the following description of a particular, non-limiting embodiment of the invention. Brief description of the drawings

[0015] Reference will be made to the attached drawings, among which:

[0016] [Fig-1] is a schematic view of an example installation allowing the implementation of work of the process of the invention;

[0017] [Fig.2] is a block diagram illustrating the process of the invention;

[0018] [Fig.3] is a perspective view of a point cloud whose positions were determined on the surface of one of the wings of the Winged Victory of Samothrace;

[0019] [Fig.4] is a perspective view of a rough mesh obtained from this point cloud (same viewing angle as [Fig.3]);

[0020] [Fig.5] is a perspective view illustrating the finalization of the mesh (same viewing angle as [Fig.4]);

[0021] [Fig.6] is a perspective view of the mesh draft obtained from the point cloud of [Fig.3] but from a different viewpoint than that of [Fig.4];

[0022] [Fig.7] is a perspective view of the final mesh (same viewing angle as [Fig.6]). DETAILED DESCRIPTION OF THE INVENTION

[0023] The invention relates to a method for creating a three-dimensional digital model representative of an object, here the terminal part of one of the wings of the The statue known as the Winged Victory of Samothrace, housed in the Louvre Museum (the statue and the wing are symbolized by V and A in [Fig. 1]). It should be noted that, at the end of the implementation of the method according to the invention, the three-dimensional digital model must be observable along a large number of axes or directions of observation (hereafter referred to as the possible directions of observation).

[0024] With reference to [Fig.1], the method according to the invention is implemented by means of an installation comprising a computer 1 and a three-dimensional measuring arm 2 connected to the computer 1.

[0025] The three-dimensional measuring arm 2 here comprises a base 20, a first segment 21 articulated on the base 20, a second segment 22 articulated on the first segment 21, a wrist 23 articulated on the second segment 22. The wrist 23 is provided with an accessory mounting plate 24 on which are fixed a LIDAR type laser transmitter / receiver 25 and a camera type image capture device 26, both pointing in the same direction.

[0026] Each articulation axis of the three-dimensional measuring arm 2 is provided with an angular encoder connected to a first bus linked to the computer 1. The angular encoders transmit to the computer 1 signals representing the relative angular positions of the elements of the arm between which they are arranged.

[0027] The laser transmitter / receiver 25 and the image capture device 26 are connected to a second bus connected to computer 1. The laser transmitter / receiver 25 transmits to computer 1 signals representing distance measurements between the laser transmitter / receiver 25 and points on wing A toward which the laser transmitter / receiver 25 is pointed. The image capture device 26 transmits to computer 1 signals representing images of wing A captured simultaneously with the distance measurements provided by the laser transmitter / receiver 25.

[0028] Computer 1 executes a program arranged to implement the process of the invention.

[0029] The computer 1 is programmed in a way known in itself to be able to calculate at each instant the position of the accessory carrier plate 24 from the signals provided by the angular encoders and to deduce therefrom, in a reference frame, the coordinates of the points of the wing A for which a distance measurement has been carried out and the orientation of the accessory carrier plate 24 for each measurement / image capture.

[0030] The three-dimensional measuring arm 2 will be manipulated and controlled by an operator to rotate around wing A and take measurements of a plurality of points distributed over the entire surface of wing A. In parallel, photographs are also captured and recorded with the coordinates of the point from which each photograph was taken (shooting point) and the axis along which each photograph was taken (shooting axis).

[0031] Computer 1 records all of this data to obtain a point cloud such as that shown in [Fig.3].

[0032] It is understood that, at this stage of the implementation of the method, the point cloud and photographs are available, for which the shooting point and axis are known. The method according to the invention aims at creating a three-dimensional model that can be observed from different angles, including angles for which no photographs exist.

[0033] From this point cloud, the program on computer 1 constructs a polygonal mesh (step 100 of the flowchart in [Fig. 2]). More precisely, the program executed by computer 1 implements a classical tetrahedralization algorithm for constructing a mesh of tetrahedra on the point cloud. Recall that a tetrahedron is a polyhedron with four triangular faces and that, in a tetrahedral mesh satisfying the Delaunay condition, all the points in the point cloud are vertices of tetrahedra such that the sphere circumscribed about each tetrahedron is empty, that is, it contains none of the points in the point cloud.

[0034] The tetrahedrization includes the operation of determining a bounding box around the point cloud and the operation of creating tetrahedra on the point cloud (including the angles of the bounding box).

[0035] It is understood that the points in the point cloud form the vertices of the triangular faces of the tetrahedra resulting from the tetrahedralization and allow us to obtain a triangular mesh (see [Fig. 4]). Alternatively, the points can be sampled so that not all points are in the mesh.

[0036] The triangular mesh is then projected onto the images (step 110). The resulting projection, known in itself, is a simple projection of a three-dimensional space onto a plane (the plane of each photograph) along the normal to the plane. This is particularly easy for photographs taken with a standard or telephoto lens. That said, projections are also known for projecting three-dimensional spaces onto photographs taken with very wide-angle or hypergonal lenses (for example, a fisheye lens).

[0037] There are necessarily triangles that do not belong to the surface of wing A, and the method of the invention preferably provides for the step of implementing an external heuristic to eliminate the triangles from the triangular mesh that are least likely to belong to the three-dimensional surface, the heuristic taking into account the conditions for obtaining the point cloud and at least one characteristic of the triangles. The implementation of the external heuristic makes it possible to reduce the number of Eligible triangles (i.e., those likely to belong to the surface of wing A) are used to refine the triangular mesh. For example, if points were measured on two opposite faces of wing A, all triangles between the two faces have no reason to exist and are therefore eliminated. As another example, if points were measured on the surface of wing A every centimeter, all triangles with sides longer than a certain threshold (for example, two centimeters) are eliminated. This external heuristic is advantageous because it saves time.

[0038] Once the triangles not eligible to belong to the surface of wing A have been eliminated, the projection is used to assign to each triangle a value of an opacity parameter and a value of a texture parameter (step 120) as will now be seen.

[0039] As a preliminary step, the program on computer 1 established, for each pixel (or image element) of each image, an ordered list of all the triangles that are crossed by one of the observation directions corresponding to each shooting direction of one of the photographs. It is understood that, for each pixel and each observation direction, there are at least two triangles: one located on the front face of wing A and the other located on the rear face of wing A. Computer 1 will then determine the order in which the triangles are crossed by following the observation direction from the theoretical observer (the camera) located at the origin of this observation direction (the shooting axis).This ordering will be used to determine the opacity of each triangle, with the understanding that the maximum opacity is theoretically assigned to the first triangle crossed by the observation direction (or the first-rank triangle, the one closest to the theoretical observer and therefore visible to said observer along that observation direction). It is understood that the order of the triangles is linked to the geometry of wing A. Opacity, commonly denoted a, is a numerical value between 0 (opaque) and 1 (transparent). Preferably, opacity is defined by a sigmoid function varying between -∞ and +∞. This determines the extent to which each triangle contributes to each pixel based on its opacity.

[0040] The texture parameter reflects the color c of the triangle and depends on the color of the triangle itself (which is in the foreground and is denoted cf = c^) but also on the background color, that is, the mixture of the colors of all the triangles visible through transparency behind the triangle in question. It is therefore understood that each triangle of rank i can thus be defined by one and / or the other of the following values:

[0041] _ (ac + fj

[0042] a < / / = a.

[0043] where and C are respectively the opacity and color parameters of the triangles. Note that, with sigmoid parameterization, the factor (1-ai) can never be negative. By applying this relation recursively from the last triangle to the first visible triangle, one can establish an effective visibility per triangle from a pixel and calculate the final color / opacity eff of the pixel A ' N rendered by the volumetric representation after traversing N triangles. It should be noted that for this calculation an automatic differentiation library is used to facilitate back-propagation (the derivatives could also be established manually). dUj, 3c; ' daf- It should be mentioned that there are variants of alpha composing and that any rendering method that is differentiable and has an aspect of opacity could be used instead.

[0044] This technique has the English name "alpha compositing" and is known in itself for example from the document HANRAHAN Pat, Image Compositing, CS 148 Introduction to Computer Graphics and Imaging, Lecture 14, Winter 2009 (https: / / graphics.stanford.edu / courses / csl48-09 / lectures / imaging.pdf) or from the document https: / / en.wikipedia.org / wiki / Alpha_compositing.

[0045] The method includes an initialization phase and an optimization phase aimed at iteratively converging the opacity and texture parameters of the triangles towards an optimal configuration.

[0046] During step 120, computer 1 places on each triangle the image(s) corresponding to each triangle to modify the texture and opacity parameters of each triangle.

[0047] During the implementation of step 120 in the initialization phase, the same texture parameter value will be applied to all triangles crossed by the same observation direction or to all triangles covered by the same image: for example, all triangles are green or all triangles are red. Alternatively, to increase the convergence speed of the texture parameters towards an optimal configuration during the optimization phase, the program in the initialization phase calculates an average of the textures of the images covering the same triangle and applies this average to the texture parameter value of that triangle. Any other arbitrary value could nevertheless be used in the initialization phase.

[0048] Similarly, during the implementation of step 120 in the initialization phase, the same opacity parameter value will be applied to all triangles: for example, the opacity parameter value is set to 0.5 because this will be closer to reality than having all triangles transparent or opaque. Choosing such an intermediate value allows the opacity parameters to converge more quickly towards an optimal configuration during the optimization phase, thus leading to a faster three-dimensional model. However, other possibilities exist. For example, the initial opacity parameter value for each triangle depends on the triangle's size: the smaller the triangle, the more points there are in the area where it lies, and the greater the likelihood that the triangle will be visible. An arbitrary value could also be chosen.

[0049] At step 120, the program on computer 1 therefore assigned to each triangle a value for the opacity parameter and a value for the texture parameter.

[0050] This gives us a rendering of the mesh forming a rough model.

[0051] The draft model is then compared to the images and a consistency score is calculated between the draft model and the images (step 130). This consistency score is, for example, a difference calculated by the least squares or MSE (mean square error) method and is then compared to a predetermined threshold corresponding to acceptable consistency, which depends on the accuracy of the three-dimensional model that is desired.

[0052] The draft resulting from an initialization phase, the consistency score is below the predetermined threshold and the program begins the optimization phase by starting a loop (140) to return to the determination of opacity / texture (step 120).

[0053] The optimization phase is arranged to modify both the texture parameter and the opacity parameter of the triangles: the idea is to obtain values ​​of the texture and opacity parameters of each triangle such that the model draft best covers the images.

[0054] The loop implements a gradient backpropagation method and corrects the parameters of each triangle according to the importance of each triangle's contribution to a pixel. This method iteratively converges the opacity and texture parameters of the triangles towards an optimal configuration. This amounts to finding a minimum of a cost function. The gradient backpropagation method is implemented here by automatic differentiation computer programs such as those found in the software library of the TORCH deep machine learning system. When errors accumulate over a series of several images, it is possible to use an optimization procedure to adjust the triangle parameters in order to minimize the error using the least-squares method.

[0055] In step 120, the program on computer 1 therefore assigned each triangle a new value for the opacity parameter and a new value for the texture parameter. A value c can thus be determined for each pixel and according to each observation direction.

[0056] This gives us a rendering of the mesh forming a new draft model: on [Fig.5], we can see, on the left part, the mesh being created and, on the right part, the draft model.

[0057] The draft model is then compared to the images and the consistency score is calculated between the draft model and the images (step 130).

[0058] If the consistency score is below the predetermined threshold, the program uses loop (140) to return to the opacity / texture determination (step 120) to modify the texture parameter and the opacity parameter of the triangles once again.

[0059] It is understood that in the process of the invention, the vertices of the triangles and the triangles themselves remain the same; only the opacity and texture parameters will be modified.

[0060] When the consistency score reaches the predetermined threshold, the surface is extracted from the draft model by choosing the triangles with the highest opacity (step 150), i.e. the lowest opacity parameter value, and this surface becomes the three-dimensional digital model of wing A.

[0061] It should be noted that the method of the invention combines known rendering techniques such as tetrahedralization and other rasterization, ray tracing, texture mapping, Y-alpha compositing...

[0062] Of course, the invention is not limited to the embodiment described but encompasses any variant falling within the scope of the invention as defined by the claims.

[0063] In particular, the installation for implementing the method of the invention may be different from that described.

[0064] Part of the processing can be done in the arm or, on the contrary, all the processing is carried out in the computer 1.

[0065] The three-dimensional measuring arm can be replaced by any means for determining the coordinates of points on the surface, by contact or without contact. For example, it is possible to use a portable lidar that incorporates means for detecting its own position (an inertial measurement unit, for example) or that is associated with tracking means.

[0066] Photographs can be taken simultaneously with distance measurements or at another time. It is simply necessary to know the point from which each photograph was taken. The photographs can be captured using a photogrammetric interval imaging (SFR) method (Structure from Motion). A three-dimensional measuring arm and a dedicated mount for an image capture device could be used, the position of which is known at each shot in a common coordinate system with the three-dimensional measuring arm.

[0067] The texture parameter may include one or more values, or refer to a texture atlas to which computer 1 has access.

[0068] The least squares method is not mandatory and any other method allowing a consistency check to be carried out is usable, such as for example an entropy minimization method.

[0069] Since it may be difficult to define a threshold corresponding to sufficient consistency between the images and the model draft, this consistency can be checked using another method.

[0070] In the process of the invention, during the initialization phase, one can choose the opacity and then project the texture to have a better initialization of the texture or, conversely, choose the texture and then project the opacity to have a better initialization of the opacity.

Claims

Demands

1. A method for numerically modeling a three-dimensional surface, comprising the steps of: determining point coordinates on the three-dimensional surface to obtain a point cloud and capturing images of the three-dimensional surface, constructing a mesh of triangles on the point cloud by tetrahedrization, projecting the mesh onto the images and simultaneously determining an opacity parameter and a texture parameter of the triangles by making at least a part of the images and the mesh coincide in order to obtain a draft three-dimensional model, comparing the draft three-dimensional model to the images and verifying consistency between the draft three-dimensional model and the images, returning to the determination of the opacity and texture parameters of the triangles if the consistency is less than an expected consistency, extracting the surface from the draft model to form the three-dimensional model otherwise.

2. A method according to claim 1, comprising an operation of ordering triangles, according to at least one observation direction and from an observation position, for each pixel of each image and an opacity calculation for each of said pixels.

3. A method according to claim 1 or 2, wherein the opacity is a sigmoid function varying between -oo and +oo.

4. A method according to any one of the preceding claims, wherein the tetrahedrization comprises the operation of determining a bounding box around the point cloud and the operation of constructing tetrahedra on the point cloud from each angle of the bounding box.

5. A method according to any one of the preceding claims, comprising a loop to return to the determination of the opacity and texture parameters of the triangles if the consistency is less than an expected consistency, the loop implementing a gradient backpropagation method to iteratively converge the opacity and texture parameters of the triangles to an optimal configuration.

6. A method according to claim 5, wherein the comparison of the model draft to the images gives rise to the calculation of a consistency score.

7. A method according to claim 6, wherein the consistency score is, for example, an error calculated between the model draft and the images by the least squares method.

8. A method according to claim 7, wherein when errors are accumulated over a series of several images, an optimization procedure is used to adjust the parameters of the triangles in order to minimize the error by least squares.

9. A method according to any one of the preceding claims, comprising the step of implementing an external heuristic to eliminate triangles from the triangular mesh least likely to belong to the three-dimensional surface, the heuristic involving the conditions for obtaining the point cloud and at least one characteristic of the triangles.

Images