Processing 3D signals of geometric attributes on real objects

Abstract

To provide a computer-implemented method for processing a 3D signal of a shape attribute over a real object.SOLUTION: A method comprises providing a graph having nodes and arcs and values representing a distribution of the shape attribute. Each node represents a respective point of a measured 3D discrete representation of the real object. Each arc connects two nodes representing neighboring points of the discrete representation. Each value is associated to a respective node of the graph and represents the shape attribute at the respective point represented by the respective node. The method further comprises modifying the set of values by minimizing an energy defined on a Markov Random Field formed on the graph. The energy penalizes, for each arc connecting a first node to a second node, the first node being associated to a first value and the second node being associated to a second value, a highness of an increasing function of: a distance between the first value and the second value, a distance between a first point and a medial geometrical element of the discrete representation, and a distance between a second point, represented by the second node, and the medial geometrical element.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present invention relates to the field of computer programs and systems, and more particularly to a method, system and program for processing 3D (three-dimensional) signals of geometric attributes on a real object. [Background technology]

[0002] Many systems and programs are offered on the market for designing, engineering and manufacturing objects. CAD is an acronym for Computer-Aided Design. It relates, for example, to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering. It relates, for example, to software solutions for simulating the physical behavior of a future product. CAM is an acronym for Computer-Aided Manufacturing. It relates, for example, to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays a key role regarding the efficiency of the technology. These techniques may be incorporated into Product Lifecycle Management (PLM) systems. PLM refers to a business strategy where companies share product data, apply common processes and leverage corporate knowledge to support the development of a product from its conception to the end of its life, across the concept of an extended enterprise. The PLM solutions offered by DassaultSystemes (trademarks CATIA, ENOVIA, DELMIA) provide an Engineering Hub to organize product engineering knowledge, a Manufacturing Hub to manage manufacturing engineering knowledge, and an Enterprise Hub to enable enterprise integration and connection to both the Engineering and Manufacturing Hubs. All of this systems provide an open object model that links products, processes and resources, enabling dynamic knowledge-based product creation and decision support to drive optimized product definition, manufacturing preparation, production and service.

[0003] In this and other contexts, processing 3D signals of shape attributes on real objects is often very useful.

[0004] Existing methods for processing 3D signals of shape attributes on real objects suffer from a lack of accuracy.

[0005] Within this context, improved methods for processing 3D signals of shape attributes on real objects are needed. [Prior art documents] [Patent documents]

[0006] [Non-Patent Document 1] Junjie Cao & al. Point Cloud Skeletons by Laplacian-Based Contraction (2010) [Non-Patent Document 2] Amenta & al., Surface reconstruction by Voronoi filtering (1999) [Non-Patent Document 3] Andrea Tagliasacchi & al, 3D Skeletons: State of the Art Report, DOI: 10.1111 / cgf.12865, EUROGRAPHICS 2016 [Non-Patent Document 4] Daniel Rebain & al, LSMAT Least Squares Medial Axis Transform, DOI:10.1111 / cgf.13599 [Non-Patent Document 5] ThomasDelame&al, Structuring 3D medial skeletons: a comparative study. Symposium on Vision, Modeling and Visualization, October 2016, Bayreuth, Germany, pp. 1-8, 10.2912 / vmv.20161336, hal-01359738 [Non-Patent Document 6] D. Thuerck & al., A fast and massively parallel solver for large scale applications [Non-Patent Document 7] D. Rebain & al, LSMAT Least Squares Medial Axis Transform, 2019 [Non-Patent Document 8] Amenta&al, Surface Reconstruction by Voronoi Filtering, 1999 Summary of the Invention [Means for solving the problem]

[0007] Thus, a computer-implemented method for processing 3D signals of shape attributes on a real object is provided. The method includes providing a graph having nodes and arcs. Each node represents a respective point of a measured 3D discrete representation of the real object. Each arc connects two nodes representing adjacent points of the discrete representation. The method further includes providing a set of values ​​representing a distribution of the shape attributes on the real object. Each value is associated with a respective node of the graph and represents the shape attribute at each point represented by the respective node. The method further includes modifying the set of values ​​by minimizing an energy defined in a Markov random field formed on the graph. The energy may be calculated as follows: for each arc connecting a first node to a second node, the first node is associated with a first value and the second node is associated with a second value: A penalty is imposed on the height of an increasing function of the distance between the first value and the second value, the distance between the first point represented by the first node and an inner geometric element of the discrete representation, and the distance between the second point represented by the second node and an inner geometric element.

[0008] The method includes one or more of the following:

[0009] The Markov Random Field has target values ​​representing a target distribution of shape attributes, and the energy further penalizes, for each node, the height of another increasing function of at least one distance between the value associated with the node and at least one target value.

[0010] - Another increasing function is: the distance between the value associated with the node and the first target value; the distance between the value associated with the node and a second target value; is an increasing function of

[0011] the at least one target value comprises at least one value of a geometrically calculated shape attribute at each point represented by the node and / or at least one value of a shape attribute inferred by a neural network at each point represented by the node.

[0012] Another increasing function also quantifies the correspondence between the values ​​contained in the at least one target value.

[0013] said another growth function is a function of the product between the correspondence and said at least one distance.

[0014] - An inner geometric element is the inner axis of a discrete representation or the inner surface of a discrete representation.

[0015] - The discrete representation is a 3D point cloud.

[0016] - The discrete representation is derived from photogrammetry and / or scanning.

[0017] - The shape attributes are the normal, curvature, or type of primitive.

[0018] - A real object contains at least one sharp feature.

[0019] The method further comprises segmenting the discrete representation based on the set of modified values.

[0020] Further provided is a computer program comprising instructions for carrying out this method.

[0021] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0022] There is further provided a system including a processor and a graphical user interface coupled to a memory, the memory having a computer program recorded thereon. [Brief description of the drawings]

[0023] Embodiments of the invention will now be described, by way of non-limiting examples, with reference to the accompanying drawings, in which: [Figure 1] FIG. 1 is an illustration showing a non-limiting example method. [Diagram 2] FIG. 1 is an illustration showing a non-limiting example method. [Diagram 3] FIG. 1 is an illustration showing a non-limiting example method. [Figure 4] FIG. 1 is an illustration showing a non-limiting method example. [Diagram 5] FIG. 1 is an illustration showing a non-limiting example method. [Figure 6] FIG. 1 is an illustration showing a non-limiting example method. [Figure 7] FIG. 1 is an illustration showing a non-limiting example method. [Figure 8] FIG. 1 is an illustration showing a non-limiting example method. [Figure 9] FIG. 1 is a block diagram illustrating an example of a system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Detailed Description of the Invention A computer-implemented method for processing 3D signals of shape attributes on a real object is proposed.

[0025] The method includes providing a graph having nodes and arcs. Each node represents a respective point of the measured 3D discrete representation of the real object. Each arc connects two nodes representing adjacent points of the discrete representation. The method further includes providing a set of values ​​representing a distribution of shape attributes on the real object. Each value is associated with a respective node of the graph and represents a shape attribute of a respective point represented by the respective node. The method further includes modifying the set of values ​​by minimizing an energy defined by a Markov random field formed on the graph. The energy penalizes, for each arc connecting a first node to a second node, the first node associated with a first value and the second node associated with a second value, the height of an increasing function of the following: the distance between the first value and the second value, the distance between the first point represented by the first node and an inner geometric element of the discrete representation, and the distance between the second point represented by the second node and an inner geometric element.

[0026] This constitutes an improved method for processing 3D signals of shape attributes on real objects.

[0027] The method receives as input a set of values ​​representing the distribution of a shape attribute for a real object. Each value is associated with a node of a graph, the node representing a respective point of a measured 3D discrete representation of the real object, and the value representing a shape attribute of each point. Thus, there is a measured 3D discrete representation of the real object and a set of values, each representing a shape attribute (e.g., a value of the shape attribute) at a respective location of the measured 3D discrete representation. In other words, the set of values ​​forms a 3D signal of the shape attribute, the 3D signal corresponding to a measured 3D discrete representation (e.g., calculated from a 3D discrete table and / or measured simultaneously), and each value representing a measurement of the shape attribute at a respective location of the measured 3D discrete representation. In other words, the 3D signal is a measure of the distribution of the shape attribute over the real object.

[0028] The 3D signal, measuring the distribution of shape attributes on real objects, can be used (e.g. processed according to the method) in many contexts, such as computer vision, 3D solid modeling (CAD), 3D reconstruction, for example in fields such as virtual reality and augmented reality, immersive experiences of any kind, video games, 3D reconstruction and modeling of machine parts, buildings and any kind of objects. The 3D signal is used, for example, in a 3D solid modeling process and / or a 3D reconstruction process, for example to segment a discrete representation based on the 3D signal. The segmentation may be followed by a 3D reconstruction process of the discrete representation (for example, when the discrete representation is a 3D point cloud) and / or by the determination of a B-rep, of a CSG construction tree and / or of a feature tree representing the real object, known per se from the fields of 3D reconstruction and 3D solid modeling.

[0029] That said, the method modifies the set of values ​​by minimizing the energy defined in a Markov Random Field (hereinafter sometimes referred to as "MRF") formed on the graph. The energy penalizes the height of an increasing function of the distance between the first and second values ​​for each arc that connects a first node associated with a first value of the set to a second node associated with a second value of the set (i.e., an adjacent node of the first node). In other words, for each couple of adjacent points, the energy penalizes the disparity (e.g., incongruity, discrepancy) between the values ​​that respectively represent the shape attribute at these adjacent points. In other words, the energy penalizes brutal local variations (e.g., sharp and / or non-smooth local variations) in the 3D signal of the shape attribute. In other words, the energy penalizes non-smooth variations of the values ​​of the shape attribute measured at adjacent points of the 3D discrete representation. Thus, the result of the minimization (i.e., the set of modified values) is a modified (and thus processed) 3D signal of the shape attribute, where at least some of the (e.g., strict) brutal local variations have been relatively smoothed (e.g., with respect to the MRF's convergence criterion). Moreover, for this method, and as will be explained further below, in regions where sharp shape / geometric variations exist in the real world, excessive smoothing of local variations in the 3D signal of the shape attribute is avoided. These regions can be considered as regions where smoothing may be counterproductive for the shape attribute values ​​of interest. This is done, in particular, by using distances to inner geometric elements, as will be explained further below. Incidentally, MRFs are powerful tools, for which a certain number of efficient and fast solvers exist.

[0030] Currently, the measured 3D discrete representation originates from measurements performed by one or more physical devices (such as sensors) suitable for this task. The brutal local variations in the 3D signal of the shape attribute typically represent noise in the 3D signal of the shape attribute, said noise resulting from the measurements. In an example, the discrete representation is noisy. By minimizing an energy that penalizes at least some of these brutal local variations, the method modifies (e.g., iteratively) a set of values ​​such that (e.g., until) the local changes in the 3D signal of the shape attribute become relatively (e.g., with respect to a convergence criterion of the MRF) smooth. In other words, the method removes noise from the 3D signal of the shape attribute.

[0031] Moreover, the increasing function is also an increasing function of the distance between said first point and the inner geometric element of the discrete representation, and the distance between said second point and the inner geometric element. Now, a point with a small distance between said point and the inner geometric element is a point corresponding to a sharp feature of the real object, for example a point located on a sharp feature of the real object or a point located near a sharp feature of the real object. Thus, if either said first point or said second point corresponds to a sharp feature of the real object, the distance between this point and the inner geometric element is relatively small (for example compared to other distances between smooth features of the real object and other points corresponding to inner geometric elements), and therefore does not contribute much to the height of the increasing function. Thus, the height of the increasing function is penalized less when minimizing the energy, compared to other couples of first and second points, for example where both points correspond to smooth features of the real object. This is even more true when both points correspond to sharp features.

[0032] Thus, minimizing the energy ensures that too strong local variations in the set of values ​​are normalized, but with much less smoothing around values ​​that correspond to sharp features of real objects. In other words, the method denoises the 3D signal of the shape attribute of the real object, but does not accommodate the noise, but preserves the variations in the 3D signal that correspond to the sharp features of the real object as in the real world. As a result, the processed 3D signal of the shape attribute (i.e. the set of modified values) accurately represents the distribution of the shape attribute across the real object. In other words, real objects often contain one or more sharp features, and the method is able to denoise the 3D signal while preserving the signal variations due to the presence of these sharp features. In other words, in the real world, the shape attribute varies very smoothly in smooth parts of the real object and varies more brutally in regions of the real object's shape where sharp features are present, and the method processes the 3D signal of the shape attribute to (e.g. until) approach the distribution of the shape attribute as in the real world with respect to a convergence criterion (e.g. of the MRF). Thus, the method is accurate and provides a realistic denoising of the 3D signal.

[0033] The method is for processing 3D signals of shape attributes on a real object. By "real object" is meant any object in the real world. Thus, a real object is a three-dimensional solid (i.e., a 3D closed volume). In an example, the real object includes at least one sharp feature. A sharp feature is a layout of material within the real object that has a sharp (i.e., non-smooth, irregular) shape.

[0034] A real object may be a (e.g., mechanical) part or assembly of parts (or equivalently an assembly of parts, which may be considered as a part itself from the perspective of the method, or the method may be applied independently to each part of the assembly), or more generally, any rigid assembly (e.g., a mobile mechanism). A real object may be any (e.g., manufactured) product from a variety of unlimited industrial sectors, including aerospace, architecture, construction, consumer goods, high tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. Thus, a real object may represent an industrial product that is any machine part, part of a ground vehicle (including, for example, cars and light truck equipment, racing cars, motorcycles, trucks and motor equipment, trucks and buses, trains), part of an aircraft (including, for example, airframe equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), part of a naval vehicle (including, for example, naval equipment, commercial vessels, offshore equipment, yachts and workboats, marine equipment), general machine parts (including, for example, industrial manufacturing machinery, large mobile machines or equipment, installation equipment, industrial equipment products, metal fabrication products, tire manufacturing products), electric machine or electronic components (including, for example, consumer electronics, security and / or control and / or instrumentation products, computing and communications equipment, semiconductors, medical equipment and instruments), consumer goods (including, for example, furniture, household goods, gardening products, leisure products, fashion products, hard goods retailer products, soft goods retailer products), packaging (including, for example, food and beverage and tobacco, beauty and personal care, household goods packaging), etc. All of these objects often have one or more sharp features.

[0035] By "shape attribute" we mean an attribute that locally describes the shape of a real object. By "3D signal of a shape attribute on a real object" we mean a measured distribution of the shape attribute on the real object, i.e. a measured field of values, each representing the value of the shape attribute at a particular location on the real object. By "measured distribution" we mean that the distribution originates from physical measurements performed on the real object (e.g. by one or more physical sensors). The distribution is in examples directly measured and in other examples calculated from the physical measurements (e.g. by calculations performed based on measured points of the measured discrete representation). As mentioned above, the set of values ​​provided forms the 3D signal of the shape attribute. Providing the set of values ​​comprises measuring the set of values ​​on the real object or calculating the set of values ​​based on the measured discrete representation (e.g. measurement points of the discrete representation). Providing the set of values ​​is further described below.

[0036] As mentioned above, a shape attribute is an attribute that locally describes the shape of an object. A shape attribute of a given location on a real object is any set of one or more parameters and / or labels that specify the characteristics of the local shape of the real object at the given location. A shape attribute is, for example, a normal vector in the normal case, i.e., for example, with respect to the outward orientation of the real object. Alternatively, a shape attribute is any curvature on a surface, for example, a Gaussian curvature, a principal curvature, or a mean curvature. Alternatively, a shape attribute is a type of primitive, i.e., a shape attribute forms the local shape of a real object and can locally describe a type of geometric primitive (e.g., cube, cylinder, sphere, cone, pyramid, ellipsoid, or torus).

[0037] Next, the provision of the graphs and sets of values ​​is described. Since a measured 3D discrete representation of a real object is involved in this provision, then the measured 3D discrete representation of the real object is discussed.

[0038] A 3D discrete representation of a real object, as used herein, is a data structure that includes a discrete set of data. Parts of the data represent respective geometric entities (e.g., points, surfaces of the 3D shape of the real object located in 3D space). Each geometric entity represents a respective position of the 3D shape (in other words, a respective portion of the material that makes up the solid represented by the 3D shape). The aggregation (i.e., combination or juxtaposition) of the geometric entities completely represents the 3D shape. Any discrete geometric representation herein includes, in examples, a number of such data that exceeds 100 (e.g., higher than 1000).

[0039] The 3D discrete representation of the real object is a 3D point cloud in an example, where each geometric entity is a point. The 3D discrete representation of the real object is a 3D mesh in another example, where each geometric entity is a mesh tile or face. The 3D mesh here is regular or irregular (i.e. composed of faces of the same type or not). Any 3D mesh herein is a polygonal mesh, for example a triangular mesh. Alternatively, any 3D mesh herein is a B-Rep. Any 3D mesh herein can be obtained from a 3D point cloud, for example by triangulating the 3D point cloud (e.g. with a 3D Delaunay triangulation and a method for extracting the associated facets).

[0040] By "measured" it is meant that the 3D discrete representation of the real object is derived from (i.e. determined from) physical measurements of the real object, for example within a 3D reconstruction process. The 3D reconstruction process includes providing a real object, providing one or more physical sensors, each configured to acquire a respective physical signal, and acquiring the one or more respective physical signals by manipulating the one or more physical sensors on the real object (i.e. scanning the real object with each sensor). The 3D reconstruction can then automatically determine a 3D point cloud and / or a 3D mesh based on the measurements according to any known technique. The one or more sensors include one or more portable tray scans. The one or more sensors additionally or alternatively include one or more metrology scans.

[0041] The method may in particular comprise providing a measured 3D discrete representation, e.g. prior to providing the graph and the set of values. Providing the 3D discrete representation may comprise performing said physical measurements on the real object, e.g. by performing the 3D reconstruction process as described above. Alternatively, providing the measured 3D discrete representation may comprise retrieving the 3D discrete representation from a (e.g. remote) memory, where the 3D discrete representation has been stored after acquisition, e.g. by performing physical measurements as described above.

[0042] Next, the provision of graphs will be described.

[0043] A measured 3D discrete representation of a real object, whether a 3D mesh or a 3D point cloud, has points each representing a specific location on the real object. The graph captures the topology of the real object represented by the discrete representation, with nodes each representing a point of the 3D discrete representation and arcs each connecting neighboring points of the discrete representation. By "topology of real object" we mean "topology with respect to the 3D discrete representation of the real object", which is different from, for example, the topology with respect to the B-rep representation. Neighbors of a point (also called "neighbors") are the points that are closest to that point, for example with respect to a given distance (e.g. a known distance in 3D). For example, neighbors of a point may be points whose distance to the point is less than a predefined threshold. The threshold may depend on the set of values ​​provided as input. For example, the threshold may depend on whether the set of values ​​provided originates from LIDAR from a scan of a small mechanical part. The threshold may actually depend on the particular case and the acquisition method of the set of values. The threshold may be related to the density of measurements (of points) provided by the acquisition method. If a first point belongs to the neighbors of a second point, then the first point and the second point are neighbors, which means that the second point belongs to the neighbors of the first point.

[0044] Providing the graph includes constructing the graph. Alternatively, providing the graph includes retrieving the graph from a (e.g., distant) memory where it was stored after the graph was created. Constructing the graph includes associating a respective node with each point of the discrete representation. Creating the graph may then include retrieving each couple of adjacent points in the discrete representation. Constructing the graph then includes connecting the nodes of the graph that each represent a couple of points by an arc in the graph.

[0045] In examples, the discrete representation is a 3D mesh, which may be vertices, edges, faces, or mesh tiles. In these examples, "two adjacent points" may represent two vertices that share an edge (i.e. are connected by an edge) and constructing the graph involves creating arcs that connect the nodes that represent the two vertices, respectively, for each couple of adjacent points. That is, in this case, the arcs in the graph correspond to the edges in the 3D mesh that are shared by the two vertices, and the arcs connect the two nodes that represent the vertices.

[0046] In other examples, the discrete representation is a 3D point cloud. In these examples, constructing the graph can consist of, for each point of the 3D point cloud, finding neighbors of the point and creating arcs between the point and at least some of the established neighbors. Finding neighbors includes using any known method suitable for finding neighbors of a point in a 3D point cloud, such as a nearest neighbor search algorithm. In these examples, the discrete representation is a 3D point cloud, constructing the graph can be performed by using any known method suitable for constructing a graph with nodes and arcs, each node representing a respective point of the 3D point cloud, and each arc connecting two nodes representing adjacent points of the 3D point cloud. For example, constructing the graph can be performed using the method described in (Non-Patent Document 1), which is incorporated herein by reference.

[0047] In an example example where the discrete representation is a 3D point cloud, constructing the graph includes connecting the points with some of their neighbors. The buildings can be constructed by estimating, for each point, its tangent plane and then projecting the neighbors of the point in that plane. The buildings can then include determining a 2D Delaunay triangulation of the projected points, generating a reduced flat mesh that includes the point and its neighboring projections. The buildings can be constructed by using that mesh to determine neighbors that connect the point. Specifically, the buildings can include connecting the nodes associated with the points to the graph nodes associated with the points of the point cloud that contributed to the projection on the tangent plane and were ultimately connected to the points of the 2D mesh embedded in the tangent plane via the 2D Delaunay triangulation. This method can be performed in another context in (Non-Patent Document 1), which is incorporated herein by reference.

[0048] Next, providing a set of values ​​will be described.

[0049] The set of values ​​represents the measured distribution of the shape attribute for the real object. As mentioned before, this means that each value in the set of values ​​represents a shape attribute at each position of the discrete representation. In other words, the values ​​are the values ​​of the shape attribute at each position of the discrete representation. The values ​​in the set of values ​​are scalar values, vector values, or label values. In the example, the MRF works on labels. In such a case, the method includes reducing the values ​​in the set of values ​​to labels by quantization, if the values ​​are scalar or vector values. If the values ​​are vector values, the minimization can be performed a number of times equal to the dimension of the corresponding vector space.

[0050] Providing the set of values ​​includes calculating a set of values ​​from the 3D discrete representation, where the calculation is performed by any suitable method. The calculation of the set of values ​​can be performed based on the discrete representation, e.g. based on the points of the discrete representation. For example, the calculation of the set of values ​​includes, for each point of at least some (e.g. all) of the discrete representation, calculating a value of a shape attribute at the point based on the point and / or neighboring points of the point.

[0051] The calculation of the value may be performed geometrically, i.e. by one or more known geometric methods. In such methods, the value is calculated by relying solely on geometry, e.g. by calculating one or more geometric formulas and / or by solving one or more geometric problems. For example, the calculation of the value may include one or more of the following:

[0052] - Fit a plane to a set of points consisting of a point and its neighbors, and use the normal of the plane as the value; - Calculate the curvature of the point as a value by applying the well-known cotangent method for calculating the curvature of a point in a 3D representation to the point (calculated as described above) and its neighbors, and / or - Fit a primitive, such as an ellipsoid, to a set of points consisting of the point and its neighbors, and calculate the point normal and principal curvatures based on the fitted primitive.

[0053] Alternatively, the value is inferred by any known neural network trained to infer the value of the shape attribute of any point in the discrete representation based on the point and / or its neighbors. In an example, the neural network is PCPNet, trained by a training supervisor based on a dataset of example 3D point clouds where the "terrain truth" for the value of the shape attribute is known.

[0054] Alternatively, providing may comprise measuring a set of values ​​of a real object, e.g. when providing a 3D discrete representation by physical measurement as described above. In either case, the set of values ​​forms a 3D signal of a geometric attribute on the real object.

[0055] Alternatively, providing the set of values ​​may comprise retrieving the set of values ​​from a stored (e.g. remote) memory, e.g. following calculation or measurement thereof, the set of values ​​forming a 3D signal of the geometric attribute on the real object.

[0056] Next, we explain the minimization of energy. Before that, we will briefly explain the concept of Markov random fields.

[0057] As is widely known, Markov Random Fields are tools used to infer the most likely configuration of labels (or discrete values) given their probability distribution on a graph, by minimizing the energy, with a smoothing term (also called "smoothness term") and a cost term (and optionally a regularization term). As such, they are a good tool to smooth out in the results of a computation some imperfections that may have been caused by excessive noise in the input of the computation. As is known per se, behind Markov Random Fields is the idea that the problem can be modeled by the probability that a node of a graph belongs to a certain class (the target class) and the probability that this node belongs to a class different from its neighbors, modeled by minimizing the following type of energy:

[0058]

number

[0059] To that end, the energy includes a cost term (first sum in equation (1)) capturing, for each node, the probability of the node belonging to a particular class (the target class), and a smoothness term (second sum in equation (1)) capturing, for each couple of neighboring nodes, the probability that these nodes belong to different classes. The scalar γ determines the amount of contribution of the smoothness term in the energy: the larger γ, the more smoothness contributes to the energy. The scalar γ is positive and, for example 1, is smaller than a predefined number. The energy minimized by this method is of type (1). In the example method, γ is equal to 1.

[0060] For this method, the problem to be modeled can be described as follows: there is a measured 3D discrete representation of a real object, a discrete representation with points, and a set of values ​​that represent the distribution of a shape attribute on the real object. Each value represents the value of the shape attribute at a respective point of the discrete representation. The minimization objective is to modify the set of values ​​such that: objective 1) the set of values ​​remains close to the set of provided values ​​(i.e. the 3D signal of the shape attribute measured / calculated from the discrete representation does not change too much), and objective 2) local strong variations in the set of values, i.e. strong variations of the values ​​representing the shape attribute at adjacent points of the discrete representation, are smoothed out except for regions of the discrete representation that represent sharp features of the 3D discrete representation. To that end, the method modifies the set of values ​​by minimizing an energy defined on a random Markov random field formed on the provided graph. The energy includes a cost term that captures objective 1) and a smoothness term that captures objective 2). The cost term of the energy is any known cost term that can capture objective 1). We now describe the smoothness term.

[0061] The energy penalizes the height of an increasing function, where for each arc connecting a first node to a second node, the first node is associated with a first value and the second node is associated with a second value, whereby the first value represents a shape attribute of a first point represented by the first node and whereby the second value represents a shape attribute of a second point represented by the second node. Also, the first point and the second point are neighbors because their representative nodes are connected by an arc in the graph. The smoothness term of the energy implements the penalty of the height of the increasing function. In other words, for each couple of neighbors represented by the first node and the second node, the smoothness term penalizes the height of the increasing function. "Penalizing the height of the growth function" means that the energy quantifies the height of the growth function (e.g., the height of the function value) for each couple of neighboring points represented by the first node and the second node, and that during the minimization, the values ​​representing the shape attribute at the neighboring points of the couple are modified if the growth function is too large (i.e., the values ​​are too large). They are modified to make the growth function smaller. Now, we will explain the growth function.

[0062] An increasing function is an increasing function of the distance between a first value and a second value. The distance can be any distance between values, such as a distance in one or more dimensions. Thus, as mentioned before, if the first and second values ​​representing the values ​​of a shape attribute at two adjacent points are too large from each other, the distance between them will be large. In such cases, the increasing function tends to be large. This means that a large distance leads to a large value of the function, but since the function depends on other parameters, as explained below, the value of the function may still be relatively small.

[0063] In fact, this function is also an increasing function of the distance between the first point (of two adjacent points represented by the first node and the second node) and an inner geometric element of the discrete representation. The inner geometric element is any inner geometric element of the discrete representation. For example, the inner geometric element is an inner axis of the discrete representation or an inner surface of the discrete representation. The distance between the first point and the inner geometric element can be any distance between the first point and the inner geometric element, such as the Euclidean distance between the first point and the inner geometric element. This distance is sometimes called the local feature size value (also called the "LFS value") of the first point.

[0064] Similarly, this function is also an increasing function of the distance between the second point (of two adjacent points represented by the first node and the second node) and an inner geometric element of the discrete representation. The distance between the second point and the inner geometric element can be any distance between the second point and the inner geometric element, such as the Euclidean distance between the second point and the inner geometric element. This distance is sometimes referred to as the local feature size value (also called the "LFS value") of the second point.

[0065] More generally, the LFS value of a point in the discrete representation is the distance between the point and an interior geometric element.

[0066] A small value of the distance between the first point and the inner geometric element, e.g. compared to other values ​​of the distance between other points and the geometric element, represents the fact that the first point corresponds to a position of the real object that belongs to a sharp part of the shape of the real object. In such a case, the distance between the first point and the inner geometric element does not contribute much to increasing the value of the function, but on the contrary, contributes to decreasing the value of the function. Similarly, a small value of the distance between the second point and the inner geometric element, e.g. compared to other values ​​of the distance between other points and the geometric element, represents the fact that the second point corresponds to a position of the real object that belongs to a sharp part of the shape of the real object. In such a case, the distance between the second point and the inner geometric element does not contribute much to increasing the value of the function, but on the contrary, contributes to decreasing the value of the function.

[0067] Therefore, it can be said that: if both the first and second points belong to a smooth part of the real object's shape, their LFS value will be large. In such a case, the height of the growth function mainly depends on the height of the distance between the first and second values. If this distance is large, it indicates that there is a large variation between the first and second values, which should not actually exist since both the first and second points belong to a smooth part of the real object's shape. This strong variation therefore corresponds to noise as mentioned before, so that the minimization will modify the first and / or second values ​​to make the distance between them smaller (e.g. with respect to the MRF's convergence criterion). In other words, if the real values ​​of the shape attribute of the neighboring points are close because they belong to a smooth part of the real object's shape, the minimization will tend to ensure that the corresponding values ​​in the set of values ​​stay close and / or similar. In other words, the strong local variations of the shape attribute in the 3D signal are smoothed when they correspond to noise.

[0068] Now, if the first (or second) point belongs to a sharp part of the shape of the real object, its LFS value will be small. In such a case, this LFS value contributes to a decrease in the value of the increasing function. Thus, the value of the increasing function will be penalized less during minimization, even if the distance between the first and second values ​​is large. If the LFS value of the second (or first) point is also small, i.e., if the second (or first) point belongs to a sharp part of the shape of the real object, the value of the increasing function will be penalized even less during minimization, even if the distance between the first and second values ​​is large. Thus, strong fluctuations between the values ​​of two neighboring points tend to be preserved (or at least smoothed less) during minimization, if at least one of the neighboring points belongs to a sharp part of the shape, and will be preserved even more (or at least smoothed less) during minimization, if both neighboring points belong to a sharp part of the shape. In other words, if the actual values ​​of a shape attribute of adjacent points are significantly different (e.g., if at least one (e.g., both) of the adjacent points belong to a sharp part of the real object's shape), minimization will tend to ensure that the corresponding values ​​in the set of values ​​will also be significantly different. In other words, local variations of strong shape attributes in the 3D signal tend to be preserved if they correspond to sharp features of the real object.

[0069] Thus, our method can be used to denoise 3D signals while preserving strong local variations in shape attributes when these strong variations correspond to sharp features of real objects in the real world.

[0070] Next, the LFS value will be further explained.

[0071] The LFS values ​​and inner geometric elements are well-known concepts and can be calculated by any known method or combination of known methods. Some of these methods are disclosed in non-patent literature 2, 3, 4, and 5, each of which is incorporated herein by reference. The method includes calculating the LFS values ​​of the points of the discrete representation by any known method, e.g., before modifying the set of values. The method further includes calculating the inner geometric elements by any known method, e.g., before calculating the LFS values.

[0072] The calculation of the LFS values ​​and / or inner geometric elements is based on the 3D discrete representation. For example, this means that the calculation of the LFS values ​​and / or inner geometric elements takes the discrete representation (e.g., points) as input and outputs the LFS values ​​and / or inner geometric elements according to any suitable known method (such as the methods described above). In yet other examples, the calculation of the LFS values ​​and / or inner geometric elements denoises (e.g., smooths, e.g., regularizes) the discrete representation and calculates the LFS values ​​and / or inner geometric elements based on the denoised discrete representation. This means that the calculation of the LFS values ​​and / or inner geometric elements takes the denoised discrete representation (e.g., points) as input and outputs the LFS values ​​and / or inner geometric elements according to any suitable known method (such as the methods described above). In these examples where the calculation of the LFS values ​​and / or inner geometric elements involves denoising the discrete representation, it should be understood that the energy minimization is performed based on the non-denoised discrete representation. By denoising the discrete representation before calculating the LFS values ​​and / or the inner geometric elements, the LFS values ​​and / or the inner geometric elements can be calculated more accurately. If one continues to perform the minimisation based on the non-denoised discrete representation, the LFS values ​​and / or the inner geometric elements used for the minimisation will be calculated more accurately, thus denoising the 3D signal of the shape attributes while more accurately preserving the sharp features as described above. Alternatively, the method can calculate the LFS values ​​without first smoothing / denoising the discrete representation. This is especially true if the calculation of the LFS values ​​is performed according to the LSMAT method described above.

[0073] Energy minimization will now be further explained.

[0074] The minimization can be performed by any known algorithm suitable for performing minimization, such as any known Markov Random Field solver. Markov Random Fields can be solved via alpha-expansion graph cuts, and solutions exist that exploit parallel processing with modern hardware, such as the fast and massively parallel solver for large scale of D. Thuerck & al, which is incorporated herein by reference. The concept of minimizing the energy defined by a Markov Random Field formed by a graph, as well as the algorithms thereof, are in fact well known in the field of Markov Random Fields and will not be described further.

[0075] Next, the smoothness term will be further explained.

[0076] In an example, the smoothness term could be the sum of all arcs in the graph of the following types of terms:

[0077]

number

[0078] where (p,q) is a couple made up of a first value p (associated with the first point) and a second value q (associated with the second point). |pq| is the distance between the first and second values. LFS p is the LFS value of the first point. q is the LFS value of the second point, where maxLFS is the maximum of all the LFS values ​​of the points in the discrete representation.

[0079] In another example, the smoothness term is the sum of all arcs in the graph of the following types of terms:

[0080]

number

[0081] where (p,q) is a couple made up of a first value p (associated with the first point) and a second value q (associated with the second point). |pq| is the distance between the first and second values. LFS p is the LFS value of the first point. q is the LFS value of the second point, where maxLFS is the maximum of all the LFS values ​​of the points in the discrete representation, and minLFS is the minimum of all the LFS values ​​of the points in the discrete representation.

[0082] Next, the cost conditions will be further explained.

[0083] As explained before, the cost term captures the requirement of the set of values ​​to be modified to remain close to the set of provided values. In other words, the smoothness term (when minimizing energy) tends to modify the set of values ​​so that local brutal fluctuations are smoothed out except in regions of the discrete representation corresponding to sharp features, whereas the cost term (when minimizing energy) tends to ensure that the set of modified values ​​stays close to the set of provided values. As mentioned before, the cost term can be any known cost term that ensures that the set of modified values ​​remains close to the set of provided values ​​during minimization.

[0084] In examples, the Markov random field has target values ​​that represent a target distribution of the shape attribute. In these examples, the energy further penalizes, for each node, the height of another increasing function of at least one distance between the value associated with the node and at least one target value.

[0085] In these examples, the target values ​​form one or more sets of target values, each of which represents a target distribution of the shape attribute. In other words, the minimization of the energy is such that the set of values ​​tends to be modified to be closer to (e.g., converge towards) one or more sets of target values. That is, the set of values ​​tends to be modified to be relatively closer to each set of one or more sets of target values. Each set of target values ​​is any set of values ​​that represents a distribution of the shape attribute that is similar to the distribution of the shape attribute represented by the set of provided values. For example, the values ​​of the set of target values ​​can represent the values ​​of the set of provided values. Additionally or alternatively, the values ​​of the set of target values ​​are calculated from a discrete representation. In such a case, the method includes, for each set of target values ​​of the one or more sets of target values, calculating the values ​​of each set of target values ​​from the discrete representation before minimizing, e.g. The calculation can be performed by a different method for each set of target values. Additionally or alternatively, the method includes sampling each target value of said set to a value that represents the value of the set of provided values. The values ​​in each set of target values ​​may be scalar values, vector values, or label values.

[0086] The calculation of each target value of the set of target values ​​may be performed based on the discrete representation, e.g., based on the points of the discrete representation. For example, the calculation of the set of target values ​​includes, for each point of at least some (e.g., all) of the discrete representation, calculating a value of a shape attribute at the point based on the point and / or the point's neighbors. The calculation of the values ​​may be performed geometrically, i.e., by one or more known geometric methods. In such methods, the values ​​are calculated by relying only on the geometry, e.g., by calculating one or more geometric formulas and / or by solving one or more geometric problems. For example, the calculation of the values ​​includes one or more of the following:

[0087] - Fit a plan to the set of points consisting of the point and its neighbors, and use the normal of the plan as the value; - Calculate the curvature of the point as a value by applying the well-known cotangent method for calculating the curvature of a point in a 3D representation to the point and its neighbors (calculated as described above), and / or - Fit a primitive, such as an ellipsoid, to a set of points consisting of the point and its neighbors, and calculate the normal or principal curvatures of the point based on the fitted primitive.

[0088] Alternatively, the value may be inferred by any known neural network trained to infer the value of a shape attribute of any point in the discrete representation based on the point and / or its neighbors.

[0089] Further in these examples, the energy further penalizes, for each node, the height of another increasing function of at least one distance between the value associated with the node and at least one target value. The cost term of the energy performs a penalty on the height of the increasing function. In other words, for each node of the graph, the cost term penalizes the height of the another increasing function. "Penalizing the height of the another increasing function" means that the energy quantifies, for each node, the height of the another increasing function (e.g., the height of the value of the another increasing function), and during minimization, if the another increasing function is too large (i.e., the value is too large), the value representing the shape attribute at the point represented by the node is modified. The another increasing function is modified to be smaller. Next, the another increasing function will be described.

[0090] The further increasing function is a function of at least one distance between the value associated with the node and at least one target value. This means that the further increasing function is a function of one or more distances, each between the value associated with the node and a respective target value (e.g. a value belonging to a respective set of the one or more sets of target values). Each of the one or more distances is any distance between values, such as any distance in one dimension or higher. Thus, for each of the one or more distances, if the value associated with the node and the respective target value are too different from each other, the distance between them is large. In such a case, the further increasing function is also large. As a result, the values ​​associated with the node are modified during the minimization such that the value of the further increasing function is reduced. This tends to ensure that the modified set of values ​​is closer to the set of values ​​provided during the minimization. The reason is that for example inconsistencies (dissimilarity, disharmony) between the modified set of values ​​and the target value are penalized because they contribute to the increase of the further increasing function.

[0091] In the examples, at least one distance consists of one single distance. In other words, the target values ​​form a single set of target values ​​representing the set of provided values, as described above, and the cost term penalized for each node is another increasing function of said single distance between the value associated with the node and the target value of the single set of target values. In these examples, the cost term may be the sum for all nodes of the graph of the following types of terms:

[0092]

number

[0093] where p is the value associated with the node. |p-target| is the single distance between the value associated with the node and the target value, where target is the target value.

[0094] In another example, the different increasing functions are an increasing function of the distance between a value associated with the node and a first target value and an increasing function of the distance between a value associated with the node and a second target value.

[0095] Such an example will now be described.

[0096] In these examples, the target values ​​form a first set of target values ​​consisting of the first target values ​​and a second set of target values ​​consisting of the second target values. As explained before, both the first and second sets of target values ​​consist of values ​​representing the set of provided values. In these examples, at least one distance consists of (e.g., is within) a first distance and a second distance. The other increasing function is, in other words, a function of a first distance between the value associated with the node and the first target value (i.e., a value belonging to the first set of target values) and a function of a second distance between the value associated with the node and the second target value (i.e., a value belonging to the second set of target values). Thus, as mentioned above, the cost term penalizes inconsistencies (dissimilarity, incongruity) during minimization, for example between the set of modified values ​​and each of the first and second sets of target values. In other words, the energy minimization measures whether the set of provided values ​​remains close to both the first and second set of target values ​​when they are changed, and tends to change the values ​​of the modified set of provided values ​​when they become too inconsistent with either the first or second set of target values. In further words, the cost term ensures that the modified set of values ​​tends to remain close to two sets that both represent the set of provided values. This means that the cost term penalizes inconsistencies between the modified set of values ​​and two different sets of values ​​during minimization, but both represent the set of provided values. This strengthens the way the method tends to ensure that the set of modified values ​​and the set of provided values ​​remain close, since they both represent the set of provided values, but both of them represent the set of provided values. This further improves the accuracy of the method.

[0097] In an example, the at least one target value includes at least one value of a geometrically calculated shape attribute at each point represented by the node and / or at least one value of a shape attribute inferred by a neural network at each point represented by the node.

[0098] By "at least one value of a geometric attribute calculated geometrically" is meant at least one value of a geometric attribute calculated by one or more geometric methods suitable for calculating values ​​of the geometric attribute, the method comprising the calculation of a target value, as previously described, comprising for each said at least one target value, geometrically calculating at least one value of the geometric attribute at each point represented by a node.

[0099] "At least one value of a shape attribute inferred by a neural network" means at least one value of a shape attribute generated by a neural network trained to calculate values ​​of a shape attribute. The neural network may be any neural network trained to calculate values ​​of a shape attribute. As mentioned above, the method includes calculating target values, which includes, for each of said at least one target value, calculating at least one value of a shape attribute at each point represented by a node by applying a neural network as previously described.

[0100] The method can therefore use as target values ​​geometrically calculated values ​​and / or values ​​inferred by a neural network. It is flexible such that the method relies on at least two different ways of calculating the target values ​​and can therefore benefit from the advantages provided by at least one of these two methods, such as accuracy, speed and / or robustness. The method can further combine these advantages.

[0101] In examples, the at least one target value includes at least one value of a geometrically calculated shape attribute at each point represented by the node and at least one value of a shape attribute inferred by a neural network at each point represented by the node. In these examples, the method uses as the target value a value resulting from a fusion of the geometrically calculated value and the value inferred by the neural network. This promotes the closeness (e.g., convergence) of the set of modified values ​​to a value that is considered more reliable than other values ​​(values ​​for which the geometric method and the neural network provide similar estimates). This improves the accuracy of the method.

[0102] In an example, the other increasing function is an increasing function of the distance between the value associated with the node and a first target value (first distance as described above) and the distance between the value associated with the node and a second target value (second distance as described above). The at least one target value includes at least one value of a geometrically calculated shape attribute at each point represented by the node and at least one value of a shape attribute inferred by a neural network at each point represented by the node. Specifically, in such a case, the first target value is at least one value of a geometrically calculated shape attribute at each point represented by the node and the second target value is at least one value of a shape attribute inferred by a neural network at each point represented by the node.

[0103] In examples, the further increasing function also quantifies the correspondence between the values ​​contained in the at least one target value. Examples of these are described below.

[0104] As mentioned above, the target values ​​may form one or more sets of target values. The other increasing function is one or more increasing functions of distances between the value associated with the node and each of the target values ​​belonging to each of the one or more sets of target values. Each of the respective target values ​​represents a value of a shape attribute at a position of the discrete representation corresponding to a point represented by the node. Now, correspondence is a term or quantity that quantifies the correspondence (e.g., similarity, e.g., substantial equivalence) between the respective target values. The term is a large term when the respective target values ​​correspond to each other, e.g., are similar, e.g., are equal or substantially equal. And it is a small term when the respective target values ​​do not correspond to each other, e.g., are too different or significantly different. This allows the distance to the target values ​​to have an additional influence when minimizing the energy. Additionally or alternatively, the term is a non-zero term when the respective target values ​​correspond to each other, e.g., are similar, e.g., are equal or substantially equal. And it is null when the respective target values ​​do not correspond to each other, e.g., are too different or significantly different. In such cases, energy may be said to capture the consideration that departures from mutually inconsistent target values ​​are not critical in anyway. The term may depend on the respective target values ​​and quantify the correspondence between the respective target values ​​based directly on the respective target values. Alternatively, the term may depend on other values ​​and quantify the correspondence between the respective target values ​​based on the other values.

[0105] Quantifying the correspondence between the values ​​contained in the at least one target value increases the robustness and accuracy of the method. Indeed, when the values ​​contained in the at least one target value do not correspond, the value of the other increasing function tends to decrease. Thus, even if at least one target value contained in the at least one target value is such that the at least one distance is large, the other increasing function does not necessarily tend to increase accordingly. This is because the lack of correspondence between the values ​​contained in the at least one target value tends to decrease the value of the function. This causes the cost term to give more strength to the target values ​​contained in the at least one target value that correspond to each other. In other words, the minimization tends to change the set of values ​​so that they remain close to these strong target values ​​(values ​​for which the stronger target values ​​are given, as described above). Here, each of these strong target values ​​relatively accurately represents the shape attribute at the location of the discrete representation. This is because the target values ​​contained in the strong target values ​​correspond to each other even if they are calculated differently, for example at least one geometrically and at least one as inferred by a neural network. In other words, the cost term promotes the closeness (e.g., convergence, e.g., compared to the convergence criterion of the MRF) of the set of modified values ​​during minimization to a set of target values ​​that adequately represents the values ​​of the set of provided values.

[0106] In an example, the other increasing function is a function of the product between the correspondence and the at least one distance. This is a particularly simple, robust and efficient way to give more strength to the target values, which are included in the at least one target value and which correspond to each other. Indeed, when the correspondence is small (e.g. null), the product tends to be similar, and so does the function. Thus, even if the at least one distance between the value associated with the node and the value of the at least one target value must be large, the value of the function is penalized less during minimization.

[0107] A further example of this method is now described.

[0108] In these examples, the other increasing function is an increasing function of a first distance between a value associated with the node and a first target value, and an increasing function of a second distance between a value associated with the node and a second target value. In these examples, the first target value is a value of a geometrically calculated shape attribute at each point represented by the node, and the second target value is a value of a shape attribute inferred by a neural network at each point represented by the node. In these examples, the other increasing function also quantifies the correspondence between the first target value and the second target value.

[0109] In these examples, the cost term tends to ensure that the set of values ​​modified during minimization remains close to both the geometrically calculated first target value and the neural network inferred second target value. Both the first target value and the second target value represent a distribution of shape attributes for real objects similar to that represented by the set of provided values. In other words, the MRF optimization according to this method uses as a target value a fusion of the geometrically calculated target value and the neural network inferred target value. This increases the robustness and accuracy of the method as at least two different types of target values, i.e., target values ​​obtained from two different computational sources and used to bring the set of modified values ​​closer to the set of provided values, are used. Furthermore, the method gives more strength to target values ​​where the two sources are consistent, i.e., target values ​​that yield similar values. This allows the benefits of the accuracy of both methods to be exploited, further increasing the robustness and accuracy of the method. In these examples, the said separate increasing function is a function of the product between the correspondence, the distance between the value associated with the node and the first target value, and the distance between the value associated with the node and the second target value. As mentioned before, this further increases the simplicity and robustness of the method.

[0110] In these examples, the cost term might be the sum of all nodes in the graph of the following types of terms:

[0111]

number

[0112] p is the value associated with the node; geomTarget is the first target value, calculated geometrically; neuralTarget is the second target value, inferred by the neural network; geomNormal is the geometrically calculated normal at the point represented by the node, and neuralNormal is the normal inferred by the neural network at the point represented by the node; |p-geomTarget| is the distance between the value associated with the node and the first target value; |p-neuralTarget| is the distance between the value associated with the node and the second target value; geomNormal.neuralNormal is the correspondence between the first target value and the second target value. Here, the shape attribute is the normal or any shape attribute closely related to the normal, calculated based on the normal, e.g. maximum curvature.

[0113] Yet in these examples, the cost term may instead be the sum of all nodes in the graph of the type term.

[0114]

number

[0115] p is the value associated with the node; geomTarget is the first target value, computed geometrically; neuralTarget is the second target value, inferred by the neural network; geomValueGaussian is a Gaussian distribution centered on geomTarget, where each of the values ​​represents a shape attribute and is computed geometrically at the point each is represented by a node; neuralValueGaussian is a Gaussian distribution centered on neuralTarget, where each of the values ​​represents a shape attribute and is inferred by the neural network at the point each is represented by a node; (i) is the bucket index of the histogram that covers the range of values ​​of the set of provided values; |p-geomTarget| is the distance between the value associated with the node and the first target value; |p-neuralTarget| is the distance between the value associated with the node and the second target value.

[0116]

number

[0117] is the correspondence between the first target value and the second target value, where the shape attribute can be any shape attribute, e.g., any shape attribute that is not related to normals.

[0118] The method can be applied in computer vision, 3D solid modeling (CAD) and / or 3D reconstruction. For example, the method can be applied in fields such as virtual reality or augmented reality, any kind of immersive experience, video games, 3D reconstruction or modeling of mechanical parts, buildings or any kind of object. The method can be included, for example, in 3D solid modeling processes and / or 3D reconstruction processes.

[0119] The method further includes, for example, segmenting the discrete representation based on the modified set of values. The segmentation of the discrete representation splits (e.g., decomposes, clusters) the discrete representation into smaller, meaningful, discrete sub-representations. The split is based on a set of values ​​of the shape attributes on the discrete representation, which is a modified set of values ​​in the case of the method. The splitting according to this method can be done by any known splitting method, is well known, and will not be further described herein. The segmentation is improved because the modified set of values ​​represents relatively accurately the distribution of the shape attributes across the real object as in the real world. The segmentation may be followed by a 3D reconstruction process of the discrete representation (e.g., if the discrete representation is a 3D point cloud), and / or by the determination of a B-rep, a CSG construction tree, and / or a feature tree representing the real object, known per se from the fields of 3D reconstruction and 3D solid modeling.

[0120] The implementation of the method will now be described.

[0121] This implementation is applicable in the fields of computer vision, 3D solid modeling (CAD) and 3D reconstruction, and is therefore useful in areas such as virtual and augmented reality (more generally, any kind of immersive experience), video games and 3D reconstruction and modeling of mechanical parts, buildings or any kind of object.

[0122] Evaluating local quantities or characteristics of noisy point clouds and meshes is a complex task. This implementation helps improve the quality of results for noisy inputs. This implementation relatively preserves detailed characteristics of regions of shapes where sharp features exist.

[0123] Markov Random Fields (MRFs) are known tools used to infer the most similar configuration of labels (variance values) given the likelihood of their distribution on a graph by minimizing the energy accompanied by a smoothing term and a cost term (and often a regularization term). This implementation involves a formulation of the MRF energy that uses local feature sizes in the smoothing term. If the input is noisy, the local feature sizes themselves will be noisy. In such cases, the local feature sizes can be calculated based on the smoothed changes in the point cloud. Even if the smoothing is a bit too strong and does not preserve the sharp geometric features of the input, this implementation still provides good results because it only uses it to identify areas worth smoothing and areas where it is dangerous to do so. Alternatively, the local feature sizes are calculated by a method that does not require the smoothed changes in the point cloud (such as the LSMAT method mentioned above). Furthermore, the cost term can use values ​​calculated via a purely geometric method and values ​​inferred by a neural network in a way that gives even stronger power to values ​​where both methods tend to match similar estimates and realize a fusion of the two estimates. Markov Random Fields can be solved via alpha-expansion graph cuts, and solutions exist that leverage parallel processing with modern hardware, such as in [6], which is incorporated herein by reference. In this implementation, any Markov Random Field solver can be used, but it is preferable to use a solver that can operate on hundreds of thousands of nodes with thousands of labels / values.

[0124] The formulation of the smoothing term in this implementation based on local feature size prevents this implementation from over-smoothing in regions where sharp geometric features are present. It therefore becomes a useful tool to smooth values ​​that are not expected to change smoothly from the sharp features, but where sharp changes are preserved in regions where sharp geometric features are present. Furthermore, when the cost term includes both geometrically calculated values ​​and values ​​inferred by the neural network, it promotes the closeness (e.g., convergence) to values ​​that are considered more reliable than others (values ​​where geometric methods and neural network inference provide similar estimates). This can be useful, for example, to improve the segmentation of point clouds, which is a classical step in the reconstruction or analysis of geometry, or models or scenes captured as Lidar PointClouds, reconstructed using measuring instruments or by photogrammetry. Figure 1 shows a screenshot of an image extracted from the documentation of a well-known computer vision library, showing such a segmentation. It is clear from Figure 1 that such segmentation can lead to primitive identifications that can be useful for model reconstruction beyond point clouds, robotic decision-making, or other forms of analysis.

[0125] Algorithms such as those used in the Point Cloud Library (PCL) (image in Fig. 1) produce poor quality results when the input point cloud is noisy. For example, a classic deficiency is to generate segmented results. In such situations, smoothing the input point cloud alone is not enough, as it may remove interesting features from the input. Markov Random Fields are also a good tool to smooth the output of algorithms that work on noisy point clouds, but care must be taken to ensure that not too much smoothing occurs in areas where no smoothing occurs. The energy formulation of this implementation is interesting from that point of view, as it ensures that much less smoothing occurs in areas where sharp shape features are present. In this implementation, we use a local feature size associated with each vertex in the energy formulation. The local feature size is the distance to the inner elements (axis or surface) of the point cloud (or mesh). At least a fairly good quality estimate of those inner elements is preferred. There are quite a few approaches focused on generating these even for noisy inputs, such as in [7], which is incorporated herein by reference. This implementation therefore extends the benefits of these regularization methods for medial axis elements to value fields where the methods operate on point clouds or meshes. In this implementation, the method uses two iterations of projections on adjacent mean planes.

[0126] Since neural networks have shown some strength in dealing with noisy inputs, it is also interesting to fuse predictions from neural networks with the output of more classical algorithms, using the aforementioned Markov Random Fields. This allows fusing the two predictions by giving more strength to matching predictions while applying less smoothing to regions corresponding to sharp features. Again, this can improve the segmentation of point clouds, for example.

[0127] This implementation can be applied to improve point cloud segmentation of noisy inputs and therefore all use cases that depend on such segmentation, such as geometry reconstruction.

[0128] The implementation will consist of four steps:

[0129] 1. Get a noisy point cloud or mesh 2. Estimate local quantities or properties (e.g. normals, curvature, surface type) 3. Calculate LFS 4. MRF Optimization As a result, this implementation generates a set of values ​​that represent the distribution of shape attributes across the real object, which is smoothed in areas where the estimate was affected by noise in the input, while preserving abrupt changes in sharp regions of the real object's shape.

[0130] 1. Obtain a noisy point cloud.

[0131] An example of such an input point cloud is shown in Figure 2, which shows a noisy point cloud.

[0132] 2. Estimate local quantities or properties (also called shape attributes).

[0133] As shown in FIG. 3, for example, we estimate normals for each point on the point cloud and show the normals inferred by a neural network.

[0134] 3. Calculate the local feature size.

[0135] Local feature size can be estimated by evaluating the distance to the pole (see, for details, "Pole Distance Estimation," IEEE Transactions on Pattern Recognition, vol. 13, No. 10, 2009, pp. 1171-1175, 2011, which is incorporated herein by reference). Figure 4 shows examples of such an evaluation, including a small local feature size (LFS) value of 40 and a high LFS value of 42. If the input is noisy, the LFS value may be inaccurate. Figure 5 shows an image showing LFS on a very noisy point cloud.

[0136] There is a long tradition of approaches focusing on the generation of medial elements (axes / surfaces) even for noisy inputs. One of the most recent is described, for example, in "Mean axis elements: a method for generating medial axis elements", IEEE Transactions on Image Processing, vol. 13, no. 1, 2011, which is incorporated herein by reference. Thus, somehow, this implementation extends the advantages of these regularization methods for medial axis elements to value fields where the methods operate on point clouds or meshes. As shown in Fig. 6, to compute the LFS seen in the image, this implementation uses two iterations of projections on adjacent mean planes.

[0137] The smoothed point cloud is only used to compute the inner elements, obviously, all computations are performed on the original point cloud. For very small sharp features that almost disappear in the point cloud noise, it is also possible that the MRF smoothing term can be very strong in those regions. For very noisy point clouds, this simple smoothing is insufficient and in the very degenerate case, this implementation can rely on approaches such as [7], which is incorporated herein by reference, which obtains the inner elements that are used to compute the local feature size used in the MRF. In some way, this kind of implementation extends the benefits of the regularization approach to compute the intermediate elements to compute the best approximation of the input.

[0138] As shown in Figure 6, the result is usable, and the smoothed point cloud is only used to calculate the LFS value. The work on the MRF is still performed on the original noisy point cloud. The calculated LFS value is only guaranteed if the smoothing term is very weak in areas where sharp geometric features exist.

[0139] The LFS value can be approximated via the distance to the nearest LFS pole, as shown in Figure 7. This forms a good approximation of the shape's inner surface and axis (again, for a more complete explanation, see "LFS Pole Distance Measurements," IEEE Transactions on Computer Vision, vol. 13, no. 10, 2003, which is incorporated herein by reference). Nevertheless, this implementation is independent of the method used to calculate the local feature size associated with the point. 4. MRF Optimization The idea behind Markov Random Fields is that we have a problem that can be modeled by the probability that a node in a graph belongs to a particular class (the target class) and the probability that the node belongs to a different class than its neighbors, by minimizing the energy:

[0140]

number

[0141] Intuitively, shape attributes vary very smoothly over smooth parts of a shape and more brutally in areas where sharp features are present. This implementation aims to smooth out the brutal variations in smooth parts of real object shapes that are a result of the noisy nature of the input. From the above discussion of the idea behind Markov Random Fields, this mathematical tool can help achieve that goal, as it is essentially a situation where the value to be assigned to a particular point is likely to be close to its current estimate, and is quite likely to be close to neighboring point values, except in parts of the shape where sharp features are present.

[0142] This implementation works specifically for point clouds. In the case of meshes, the graph used is either the graph formed by the vertices of the mesh or the graph formed by the faces of the mesh. In the case of point clouds, the graph is created by connecting the point with some neighbors. For each point, a tangent plane is estimated and the planes of the point's neighbors are projected onto that plane. Then, a 2D Delaunay of the projected points gives a reduced flat mesh that meshes the projection of the point and its neighbors. That mesh is used to decide which neighbors to connect the point to. In fact, the relevant nodes are connected to the graph nodes associated with the point of the point cloud, and these nodes contribute to the projection at the tangent location and are finally connected to the point of the 2D mesh constructed on the tangent plane via Delaunay. This method is implemented in (Non-Patent Document 1, incorporated herein by reference).

[0143] This gives a graph structure to the discrete representation. Figure 4 shows the local feature sizes and it seems desirable to smooth the values ​​in regions of high LFS 42 while preserving the strong fluctuations in regions of low LFS values ​​40. It is therefore interesting to formulate the smoothness term in such a way that it remains small in the blue regions even when the target classes of p and q are very different. The scalar values ​​are discretized and each discrete value is indeed one of the classes referenced in this implementation.

[0144] The formulation of the smoothness term is as follows:

[0145]

number

[0146] Or,

[0147]

number

[0148] Additionally, when using multiple inputs, more strength should be given to values ​​where the inputs match. The set of values ​​provided in this implementation is a scalar field of type "normal_x, normal_y, normal_z". The normals are discretized with 100 (or up to 1000) other indexed discrete values ​​that are different classes that can be assigned to a particular node.

[0149]

number

[0150] If only one input is used, the target energy term becomes (in the two case, equation 3 above actually replaces equation 4 below):

[0151]

number

[0152] The above formulation for the two input case can be applied to the case of normal fields "normal_x, normal_y, normal_z". Alternatively, the implementation can use it to work with other scalar fields, where estimating how similar the normals are indicates how much the input values ​​associated with a particular point or vertex are valued. Alternatively, for scalar fields not directly related to the normal, the implementation creates two Gaussians centered on the two input values ​​and takes the maximum of the multiplication of the two Gaussians.

[0153]

number

[0154] geomValueGaussian and geomTarget are functions related to the same value field defined on a point cloud or mesh, just as neuralValueGaussian and neuralTarget are also related to the same value field, and i is the bucket index of a histogram covering the range of values ​​in the provided set of values ​​(again, the values ​​are discretized and each discrete value is assimilated to a different class in a Markov random field).

[0155] FIG. 8 shows the results of the MRF.

[0156] The method is computer-implemented. This means that the steps of the method (or substantially all steps) are performed by at least one computer or any system. Thus, the steps of the method are computer-implemented, possibly fully automatically, or semi-automatically. For example, triggering of at least some steps of the method is performed via user-computer interaction. The level of user-computer interaction required depends on the level of automation anticipated and may be balanced against the need to implement user wishes. In examples, this level may be user-defined and / or predefined.

[0157] A typical example of a computer implementation of the method is executing the method in a system adapted for this purpose. The system includes a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program including instructions for carrying out the method. The memory may also store a database. The memory is hardware adapted for such storage, possibly consisting of several physically distinct parts (e.g. one for the program and one for the database).

[0158] FIG. 9 shows an example of a system where the system is a client computer system, eg, a user's workstation.

[0159] The client computer of this example comprises a central processing unit (CPU) 1010 connected to an internal communication bus 1000. A random access memory (RAM) 1070 is also connected to the bus. The client computer further comprises a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for embodying computer program instructions and / or data include all forms of non-volatile memory including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented or incorporated by specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer also includes a haptic device 1090 such as a cursor control device, a keyboard, and the like. A cursor control device is used in the client computer to allow a user to selectively place a cursor at any desired location on the display 1080. Additionally, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes some signal generating devices for inputting control signals to the system. Typically, the cursor control device is a mouse, and the buttons of the mouse are used to generate the signals. Alternatively, or in addition, the client computer system includes a sensitive pad and / or a sensitive screen.

[0160] The computer program includes instructions executable by a computer, the instructions including the means for causing the system described above to carry out the method. The program can be recorded on any data storage medium, including the memory of the system. The program can be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program is implemented as an apparatus, for example an article of manufacture tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be executed by a programmable processor executing a program of instructions to perform the functions of the method by acting on input data and producing output. Thus, the processor is programmable and coupled to receive data and instructions from, and to transmit data and instructions to, the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. Application of the program to the system in either case provides instructions for carrying out the method.

Claims

1. 1. A computer-implemented method for processing 3D signals of shape attributes on a real object, the shape attributes being normals, curvatures, or types of primitives, the computer-implemented method comprising: - providing a graph having nodes and arcs, each node representing a respective point of a measured 3D discrete representation of a real object, and each arc connecting two nodes representing adjacent points of said discrete representation; providing a set of values ​​representing a distribution of the shape attribute on the real object, each value being associated with a respective node of the graph and representing the shape attribute at the respective point represented by the respective node; Modifying the set of values ​​by minimizing an energy defined by a Markov random field formed on the graph, the energy being defined as, for each arc connecting a first node to a second node, the first node being associated with a first value and the second node being associated with a second value, the energy being: A distance between the first value and the second value; and a distance between a first point represented by the first node and an interior geometric element of the discrete representation; the modifying step penalizing the height of an increasing function of the distance between a second point represented by the second node and the inner geometric element; 23. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the Markov random field has target values ​​representing a target distribution of the shape attribute, and the energy further penalizes, for each node, the height of another increasing function of at least one distance between a value associated with the node and at least one target value.

3. The other increasing function is as follows: a distance between a value associated with the node and a first target value; a distance between a value associated with the node and a second target value; 3. The computer-implemented method of claim 2, wherein the function is an increasing function of .

4. 4. The computer-implemented method of claim 2 or 3, wherein the at least one target value comprises at least one value of the shape attribute geometrically calculated at the respective point represented by the node and / or at least one value of the shape attribute inferred by a neural network at the respective point represented by the node.

5. 5. The computer-implemented method of claim 2, wherein the further increasing function also quantifies a correspondence between values ​​included in the at least one target value.

6. 6. The computer-implemented method of claim 5, wherein the another increasing function is a function of a product between the correspondence and the at least one distance.

7. 7. A computer-implemented method according to claim 1, wherein the inner geometric element is an inner axis of the discrete representation or an inner surface of the discrete representation.

8. 8. A computer-implemented method according to claim 1, wherein the discrete representation is a 3D point cloud.

9. 9. A computer-implemented method according to any one of claims 1 to 8, wherein the discrete representation is derived from photogrammetry and / or scanning.

10. 10. The computer-implemented method of claim 1, wherein the real object comprises at least one sharp feature, the sharp feature being a layout of materials of the real object having an irregular shape.

11. 11. The computer-implemented method of claim 1, further comprising: segmenting the discrete representation based on the set of modified values.

12. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 11.

13. 13. A computer readable storage medium having the computer program of claim 12 recorded thereon.

14. 13. A system comprising a processor coupled to a memory and a graphical user interface, the memory having recorded thereon the computer program of claim 12.

Images