3D signals that process shape attributes on real objects

Abstract

This invention particularly relates to a computer-implemented method for processing 3D signals of shape attributes on a real object. The method includes providing a graph having nodes and arcs. Each node represents a corresponding point in a measured 3D discrete representation of the real object. Each arc connects two nodes representing adjacent points in the discrete representation. The method further includes providing a set of values ​​representing the distribution of shape attributes on the real object. Each value is associated with a corresponding node in the graph and represents the shape attribute at the corresponding point represented by that node. The method further includes modifying the set of values ​​by minimizing an energy defined on a Markov random field formed on the graph.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly to a method, system, and program for processing 3D signals of shape attributes on real objects. Background Technology

[0002] A variety of systems and programs are available on the market for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, which, for example, involves software solutions for designing objects. CAE stands for Computer-Aided Engineering, which, for example, involves software solutions for simulating the physical behavior of future products. CAM stands for Computer-Aided Manufacturing, which, for example, involves software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays a crucial role in the efficiency of the technology. These technologies can be embedded within Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge for product development from concept to its end-of-life (across extended enterprise concepts). PLM solutions offered by Dassault Systèmes (traded as CATIA, ENOVIA, and DELMIA) provide an engineering center for organizing product engineering knowledge, a manufacturing center for managing manufacturing engineering knowledge, and an enterprise center that enables the enterprise to integrate and connect to both the engineering and manufacturing centers. The system delivers an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, thereby driving optimized product definition, manufacturing preparation, production, and service.

[0003] In this context and other contexts, processing 3D signals of shape properties on real objects is often very useful.

[0004] Existing methods for processing 3D signals of shape attributes on real objects suffer from a lack of accuracy.

[0005] In this context, an improved method is needed for processing 3D signals of shape attributes on real objects. Summary of the Invention

[0006] Therefore, a computer-implemented method for processing 3D signals of shape attributes on a real object is provided. The method includes providing a graph with nodes and arcs. Each node represents a corresponding point in a measured 3D discrete representation of the real object. Each arc connects two nodes representing adjacent points in the discrete representation. The method further includes providing a set of values ​​that represent the distribution of shape attributes on the real object. Each value is associated with a corresponding node in the graph and represents the shape attribute at the corresponding point represented by that node. The method further includes modifying the set of values ​​by minimizing an energy defined on a Markov random field formed on the graph. For each arc connecting a first node to a second node, the energy penalizes the height of an increasing function of the following terms to the first node associated with the first value and the second node associated with the second value: the distance between the first and second values, the distance between the first point represented by the first node and an intermediate geometric element in the discrete representation, and the distance between the second point represented by the second node and an intermediate geometric element.

[0007] The method may include one or more of the following:

[0008] - A Markov random field has a target value representing a target distribution of shape properties, and for each node, the energy further penalizes the height of another increasing function associated with the value of that node and at least one distance between at least one target value;

[0009] - The other increasing function is the increasing function of the following terms:

[0010] o is associated with the distance between the value of this node and the first target value; and

[0011] o is associated with the distance between the value of this node and the second target value;

[0012] - At least one target value includes at least one value of a shape property that is geometrically calculated at the corresponding point represented by the node, and / or at least one value of a shape property that is inferred by a neural network at the corresponding point represented by the node;

[0013] -The other increasing function also quantifies the correspondence between values ​​included in at least one target value;

[0014] - The other increasing function is a function of the product between the correspondence and the at least one distance;

[0015] - Intermediate geometric elements are intermediate axes or intermediate surfaces in a discrete representation;

[0016] - Discrete representation is a 3D point cloud;

[0017] - Discrete representations originate from photogrammetry and / or scanning;

[0018] - The shape attribute is the type of normal, curvature, or primitive;

[0019] - Real objects include at least one sharp feature; and / or

[0020] - This method further includes segmenting the discrete representation based on the set of modified values.

[0021] A computer program is further provided, which includes instructions for performing the method.

[0022] A computer-readable storage medium having the computer program recorded thereon is further provided.

[0023] A system is further provided that includes a processor coupled to a memory and a graphical user interface on which the computer program is recorded. Attached Figure Description

[0024] Embodiments of the invention will now be described by way of non-limiting examples and with reference to the accompanying drawings, wherein:

[0025] - Figures 1 to 8 The method is shown; and

[0026] - Figure 9 An example of the system is shown. Detailed Implementation

[0027] A computer-implemented method for processing 3D signals of shape attributes on real-world objects is proposed. The method includes providing a graph with nodes and arcs. Each node represents a corresponding point in a measured 3D discrete representation of the real-world object. Each arc connects two nodes representing adjacent points in the discrete representation. The method further includes providing a set of values ​​representing the distribution of shape attributes on the real-world object. Each value is associated with a corresponding node in the graph and represents the shape attribute at the corresponding point represented by that node. The method further includes modifying the set of values ​​by minimizing an energy defined on a Markov random field formed on the graph. For each arc connecting a first node to a second node, the energy penalizes the height of an increasing function of the following terms to the first node associated with the first value and the second node associated with the second value: the distance between the first and second values, the distance between the first point represented by the first node and an intermediate geometric element in the discrete representation, and the distance between the second point represented by the second node and an intermediate geometric element.

[0028] This constitutes an improved method for processing 3D signals of shape attributes on real objects.

[0029] This method takes as input a set of values ​​representing the distribution of shape attributes on a real object. Each value is associated with a node in a graph, which represents a corresponding point in a measured 3D discrete representation of the real object, and the value represents the shape attribute at that point. Therefore, there exists a set of measured 3D discrete representations of the real object and values ​​that each represent the shape attribute at a corresponding location in the measured 3D discrete representation (e.g., the value of ). In other words, this set of values ​​forms a 3D signal of shape attributes that corresponds to the measured 3D discrete representation (e.g., calculated based on and / or measured simultaneously with the measured 3D discrete representation), with each value representing a measure of the shape attribute at a corresponding location in the measured 3D discrete representation. In other words, the 3D signal is a measure of the distribution of shape attributes on the real object.

[0030] 3D signals measuring the distribution of shape attributes on real-world objects can be used in many contexts (e.g., computer vision, 3D solid modeling (CAD), and / or 3D reconstruction) (e.g., once processed according to this method). For example, it can be used in domains such as virtual reality and augmented reality, or any kind of immersive experience, video games and mechanical parts, architecture, or any kind of object 3D reconstruction and modeling. The 3D signal can be used, for example, in 3D solid modeling and / or 3D reconstruction processes, for instance, for segmenting discrete representations based on the 3D signal. Segmentation can be followed by a 3D reconstruction process of the discrete representation (e.g., when the discrete representation is a 3D point cloud) and / or by determining a B-rep, CSG construction tree, and / or a feature tree representing the real-world object, as is known from the domains of 3D reconstruction and 3D solid modeling themselves.

[0031] That is, this method modifies the set of values ​​by minimizing the energy defined on a Markov random field (hereinafter sometimes referred to as "MRF") formed on the graph. For each arc connecting a first node associated with a first value of the set to a second node associated with a second value of the set (i.e., a neighboring node of the first node), the energy penalizes the height of an increasing function of the distance between the first and second values. In other words, for each pair of neighboring points, the energy penalizes the differences (e.g., inconsistencies, discrepancies) between the values ​​of the shape attribute representing those neighboring points, respectively. In other words, the energy penalizes drastic local variations (e.g., sharp and / or non-smooth local variations) within the 3D signal of the shape attribute. In other words, the energy penalizes non-smooth variations in the values ​​of the shape attribute measured at neighboring points in the 3D discrete representation. Thus, the minimized result (i.e., the set of modified values) is a modified (and thus processed) 3D signal of the shape attribute where at least (e.g., strictly) a portion of the drastic local variations are relatively smooth (e.g., relative to the convergence criterion of the MRF). Furthermore, in this approach, and as further discussed below, over-smoothing of local variations within the 3D signal of shape attributes is avoided in regions with sharp shape / geometric changes in the real world. These regions can be considered as areas where smoothing might be counterproductive to the shape attribute values ​​of interest. This is achieved, in particular, by using distances to intermediate geometric elements, as further discussed below. Incidentally, MRF is a powerful tool, and a number of efficient and fast solvers exist for it.

[0032] The measured 3D discrete representation originates from measurements performed by one or more physical devices (e.g., sensors) suitable for this task. Drastic local variations within the 3D signal of shape attributes can typically represent noise within the 3D signal of shape attributes, generated by the measurement. In this example, the discrete representation is noisy. By minimizing energy (which penalizes at least a portion of these drastic local variations), the method modifies (e.g., iteratively) the set of values ​​such that (e.g., until) the local variations within the 3D signal of shape attributes are relatively smooth (e.g., relative to the convergence criterion of the MRF). In other words, the method denoises the 3D signal of shape attributes.

[0033] Furthermore, the increasing function is also an increasing function of the following: the distance between the first point and the intermediate geometric element of the discrete representation, and the distance between the second point and the intermediate geometric element. Now, a point (where the distance between the point and the intermediate geometric element is small) corresponds to a sharp feature of the real object, for example, a point located on or near a sharp feature of the real object. Therefore, when the first or second point corresponds to a sharp feature of the real object, the distance between that point and the intermediate geometric element is relatively small (e.g., compared to other distances between the first and second points: other points corresponding to smooth features of the real object, and the intermediate geometric element), and therefore does not contribute much to the height of the increasing function. Thus, the height of the increasing function is less penalized when minimizing energy, for example, compared to other pairs of the first and second points (where both points correspond to smooth features of the real object). This is even more true when both points correspond to sharp features.

[0034] Therefore, minimizing energy necessarily adjusts for overly strong local variations within the set of values, but much less smoothing occurs around values ​​corresponding to the sharp features of a real object. In other words, the method denoises the 3D signal of the shape attributes of a real object, but preserves the variations within the 3D signal that do not correspond to noise, but rather to the shape features of the real object as they are in the real world. Thus, the processed 3D signal of the shape attributes (i.e., the set of modified values) accurately represents the distribution of shape attributes on the real object. In other words, real objects often include one or more sharp features, and the method allows for denoising the 3D signal while preserving the variations in the signal resulting from the presence of these sharp features. In other words, in the real world, shape attributes vary very smoothly on the smooth parts of a real object and more dramatically in the regions of the shape of a real object where sharp features exist, and the method processes the 3D signal of the shape attributes such that (e.g., until) it approximates the distribution of shape attributes as they are in the real world, for example, relative to (e.g., the convergence criterion of MRF). The method is therefore accurate and provides realistic denoising of the 3D signal.

[0035] This method is used to process 3D signals of shape attributes on real objects.

[0036] A "real object" is any object in the real world. A real object is therefore a three-dimensional entity (i.e., a closed 3D volume). In this example, a real object includes at least one sharp feature. A sharp feature is the layout of material within a real object that has a sharp (i.e., non-smooth, irregular) shape.

[0037] The real object can be a component (e.g., mechanical) or an assembly of components (or equivalently, an assembly of components, since from a methodological perspective, an assembly of components can be considered as the component itself, or the method can be applied independently to each component of the assembly), or more generally, any rigid body assembly (e.g., a moving mechanism). The real object can be any (e.g., manufactured) product from a variety of unrestricted industrial sectors, including: aerospace, construction, building, consumer goods, high-tech equipment, industrial equipment, transportation, marine and / or offshore oil / gas production or transportation. Real-world objects can therefore represent industrial products, which can be any mechanical component, such as components of ground vehicles (including, for example, automobiles and light truck equipment, racing cars, motorcycles, trucks and motor vehicles, trucks and buses, trains), components of aircraft (including, for example, fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), components of marine vehicles (including, for example, naval equipment, commercial vessels, marine equipment, yachts and workboats, navigation equipment), general mechanical components (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installed equipment, industrial equipment products, metal products, tire manufacturing products), electromechanical or electronic components (including, for example, consumer electronics, safety and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical devices and equipment), consumer goods (including, for example, furniture, home and garden products, leisure goods, fashion products, products of hard goods retailers, products of soft goods retailers), and packaging (including, for example, food and beverage and tobacco, beauty and personal care, household product packaging). All these objects often include one or more sharp features.

[0038] The term "shape attribute" refers to any attribute that locally describes the shape of a real object. The term "3D signal of shape attributes on a real object" refers to the measured distribution of shape attributes on a real object; that is, a measured field representing the values ​​of the shape attributes at a given location on the real object. The term "measured distribution" refers to a distribution derived from physical measurements performed on the real object (e.g., by one or more physical sensors). In this example, the distribution is directly measured, and in other examples, it is calculated based on physical measurements, such as by calculations performed based on measured points in a discrete representation of the measurements. As described above, the set of values ​​provided forms the 3D signal of the shape attributes. The set of values ​​provided can include a set of values ​​measured on the real object or a set of values ​​calculated based on a discrete representation of the measurements (e.g., measured points). The set of values ​​provided is discussed further below.

[0039] As described above, a shape attribute is an attribute that locally describes the shape of an object. A shape attribute at a given location on a real object can be any set of one or more parameters and / or labels that specify the characteristics of the local shape of the real object at that location. A shape attribute can be, for example, a normal (i.e., a normal vector), such as an outward orientation relative to the real object. Alternatively, a shape attribute can be curvature, such as any curvature on a surface (e.g., Gaussian curvature, principal curvature, or mean curvature). Alternatively, a shape attribute can be the type of a primitive; that is, a shape attribute can locally describe the type of a geometric primitive (e.g., a cube, cylinder, sphere, cone, pyramid, ellipsoid, or torus) that forms the local shape of the real object.

[0040] We will now discuss the provision of a set of graphs and values. Since this provision involves the measured 3D discrete representation of real objects, we will now discuss the measured 3D discrete representation of real objects.

[0041] The discrete 3D representation of a real-world object in this paper is a data structure comprising a discrete set of data fragments. Each data fragment represents a corresponding geometric entity (e.g., a point, a surface) of the 3D shape of a real-world object located in 3D space. Each geometric entity represents a corresponding location of the 3D shape (in other words, a corresponding portion of the material composition of the entity represented by the 3D shape). The aggregation (i.e., union or juxtaposition) of geometric entities together represents the 3D shape. Any discrete geometric representation in this paper may include, in examples, a plurality of such data fragments greater than 100 (e.g., greater than 1000).

[0042] In this example, the 3D discrete representation of a real-world object is a 3D point cloud, where each geometric entity is a point. In other examples, the 3D discrete representation of a real-world object is a 3D mesh, where each geometric entity is a mesh tile or face. Any 3D mesh can be regular or irregular (i.e., composed of faces of the same type or not). Any 3D mesh can be a polygonal mesh (e.g., a triangular mesh). Alternatively, any 3D mesh can be a B-Rep. Any 3D mesh can be obtained from a 3D point cloud, for example, by triangulating the 3D point cloud (e.g., using 3D Delaunay triangulation and methods for extracting relevant facets).

[0043] The term "measured" refers to a 3D discrete representation of a real object derived from physical measurements of the real object (i.e., determined from physical measurements), for example, within a 3D reconstruction process. The 3D reconstruction process may include providing a real object, providing one or more physical sensors configured to acquire corresponding physical signals, and acquiring one or more corresponding physical signals by operating the one or more physical sensors on the real object (i.e., scanning the real object with each sensor). According to any known technique, 3D reconstruction can then automatically determine a 3D point cloud and / or a 3D mesh based on the measurements. The one or more sensors may include one or more portable tray scanners. The one or more sensors may additionally or alternatively include one or more metrological scans.

[0044] This method may in particular include providing a measured 3D discrete representation, for example, before providing a set of graphs and values. Providing the 3D discrete representation may include, for example, performing the previously discussed physical measurements on a real object by performing a previously discussed 3D reconstruction process. Alternatively, providing the measured 3D discrete representation may include retrieving the 3D discrete representation from (e.g., remote) memory, wherein the 3D discrete representation has already been stored after it has been obtained by performing physical measurements, as previously discussed.

[0045] Now let's discuss the provision of the diagram.

[0046] Whether it's a 3D mesh or a 3D point cloud, the measured 3D discrete representation of a real object has points that all represent a given location on the real object. The graph captures the topology of the real object represented by the discrete representation through the corresponding points that all represent the 3D discrete representation and the arcs that all connect the adjacent points of the discrete representation. The term "topology of the real object" refers to "the topology in relation to the 3D discrete representation of the real object," which differs, for example, from the topology in relation to the B-rep representation. A point's neighbor (also called a "neighbor") is the point closest to that point, for example, relative to a given distance (e.g., any known distance in 3D). For example, a point's neighbor could be a point whose distance to that point is less than a predefined threshold. The threshold can depend on the set of values ​​provided as input. For example, the threshold can depend on whether the set of values ​​provided originates from LiDAR or from scanning of small mechanical parts. The threshold can, in fact, depend on the specific circumstances and the method of acquiring the set of values. The threshold can be related to the density of measurements (of points) provided by the acquisition method. When a first point is a neighbor of a second point, the first and second points are neighbors, implying that the second point is a neighbor of the first point.

[0047] Providing a graph may include constructing the graph. Alternatively, providing a graph may include retrieving the graph from (e.g., remote) memory, where the graph has already been stored after it has been constructed. Constructing the graph may include assigning corresponding nodes to each point in the discrete representation. Constructing the graph may then include searching for each pair of adjacent points in the discrete representation. Constructing the graph may then include connecting the nodes of the graph by arcs in the graph, each node representing a point in that pair.

[0048] In the examples, the discrete representation is a 3D mesh with vertices, edges, and faces or mesh blocks. In these examples, "two adjacent points" can represent two vertices sharing an edge (i.e., connected by that edge), and constructing the graph can include creating arcs connecting the nodes that represent the two vertices respectively, and this is true for every pair of adjacent points. In other words, in this case, the arc in the graph corresponds to an edge in the 3D mesh shared by the two vertices, and the arc connects the two nodes representing the vertices.

[0049] In other examples, the discrete representation is a 3D point cloud. In these examples, for each point in the 3D point cloud, graph construction may include finding the neighbors of that point and creating an arc between that point and at least some of the found neighbors. Finding neighbors may include using any known method suitable for finding the neighboring points of a point in a 3D point cloud, such as a nearest neighbor search algorithm. In these examples where the discrete representation is a 3D point cloud, graph construction can be performed using any known method suitable for constructing a graph with nodes and arcs, where each node represents a corresponding point in the 3D point cloud, and where each arc connects two nodes representing neighboring points in the 3D point cloud. For example, graph construction can be performed using the method described in Junjie Cao et al.: PointCloud Skeletons via Laplacian-Based Contraction (2010) (which is incorporated herein by reference).

[0050] In an example where the discrete representation is a 3D point cloud, the construction of the graph may include connecting points to some of their neighbors. The construction may include: for each point, estimating its tangent plane, and then projecting the point's neighbors onto that plane. The construction may then include determining a 2D Delaunay triangulation of the projected points, producing a streamlined planar mesh that includes the point and the projections of its neighbors. The construction may include using this mesh to determine which neighbors the point should connect to. Specifically, the construction may then include connecting nodes associated with the points to graph nodes associated with the points in the point cloud that do contribute projections in the tangent plane and do ultimately connect to the points in the 2D mesh constructed in the tangent plane via 2D Delaunay triangulation. This approach can be performed in another context: Junjie Cao et al.: Point Cloud Skeletons via Laplacian-BasedContraction (2010) (which is incorporated herein by reference).

[0051] Now let's discuss sets that provide values.

[0052] The set of values ​​represents the measured distribution of shape attributes on a real object. As previously mentioned, this means that each value in the set of values ​​represents the shape attribute at the corresponding location in the discrete representation. In other words, the value is the value of the shape attribute at the corresponding location in the discrete representation. The values ​​in the set of values ​​can be scalar values, vector values, or label values. In the example, MRF operates on labels. In this case, if the values ​​are scalar or vector values, the method can include reducing the values ​​in the set of values ​​to labels through quantization. If the values ​​are vector values, multiple minimizations can be performed, the number of which is equal to the dimension of the corresponding vector space.

[0053] Providing the set of values ​​may include calculating a set of values ​​based on a 3D discrete representation, the calculation being performed by any suitable method. The calculation of the set of values ​​may be performed based on the discrete representation (e.g., based on points in the discrete representation). For example, calculating the set of values ​​may include, for each point in at least a portion (e.g., all) of the discrete representation, calculating a value for a shape attribute at that point based on that point and / or its neighboring points.

[0054] The calculation of this value can be performed geometrically, that is, through one or more known geometric methods. This method calculates the value solely by relying on geometric structures, for example, by calculating one or more geometric formulas and / or by solving one or more geometric problems. For example, the calculation of this value may include one or more of the following operations:

[0055] - Fits the plane to the set of points consisting of the point and its neighbors, and uses the plane's normal as the value;

[0056] - The curvature at a point is calculated as a value by applying the well-known cotangent method used to calculate the curvature at a point in a 3D representation to the point and its neighbors (as described above); and / or

[0057] - Fit a primitive (e.g., an ellipsoid) to a set of points consisting of the point and its neighbors, and calculate the normal and principal curvature at that point based on the fitted primitive.

[0058] Alternatively, the value can be inferred by any known neural network that is learned to infer the value of a shape attribute on any point in a discrete representation based on that point and / or its neighboring points. In the example, the neural network could be PCPNet, which is learned through supervised learning on a dataset of examples of 3D point clouds, for which the "terrain truth" of the shape attribute value is known.

[0059] Alternatively, for example, when providing a 3D discrete representation via physical measurements as discussed earlier, the provision may include a set of values ​​measured on a real object. In both cases, the set of values ​​forms a 3D signal of the shape properties on the real object.

[0060] Alternatively, providing the set of values ​​may include retrieving the set of values ​​from (e.g., remote) memory, where the set of values ​​has already been stored, for example, after its calculation or measurement, the set of values ​​still forms a 3D signal of shape attributes on a real object.

[0061] We will now discuss energy minimization. Before that, we will briefly discuss the concept of Markov random fields.

[0062] Markov random fields (MRFs) are well-known tools used to infer the most probable configurations of labels (or discrete values) given a probability distribution of labels (or discrete values) on a graph, by minimizing energy using a smoothing term (also known as a "smoothing term") and a cost term (and optionally a regularization term). Therefore, they are excellent tools for eliminating some defects in the computational results that may have been caused by excessive noise in the computational input. It is well-known that the idea behind MRFs is that any problem can be modeled by the probability that a node in a graph belongs to a specific class (the target class), and the probability that that node belongs to a class different from its neighbors can be modeled by minimizing energy of the following type:

[0063]

[0064] For this purpose, the energy comprises: a cost term (the first sum in Equation (1)) which captures the probability that a node belongs to a specific class (target class) for each node; and a smoothing term (the second sum in Equation (1)) which captures the probability that each pair of adjacent nodes belongs to a different class. A scalar γ determines the amount of contribution of the smoothing term to the energy: the larger γ is, the greater the contribution of the smoothing term to the energy. The scalar γ is positive and less than a predefined number, such as 1. The energy minimized by this method belongs to type (1). In the example of this method, γ equals 1.

[0065] In this approach, the problem to be modeled can be described as follows: there exists a measured 3D discrete representation of a real object, a discrete representation with points, and a set of values ​​representing the distribution of shape attributes on the real object, each value representing the value of the shape attribute at the corresponding point in the discrete representation. The goal of minimization is to modify the set of values ​​such that: 1) the set of values ​​remains close to the provided set of values ​​(i.e., there is not much modification to the 3D signal of the shape attribute measured / computed according to the discrete representation), and 2) strong local variations within the set of values ​​are eliminated except in regions of the discrete representation representing sharp features of the 3D discrete representation, i.e., strong variations in the values ​​of the shape attribute at neighboring points in the discrete representation. To this end, the method modifies the set of values ​​by minimizing an energy defined on a random Markov field formed on the provided graph, which has a cost term for capturing target 1) and a smoothing term for capturing target 2). The cost term of the energy can be any known cost term capable of capturing target 1). The smoothing term is now discussed.

[0066] For each arc connecting the first node to the second node, the energy penalizes the height of the increasing function to the first node associated with a first value and the second node associated with a second value. The first value thus represents the shape attribute at the first point represented by the first node, and the second value thus represents the shape attribute at the second point represented by the second node. Furthermore, the first and second points are adjacent points because their representative nodes are connected by arcs in the graph. The smoothing term of the energy performs the penalty on the height of the increasing function. In other words, for each pair of adjacent points represented by the first and second nodes, the smoothing term penalizes the height of the increasing function. The term "penalize the height of the increasing function" means that for each pair of adjacent points represented by the first and second nodes, the energy quantizes the height of the increasing function (e.g., the height of the function's value), and during minimization, if the value of the shape attribute representing that pair of adjacent points makes the increasing function too large (i.e., has a very large value), then they are modified. They are modified to make the increasing function smaller. The increasing function will now be discussed.

[0067] An increasing function is an increasing function of the distance between the first and second values. The distance can be any distance between the values, such as any distance in one dimension or higher. Therefore, if the first and second values ​​(as mentioned above, which represent the values ​​of shape properties at two adjacent points) are too different from each other, the distance between them will be large. In this case, the increasing function tends to be large. This means that a large distance helps to increase the value of the function; however, because the function depends on other parameters, the value of the function may still be relatively small, as discussed below.

[0068] In fact, this function is also an increasing function of the distance between the first point (of the pair of adjacent points represented by the first and second nodes) and the intermediate geometric element of the discrete representation. The intermediate geometric element can be any intermediate geometric element of the discrete representation. In the example, the intermediate geometric element is the intermediate axis or intermediate surface of the discrete representation. The distance between the first point and the intermediate geometric element can be any distance between the first point and the intermediate geometric element (e.g., the Euclidean distance between the first point and the intermediate geometric element). This distance can also be referred to as the local feature size value of the first point (also known as the "LFS value").

[0069] Similarly, this function is also an increasing function of the distance between the second point (of the pair of adjacent points represented by the first and second nodes) and the intermediate geometric element in the discrete representation. The distance between the second point and the intermediate geometric element can be any distance between the second point and the intermediate geometric element, such as the Euclidean distance between the second point and the intermediate geometric element. This distance can also be referred to as the local feature size value of the second point (also known as the "LFS value").

[0070] More generally, the LFS value of a discretely represented point is the distance between that point and the intermediate geometric element.

[0071] For example, a smaller distance between the first point and the intermediate geometric element compared to other values ​​of other distances between other points and geometric elements indicates that the first point corresponds to the position of the real object belonging to the sharp part of the shape of the real object. In this case, the distance between the first point and the intermediate geometric element does not contribute much to increasing the value of the function; on the contrary, it helps to decrease the value of the function. Similarly, for example, a smaller distance between the second point and the intermediate geometric element compared to other values ​​of other distances between other points and geometric elements indicates that the second point corresponds to the position of the real object belonging to the sharp part of the shape of the real object. In this case, the distance between the second point and the intermediate geometric element does not contribute much to increasing the value of the function; on the contrary, it helps to decrease the value of the function.

[0072] Therefore, it can be said that if both the first and second points belong to the smooth portion of the shape of the real object, their LFS values ​​are large. In this case, the height of the increasing function depends primarily on the height of the distance between the first and second values. If this distance is large, it indicates a strong change between the first and second values, whereas there should actually be no change because both the first and second points belong to the smooth portion of the shape of the real object. Therefore, as discussed earlier, this strong change corresponds to noise, and thus, minimization will modify the first and / or second values ​​so that the distance between them becomes smaller (e.g., regarding the convergence criterion of MRF). In other words, when the true values ​​of the shape properties of adjacent points are close because they belong to the smooth portion of the shape of the real object, minimization tends to ensure that the corresponding values ​​in the set of values ​​are also and / or remain close. In other words, strong local changes in shape properties within the 3D signal correspond to noise, and these are eliminated.

[0073] Now, if the first (and correspondingly the second) point belongs to a sharp portion of the shape of the real object, its LFS value is small. In this case, the LFS value helps to reduce the value of the increasing function. Therefore, even if the distance between the first and second values ​​is large, the value of the increasing function will be less penalized during minimization. If the LFS value of the second (and correspondingly the first) point is also small, meaning that the second (and correspondingly the first) point belongs to a sharp portion of the shape of the real object, then even if the distance between the first and second values ​​is large, the value of the increasing function will be less penalized during minimization. Therefore, when at least one of the adjacent points belongs to a sharp portion of the shape, strong variations between the values ​​of the two adjacent points tend to be preserved (or at least not over-smoothed) during minimization, and when both adjacent points belong to sharp portions of the shape, strong variations between the values ​​of the two adjacent points tend to be preserved more (or at least less smoothed) during minimization. In other words, when the true values ​​of the shape properties of adjacent points are very different because at least one (e.g., two) of the adjacent points belong to sharp portions of the shape of the real object, minimization tends to ensure that the corresponding values ​​in the set of values ​​are also very different. In other words, strong local variations in shape properties within a 3D signal tend to be preserved when they correspond to sharp features of a real object.

[0074] Therefore, when these strong variations correspond to the sharp features of real objects in the real world, this method allows for denoising of 3D signals while preserving strong local variations in the shape properties within the 3D signal.

[0075] Now let's discuss LFS values ​​further.

[0076] LFS values ​​and intermediate geometric elements are well-known concepts and can be calculated using any known method or combination of known methods. Some of these methods are publicly available: Amenta et al., Surface Reconstruction by Voronoi Filtering (1999); Andrea Tagliasacchi et al., 3D Skeleton: A State-of-the-Art Report, DOI: 10.1111 / cgf.12865, EUROGRAPHICS 2016; Daniel Rebai et al., LSMAT Least Squares Medial Axis Transform, DOI: 10.1111 / cgf.13599; and Thomas Delame et al., Structuring 3D Medial Skeletons: A Comparative Study. Symposium on Vision, Modeling and Visualization, Oct 2016, Bayreuth, Germany.pp.1-8, 10.2312 / vmv.20161336, hal-01359738, all of which are incorporated herein by reference. This method may include computing the LFS values ​​of the discretely represented points using any known method, for example, before modifying the set of values. The method may also include computing intermediate geometric elements using any known method, for example, before computing the LFS values.

[0077] The computation of LFS values ​​and / or intermediate geometric elements is based on a 3D discrete representation. In the example, this means that the computation of LFS values ​​and / or intermediate geometric elements takes a discrete representation (e.g., points of ) as input and outputs LFS values ​​and / or intermediate geometric elements according to any suitable known method (e.g., the methods discussed earlier). In other examples, the computation of LFS values ​​and / or intermediate geometric elements involves denoising the discrete representation (e.g., smoothing, or regularizing), and then computing the LFS values ​​and / or intermediate geometric elements based on the denoised discrete representation. Thus, it means that the computation of LFS values ​​and / or intermediate geometric elements takes a denoised discrete representation (e.g., points of ) as input and outputs LFS values ​​and / or intermediate geometric elements according to any suitable known method (e.g., the methods discussed earlier). It should be understood that in those examples where the computation of LFS values ​​and / or intermediate geometric elements involves denoising the discrete representation, energy minimization is still performed based on the undated discrete representation. Denoising the discrete representation before calculating the LFS values ​​and / or intermediate geometric elements allows for higher accuracy in calculating them. Performing minimization based on the undennoised discrete representation still allows for higher accuracy in denoising the 3D signal of shape properties while preserving sharp features, as discussed earlier, because the LFS values ​​and / or intermediate geometric elements used in minimization are calculated with higher accuracy. Alternatively, the method can calculate the LFS values ​​without first smoothing / denoising the discrete representation. This is especially true when calculating LFS values ​​according to the previously cited LSMAT method.

[0078] Now let's discuss energy minimization further.

[0079] Minimization can be performed using any known algorithm suitable for performing the minimization (e.g., any known Markov random field solver). Markov random fields can be solved via alpha-extended graph cutting, and solutions exist that leverage the parallelism of modern hardware, such as in D. Thuerck et al.: A Fast, Massively Parallel Solver for Large (incorporated hereby by reference). The idea of ​​minimizing the energy defined on a graph-formed Markov random field and its algorithms are well-known in the field of Markov random fields and will not be discussed further.

[0080] Now let's discuss the smoothing term further.

[0081] In the example, the smoothing term can be the sum of all arcs in the graph of the following types of terms:

[0082]

[0083] Where (p,q) is a pair consisting of the first value p (associated to the first point) and the second value q (associated to the second point), where |pq| is the distance between the first and second values, and LFS... p It is the LFS value of the first point, where LFS p Let be the LFS value of the second point, where maxLFS is the maximum value of all LFS values ​​of the discretely represented point.

[0084] In other examples, the smoothing term can be the sum of all arcs in the graph of the following types of terms:

[0085]

[0086] Where (p,q) is a pair consisting of the first value p (associated to the first point) and the second value q (associated to the second point), where |pq| is the distance between the first and second values, and LFS... p It is the LFS value of the first point, where LFS p Let be the LFS value of the second point, where maxLFS is the maximum value among all LFS values ​​of the discretely represented point, and minLFS is the minimum value among all LFS values ​​of the discretely represented point.

[0087] Now let's discuss the cost items further.

[0088] As discussed earlier, the cost term captures the requirement that the set of values ​​being modified remains close to the set of values ​​provided. In other words, while the smoothing term tends to modify (when minimizing energy) the set of values ​​so that drastic local variations outside the regions corresponding to the discrete representations of sharp features are eliminated, the cost term tends to ensure that (when minimizing energy) the set of modified values ​​remains close to the set of values ​​provided. As mentioned above, the cost term can be any known cost term that ensures the set of values ​​being modified remains close to the set of values ​​provided during minimization.

[0089] In the examples, the Markov random field has a target value representing a target distribution of shape properties. In these examples, for each node, the energy is further penalized by the height of another increasing function associated with at least one distance between the value of that node and at least one target value.

[0090] In these examples, the target values ​​form a set of one or more target values, each set representing a target distribution of the shape property. In other words, minimizing the energy causes the set of values ​​to be modified to approach one or more sets of target values ​​(e.g., to converge toward one or more sets of target values), that is, to cause the set of values ​​to be modified to become relatively close to each of the one or more sets of target values. Each set of target values ​​can be any set of values ​​representing a distribution of the shape property that is similar to the distribution of the shape property represented by the set of provided values. For example, the values ​​in the set of target values ​​can represent the values ​​in the set of provided values. Alternatively or alternatively, the values ​​in the set of target values ​​can be computed according to a discrete representation. In this case, the method can include: compute the values ​​in the corresponding set of target values ​​according to the discrete representation for each corresponding set of target values ​​in the one or more sets of target values, for example, before minimization. The computation can be performed by a correspondingly different method for each corresponding set of target values. Alternatively or alternatively, the method can include sampling the target value of each of the sets as a value (which represents the value in the set of provided values). The values ​​in each set of target values ​​can be scalar values, vector values, or label values.

[0091] The calculation of the target value for each of the set of target values ​​can be performed based on a discrete representation (e.g., points in the discrete representation). For example, the calculation of the target value for the set may include: for each point in at least a portion (e.g., all) of the discrete representation, calculating a value of a shape attribute at that point based on the point and / or its neighboring points. This calculation can be performed geometrically (i.e., by one or more known geometric methods). This method calculates the value solely by relying on a geometric structure (e.g., by calculating one or more geometric formulas and / or by solving one or more geometric problems). For example, the calculation of the value may include one or more of the following operations:

[0092] - Fits the plane to the set of points consisting of the point and its neighbors, and uses the plane's normal as the value;

[0093] - The curvature at a point is calculated as a value by applying the well-known cotangent method for calculating the curvature of a point in a 3D representation to the point and its neighbors (calculated as described above); and / or

[0094] - Fit a primitive (e.g., an ellipsoid) to a set of points consisting of the point and its neighbors, and calculate the normal or principal curvature at that point based on the fitted primitive.

[0095] Alternatively, the value can be inferred by any known neural network that is learned to infer the value of a shape attribute on any point in a discrete representation based on the point and / or its neighboring points.

[0096] Still in these examples, for each node, the energy further penalizes the height of another increasing function associated with at least one distance between the node's value and at least one target value. The cost term of the energy performs the penalty for the height of the increasing function. In other words, for each node in the graph, the cost term penalizes the height of the other increasing function. "Penalizing the height of the other increasing function" means that for each node, the energy quantizes the height of the other increasing function (e.g., the height of the value of the other increasing function), and during minimization, if the value of the shape attribute representing the point represented by the node makes the other increasing function too large (i.e., has a very large value), then that value is modified. It is modified to make the other increasing function smaller. Now let's discuss the other increasing function.

[0097] The other increasing function is a function of at least one distance between the value associated with the node and at least one target value. This means that the other increasing function is a function of one or more distances, each distance being between the value associated with the node and the corresponding target value (e.g., the corresponding set belonging to the one or more sets of target values). Each corresponding distance in the one or more distances can be any distance between values ​​(e.g., any distance in one or higher dimensions). Therefore, for each of the one or more distances, if the value associated with the node and the corresponding target value differ too much from each other, the distance between them is large. In this case, the other increasing function also becomes large. Therefore, the values ​​associated with the node will be modified during minimization to reduce the value of the other increasing function. This tends to ensure that the set of modified values ​​remains close to the set of provided values ​​during minimization because differences (e.g., discrepancies, inconsistencies) between the set of modified values ​​and the target values ​​are penalized because they contribute to increasing the other increasing function.

[0098] In the examples, at least one distance comprises a single distance. In other words, the target value forms a single set of target values ​​representing the set of provided values, as discussed previously, and for each node, the cost term penalty is associated with another increasing function of the single distance between the node's value and the target value of the single set of target values. In these examples, the cost term could be the sum of all nodes in the graph of terms of the following types:

[0099] E target (p)=|p-target|

[0100] Where p is the value associated with the node, |p-target| is the single distance between the value associated with the node and the target value, and target is the target value.

[0101] In other examples, the other increment function is an increment function for the following: the distance between the value associated with the node and the first target value, and the distance between the value associated with the node and the second target value.

[0102] Now let's discuss such an example.

[0103] In these examples, the target values ​​form a first set of target values ​​consisting of a first target value and a second set of target values ​​consisting of a second target value. As discussed earlier, both the first and second sets of target values ​​consist of values ​​representing the sets of provided values. In these examples, at least one distance includes (e.g., consists of) a first distance and a second distance. In other words, the other increasing function is a function of the first distance between the value associated with the node and the first target value (i.e., belonging to the first set of target values) and the second distance between the value associated with the node and the second target value (i.e., belonging to the second set of target values). Therefore, as mentioned earlier, during minimization, the cost term penalizes the difference (e.g., discrepancy, inconsistency) between the set of modified values ​​and each of the first and second sets of target values. In other words, energy minimization tends to measure whether the set of provided values, when modified, is still close to both the first and second sets of target values, and to modify them when the values ​​of the set of provided values ​​being modified become highly inconsistent with the values ​​of the first or second set of target values. In other words, the cost term ensures that the set of modified values ​​tends to remain close to both sets that represent the sets of provided values. This is equivalent to saying that the cost term, during minimization, penalizes the difference between the set of modified values ​​and two distinct sets of values, both of which represent the set of provided values. Since the method tends to ensure proximity to the target value from two distinct sets, both representing the set of provided values, this enhances the way the method ensures that the set of modified values ​​and the set of provided values ​​remain close. This further improves the accuracy of the method.

[0104] In the example, at least one target value includes at least one value of a shape property that is geometrically calculated at the corresponding point represented by the node, and / or at least one value of a shape property that is inferred by the neural network at the corresponding point represented by the node.

[0105] The term "at least one value of a shape attribute calculated geometrically" refers to at least one value of a shape attribute calculated by one or more geometric methods suitable for calculating the value of a shape attribute. As described above, the method may include the calculation of target values, which may include: for each of the at least one target value, geometrically calculating at least one value of the shape attribute at a corresponding point represented by a node, as discussed above.

[0106] The term "at least one value of a shape attribute inferred by a neural network" refers to at least one value of a shape attribute generated by a neural network learned to compute the value of the shape attribute. The neural network can be any neural network learned to compute the value of the shape attribute. As described above, the method may include the computation of target values, which may include: for each of the at least one target value, applying the neural network to compute at least one value of the shape attribute at a corresponding point represented by a node, as discussed previously.

[0107] This method can therefore use geometrically computed values ​​and / or values ​​inferred by a neural network as target values. This makes the method flexible, as it can rely on at least two different methods for computing the target value, and thus benefit from the advantages (e.g., accuracy, speed, and / or robustness) arising from at least one of these methods. These advantages can be further combined.

[0108] In the examples, at least one target value includes at least one value of a shape property geometrically computed at the corresponding point represented by a node, and at least one value of a shape property inferred by a neural network at the corresponding point represented by a node. In these examples, the method uses a value derived from the fusion of the geometrically computed value and the value inferred by the neural network as the target value. This facilitates the proximity (e.g., convergence) of the set of modified values ​​to values ​​considered more reliable than others (for which geometric methods and neural networks provide similar estimates), thus improving the accuracy of the method.

[0109] In an example where the other increasing function is an increasing function relating the distance between the node's value and a first target value (the first distance, as previously discussed) and the distance between the node's value and a second target value (the second distance, as previously discussed), at least one target value may include at least one value of a shape attribute geometrically computed at the corresponding point represented by the node, and at least one value of a shape attribute inferred by the neural network at the corresponding point represented by the node. Specifically, in this case, the first target value is at least one value of a shape attribute geometrically computed at the corresponding point represented by the node, and the second target value is at least one value of a shape attribute inferred by the neural network at the corresponding point represented by the node.

[0110] In the example, the other incrementing function also quantifies the correspondence between values ​​included in at least one target value. These examples will now be discussed.

[0111] As previously mentioned, the target values ​​can form one or more sets of target values. The other increasing function can be an increasing function of one or more distances, respectively, between the value associated with the node and the corresponding target value (which belongs to the corresponding one of the one or more sets of target values). Each of the corresponding target values ​​represents the value of a shape attribute at a discretely represented location corresponding to a point represented by a node. Now, the correspondence is a term or number that quantifies the correspondence (e.g., similarity, e.g., substantial equality) between each of the corresponding target values. This term can be such that it is large when the corresponding target values ​​correspond to each other (e.g., are similar, e.g., are equal or substantially equal), and small when the target values ​​do not correspond to each other (e.g., are very different or substantially very different). Therefore, the distance to the target value has all the greater influence when minimizing energy. Alternatively, a term can be one that is non-zero when the corresponding target values ​​correspond to each other (e.g., are similar, equal, or substantially equal), and null when the target values ​​do not correspond to each other (e.g., are very different or substantially very different). In this case, it can be said that energy capture takes into account that deviations from target values ​​are not serious and that these target values ​​are not consistent with each other in any way. A term can depend on the corresponding target value, and the correspondence between the corresponding target values ​​can be quantified directly based on the corresponding target value. Alternatively, the term can depend on other values, and the correspondence between the corresponding target values ​​can be quantified based on the other values.

[0112] Quantizing the correspondence between values ​​included in at least one target value improves the robustness and accuracy of the method. In fact, the value of the other increasing function tends to decrease when the values ​​included in at least one target value do not correspond. Therefore, even if at least one target value included in at least one target value makes the at least one distance large, the other increasing function does not necessarily tend to be correspondingly large, because the lack of correspondence between the values ​​included in at least one target value tends to decrease the value of the function. Therefore, the cost term tends to give more strength to the target values ​​included in at least one corresponding target value. In other words, minimizing the set of values ​​tends to modify the value so that it remains close to these strong target values ​​(as mentioned above, the target values ​​given more strength). Now, each of these strong target values ​​relatively accurately represents the shape property at the location of the discrete representation because the target values ​​included in the strong target values ​​correspond to each other, even if they are calculated differently (e.g., at least one is calculated geometrically and at least one is inferred by a neural network). In other words, the cost term promotes proximity (e.g., convergence, such as the convergence criterion relative to the MRF) of the set of modified values ​​during minimization, and the set of target values ​​well represents the values ​​in the set of provided values.

[0113] In the example, the other increasing function is a function of the product between the correspondence and the at least one distance. This is a particularly simple, robust, and efficient way to give more strength to the target values ​​that are included in the at least one target value and correspond to each other. In fact, if the correspondence is small (e.g., empty), the product tends to be as strong, and so does the function. Therefore, even though the at least one distance between the value associated with the node and the value in the at least one target value should be large, a small penalty is applied to the value of the function during minimization.

[0114] Now let's discuss further examples of this method.

[0115] In these examples, the other increasing function is an increasing function of the first distance between the value associated with the node and a first target value, and a second distance between the value associated with the node and a second target value. In these examples, the first target value is the value of a shape attribute geometrically calculated at the corresponding point represented by the node, and the second target value is the value of a shape attribute inferred by the neural network at the corresponding point represented by the node. In these examples, the other increasing function also quantifies the correspondence between the first target value and the second target value.

[0116] In these examples, the cost term tends to ensure that the set of modified values ​​remains close to both the geometrically computed first target value and the second target value inferred by the neural network during minimization. Both the first and second target values ​​represent the distribution of shape properties on the real object, similar to the distribution represented by the set of provided values. In other words, the MRF optimization according to this method uses a fusion of the geometrically computed target value and the target value inferred by the neural network as the target value. This improves the robustness and accuracy of the method because at least two different types of target values ​​(i.e., target values ​​from two different computational sources) are used to make the set of modified values ​​close to the set of provided values. Furthermore, the method gives more strength to target values ​​that are consistent between the two sources (i.e., produce similar values). This allows benefiting from the advantages of both methods in terms of accuracy and further improves the robustness and accuracy of the method. In these examples, the other increasing function can be a function of the product of the correspondence between the value associated with the node and the first target value, and the correspondence between the value associated with that node and the second target value. As previously mentioned, this further improves the simplicity and robustness of the method.

[0117] In these examples, the cost term can be the sum of all nodes in the graph for the following types of terms:

[0118]

[0119] Where p is the value associated with the node, geomTarget is the first target value calculated geometrically, neuralTarget is the second target value inferred by the neural network, geomNormal is the geometrically calculated normal at the point represented by the node, and neuralNormal is the normal inferred by the neural network at the point represented by the node. |p-geomTarget| is the distance between the value associated with the node and the first target value. |p-neuralTarget| is the distance between the value associated with the node and the second target value. geomNormal.neuralNormal is the correspondence between the first target value and the second target value. The shape property here is the normal or any shape property closely related to the normal, such as shape properties calculated based on the normal (e.g., maximum curvature).

[0120] Still in these examples, the cost item can alternatively be the sum of all nodes in the graph of the following types of items:

[0121]

[0122] Where p is the value associated with the node, geomTarget is the first target value calculated geometrically, neuralTarget is the second target value inferred by the neural network, geomValueGaussian is a Gaussian distribution centered on geomTarget, representing the geometrically calculated values ​​at the points represented by the node, and neuralValueGaussian is a Gaussian distribution centered on neuralTarget, representing the neural network inferred values ​​at the points represented by the node, representing the shape attribute. (i) is the bucket index of the histogram covering the range of values ​​of the provided set of values. |p-geomTarget| is the distance between the value associated with the node and the first target value. |p-neuralTarget| is the distance between the value associated with the node and the second target value. This represents the correspondence between the first target value and the second target value. The shape attribute here can be any shape attribute, such as any shape attribute independent of the normal.

[0123] This method can be applied to computer vision, 3D solid modeling (CAD), and / or 3D reconstruction. For example, it can be applied to areas such as virtual reality and augmented reality, or any kind of immersive experience, video games and mechanical parts, architecture, or 3D reconstruction and modeling of any kind of object. This method can be included, for example, in the 3D solid modeling process and / or the 3D reconstruction process.

[0124] This method may, for example, further include segmenting the discrete representation based on a set of modified values. Segmenting the discrete representation involves partitioning (e.g., decomposing, clustering) the discrete representation into smaller and more meaningful discrete sub-representations. The partitioning is based on a set of values ​​representing shape attributes on the discrete representation, which, in the case of this method, is the set of modified values. Segmentation according to this method can be accomplished by any known segmentation methods, which are well-known and will not be discussed further herein. Segmentation is improved because the set of modified values ​​relatively accurately represents the distribution of shape attributes on a real object (as it is in the real world). Segmentation can be followed by a 3D reconstruction process of the discrete representation (e.g., when the discrete representation is a 3D point cloud) and / or by determining a B-rep, CSG construction tree, and / or a feature tree representing the real object, as the domain of 3D reconstruction and 3D entity modeling is known itself.

[0125] We will now discuss how to implement this method.

[0126] This implementation can be applied to the fields of computer vision, 3D solid modeling (CAD), and 3D reconstruction. Therefore, this implementation can be useful for fields such as virtual reality and augmented reality (more generally, any kind of immersive experience), video games and mechanical parts, architecture, or 3D reconstruction and modeling of any kind of object.

[0127] Evaluating local quantity or characteristics on noisy point clouds and meshes is a complex task. This implementation helps improve the quality of results for noisy inputs. It relatively preserves the detailed characteristics of regions with sharp features.

[0128] Markov random fields (MRFs) are known tools used to minimize the most probable configuration of labels (or discrete values) given a probability distribution of labels (or discrete values) on a graph, by minimizing the energy of a smoothing term and a cost term (and often a regularization term). This implementation includes a formula for the MRF energy using the local feature size in the smoothing term. In the case of noisy inputs, the local feature size itself can be noisy. In those cases, the local feature size can be computed based on the smoothed variation of the point cloud. This implementation still provides good results even if the smoothing is somewhat strong and does not preserve the sharp geometry of the input, since it is only used to identify regions worth smoothing and regions where doing so is riskier. Alternatively, the local feature size can be computed by methods that do not require the smoothed variation of the point cloud (e.g., the previously cited LSMAT method). Furthermore, the cost term can use values ​​computed via purely geometric methods as well as values ​​inferred by neural networks in such a way that the two methods tend to agree on similar estimates and achieve fusion of the two estimates. Markov random fields can be solved via alpha-spread graph cutting, and solutions exist to leverage the parallelism of modern hardware (such as Thuerck et al.: A Fast, Massively Parallel Solver for Large, which is incorporated herein by reference). In this implementation, any Markov random field solver can be used, but those that operate on hundreds of thousands of nodes with thousands of labels / values ​​are preferred.

[0129] Using a formula for the smoothing term in this implementation based on local feature dimensions, this implementation allows for non-oversmoothing in regions with sharp geometric features. Therefore, it becomes a convenient tool for smoothing values ​​that are not expected to change smoothly from sharp features, but for preserving sharp variations in regions where sharp geometric features exist. Additionally, when the cost term involves both geometrically calculated values ​​and values ​​inferred by neural networks, it promotes proximity (e.g., convergence) to values ​​considered more reliable than others (values ​​for which geometric methods and neural network inferences provide similar estimates). This can be useful, for example, for improving the segmentation of point clouds or models or scenes captured as LiDAR point clouds reconstructed using metrological equipment or photogrammetry, where point cloud segmentation is a classic step for geometric reconstruction or analysis. Figure 1 The image shown is a screenshot extracted from documentation of a well-known computer vision library illustrating this segmentation. Figure 1 It is clear from this that such segmentation can lead to primitive recognition, which is useful for reconstructing models beyond point clouds, robot decision-making, or any other form of analysis.

[0130] When the input point cloud is noisy, the algorithm (like the Point Cloud Library (PCL)) Figure 1 The algorithm used in the image (on the image) yields lower quality results. A typical drawback is, for example, producing oversegmented results. In this type of case, smoothing the input point cloud is insufficient, as it may remove interesting features from the input. Markov random fields are also a good tool for smoothing the output of any algorithm that will work on noisy point clouds, but care must be taken to ensure that too much smoothing does not occur in regions where smoothing is not performed. From this perspective, the energy formula of this implementation is of concern because it ensures that much less smoothing will occur in regions where sharp geometric features appear. In this implementation, the local feature size associated with each vertex in the energy formula is used. The local feature size is the distance to the intermediate elements (axis or surface) of the point cloud (or mesh). At least reasonably high quality estimates of those intermediate elements are preferred. There are numerous methods focused on generating these, even for noisy inputs, such as D. Rebain et al.: LSMAT Least Square Medial Axis Transform, 2019, which is incorporated herein by reference. Therefore, this implementation extends the advantages of those regularization methods for intermediate axis elements to the field of values ​​on point clouds or meshes. In this implementation, the method uses two iterations of projection onto the mean plane of the neighbors.

[0131] Since neural networks have proven quite strong in handling noisy inputs, it is also interesting to use the aforementioned Markov random fields to fuse predictions from neural networks with the outputs of more classical algorithms. This allows for fusing two predictions by giving more advantage to consistent predictions, while still applying less smoothing in regions corresponding to sharp features, thus giving more strength to consistent predictions. Furthermore, this could, for example, improve point cloud segmentation.

[0132] This implementation can be applied to improve point cloud segmentation of noisy inputs, and thus improve all use cases that rely on such segmentation, such as geometric reconstruction.

[0133] This implementation method includes the following four steps:

[0134] 1. Use noisy point clouds or meshes

[0135] 2. Estimate some local quantities or properties (e.g., normals, curvature, surface type).

[0136] 3. Calculate LFS

[0137] 4. MRF Optimization

[0138] Therefore, this implementation produces a set of values ​​representing the distribution of shape attributes on a real object that has been smoothed in the region estimated to be affected by noise in the input, while preserving sharp variations in the sharp regions of the real object's shape.

[0139] 1. Take noisy point clouds

[0140] Figure 2 The example given is a noisy point cloud.

[0141] 2. Estimating a local quantity or property (also known as shape attribute)

[0142] For example, estimating the normal to each point on the point cloud, as shown in the example of the normal inferred by the neural network. Figure 3 As shown.

[0143] 3. Calculate the local feature dimensions

[0144] The size of local features can be approximated by evaluating the distance to the poles (for further details, see Amenta et al.: Surface Reconstruction by Voronoi Filtering, 1999, which is incorporated herein by reference). Figure 4 An example of this evaluation is shown, which includes a small Local Feature Size (LFS) value of 40 and a high LFS value of 42. The LFS values ​​may be inaccurate if the input is noisy. Figure 5 An image showing LFS on a very noisy point cloud is shown.

[0145] There is a long tradition of methods focused on generating intermediate elements (axis / surfaces) (even for noisy inputs), such as one of the more recent methods described in D. Rebain et al.: LSMAT Least Square Medial Axis Transform, 2019 (which is incorporated herein by reference). Therefore, in some way, this implementation extends the advantages of those regularization methods used for intermediate axis elements to fields of values ​​that this method applies to point clouds or meshes. To compute in Figure 6 The LFS shown in the image illustrates an implementation that uses two iterations of projection onto the mean plane of the neighboring plane.

[0146] The smoothed point cloud is only used to compute intermediate elements; obviously, all computations are still performed on the original point cloud. For very small, sharp features that almost disappear in point cloud noise, the MRF smoothing term can also be very strong in those regions. For very noisy point clouds, this simple smoothing will not be sufficient, and in those highly degenerate cases, this implementation can rely on methods such as D. Rebain et al.: LSMAT Least Square Medial AxisTransform, 2019 (incorporated herein by reference) to obtain intermediate elements that will be used to compute local feature sizes, which will then be used in the MRF. In any case, this implementation slightly extends the benefits of any regularization method used to compute intermediate elements to compute the best approximation of the input.

[0147] like Figure 6 As shown, the results are available, and the smoothed point cloud is used only to calculate the LFS value. The work using MRF will still be done on the original noisy point cloud, and the calculated LFS value will only ensure that the smoothing term is very weak in areas where there are sharp geometric features.

[0148] The LFS value can be approximated by the distance to the nearest LFS pole, such as Figure 7 As shown, this forms a good approximation of the intermediate surface and axis of the shape (see again Amenta et al.: Surface Reconstruction by Voronoi Filtering, 1999, which is incorporated herein by reference for a more complete description). This implementation remains independent of the method used to calculate the local feature dimensions associated with points.

[0149] 4. MRF Optimization

[0150] The underlying idea behind Markov random fields is that any problem that can be modeled by the probability of a node in a graph belonging to a specific class (the target class) and by the probability of that node belonging to a class different from its neighbors can be modeled by minimizing the following energy:

[0151]

[0152] Intuitively, shape properties change very smoothly in the smooth parts of a shape and more dramatically in regions where sharp features exist. The aim of this implementation is to eliminate dramatic changes in the smooth parts of the shape of a real object, which would be a result of noisy properties of the input. Based on the above discussion of the ideas behind Markov random fields, it seems that this mathematical tool could be useful in achieving this goal, since the fundamental situation is that the value that should be assigned to a particular point is more likely to be close to the current estimate and has a reasonable probability of being close to the values ​​of its neighbors (except in the parts of the shape where sharp features exist).

[0153] This implementation works particularly well on point clouds. In the case of meshes, the graph to be used is either a graph formed by mesh vertices or a graph formed by mesh faces. For point clouds, the graph is constructed by connecting these points to some of their neighbors. For each point, a tangent plane is estimated, and then the point's neighbors are projected onto that plane. The 2D Delaunay of the projected points then gives a simplified planar mesh that is built with the projections of the point and its neighbors. This mesh is used to determine which neighbors the point should be connected to. In fact, its associated nodes are connected to the graph nodes associated with points in the point cloud that do contribute to the projection at tangent locations and are indeed ultimately connected via Delaunay to points in the 2D mesh built in the tangent plane. This method can be performed in another context (Junjie Cao et al.: Point Cloud Skeletons via Laplacian-Based Contraction, 2010, which is incorporated herein by reference).

[0154] Now we will give the graph structure a discrete representation. Figure 4 The local feature sizes are shown, and it seems desirable to smooth the values ​​in regions with high LFS values ​​of 42 while preserving the strong variations in regions with low LFS values ​​of 40. Therefore, it is of interest to formulate the smoothing term in a way that the smoothing term remains small in the blue regions even though p and q have very different target classes. The scalar values ​​are discretized, and each discrete value is, in fact, one of those classes referenced in this implementation.

[0155] The formula for the smoothing term can be:

[0156]

[0157] Alternatively, it can be:

[0158]

[0159] Furthermore, when using multiple inputs, more intensity is given to the values ​​(that are indeed consistent with their inputs). The set of values ​​provided in this implementation is a scalar field of type "normal_x, normal_y, normal_z". The normal is discretized into discrete values ​​of 100 (or up to 1000) other indices, which are different classes that can be assigned to a particular node:

[0160]

[0161] Note that when using only one input, the target energy term is (in the case of two inputs, Equation 3 above actually replaces Equation 4 below):

[0162] E target (p)=|p-target| (4)

[0163] When dealing with the normal fields "normal_x, normal_y, normal_z", the above formula applies to the case of two inputs. Alternatively, the implementation can use it to work on other scalar fields, provided that the similarity of the estimated normals still indicates how large the values ​​of the input values ​​associated with a particular point or vertex are. Conversely, if the scalar fields are not directly related to the normals, this implementation creates two Gaussian numbers centered on the two input values ​​and takes the maximum of the product of the two Gaussian numbers:

[0164]

[0165] geomValueGaussian and geomTarget are functions relative to the same value field defined on the point cloud or grid, just as neuralValueGaussian and neuralTarget are also related to the same value field. i is the bucket index of the histogram, which will cover the range of the set of provided values ​​(again, these values ​​are discretized, and each discrete value is absorbed into a different class for the Markov random field).

[0166] Figure 8 The results of MRF are shown.

[0167] This method is implemented by a computer. This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the execution of the steps by a computer may be fully automatic or semi-automatic. In the example, the triggering of at least some of the steps of the method can be performed through user-computer interaction. The required level of user-computer interaction may depend on the anticipated level of automation and be balanced with the need to fulfill the user's wishes. In the example, this level may be user-defined and / or predefined.

[0168] A typical example of a computer-based implementation of the method is to execute the method using a system suitable for this purpose. This system may include a processor coupled to memory and a graphical user interface (GUI) on which a computer program containing instructions for performing the method is stored. The memory may also store a database. The memory is any hardware suitable for such storage and may comprise several physically distinct parts (e.g., one for the program and one possibly for the database).

[0169] Figure 9 An example of a system is shown, where the system is a client computer system (e.g., a user's workstation).

[0170] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and random access memory (RAM) 1070 also connected to the bus. The client computer is further provided with a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard disk drive 1030. Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the foregoing may be supplemented by or included therein by a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a tactile device 1090 (e.g., a cursor control device, keyboard, etc.). The cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, with mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0171] A computer program may include computer-executable instructions, which include units for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, as digital electronic circuitry, or as computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be executed by a programmable processor executing instructions for performing the function of the method by manipulating input data and generating output. The processor may therefore be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly language or machine language, if desired. In any case, the language may be a compiled language or an interpreted language. The program may be a complete installation program or update program. The application of the program on the system in any case receives instructions for performing the method.

Claims

1. A computer-implemented method for processing 3D signals of shape attributes on a real object, the method comprising: -supply: • A graph with nodes and arcs, where each node represents a corresponding point in a measured 3D discrete representation of the real object, and each arc connects two nodes representing adjacent points in the discrete representation; and • A set of values ​​representing the distribution of the shape attributes on the real object, each value being associated with a corresponding node in the graph and representing the shape attribute at the corresponding point represented by the corresponding node; as well as - The set of values ​​is modified by minimizing the energy defined on the Markov random field formed on the graph, wherein for each arc connecting the first node to the second node, the energy penalizes the height of an increasing function of the following terms to the first node associated with the first value and the second node associated with the second value: • The distance between the first value and the second value • The distance between the first point represented by the first node and the intermediate geometric elements of the discrete representation, and • The distance between the second point represented by the second node and the intermediate geometric element.

2. The method according to claim 1, wherein, The Markov random field has target values ​​representing the target distribution of the shape property, and for each node, the energy further penalizes the height of another increasing function associated with the value of that node and at least one distance between at least one target value.

3. The method according to claim 2, wherein, The other increasing function is the increasing function of the following terms: - The distance between the value associated with this node and the first target value; and - The distance between the value associated with this node and the second target value.

4. The method according to claim 2, wherein, The at least one target value includes at least one value of a shape attribute calculated geometrically at the corresponding point represented by the node, and / or at least one value of a shape attribute inferred by a neural network at the corresponding point represented by the node.

5. The method according to claim 2, wherein, The at least one target value includes multiple target values, and the other incrementing function further quantifies the correspondence between the various target values ​​included in the at least one target value.

6. The method according to claim 5, wherein, The other increasing function is a function of the product between the correspondence and the at least one distance.

7. The method according to any one of claims 1 to 6, wherein, The intermediate geometric element is either the intermediate axis of the discrete representation or the intermediate surface of the discrete representation.

8. The method according to any one of claims 1 to 6, wherein, The discrete representation is a 3D point cloud.

9. The method according to any one of claims 1 to 6, wherein, The discrete representation is derived from photogrammetry and / or scanning.

10. The method according to any one of claims 1 to 6, wherein, The shape attribute is normal, curvature, or the type of the primitive.

11. The method according to any one of claims 1 to 6, wherein, The real object includes at least one sharp feature.

12. The method according to any one of claims 1 to 6, wherein, The method further includes segmenting the discrete representation based on a set of modified values.

13. A computer program product, comprising: Instructions for performing the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having instructions recorded thereon for performing the method according to any one of claims 1 to 12.

15. A system comprising a processor coupled to a memory and a graphical user interface, the memory having instructions thereon stored for performing the method according to any one of claims 1 to 12.

Images