Transformative basic learning

Abstract

The invention relates in particular to a computer-implemented machine learning method. The method comprises providing a dataset of 3D modeled objects. The method further comprises learning a neural network. The neural network is configured for inferring a deformation basis of an input 3D modeled object. This constitutes an improved machine learning method.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for machine learning to deform 3D modeled objects. Background Technology

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, which involves software solutions for designing objects. CAE stands for Computer-Aided Engineering, which involves software solutions for simulating the physical behavior of future products. CAM stands for Computer-Aided Manufacturing, which involves software solutions for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play a crucial role in technical efficiency. These technologies can be embedded within Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge for product development across extended enterprise concepts, from ideation to the end of the product lifecycle. PLM solutions offered by Dassault Systèmes (traded as CATIA, ENOVIA, and DELMIA) provide: an Engineering Center, which organizes product engineering knowledge; a Manufacturing Center, which manages manufacturing engineering knowledge; and an Enterprise Center, which enables enterprise integration and connectivity between the Engineering Center and the Manufacturing Center. The system delivers an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, thereby driving optimized product definition, manufacturing preparation, production, and service.

[0003] In this context and others, deforming 3D modeled objects is gaining widespread importance.

[0004] The following papers relate to this field and are mentioned below:

[0005] [1]W.Wang, D.Ceylan, R.Mech, and U.Neumann.3dn: 3d deformation network. InConference on Computer Vision and Pattern Regognition (CVPR), 2019.

[0006] [2]T.Groueix,M.Fisher,VGKim,B.Russell,and M.Aubry.AtlasNet:APapier- Approach to Learning 3D Surface Generation.In Conference onComputer Vision and Pattern Regognition(CVPR),2018.

[0007] [3]Reconstructing a 3D Modeled Object.Patent US9978177B2,grantedin2018. Mehr and Vincent Guitteny.

[0008] [4]D.Jack,J.K.Pontes,S.Sridharan,C.Fookes,S.Shirazi,F.Maire,andA.Eriksson.Learning free-form deformations for 3d object reconstruction.InAsian Conference on Computer Vision(ACCV),2018.

[0009] [5]I.Kokkinos,and A.Yuille.Unsupervised Learning of ObjectDeformation Models.In International Conference on Computer Vision(ICCV),2007.

[0010] [6]J.Mairal,F.Bach,and J.Ponce.Sparse Modeling for Image and VisionProcessing.New Foundations and Trends,2014.

[0011] [7]V.Blanz and T.Vetter.A morphable model for the synthesis of 3dfaces.In SIGGRAPH,1999.

[0012] [8]C.Qi,H.Su,K.Mo,and L.Guibas.Pointnet:Deep learning on point sets for 3d classification and segmentation.In Conference on Computer Vision andPattern Regognition(CVPR), 2017.

[0013] [9] R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-Or. Meshcnn: A network with an edge. In SIGGRAPH, 2019.

[0014]

[10] Eloi Mehr, Ariane Jourdan, Nicolas Thome, Matthieu Cord, and Vincent Guitteny. DiscoNet: Shapes Learning on Disconnected Manifolds for 3D Editing. InICCV 2019.

[0015] However, there remains a need for improved solutions for deforming 3D modeled objects. Summary of the Invention

[0016] Therefore, a computer-implemented machine learning method is provided. This method includes providing a dataset of 3D modeled objects. The method further includes learning a neural network. The neural network is configured to infer the deformation basis of the input 3D modeled objects. This method may be referred to as a "learning method".

[0017] This constitutes an improved solution for deforming 3D modeled objects.

[0018] Significantly, the neural network learned through this learning method is configured to take a 3D modeled object as input and output a deformation basis for that object. The deformation basis comprises vectors that each form a deformation of the input 3D modeled object. This allows for numerous deformations of the 3D modeled object because the vectors of the basis can be linearly combined, with each possible linear combination producing a different deformation of the 3D modeled object. This learning method thus allows for an increase in the possibilities of deformations of the 3D modeled object. Furthermore, this learning method allows for the computation of the deformation basis once and for all, through a single application of the learned neural network, so that only linear combinations need to be performed thereafter. This allows for efficient computation of deformations, especially in terms of computation time and resource usage. It should also be noted that the neural network does not require a target deformation of the 3D modeled object to generate the deformation basis: the neural network generates the basis of deformation vectors independently, which can then be linearly combined to achieve the target deformation of the input 3D modeled object. This learning method therefore has strong generalization capabilities and maximizes the expressiveness of the input 3D modeled object.

[0019] Neural networks can infer a basis for local deformation, which can include local deformation vectors, each representing a local (e.g., small) deformation of the input 3D modeled object. This allows the 3D modeled object to be practically deformed in its local neighborhood (i.e., in the vicinity of the 3D modeled object). In other words, the neural network outputs a satisfactory basis in this sense: a basis that allows the input object to be deformed toward a proximate object in a physically acceptable manner.

[0020] Furthermore, a neural network is learned on 3D modeled objects in the dataset, which are at least mostly (e.g., all) plausible 3D modeled objects. The neural network thus generates the basis of plausible vectors for the deformation of the input 3D modeled objects. As previously discussed, these vectors can be linearly combined, thereby generating plausible deformations of the input 3D modeled objects, for example, in their local neighborhoods.

[0021] Furthermore, the provided dataset does not need to consist of objects of the same category (e.g., all chairs) for the neural network to learn the basis of reasonable deformation vectors for the input 3D modeled objects. Specifically, if the provided dataset consists of, or substantially consists of, a large number of 3D modeled objects of the same category (e.g., all chairs), the neural network will reasonably infer the physically realistic deformation basis of objects of this category (e.g., chairs). However, the neural network can do so even if the 3D modeled objects in the provided dataset form different categories (e.g., chairs and benches), and even if these categories are not represented by a large number of 3D modeled objects within the provided dataset. Significantly, this learning does not require annotating the objects in the provided dataset with labels (if any) indicating their corresponding categories. Furthermore, the learning does not require modeling objects using templates, such as the average shape of object categories. Additionally, the provided dataset does not need to be aggregated according to the object categories. This further improves the generalization ability of the neural network.

[0022] However, the generalization ability of neural networks is further enhanced when the provided dataset includes 3D modeled objects of two or more categories forming a manifold (e.g., chairs and benches). In fact, a manifold can be formed by objects of different categories with at least two-to-two connectivity, i.e., a certain amount of shared features / properties (e.g., this is also true for chairs and benches). Therefore, for many object categories within the manifold, a single vector of the deformation basis can be a meaningful and reasonable deformation (e.g., the seats of chairs and benches can be deformed similarly). In other words, if the provided dataset forms a manifold of 3D modeled objects (e.g., chairs and benches), the neural network learns in this way: it correlates the deformations of objects of different categories with each other. Even if the provided dataset consists of a small number of modeled objects of different categories forming the manifold, this allows the neural network to learn to infer physically realistic / acceptable deformation bases. This improves the generalization ability of the neural network.

[0023] Generally, the generalization ability that learning gives neural networks means that the training dataset (i.e., the dataset provided) should contain fewer 3D modeled objects per category compared to other deep learning frameworks, without affecting the neural network's ability to infer the basis of deformations. Specifically, fewer objects are needed per category because neural networks can correlate deformations learned across different categories. However, if the provided dataset only includes one of two very different categories of objects, a much larger number of objects may be required.

[0024] The deformations obtained by using the deformation basis derived from the neural network output can be used in 3D modeling, such as for 3D editing applications. The basis inferred by the neural network can also be used in other applications, such as shape matching and nonlinear deformation inference, as discussed further below. Furthermore, the neural network provides linear deformations of the input 3D modeled object, which is efficient for real-time applications.

[0025] This learning method may include one or more of the following:

[0026] - Neural networks include:

[0027] An encoder, configured to take a 3D modeled object as input and output a latent vector representing the input 3D modeled object, and

[0028] A deep feedforward neural network is configured to take the latent vectors output by the encoder as input and output the deformation basis of the 3D modeled object represented by the latent vectors.

[0029] - The learning process includes: minimizing the loss for at least a portion of the dataset for each 3D modeling object and penalizing the distance between the deformation of a 3D modeling object by a linear combination of vectors and another 3D modeling object for each candidate deformation basis with vectors.

[0030] - Learning is performed on a mini-batch basis and includes minimizing the loss for each mini-batch;

[0031] - The learning process includes: selecting another 3D modeling object from at least a portion of the 3D modeling objects in the dataset based on the distance to at least a portion of the 3D modeling objects in the dataset;

[0032] -The other 3D modeling object is the 3D modeling object that is closest to at least a portion of the 3D modeling objects in the dataset;

[0033] - The learning is performed on a mini-batch basis and includes: for each mini-batch, minimizing the loss and selecting the closest 3D modeling object among the 3D modeling objects in the mini-batch;

[0034] - The minimum distance between the loss-penalty 3D modeling object and the other 3D modeling object, calculated by a linear combination of vectors;

[0035] -Loss can be of the following types:

[0036]

[0037] in:

[0038] ·e1,...,e N It refers to 3D modeled objects that constitute at least a portion of the dataset, where N is the number of objects in at least a portion of the dataset.

[0039] ·p1,...,p N These are respectively from e1,..., e N The obtained point cloud,

[0040] For each 3D modeling object e i g w (f w (e i ),p i )1,...,g w (f w (e i ),p i ) n is the vector of the candidate deformation basis of the 3D modeled object, and n is the size of the candidate deformation basis.

[0041] · It is a 3D modeling object e i According to the transformation of linear combination of vectors, α1,...,α n These are the coefficients of the linear combination.

[0042] ·q i The point cloud is obtained from the other 3D modeled object.

[0043] ·d CH It's distance.

[0044] Neural networks have weights, where w represents the weights of the neural network.

[0045] ·f w It is an encoder, configured to take a 3D modeled object as input and output a latent vector representing the input 3D modeled object, and

[0046] ·g w It is a deep feedforward neural network, which is configured to take the latent vectors output by the encoder as input and output the deformation basis of the 3D modeled object represented by the latent vectors;

[0047] - The loss penalty further penalizes the coefficients of the linear combination as the sparsity-inducing function of the input; and / or

[0048] - The loss further rewards the orthonormality of the candidate deformation basis.

[0049] Furthermore, a neural network capable of learning according to a learning method is provided, such as a neural network that learns directly (i.e., has already been learned) through a learning method.

[0050] A further method using neural networks is provided.

[0051] The method of use may include: providing a neural network (e.g., by performing a learning method), and applying the neural network to one or more 3D modeled objects (i.e., using the neural network to infer one or more deformation bases, each corresponding to a corresponding input 3D modeled object). The method of use can form an application of the learning method. This application may be deep frame reconstruction, shape matching, or 3D editing. The method of use can be integrated into the learning method, for example, as a step performed after learning the neural network; in this case, the learning method and the method of use form a computer-implemented method for inferring deformations of 3D modeled objects.

[0052] A computer program including instructions is further provided, which, when executed on a computer, cause the computer to perform a learning method and / or a method of use.

[0053] A computer-readable data storage medium is further provided on which computer programs and / or neural networks are recorded.

[0054] A computer is further provided, which includes a processor coupled to a memory on which computer programs and / or neural networks are stored. Attached Figure Description

[0055] Embodiments of the invention will now be described by way of non-limiting examples and with reference to the accompanying drawings, wherein:

[0056] Figure 1 An example of a computer is shown; and

[0057] Figures 2 to 4 The method is shown. Detailed Implementation

[0058] The learning methods and the methods used are computer-implemented methods.

[0059] This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the execution of the method's steps by a computer may be fully automatic or semi-automatic. In the example, at least some of the steps that trigger the method can be executed through user-computer interaction. The required level of user-computer interaction can depend on the anticipated level of automation and be balanced with the need to fulfill the user's intentions. In the example, this level can be user-defined and / or predefined.

[0060] A typical example of a computer implementation of the method is to execute the method using a system suitable for this purpose. This system may include a processor coupled to memory and a graphical user interface (GUI) on which a computer program is recorded, the computer program including instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for such storage and may include several physically distinct components (e.g., one for the program and possibly one for the database).

[0061] These methods typically manipulate modeling objects. A modeling object is any object defined by data stored, for example, in a database. By extension, the expression "modeling object" specifies the data itself. Depending on the type of system, modeling objects can be defined using different kinds of data. This system can actually be any combination of CAD, CAE, CAM, PDM, and / or PLM systems. In those different systems, modeling objects are defined by corresponding data. Therefore, one can refer to CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, and CAE data. However, these systems are not mutually exclusive, because modeling objects can be defined using data corresponding to any combination of these systems. Therefore, a system can also be both a CAD and a PLM system simultaneously.

[0062] The term "CAD system" also implies any system suitable for designing modeled objects based on a graphical representation of those objects, such as CATIA. In this context, the data defining the modeled object includes data that allows for the representation of the object. CAD systems can provide a representation of CAD modeled objects, for example, using edges or lines (and in some cases, faces or surfaces). Lines, edges, or surfaces can be represented in various ways, such as non-uniform rational B-splines (NURBS). Specifically, CAD files contain specifications from which geometry can be generated, which in turn allow for the generation of representations. The specifications of the modeled object can be stored in a single CAD file or multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of one megabyte per part. And modeled objects can often be assemblies of thousands of parts.

[0063] In the context of CAD, modeling objects can typically be 3D modeling objects, such as representations of products (e.g., parts or assemblies of parts, or possibly assemblies of products). The term "3D modeling object" means any object modeled from data that allows for its 3D representation. 3D representation allows parts to be viewed from all angles. For example, a 3D modeling object, when represented in 3D, can be manipulated and rotated about any of its axes or about any axis on the screen displaying the representation. This specifically excludes 2D icons that are not 3D modeled. The display of 3D representations facilitates design (i.e., increases the speed at which designers can statistically complete their tasks). Since product design is part of the manufacturing process, this accelerates the manufacturing process in the industry.

[0064] 3D modeling objects can represent the geometry of a product to be manufactured in the real world after its virtual design has been completed using, for example, CAD software solutions or CAD systems. Examples include (e.g., mechanical) parts or assemblies of parts (or equivalent assemblies of parts, since from a methodological perspective, an assembly of parts can be viewed as the parts themselves, or these methods can be applied independently to each part of the assembly), or more generally, any rigid body assembly (e.g., a moving mechanism). CAD software solutions allow for product design in a wide range of unrestricted industrial sectors, including: aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine and / or offshore oil and gas production or transportation. Any 3D modeling object involved in these methods can therefore represent an industrial product, which can be any mechanical part, such as: parts of land vehicles (including, for example, automobile and light truck equipment, racing cars, motorcycles, truck and motor vehicle equipment, trucks and buses, trains), parts of aircraft vehicles (including, for example, fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), parts of marine vehicles (including, for example, naval equipment, commercial ships, marine equipment, yachts and workboats, ship equipment), general mechanical parts (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installed equipment, industrial equipment products, metal products, tire products), electromechanical or electronic parts (including, for example, consumer electronics products, safety and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical devices and equipment), consumer goods (including, for example, furniture, home and garden products, leisure products, fashion products, products of hard goods retailers, products of soft goods retailers), and packaging (including, for example, food and beverage and tobacco, beauty and personal care, and household product packaging).

[0065] Figure 1 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0066] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication BUS 1000 and random access memory (RAM) 1070 also connected to the BUS. The client computer is further provided with a graphics processing unit (GPU) 1110 associated with a video RAM 1100 connected to the BUS. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device (e.g., a hard disk drive 1030). Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, as examples; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the above can be supplemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a tactile device 1090, such as a cursor control device, a keyboard, etc. The cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, whose buttons are used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0067] The computer program may include computer-executable instructions, including units for causing the system to perform the methods. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, as digital electronic circuitry or as computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus (e.g., tangibly embodied in a machine-readable storage device for use in a product executed by a programmable processor). The method steps can be executed by the programmable processor executing the instruction program to perform the function of the method by manipulating input data and generating output. The processor can therefore be programmable and coupled to receive data and instructions from the data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or, if desired, in assembly or machine language. In any case, the language may be a compiled language or an interpreted language. The program may be a complete installer or updater. In any case, the application of the program on the system generates instructions for performing these methods.

[0068] We will now discuss the dataset that provides the 3D modeled objects. Before proceeding with this discussion, we will now discuss the data structures involved.

[0069] Any 3D modeling object in this paper can form a discrete geometric representation of a 3D shape, such as representing objects from the real world, like the mechanical parts discussed earlier. A discrete geometric representation in this paper is a data structure comprising a discrete set of data segments. Each data segment represents a corresponding geometric entity located in 3D space. Each geometric entity represents a corresponding location of the 3D shape (in other words, a corresponding portion of material composed of entities represented by the 3D shape). The aggregation (i.e., union or juxtaposition) of geometric entities collectively represents the 3D shape. Any discrete geometric representation in this paper may include more than 100, 1000, or 10000 such data segments in the examples.

[0070] Any discrete geometric representation in this paper may be, for example, a 3D point cloud, where each geometric entity is a point. Alternatively, any discrete geometric representation in this paper may be a 3D mesh, where each geometric entity is a mesh tile or face. Any 3D mesh in this paper may be regular or irregular (i.e., composed of faces of the same type or not). Any 3D mesh in this paper may be a polygonal mesh, such as a triangular mesh. Any 3D mesh in this paper may alternatively be a B-Rep. Any 3D mesh in this paper may be obtained from a 3D point cloud, for example, by triangulating the 3D point cloud (e.g., using Delaunay triangulation). Any 3D point cloud in this paper may be determined, for example, within a 3D reconstruction process based on physical measurements of a real object. The 3D reconstruction process may include: providing a real object; providing one or more physical sensors, each configured to acquire a corresponding physical signal; and acquiring one or more corresponding physical signals by operating the one or more physical sensors on the real object (i.e., scanning the real object with each sensor). According to any known technique, the 3D reconstruction may then automatically determine the 3D point cloud and / or 3D mesh based on the measurement results. One or more sensors may include multiple (e.g., RGB and / or image or video) cameras, and determination may include structure-from-motion analysis. One or more sensors may alternatively or additionally include one or more depth sensors (e.g., on an RGB depth camera), and determination may include 3D reconstruction based on depth data. One or more depth sensors may, for example, include lasers (e.g., LiDAR) or ultrasonic transmitter-receiver pairs.

[0071] Any 3D point cloud or 3D mesh described herein can be alternatively obtained from a 3D modeling object representing the skin (i.e., the outer surface) of an entity (e.g., a B-Rep model corresponding to the skin (i.e., the exact surface)) by, for example, by ray casting onto the 3D modeling object or by subdividing the 3D modeling object. Subdivision can be performed according to any 3D modeling object rendering process. Such a rendering process can be coded on any CAD system to display a graphical representation of the 3D modeling object. The 3D modeling object can be designed by the user using a CAD system or has already been designed by the user using a CAD system.

[0072] Providing a dataset can include, for example, forming the dataset by creating 3D modeled objects. Alternatively, providing a dataset can include retrieving the dataset from (e.g., remote) memory, where the dataset has been stored after its creation. The 3D modeled objects of the dataset can all be 3D point clouds, generated, for example, by preprocessing 3D meshes. Alternatively, they can all be 3D meshes. In this case, the learning method can include preprocessing these meshes, which samples them into the 3D point cloud. The preprocessing can then include centering each 3D mesh. The preprocessing can then include independently resizing each 3D mesh such that the mesh vertices fit exactly into a unit cube. The preprocessing can then include, for example, extracting a dense point cloud from each normalized shape in the dataset by ray casting each normalized shape on six orthographic views. The preprocessing can then include uniformly subsampling each point cloud (e.g., by downsampling each point cloud to the same size). Subsampling can start from random points in the point cloud and iteratively select the points in the point cloud furthest from the already selected points to reach the desired number of points.

[0073] The 3D modeling objects in the dataset can be plausible (e.g., realistic) 3D modeling objects. A plausible 3D modeling object can specify a 3D modeling object representing a real-world object, such as a plausible mechanical part. A plausible mechanical part can specify a mechanical part that can be practically manufactured in a real-world industrial manufacturing process. A plausible mechanical part can refer to a mechanical part that complies with all the constraints that must be followed in order to actually manufacture that mechanical part in a real-world industrial manufacturing process. Constraints can include one or more of the following: mechanical constraints (e.g., constraints derived from classical laws of mechanics), functional constraints (e.g., constraints relating to one or more mechanical functions that the mechanical part will perform once manufactured), manufacturing constraints (e.g., constraints relating to the ability to apply one or more manufacturing tools to the mechanical part during one or more manufacturing processes used to manufacture the mechanical part), structural constraints (e.g., constraints relating to the strength and / or resistance of the mechanical part), and / or assembly constraints (e.g., constraints defining how the mechanical part can be assembled with one or more other mechanical parts).

[0074] As previously discussed, the 3D modeled objects in the provided dataset may all or substantially all belong to a single category of 3D modeled objects. Alternatively, they may form different categories of 3D modeled objects, such as categories forming a manifold. In any case, the 3D modeled objects in the dataset each represent a corresponding object from the real world, such as the mechanical parts discussed previously.

[0075] Now let's discuss learning through neural networks.

[0076] As is known from the field of machine learning itself, the processing of input by a neural network involves applying operations to the input, which are defined by data including weight values. Learning a neural network therefore involves determining the weight values ​​based on a dataset configured for this learning, which may be called a learning dataset or training dataset. For this purpose, the dataset includes data segments, each forming a corresponding training sample. The training samples represent the diversity of situations in which the neural network will be used after being learned. Any dataset cited in this paper may include more than 1,000, 10,000, 100,000, or 1,000,000 training samples. In this paper, the neural network is learned on the provided dataset, meaning that the provided dataset is the learning / training dataset for the neural network. This learning can be performed using any suitable known method.

[0077] A neural network is configured to infer the deformation basis of an input 3D modeled object. In other words, the neural network takes a 3D modeled object as input and outputs the deformation basis of that object. For this purpose, the input 3D modeled object has the same data type as the 3D modeled objects in the dataset. For example, if the 3D modeled objects in the dataset are 3D meshes, then the input 3D modeled object is also a 3D mesh. Alternatively, if the 3D modeled objects in the dataset are 3D point clouds, then the input 3D modeled object is also a 3D point cloud. Nevertheless, the 3D modeled object can be a 3D point cloud generated from sampling of a mesh. The deformation basis is a set of vectors, each vector being the direction of deformation. Vectors can be linearly combined to deform the input 3D modeled object; a linear combination of vectors with their set of coefficients (also called magnitudes) produces a deformation. Vectors can be linearly combined in such a way that if the linear combination is small enough, the deformed input 3D modeled object is close to the original input 3D modeled object. In this case, the deformation is real. Mathematically, the deformation is real as long as its manifold remains sufficiently close to the deformable position in the tangent space defined by the deformation basis. The basis of deformations can be a basis in the linear algebraic sense, i.e., a set of linearly independent vectors, which may be normalized, for example, orthogonal. Specifically, the learning objective is to enable the neural network to infer, or at least tend to infer, the basis of linearly independent (e.g., uncorrelated and / or orthogonal, as discussed further below) deformation vectors. The basis of deformations can have a fixed size (e.g., between 2 and 10 vectors, or more than 10 vectors). In other words, the neural network can always (i.e., for each input 3D modeling object) output a basis of deformations with the same fixed number of vectors. Given a dataset consisting of reasonable 3D modeling objects, the neural network infers the basis of reasonable deformation vectors for the 3D modeling objects because it is learned to do so.

[0078] The neural network has an architecture configured to take a 3D modeled object as input and output its deformable basis. The neural network may include an encoder and a deep feedforward neural network. The encoder is configured to take a 3D modeled object as input and output a latent vector representing the input 3D modeled object. The encoder may therefore be configured to take a 3D mesh or, for example, a 3D point cloud sampled from a 3D mesh as input. The deep feedforward neural network is configured to take the latent vector output by the encoder as input and output the deformable basis of the 3D modeled object represented by the latent vector. The encoder architecture may be based on PointNet (e.g., described in previously cited reference [8], which is incorporated herein by reference) or any extension thereof. Alternatively, the encoder architecture may use a mesh topology (i.e., if the 3D modeled objects of the dataset are 3D meshes), such as that done in MeshCNN (e.g., see previously cited reference [9], which is incorporated herein by reference).

[0079] The learning may include: minimizing the loss for at least a portion of the dataset; and penalizing the distance between another 3D modeling object and the deformation of the 3D modeling object by a linear combination of vectors for each 3D modeling object in the dataset and for each candidate deformation basis having vectors.

[0080] Now let's discuss minimizing the loss.

[0081] At least a portion of the dataset consists of samples of 3D modeled objects. For example, at least a portion of the dataset may be mini-batches, in which case learning is performed sequentially on a mini-batch basis and includes minimizing the loss for each mini-batch. The concept of sequential mini-batch learning is known from the field of machine learning itself. For example, learning can implement any known mini-batch stochastic optimization method, such as mini-batch stochastic gradient descent. As is known from the field of machine learning itself, sequential mini-batch learning improves learning efficiency.

[0082] For each 3D modeled object in at least a portion of the dataset, a deformable basis of the modeled object is learned and evaluated by the neural network using its current weight values. This computed deformable basis forms candidate deformable bases for the 3D modeled object. The loss penalizes the distance between the deformable 3D modeled object, calculated as a linear combination of these candidate basis vectors, and another 3D modeled object. This means that the loss tends to increase when the distance between the deformable 3D modeled object, calculated as a linear combination of these candidate basis vectors, and the other 3D modeled object is large. If this is the case, the learning process that minimizes the loss modifies the weights of the neural network to reduce the loss value, thus generating new candidates. This process is repeated (e.g., mini-batch) until the loss reaches its minimum or is at least sufficiently small (e.g., relative to a convergence criterion).

[0083] The other 3D modeling object can be a target object, i.e., where a linear combination of vectors is used to deform the 3D modeling object. This allows training a neural network to infer the basis of the deformation. For example, the other 3D modeling object can be: among 3D modeling objects in at least a portion of the dataset, the 3D modeling object that is close to the 3D modeling objects in at least a portion of the dataset. In other words, learning can significantly include: selecting the other 3D modeling object among the 3D modeling objects in at least a portion of the dataset based on the distance to the 3D modeling objects in at least a portion of the dataset. Selection can include: calculating the distance between each 3D modeling object in at least a portion of the dataset and the 3D modeling objects in at least a portion of the dataset, and evaluating which 3D modeling objects are close, i.e., for which 3D modeling objects the distance is small. Then, selection can choose the 3D modeling object with the small distance. In the example, selection can evaluate which 3D modeling object is the closest, i.e., for which 3D modeling object the distance is minimum. In other words, selection calculates the minimum distance and evaluates which 3D modeling object produces this minimum value. In these examples, the other 3D modeling object is the 3D modeling object that is closest to at least a portion of the 3D modeling objects in the dataset. In the examples, learning can be performed mini-batch by mini-batch, and the learning includes: for each mini-batch, minimizing the loss and selecting the closest 3D modeling object from the 3D modeling objects in the mini-batch. In other words, the other 3D modeling object is selected from the objects in the mini-batch, where at least a portion of the dataset is a mini-batch.

[0084] The distance to at least a portion of the 3D modeled objects in the dataset can be any distance, such as the distance between 3D modeled objects, or the 3D distance between point clouds or 3D meshes. In the example, this distance can be a distance in the latent space, as discussed now.

[0085] Specifically, in these examples, the learning method may include, for example, providing another encoder before learning, such as at the initial stage of the learning method, which is configured to take 3D modeled objects of the provided dataset as input and output a latent vector encoding these 3D modeled objects. This other encoder may be trained in an autoencoder framework, in a classification task, or through any other machine learning method to learn a meaningful latent space of the 3D modeled objects, as known from the field of machine learning itself. Providing the other encoder may include training the other encoder. Alternatively, the other encoder may be pre-trained. In these examples, for example, the distance between a first 3D modeled object and a second 3D modeled object may be the distance in the latent space of the other encoder between the result of applying the other encoder to the first 3D modeled object and the result of applying the other encoder to the second 3D modeled object. This latent space distance allows training a neural network to infer the basis of deformations with higher accuracy because the latent vector output by the other encoder captures the semantics encoding the 3D modeled objects. In other words, the encoder implicitly aggregates the dataset with respect to semantics. This allows the other 3D modeled object to be identified as the 3D modeled object most similar to at least a portion of the 3D modeled objects in the dataset. For example, using this latent spatial distance improves the accuracy and efficiency of learning compared to distances in the 3D modeling object space (which may cost more to compute).

[0086] The selection of the other 3D modeling object can be performed before minimization. For example, the result of applying the other encoder to the 3D modeling object of the dataset can be pre-computed, i.e., computed before minimization.

[0087] Linear combinations have coefficients, each corresponding to the contribution of the corresponding vector in the combination. These coefficients can be computed via a neural network during learning. Alternatively, they can be the result of optimization performed during learning.

[0088] The loss can penalize the minimum distance between the deformation of a 3D modeled object as a linear combination of vectors and the other 3D modeled object. This selects the optimal deformation as a linear combination, which minimizes the distance while minimizing the loss. The minimum can be, for example, the minimum distance between the deformation of the 3D modeled object as a linear combination of vectors and the other 3D modeled object among all possible values ​​of the coefficients of the linear combination. The loss penalizes the minimum because the loss tends to increase when the minimum is large. If this is the case, the learning to minimize the loss modifies the weights of the neural network to reduce the loss value. The learning does this (e.g., mini-batch) until the loss reaches its minimum or at least a sufficiently small value (e.g., relative to a convergence criterion).

[0089] Losses can be of the following types:

[0090]

[0091] in:

[0092] ·e1,...,e N It refers to 3D modeled objects that constitute at least a portion of the dataset, where N is the number of objects in at least a portion of the dataset.

[0093] ·p1,...,p N These are respectively from e1,...,e N The obtained point cloud,

[0094] For each 3D modeling object e i g w (f w (e i ),p i )1,...,g w (f w (e i ),p i ) n It is the vector of candidate deformation bases for the 3D modeled object, where n is the size of the candidate deformation base.

[0095] · It is a 3D modeling object e i According to the transformation of linear combination of vectors, α1,...,α n These are the coefficients of the linear combination.

[0096] ·q i The point cloud is obtained from the other 3D modeled object.

[0097] ·d CH It's distance.

[0098] Neural networks have weights, where w represents the weights of the neural network.

[0099] ·f w It is an encoder, configured to take a 3D modeled object as input and output a latent vector representing the input 3D modeled object, and

[0100] ·g w It is a deep feedforward neural network that is configured to take the latent vectors output by the encoder as input and output the deformation basis of the 3D modeled object represented by the latent vectors.

[0101] e1,...,e N It may already be a 3D point cloud. In this case, e1 = p1,...,e N =pN Alternatively, e1,...,e N It can be a 3D mesh, in which case p1,...,p N It can be derived from the pairs e1,...,e N The sampling is generated. d CH This can be any 3D distance between 3D point clouds, such as earth-mover distance or chamfer distance. CH In the example, it can be of the following types:

[0102]

[0103] q i The calculation can be performed as follows. Let f′ be another encoder discussed earlier. Then, for each i, the learning method is determined. Then, q i By q i =p i′ Given. As discussed previously, the vector f′(e) can be pre-computed. i ).

[0104] For each 3D modeling object and each candidate deformation basis, the loss can be further penalized by a sparse induced function (e.g., its minimum), which takes the coefficients of a linear combination as input. As previously discussed, the sparse function can be, for example, the L1 norm of the coefficients of the linear combination (e.g., see reference [6], which is incorporated herein by reference). In other words, the loss tends to increase when the sparse induced function (e.g., its minimum) is large. If this is the case, the learning to minimize the loss modifies the weights of the neural network to reduce the loss value. The learning does this (e.g., mini-batch) until the loss reaches its minimum or at least a sufficiently small value (e.g., relative to a convergence criterion). In this case, the loss can be of the following types:

[0105]

[0106] λ is a tradeoff parameter (a well-known concept in machine learning), and φ is a sparse induced function, such as L0. 1 Norms (e.g., see previously cited reference [6], which is incorporated herein by reference), (α1,...,α) p ) is a vector of coefficients, also known as a vector of magnitudes.

[0107] This allows for sparse coefficients / amplitudes during learning. Ultimately, this allows the neural network to learn uncorrelated deformation basis vectors for its output, which further improves the generalization ability of the learning method. Significantly, neural networks learned in this way output satisfactory basis vectors because their sparsity allows for the computation of deformations of input 3D modeled objects as linear combinations of basis vectors using a small number of vectors. Furthermore, the output basis is, or at least tends to be, a basis vector in the linear algebraic sense.

[0108] For each 3D modeled object and each candidate deformable basis, the loss can further reward the orthogonality of the candidate deformable basis. In other words, the loss can further include a term that captures, for example, the orthogonality of the deformable basis of the 3D modeled object inferred by the neural network using its current weight values, which tends to be smaller as the deformable basis tends to be orthogonal. In this case, the loss can be of the following type:

[0109]

[0110] δ is a tradeoff parameter (a concept well-known from the field of machine learning). This further allows neural networks to learn to output uncorrelated deformable fundamental vectors, as they tend to be orthogonal. Thus, this foundation is, or at least tends to be, a foundation in the sense of linear algebra.

[0111] The loss can be further included with any suitable regularization function, as known from the field of machine learning itself.

[0112] Now we will discuss how to implement the learning methods.

[0113] This implementation brings improvements to the fields of 3D modeling and 3D machine learning. The results of this implementation can be used in fields such as virtual reality and augmented reality (more generally, any kind of immersive experience), video games, manufacturing and 3D printing, or 3D modeling. This implementation provides a solution for calculating the realistic sparse deformation linear basis of any 3D modeled object, which can then be used to perform shape synthesis, shape reconstruction from an image, or shape matching.

[0114] This implementation allows for a reasonable linear deformation basis for any 3D modeled object, enabling the model to be practically deformed in a local neighborhood by any linear combination of the deformation vectors constituting its basis. This deformation can be used in 3D modeling software, particularly for 3D editing applications. That is, it can be used for large nonlinear deformations and shape matching inference.

[0115] This implementation trains a neural network (a learning-in-the-learning neural network) that learns a linear 3D deformation field with sparse constraints to obtain reasonable deformations that are independent of each other. Starting with a provided dataset (a dataset of unlabeled 3D modeled objects), this implementation (in the learning process) learns a neural network that infers the deformation basis at each point of the input 3D modeled object model. To learn this network, learning is performed in mini-batches, as previously discussed:

[0116] 1) Match each 3D modeling object in each batch with the closest 3D modeling object in the same batch. In other words, as previously discussed, select the other 3D modeling object that is closest to the 3D modeling object in that batch.

[0117] 2) Using an additional sparse regularization term, compute a linear combination of the predicted deformation vectors of the 3D modeled object that minimizes the distance between: the deformation of the 3D modeled object as a linear combination of deformation vectors inferred by the neural network using its current weights, and the 3D modeled object that is closest to it in the batch.

[0118] 3) The same loss (the distance between the deformed 3D modeling object and its nearest 3D modeling object) is minimized by adjusting the weights of the neural network to optimize the prediction deformation basis.

[0119] This implementation does not require aggregating different types of 3D modeled objects to learn each aggregation individually. Instead, it is suitable to learn directly from the entire dataset together. This implementation learns unrelated deformation bases, where each deformation is inherently plausible. The inferred deformations are linear and therefore usable in real-time applications. Deformations are inferred as 3D deformation fields that maximize the expressiveness of the input 3D modeled object.

[0120] When providing a dataset, if the dataset consists of 3D meshes, this implementation can perform preprocessing as previously discussed. Preprocessing includes centering each mesh and independently resizing it so that the mesh vertices fit precisely into a unit cube. Preprocessing then includes extracting a dense point cloud from each normalized shape in the dataset by ray casting onto each normalized shape on six orthogonal views. Preprocessing then includes uniformly subsampling each point cloud (e.g., by downsampling each point cloud to the same size). For this purpose, preprocessing can begin with random points in the point cloud and iteratively select the points in the point cloud furthest from the already selected points until a predefined (e.g., desired) number of points is reached.

[0121] We will now discuss the architecture of the neural network according to the implementation method discussed here.

[0122] In this implementation, the neural network infers (e.g., for any input 3D modeling object) the same fixed number of deformable basis vectors. Let n be the fixed, same size of the inferred deformable basis. This implementation includes designing the encoder architecture f. w It takes a 3D point cloud or 3D mesh as input and outputs a latent vector representing the input 3D point cloud or 3D mesh. For example, such encoder architectures can be based on PointNet (e.g., see previously cited reference [8], which is incorporated herein by reference) or many of its extensions, or they can use mesh topologies such as those done in MeshCNN (e.g., see previously cited reference [9], which is incorporated herein by reference). Therefore, f w It can be designed to take a 3D mesh x or a sampled 3D point cloud y as input.

[0123] Attached to f w This implementation includes designing a deep feedforward neural network g. w The deep feedforward neural network g w Take the latent vector h of a 3D point cloud or 3D mesh and a 3D point x as input, and output its deformation basis at point x. To further simplify the representation, for any point cloud or mesh containing m points or vertices... Let g w (h,p) j ,j∈{1,...,n} as vectors therefore, This is based on calculations at all points of X. Furthermore, this implementation can also construct g. w , so that ||g w (h,X) j ||2=1, where all j∈{1,...,n}. w are the trainable weights of neural networks f and g.

[0124] This implementation may include providing something similar to f w Another encoder architecture, f′, is used. As previously discussed, f′ is trained in an autoencoder framework, or in a classification task, or in any other machine learning method, to learn a meaningful latent space for 3D objects. This implementation uses this other encoder to train f′. w During this process, 3D modeling objects (3D meshes or 3D point clouds) are efficiently matched with their closest 3D modeling objects.

[0125] This implementation performs unsupervised learning when learning a neural network. This will now be discussed.

[0126] This learning is achieved by applying a set of N point clouds p1,...,p N Minimize the mini-batch loss E(w) (also known as "energy"), and learn the network f through any mini-batch stochastic optimization (e.g., mini-batch stochastic gradient descent). w and g w Both. Let e1,...,e N f w The corresponding 3D modeling objects in the input space, that is, if these inputs are point clouds e i =p i And if they are grids, then e i It is used for p i The source grid from which sampling is performed.

[0127] For each i, let And q i =p i′ The vector f′(e) can be pre-calculated. i That is, for example, this implementation includes pre-computation of vectors before learning.

[0128] The loss is then given by the following formula:

[0129]

[0130] d CH Here it is the beveled 3D distance, but it can be replaced with any other 3D loss (e.g., seismic distance):

[0131]

[0132] φ is a sparse induced function, such as L 1 Norm [6]. Let (α1,...,α p This is referred to as the amplitude vector. This implementation attempts to make the amplitude sparse in order to make the deformation as uncorrelated as possible, so that any deformation vector is reasonable on its own. For the same reason, this implementation forces the deformation basis to be orthogonal to the δ-penalty standard, before the term capturing the orthogonality of the basis, as previously discussed.

[0133] σ(e i ,e i +v) can be any regularization function, which can include, but is not limited to, combinations of the following:

[0134] -to e i and e i The penalty for the difference in edge length between +v (if ei is a point cloud, this implementation may include creating edges on its points using a k nearest neighbor graph), and

[0135] -to ei and e i The penalty for the difference between the Laplace coordinates of +v, i.e., L(e i ) and L(e i The difference between +v), where L is e i The Laplace operator.

[0136] λ, δ, and γ are trade-off parameters.

[0137] To minimize E(w), learning involves computing the magnitude (α1,...,α) of each input in the batch. p The minimum value of E(w) is obtained, and the gradient of E(w) is calculated.

[0138] For each input i, It can be computed using standard gradient descent or any continuous optimization algorithm.

[0139] However, gradient This may seem difficult to handle, due to reaching the minimum value. amplitude It depends on w.

[0140] make

[0141]

[0142] and

[0143]

[0144]

[0145] in

[0146]

[0147] The envelope theorem makes The gradient of is easy to handle (and therefore the gradient of E(w) is also easy to handle), because it equals The gradient of , where α1,...,α p It is considered constant with respect to w, that is, the gradient calculation will not be affected by the magnitude α1,...,α p The calculation is backpropagated.

[0148] Right now,

[0149]

[0150] This can be followed The definition is proved using the following facts:

[0151] Figure 2 A deep model architecture of a neural network learned according to the implementation method discussed here is shown.

[0152] Now let's discuss the application of learning methods.

[0153] The term "application of learning methods" refers to the use of computer-implemented methods that utilize neural networks that can learn according to learning methods, such as neural networks that have been directly learned (i.e., already learned) by learning methods. The methods used may include: using neural networks to infer one or more deformation bases, each corresponding to a given input 3D modeling object, and using the inferred one or more deformation bases. Learning methods may include methods used after learning, such as further steps within the learning process.

[0154] The first example of the application of the learning method is deep frame reconstruction (see, for example, the previously cited reference [3], which is incorporated herein by reference). Deep frame reconstruction involves providing a 3D mesh and a depth map of another object. The other object is approximated by the provided 3D mesh. Deep frame reconstruction then involves applying a neural network to the 3D mesh, which produces a deformable basis for the 3D mesh. Deep frame reconstruction uses this deformable basis to deform the provided 3D mesh into another object represented by the depth map. Deep frame reconstruction then further involves optimization on the deformable basis to fit the depth map, i.e., deep frame reconstruction involves optimizing the coefficients of a linear combination of the deformable basis vectors such that the 3D mesh is deformed in a linear combination to fit the depth map. The goal of the reconstruction is to obtain a 3D modeled object corresponding to the depth map.

[0155] A second example of the application of the learning method is shape matching. Shape matching involves providing two close 3D meshes e1 and e2. Shape matching then involves computing the deformation basis of the first 3D mesh e1, and optimizing the coefficients of a linear combination of the deformation basis vectors to deform the first 3D mesh e1 such that it matches the second 3D mesh e2. For example, shape matching can compute:

[0156]

[0157] in

[0158]

[0159] Where p1 is the point cloud sampled from e1, and p2 is the point cloud sampled from e2.

[0160] Figure 3A first example of shape matching is shown. The first 3D mesh is chair 30, and the second 3D mesh is chair 36, to which chair 30 will be matched. Chair 34 shows the result of the optimization of the coefficients, while chair 32 shows the intermediate chair calculated during optimization.

[0161] Figure 4 A first example of shape matching is shown. The first 3D mesh is chair 40, and the second 3D mesh is chair 46, to which chair 40 will be matched. Chair 44 shows the result of the optimization of the coefficients, while chair 42 shows the intermediate chair calculated during optimization.

[0162] A third example application of the learning method is 3D editing. 3D editing may include providing a first 3D modeling object and iteratively transforming the first 3D modeling object into a second 3D modeling object by iteratively applying a neural network to iterative deformations of the first 3D modeling object, starting from the first 3D modeling object.

Claims

1. A computer-implemented machine learning method, the method comprising: Provides a dataset of 3D modeled objects; as well as A learning neural network is configured to infer the deformation basis of an input 3D modeling object, wherein the inferred deformation basis of the input 3D modeling object comprises a set of vectors, each vector in the set being a direction of deformation of the input 3D modeling object, and the vectors in the set can be linearly combined, wherein each possible linear combination produces a different deformation of the input 3D modeling object, wherein the neural network comprises: - An encoder configured to take a 3D modeled object as input and output a latent vector representing the 3D modeled object, and - A deep feedforward neural network, configured to take the latent vectors output by the encoder as input and output a deformation basis of the 3D modeled object represented by the latent vectors, and The learning includes: minimizing a loss for at least a portion of the dataset, the loss being applied to each 3D modeling object of the at least a portion of the dataset and to each candidate deformation basis having vectors, penalizing the distance between the deformation of the 3D modeling object by a linear combination of the vectors of the candidate deformation basis and another 3D modeling object, wherein the other 3D modeling object is selected from the 3D modeling objects of the at least a portion of the dataset and is the 3D modeling object closest to the 3D modeling object.

2. The method according to claim 1, wherein, The learning is performed on a mini-batch basis and includes minimizing the loss for each mini-batch.

3. The method according to claim 1, wherein, The learning is performed on a mini-batch basis and includes: for each mini-batch, minimizing the loss and selecting the closest 3D modeling object among the 3D modeling objects in the mini-batch.

4. The method according to any one of claims 1 to 3, wherein, The loss penalty is the minimum distance between the deformation of the 3D modeling object according to the linear combination of the vectors and the other 3D modeling object.

5. The method according to claim 4, wherein, The loss is of the following type: ， in: The 3D modeled object is at least a portion of the dataset. It is the number of objects in at least a portion of the dataset. They are respectively from The obtained point cloud, For each 3D modeling object , The vector is the candidate deformation basis of the 3D modeled object. The size of the candidate deformation basis. It is the 3D modeling object The deformation according to the linear combination of the vectors, These are the coefficients of the linear combination. The point cloud is obtained from the other 3D modeled object. It is the distance mentioned. The neural network has weights. The weights represent the weights of the neural network. It is an encoder configured to take a 3D modeled object as input and output a latent vector representing the input 3D modeled object, and It is a deep feedforward neural network, which is configured to take the latent vector output by the encoder as input and output the deformation basis of the 3D modeled object represented by the latent vector.

6. The method according to any one of claims 1 to 3, wherein, The loss further penalizes the sparse induced function that takes the coefficients of the linear combination as input.

7. The method according to any one of claims 1 to 3, wherein, The loss further rewards the orthogonality of the candidate deformation basis.

8. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 7.

9. A computer-readable data storage medium having instructions recorded thereon, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 7.

10. A computer comprising a processor coupled to a memory having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform the method according to any one of claims 1 to 7.

Images