A neural network that outputs a parameterized 3D model

Abstract

To provide a method executed by a computer for machine learning.SOLUTION: The method uses a 2D sketch as an input, and includes the step of providing an architecture for a neural network formed to output a 3D model expressed by the 2D sketch. The 3D model is a 3D model converted to a parameter defined by a set of parameters including a first subset of at least one parameter and a second subset of at least one parameter. The neural network is formed to selectively output a value for a set, uses the value for the first subset from a user as an input, and outputs a value for the second subset. The method for machine learning also includes the step of learning a neural network.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and computer systems, and more particularly to methods, programs, data structures, and systems relating to neural networks configured to take a 2D sketch as input and output a 3D model represented by the 2D sketch. [Background technology]

[0002] Several systems and programs are available on the market for designing, engineering, and manufacturing objects. CAD stands for Computer-Aided Design, referring to software solutions for designing objects. CAE stands for Computer-Aided Engineering, referring to software solutions for simulating the physical behavior of a resulting product. CAM stands for Computer-Aided Manufacturing, referring to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays a key role in the efficiency of the technology. These technologies may be incorporated into product lifecycle management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge for the development of products from conception to the end of their lifecycle across an extended enterprise. PLM solutions offered by Dassault Systèmes (under the trademarks "CATIA," "ENOVIA," and "DELMIA") provide an Engineering Hub that configures product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to both the Engineering Hub and the Manufacturing Hub. Together, the systems create an open object model that connects products, processes, and resources to enable dynamic, knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this context, researchers have explored solutions for computing a 3D model represented by an input 2D sketch. This is a complex problem, and depending on the view angle selected for the input 2D sketch, it can be difficult for the user to discover the intended 3D model. For example, a 2D sketch of a rectangle may be a representation of either a 3D cylinder (from a frontal view) or a cube (from either canonical view). Furthermore, 2D sketches may be drawn inaccurately, such that their lines may not be flat or straight, or their perspective may not be true to life. These inaccuracies make this problem more difficult to solve. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] DELANOY et al., 3D Sketching using Multi-View Deep Volumetric Prediction, Proceedings of the ACM on Computer Graphics and Interactive Techniques 1.1, 2018, pages 1-22 [Non-patent document 2] WAILY and BOUSSEAU, Sketch-Based Modeling of Parametric Shapes, ACM / EG Expressive Symposium - Posters, Demos, and Artworks, 2019 [Non-patent document 3] LI et al., SweepCanvas: Sketch-based 3D prototyping on an RGB-D image, 2017, pages 387-399 [Non-patent document 4] EITZ et al., Sketch-Based Shape Retrieval, ACM Transactions on Graphics (Proc. SIGGRAPH), 2012, Vol. 31, No. 4, pages 31:1 - 31:10 Summary of the Invention [Problem to be solved by the invention]

[0005] In the prior art, there are different approaches to solving this problem.

[0006] [Non-Patent Document 1] proposes a data-driven approach by training a deep convolutional neural network (CNN) to reconstruct 3D shapes from one or more drawings. The CNN predicts the occupancy of the voxel grid from line drawings, resulting in an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. This single-viewpoint network is complemented by an updater CNN that refines the existing predictions given new drawings of the shape made from novel viewpoints.

[0007] Non-Patent Document 2 proposes a sketch-based modeling system that uses deep convolutional networks to interpret sketches of parametric shapes. The proposed method focuses on creating simple shapes (cubes, cylinders, cones, spheres, and pyramids) that rely on two types of deep convolutional networks for sketch interpretation. First, a classification network recognizes which shape is being drawn. Then, a parameter estimation network predicts the location and dimensions of the recognized shape. Since different categories have different parameters, there is one such network per shape category. Furthermore, to enable the method to disambiguate the elevation of shapes relative to the ground, the shadows cast by each shape on the ground when illuminated from above are rendered and provided to the deep network.

[0008] Non-Patent Document 3 proposes a sketch-based interactive tool for exploratory 3D modeling on top of RGBD images to provide end users with a way to create 3D models related to the image. The tool includes a sketch-based modeling interface that takes pairs of user strokes as input and instantly generates curved 3D surfaces by sweeping one stroke according to the other. The proposed method applies an optimization procedure based on Markov Random Fields (MRFs) that extracts pairs of spatial planes from the context to position and sweep the strokes.

[0009] Non-Patent Document 4 proposes a method for 3D object retrieval based on sketched feature lines as input, using a bag-of-features (BOF) approach to computer-generated line drawings of objects. The method proposes a set of matching models from a dataset to a user based on geometric similarity.

[0010] Within this context, there remains a need for an improved solution that takes an input 2D sketch and outputs a 3D model represented by the 2D sketch. [Means for solving the problem]

[0011] Accordingly, a machine learning computer-implemented method is provided. The method includes providing an architecture for a neural network configured to take a 2D sketch as input and output a 3D model represented by the 2D sketch. The 3D model is a parameterized 3D model defined by a set of parameters consisting of a first subset of one or more parameters and a second subset of one or more parameters. The neural network is configured to selectively output values ​​for the set, take values ​​for the first subset from a user as input, and output values ​​for the second subset. The machine learning method also includes training the neural network.

[0012] The machine learning method is - the architecture includes pieces of data, and the neural network is configured to generate first values ​​for the pieces of data, determine values ​​for a first subset based on said first values ​​for the pieces of data, and determine values ​​for a second subset based on said first values ​​for the pieces of data, to selectively output values ​​for a set, and to selectively take values ​​for the first subset from a user as input, generate second values ​​for the pieces of data based on the values ​​for the first subset, and output values ​​for the second subset based on the second values ​​for the pieces of data, to output values ​​for the second subset; the values ​​for the first subset correspond to an image of a function of the first values ​​for the data piece, and the second values ​​are a preimage of the values ​​for the first subset according to the function; the piece of data is a vector representing a probability distribution over the first subset, the function is an argmax function; the second value of the data piece is a vector having a value of 1 for the coordinate corresponding to the value for the first subset and a value of 0 for all other coordinates; - the 3D model is defined by sections and extrusions, and the first subset is made up of numbers representing types of sections; - the number represents the number of sides forming the section; the second subset includes parameters defining the extrusion and / or parameters further defining the section; the parameters further defining section includes a list of position parameters, each position parameter optionally including a pair of 3D points; the neural network includes a recurrent neural network (RNN) configured to output a value for the list, the neural network optionally being further configured to output a value for the extrusion based on a final state of the RNN; and may include one or more of:

[0013] A computer-implemented method of 3D design is also provided, the method including providing a neural network trained according to a machine learning method, the method further including providing a 2D sketch, providing values ​​for a first subset by a user, and applying the neural network to the 2D sketch, taking as input the values ​​for the first subset from the user, to output values ​​for a second subset.

[0014] The 3D design method may further include applying a neural network to the 2D sketch to output one or more respective values ​​for each set before a user provides values ​​for the first subset; and displaying one or more 3D models represented by the 2D sketch, each defined by the respective values ​​output for the set, and the user providing values ​​for the first subset while the one or more 3D models are displayed.

[0015] There is also provided a computer program comprising instructions which, when executed on a computer system, cause the computer system to perform the method.

[0016] A data structure representing the trained neural network is also provided.

[0017] Further provided is a device including a memory having stored thereon computer programs and / or data structures, which may alternatively form or serve as a non-transitory computer-readable medium or a computer system including a processor coupled to the memory.

[0018] Embodiments will now be described, by way of non-limiting example and with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0019] [Figure 1] 1 illustrates an embodiment of a system. [Figure 2] 1 illustrates an example of a graphical user interface for the system. [Figure 3] Here's how. [Figure 4] Here's how. [Figure 5A] Here's how. [Figure 5B] Here's how. [Figure 6] Here's how. [Figure 7] Here's how. [Figure 8] Here's how. [Figure 9] Here's how. [Figure 10] Here's how. [Figure 11] Here's how. [Figure 12] Here's how. DETAILED DESCRIPTION OF THE INVENTION

[0020] A computer-implemented method of machine learning is proposed below. The machine learning method includes providing an architecture for a neural network. The neural network is configured to take a 2D sketch as input and output a 3D model represented by the 2D sketch. The machine learning method also includes training the neural network. The neural network is configured to output a 3D model represented by a single 2D sketch. This represents high ergonomics for a user to infer, i.e., calculate, a 3D model based on a single corresponding 2D sketch.

[0021] As is known from the field of machine learning, a neural network is a function that includes operations according to an architecture, where each operation is defined by data including weight values. The architecture of a neural network defines the relationship between the operands of each operation and the weight values. Thus, training a neural network includes determining the values ​​of the weights based on a dataset configured for such training. In that regard, a dataset includes data pieces that each form a respective training sample. The training samples represent the diversity of situations in which the neural network will be used after training. Any dataset referred to herein may include a number of training samples, such as 1,000, 10,000, 100,000, or more than 1,000,000.

[0022] The machine learning method may be part of a machine learning process that includes a computer-implemented method for forming such a dataset, which may be referred to as a "dataset formation method." The dataset formation method may be performed before the machine learning method, or alternatively, may be combined with the machine learning method. The dataset formation method includes providing (e.g., synthesizing) one or more 3D models. The dataset formation method may further include determining (e.g., synthesizing) one or more respective 2D sketches that each represent a respective 3D model (e.g., based on the generated 3D models), and inserting one or more training samples into the dataset, each of the one or more training samples including the (e.g., synthesized) 3D model and a respective (e.g., synthesized) 2D sketch.

[0023] Training of the neural network method may, in embodiments, be performed after the dataset formation method and based at least in part on the dataset formed by the dataset formation method, such machine learning processes being particularly efficient and resulting in improved accuracy.

[0024] Alternatively, the machine learning process may include performing the dataset formation method and any other computer-implemented method (other than the proposed training method) that trains a neural network based on the dataset. Alternatively, the machine learning process may include performing the training method on a dataset provided by any other computer-implemented method (other than the proposed dataset formation method), such as forming the dataset or another method of deriving the dataset.

[0025] A computer-implemented method of 3D design is also proposed. The method of 3D design includes providing a neural network trained according to a machine learning method. The method of 3D design also includes providing a 2D sketch. The method of 3D design then includes applying the neural network to the 2D sketch to output a 3D model represented by the 2D sketch. The 3D design process specifies any action or series of actions that are at least part of the process of elaborating the 3D model. Thus, the method may include creating a 3D model from scratch. The method ultimately allows a user to input a 2D sketch and output one or more 3D models, the one or more 3D models being an ergonomic method for performing 3D design.

[0026] 3D design methods provide improved solutions for CAD design. In particular, trained neural networks can be used to convert 2D sketches into solid CAD. In the context of CAD, performing such conversions using trained neural networks enables achieving advantages offered by the field of machine learning, such as fast or real-time execution, the absence of tedious traditional manual programming / coding requirements, and high accuracy of results. Herein, accuracy refers to the 3D similarity value evaluated between a 3D model and the 3D shape intended by the user when sketching the 2D sketch, optionally measured as the 2D similarity value evaluated between a 2D projection of the 3D model (e.g., its characteristic edges) and the 2D sketch. Thus, learning methods improve ergonomics in the context of 3D design.

[0027] The 3D design method may output to the user one single 3D model or several propositions about 3D models. In the latter case, the user may select one according to the user's intention in the design process. In an embodiment, the 3D design method may include displaying a 3D graphical representation of one or more 3D models represented by 2D sketches.

[0028] The 3D design method may be iterative. In such cases, the 3D design method includes providing a neural network and iteratively providing 2D sketches, applying the neural network to each 2D sketch to each time (e.g., automatically) output a new 3D model represented by the 2D sketch. The method may include adding each newly obtained 3D model to the current design, thereby iteratively / incrementally building the (final) 3D model object included in each output 3D model object. Providing each 2D sketch may be performed by a user, for example, by sketching (e.g., via touch technology such as a touchscreen or touchpad). The final 3D model object may represent a mechanical part of a mechanical assembly, as described further below.

[0029] The 3D design method may include a machine learning method, or it may be subsequently performed, for example as part of a machine learning process.

[0030] A modeled object is any object defined by data stored, for example, in a database. By extension, the expression "modeled object" specifies the data itself. Depending on the type of system used to specify the modeled object, the modeled object may be defined by different kinds of data. In the context of CAD, a modeled object may typically be a 3D modeled object (or 3D model) representing a product, such as a part, an assembly of parts, or possibly an assembly of a product. A 3D model is a modeled object that forms a 3D representation of an individual, for example, a 3D spatial distribution of materials made up of a real-world object. The real-world object may be a machine part, an assembly of parts, or possibly an assembly of a product in a mechanical design process. The 3D representation allows the individual to be viewed from all angles. For example, the 3D model may be manipulated and rotated around any of its axes or around any axis within the screen on which the representation is displayed. The 3D model may represent the skin (i.e., exterior surface) of the individual. The 3D model may be provided to a CAD system for display to a user and / or for the user to perform one or more CAD design operations on the 3D model. This specifically eliminates 2D icons that are not 3D modeled. Displaying 3D representations expedites design (i.e., increases the speed at which designers statistically accomplish their tasks). This accelerates the manufacturing process in industry, since the design of a product is part of the manufacturing process.

[0031] A system may actually be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these different systems, modeled objects are defined by corresponding data. Thus, one can speak of CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, and CAE data. However, since a modeled object can be defined by corresponding data in any combination of these systems, one of these systems does not exclude the other. Thus, a system may well be both a CAD system and a PLM system.

[0032] By CAD system, it also means any system at least adapted to design a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, data defining the modeled object includes data enabling the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object using edges or lines, for example, in the particular case of faces or surfaces. Lines, edges, or surfaces may be represented in various manners, for example, in non-uniform rational B-splines (NURBS). In particular, a CAD file may include specifications from which geometry can be generated, which then enable a representation to be generated. The specifications of a modeled object may be stored in a single CAD file or multiple CAD files. Typical sizes of files representing modeled objects in a CAD system are in the range of one megabyte per part, and a modeled object may typically be an assembly of thousands of parts. In the context of CAD, a modeled object may typically represent, for example, a part, an assembly of parts, or possibly a product, such as an assembly of products.

[0033] A 2D sketch is a 2D image representing a 3D model on a 2D plane. The 2D sketch may be established by a user. In particular, the 2D sketch may be a hand-drawn sketch. Alternatively, the 2D sketch may be synthesized, i.e., artificially and automatically synthesized by a computer system. In particular, the hand-drawn drawing in the dataset formation method may be synthetic. Thus, a 2D sketch is a 2D image data structure including one or more (2D plane) strokes representing a 3D shape from each perspective from which the 3D shape is visible. The one or more strokes are defined in the same plane and represent the entire view of the 3D shape from each perspective. Each stroke is a continuous or substantially continuous surface defined in the plane.

[0034] In an embodiment, for any 2D sketch herein, the background of the 2D image may be a single color, or alternatively may not be a single color. The background may be, for example, a photograph onto which a user can add strokes. The strokes may be distinguishable, for example, based on a contrast with the remainder of the photograph that is higher than a predetermined threshold. In the machine learning process, the background may be replaced with a single color background of a first color, and the identified strokes of the 2D sketch may be rendered in a second color. In an embodiment, the first color may be black, and the second color may be white. In other embodiments, the first color may be white, and the second color may be black. All of the methods of the machine learning process, including the machine learning method, the dataset formation method, and the 3D design method, may use the same first and second colors. Alternatively, the methods of the machine learning process and the 3D design method may use different first and / or second colors in combination with one or more transformation steps to maintain consistency of their respective inputs and outputs.

[0035] In an example, for any 2D sketch herein, one or more (e.g., all) strokes may each have an endpoint that has a distance from another stroke (e.g., from the extremity point of another stroke) that is less than a predetermined threshold, e.g., 10 millimeters or 5 millimeters. In an example, for any 2D sketch herein, the strokes represent free 3D shapes. This means that strokes can be defined in any order and without predefined constraints (e.g., having necessarily parallel and / or orthogonal strokes), and they are not organized into clusters (e.g., obtainable based on distance), each of which represents a solid plane. In an example, any 2D sketch herein may instead or additionally be provided in a raw format, e.g., without any labels associated with the strokes (e.g., annotations of actions such as creation, extrusion, bending, etc.) and / or without any additional data (e.g., pictures, user definition of parallel lines, depth images).

[0036] Sketching a 2D sketch may be performed through graphical interaction with a computer screen by sequentially (e.g., substantially continuously) activating a series of pixels on the screen that define strokes. Activation may be performed through a haptic device that acts on the pixels, such as a mouse (e.g., cursor movement that defines strokes), or through user interaction with a touchscreen or touchpad (e.g., interaction that involves physically touch-drawing strokes on the touchscreen or touchpad). This is known in the CAD field as sketch design or stroke design. Alternatively, a scan of a drawing sketched on paper or any other image may be input to the inference method. Alternatively, a handwritten drawing, for example, generated by another user and / or by any of the previously described techniques, may be retrieved from memory or received from a remote computer system.

[0037] Any of the methods herein may be included in a manufacturing process, which may include, after performing the method, creating a physical product corresponding to the modeled object. In either case, the modeled object designed by the method may represent a manufactured object. Thus, the modeled object may be a modeled solid (i.e., a modeled object representing a solid). The manufactured object may be a product, such as a part or an assembly of parts. Because the method improves the design of the modeled object, the method may also improve the manufacturing of the product, thereby increasing the productivity of the manufacturing process.

[0038] Any 3D shape herein may represent the geometry of a product to be manufactured in the real world, such as a (e.g., mechanical) part or assembly of parts (or equivalently, an assembly of parts, since from the perspective of the method, the assembly of parts can be viewed as the part itself, or the method can be applied independently to each part of the assembly), or more generally, any rigid assembly (e.g., a moving mechanism). The product may be one of a variety of products in an unlimited number of industries, including aerospace, architecture, construction, consumer products, high-tech devices, industrial equipment, transportation, maritime, and / or offshore oil / gas production, or transportation. Thus, the 3D shapes may represent industrial products, including parts for ground vehicles (including, for example, automobile equipment, light truck equipment, race car equipment, motorcycle equipment, truck equipment, power equipment, trucks, buses, and trains), parts for air vehicles (including, for example, airframe equipment, aerospace equipment, propulsion equipment, defense products, airline equipment, space equipment), parts for naval vehicles (including, for example, naval equipment, commercial vessels, offshore equipment, yacht equipment, workboat equipment, and marine equipment), general machinery parts (including, for example, industrial manufacturing machines, heavy duty machines or equipment, installation equipment, industrial machinery products, processing equipment, etc.). The material may be any mechanical part such as a machined metal product, a machined metal product, a tire manufacturing product, an electromechanical or electrical component (including, for example, consumer electronics, security products, control products, and / or measurement products, computing and communication equipment, semiconductor devices and equipment, medical devices and equipment), consumer product (including, for example, furniture, home products, garden products, leisure goods, fashion products, hard goods retailer products, soft goods retailer products), packaging (including, for example, packaging for food, beverage, tobacco, beauty, health, and household products), etc.

[0039] Any of the methods are computer-implemented. This means that the steps of the method (or substantially all steps) are performed by at least one computer, or any similar system. Thus, the method steps are performed by the computer, possibly fully automatically, or semi-automatically. In embodiments, triggering at least some of the method steps is performed through user-computer interaction. The level of user-computer interaction required may depend on the expected level of automation and may balance the need to implement user requirements. In embodiments, this level may be user-defined and / or pre-defined.

[0040] Also proposed is a computer program comprising instructions that, when executed on a computer system (i.e., a system having a processor coupled to a memory), cause the computer system to perform any of the methods. The computer program may include computer-executable instructions, the instructions including means for causing the system to perform the methods. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, for example, an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions to perform the functions of the method by operating on input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language, as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. The application of the program on a system, in either case, carries instructions for performing the method.

[0041] Also proposed is a data structure for representing neural networks trained according to machine learning methods and / or 3D design methods.

[0042] Also proposed is a device including a memory storing computer programs and / or data structures. The device may form or serve as a non-transitory computer-readable medium, such as on a SaaS (Software as a Service), another server, or a cloud-based platform. Alternatively, the device may include a processor coupled to the memory. Thus, the device may form a system in whole or in part (e.g., the device is a subsystem of an overall system). The system may further include a graphical user interface coupled to the processor.

[0043] The computer system may include a processor and a graphical user interface (GUI) coupled to a memory, the memory storing a computer program including instructions for performing a method. The memory may also store a database. The memory is any hardware adapted for such storage, possibly including several physically distinct parts (e.g., one for the program and possibly one for the database).

[0044] FIG. 1 shows an embodiment of a computer system.

[0045] The client computer of the embodiment includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, with random access memory (RAM) also connected to the bus. The client computer is further provided with a graphical processing unit (GPU) 1110 associated with video random access memory 1100 connected to the bus. Video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass memory devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated in specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device or a haptic device 1090, such as a keyboard. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. In addition, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse, and the buttons on the mouse are used to generate the signals. Alternatively or in addition, the client computer system may include a pressure-sensitive pad and / or a pressure-sensitive screen.

[0046] FIG. 2 shows an example of a GUI for one of the systems herein, where the system is a CAD system.

[0047] The GUI 2100 may be a typical CAD-like interface, with standard menu bars 2110, 2120 as well as a bottom toolbar 2140 and a side toolbar 2150. Such menu bars and toolbars include a set of user-selectable icons, each associated with one or more operations or functions, as is known in the art. Some of the icons are associated with software tools adapted to edit and / or act on the 3D modeled object 2000 displayed in the GUI 2100. The software tools may be grouped into workbenches. Each workbench includes a subset of software tools. In particular, one of the workbenches is an editing workbench, suitable for editing geometric features of the modeled product 2000. In an operation, the designer may, for example, pre-select a portion of the object 2000 and then initiate an operation (e.g., change dimensions, color, etc.) or edit geometric constraints by selecting the appropriate icon. For example, a typical CAD operation is punching or folding a 3D modeled object displayed on the screen. The GUI may, for example, display data 2500 related to the displayed product 2000. In the illustrated example, the data 2500 displayed as a "feature tree" and their 3D representation 2000 relate to a brake assembly including a brake caliper and a brake disc. The GUI may further show various types of graphic tools 2130, 2070, 2080, for example, to facilitate 3D orientation of the object, trigger a simulation of the operation of the edited product, or render various attributes of the displayed product 2000. A cursor 2060 may be controlled by a haptic device to allow the user to interact with the graphic tools.

[0048] Aspects of machine learning methods and 3D design methods are discussed herein.

[0049] According to a first aspect, in the machine learning method, the architecture provided for the neural network is a variational autoencoder (VAE) architecture. In such a case, the neural network is configured to consider random variables as additional inputs. This forms an improved solution for outputting a 3D model represented by an input 2D sketch from an input 2D sketch.

[0050] In particular, the variational autoencoder, which includes input random variables, improves the robustness of the learning method with respect to inaccuracies in the neural network's input data. In practice, the input 2D sketch may be inaccurate (e.g., when the 2D sketch contains gaps between strokes or when the 2D sketch is handwritten), and the VAE features improve the method's ability to output a 3D model with inaccurate input data.

[0051] According to a first aspect, in a 3D design method, a neural network trained according to the first aspect is applied to a 2D sketch and random variables to output a 3D model represented by the 2D sketch. This feature improves the ergonomics of the 3D design method, such as improving the ability of the design method to accurately output the 3D model when the input 2D sketch has some inaccuracy or when the input 2D sketch is incomplete (e.g., when the user intends to calculate the 3D model more quickly and / or when the user is currently drawing).

[0052] Inputting random variables into the VAE also enables the 3D design method according to the first aspect to output multiple 3D models from a single 2D sketch. Each output 3D model corresponds to one random variable. This provides an improved solution for outputting a 3D model represented by a 2D sketch selected by a user from multiple 3D models proposed to the user from an input 2D sketch. This is particularly efficient when the input 2D sketch has some inaccuracies, is incomplete, or does not clearly represent a single 3D model. Referring to FIG. 3 , the VAE allows for obtaining a regularized latent space and introducing randomness into the output. As previously mentioned, a 2D rectangle can represent a 3D cylinder from a frontal view or a 3D box from any typical view, and thus the method can propose both a cylinder and a box. The figure shows a representation of the regularized latent space 310 and the unregularized latent space 320. Further details regarding regularization are provided below. One can observe that color regions are closer to each other in the normalized latent space 310 compared to the unnormalized latent space 320. Thus, selecting a latent vector near the blue region 312 (e.g., representing a cylinder) can yield a latent vector in the green region (e.g., representing a box). In contrast, a latent vector near the blue region 322 may not yield a latent vector in the green region 324. Thus, when the 2D sketch is uncertain, the VAE method can propose multiple different solutions, i.e., 3D models corresponding to different regions in the normalized latent space 310, e.g., one 3D model corresponding to region 312 and another 3D model corresponding to region 314.

[0053] According to a second aspect, the 3D model is a parameterized 3D model defined by a set of parameters. The set of parameters is composed of a first subset of one or more parameters and a second subset of one or more parameters. According to the second aspect, in the machine learning method, the neural network is configured to selectively (1) output values ​​for the set, and (2) take values ​​for the first subset from a user as input and output values ​​for the second subset (in other words, the neural network may alternatively be selectively used according to option (1) or option (2)). This forms an improved solution for outputting a 3D model represented by an input 2D sketch from the input 2D sketch.

[0054] Unlike fixed representations of 3D models, such as mesh representations, voxel occupied grids, or point clouds, a (e.g., fully) parameterized 3D model of a 2D sketch can be easily edited by modifying a few parameters. In an embodiment, the number of parameters for parameterizing a 3D model is less than the number of faces of its tessellated mesh, the number of occupied cells of its voxel grid, or the number of marked points in its point cloud by a factor of more than 5 (e.g., more than 10). Each parameterized 3D model may be defined by fewer than 100 (e.g., fewer than 50) parameters. Thus, the parameterization mechanism represents the 3D model in a compact and organized manner. Also, as is known in the field of CAD, acting on a single parameter can have a global effect, while acting on a single geometric entity, in the case of a mesh, voxel grid, or point cloud, only has a local effect. For example, any 3D model herein may be configured to scale up its occupied volume by modifying a few parameter values ​​less than 5 (e.g., less than 3), as opposed to moving / increasing a large number of mesh triangles. Additionally, parameterized 3D models are more semantically useful to users than meshes, voxel grids, or point clouds.

[0055] According to a second aspect, a user can selectively fix portions of the data that define the 3D model to be output, thereby efficiently training the neural network to output a particular type of 3D model characterized by having particular values ​​for a first subset of one or more parameters, thereby improving training.

[0056] In embodiments, a 3D model may be parameterized by one or more shape parameters, including, for example, one or more positioning parameters (e.g., 3D coordinates) for one or more bounding points (e.g., corner points when the 3D model represents a prism). The shape parameters may additionally or alternatively include one or more lengths corresponding to one or more dimensions (e.g., edge lengths for a cube and / or heights of prisms). Any 3D model herein may alternatively or additionally be defined by a type parameter (e.g., type of section when the 3D model represents a prism). Any 3D model herein may alternatively or additionally be defined by additional shape parameters according to its type, optionally including one or more positioning parameters (e.g., 3D coordinates of corner points of a section when the 3D model represents a prism). Such parameterization of a 3D model is particularly ergonomic for a user to edit.

[0057] In the following, the values ​​for the set of parameters consist of values ​​for the first subset and values ​​for the second subset. Selectively outputting values ​​for the set means that the user can choose either to output values ​​for the (entire) set of parameters or to provide values ​​for the first subset of parameters as input to the neural network. In the latter case, the values ​​for the first subset and the values ​​for the second subset are not input by the user, but are both calculated (automatically) by the neural network. In this case, the 2D sketch is only an input to the neural network, and the output values ​​for the first subset and the output values ​​for the second subset are calculated by the neural network. In the latter case, the input values ​​of the first set of parameters of the 3D model may be used by the neural network to output values ​​for the second subset.

[0058] In an embodiment, the first subset of the one or more parameters may consist of one or more of the type parameters, and the second subset of the one or more parameters may consist of one or more of the shape parameters. Thus, a user can restrict the output of a neural network to a particular type (e.g., a prism with triangular sections) while still allowing the neural network to calculate any shape.

[0059] According to a second aspect, the 3D design method selectively provides values ​​for a first subset of one or more parameters to output values ​​for a second subset of one or more parameters. Thus, a user can set a portion of the parameters that define the 3D model as model inputs and obtain the remaining parameters. This feature improves the ergonomics of the 3D design method by allowing the user to output 3D models of a specific category, i.e., with some predefined set of one or more parameters that characterize the category. In an embodiment of the 3D design method, the user may have already made decisions about values ​​for one or more parameters of the first subset, or there may be some practical constraints on their selection.

[0060] According to a third aspect, in the machine learning method, the 3D model is a parameterized 3D model defined by a set of parameters composed of sections and extrusions, which results in a simple and compact editable parameterization of the 3D model, forming an improved solution for, from an input 2D sketch, outputting a 3D model represented by the 2D sketch.

[0061] In an embodiment, the set of parameters may be common among different categories of 3D models, and each category may be characterized by one or more parameters of the set of parameters. Machine learning according to the third aspect trains a single network, which is then used to generate 3D models of several categories. By single network, it is meant that a single set of operations and neural network weights are jointly trained for 3D models of different categories in the machine learning method. In other words, training involves a single minimization of a single loss, possibly even mini-batch by mini-batch, e.g., a single stochastic gradient descent. Thus, a single network differs from a combination of several neural networks, each of which may be trained for 3D models of one or more categories, connected together by a selector for each category.

[0062] In an embodiment, a section may be defined by a type parameter that describes a geometric type of section and one or more (e.g., two or more) shape parameters. The shape parameter may be a list of position parameters. In an embodiment, an extrusion may be defined by a line in 3D space. In an embodiment, the extrusion line may be a straight line or a curved line. The straight line may be perpendicular to the section, and thus the extrusion may be represented by an extrusion length.

[0063] According to a third aspect, in a 3D design method, the trained neural network is applied to a 2D sketch to output values ​​of a set of parameters configured from sections and extrusions. This feature improves the ergonomics of the 3D design method, providing the user with a simple and compact editable parameterization of the 3D model.

[0064] Different aspects of the method may be combined together.

[0065] For example, the first aspect may be combined with the second aspect. In such a case, the 3D model is defined by a set of parameters consisting of a first subset of one or more parameters and a second subset of one or more parameters, and the neural network has a VAE architecture. The neural network is configured to consider a random variable as an additional input. The neural network is further configured to selectively output values ​​for the set, consider values ​​for the first subset from a user as input, and output values ​​for the second subset. This forms an improved solution for outputting a 3D model represented by the 2D sketch from an input 2D sketch.

[0066] As a result, a 3D design method is applied to a 2D sketch and random variables to output a 3D model represented by the 2D sketch, and selectively provides values ​​for a first subset of one or more parameters to output values ​​for a second subset of one or more parameters. In such a selection, the neural network VAE architecture can be a conditional VAE based on the input values ​​for the first subset. Such an architecture is known as a conditional variational autoencoder (CVAE).

[0067] As another example, the second aspect may be combined with the third aspect. In such a case, the 3D model is defined by sections and extrusions, and the sections and extrusions are defined by a set of parameters composed of a first subset of one or more parameters and a second subset of one or more parameters. The first subset may then be composed of numbers representing the type of section. In an example, the numbers may represent numbers representing sides forming the section. Each side may be either a straight line (segment) or a curved line (arc). The second subset may include parameters defining the extrusions and / or parameters further defining the sections.

[0068] As yet another example, the first aspect may be combined with the third aspect. In such a case, the 3D model is a parameterized 3D model defined by a set of parameters composed of sections and extrusions. The 3D design method outputs one or more 3D models, possibly with different types of sections.

[0069] Finally, the first, second and third aspects may be combined together, in which case the 3D design method is applied to a 2D sketch and random variables to output a 3D model represented by the 2D sketch, and selectively provides values ​​for extrusion and / or type of section to output parameters further defining the section.

[0070] The first subset may then consist of numbers representing the type of section. In an embodiment, the numbers may represent numbers representing sides forming the section. Each side may be either a straight line (segment) or a curved line (arc). The second subset may include parameters that define the extrusion and / or parameters that further define the section.

[0071] An example of a method for forming a data set will now be discussed.

[0072] In an embodiment, the dataset formation method may include generating random values ​​for one or more of the training samples. The generation of the random values ​​corresponding to one or more of the training samples may be part of a learning method, e.g., the random values ​​may be generated "on the fly." The random values ​​may be selected according to a probability distribution (e.g., a normal distribution), possibly with predefined parameters (e.g., a standard normal distribution).

[0073] In an embodiment, the 3D model may be a parameterized 3D model defined by a set of parameters consisting of a first subset of one or more parameters and a second subset of one or more parameters. The dataset formation method may include providing initial data including a first discrete subset of one or more parameter domains corresponding to the first subset of one or more parameters and a second discrete subset of one or more parameter domains corresponding to the second subset of one or more parameters. Synthesizing the 3D model may be performed based on such initial data based on selected values ​​of the first and / or second subsets of parameter domains in their respective parameter domains.

[0074] Any selection performed when synthesizing a 3D model may be performed via (e.g., random) sampling. This allows for the generation of data that better represents the diversity of real-world situations, ultimately leading to accurate learning. Any (e.g., random) sampling may include uniform (e.g., random) sampling within a set of values ​​that the variable to be selected takes on (e.g., random uniform sampling is random sampling according to a uniform probability distribution). Alternatively, sampling may include random non-uniform sampling, in which the probability of selecting some values ​​is higher than others. Higher probabilities may be attributed to values ​​that appear more frequently in the application of the 3D design method.

[0075] In certain embodiments, the dataset formation method may include providing initial data including a section type from a discrete set of section types and, together with the section type, a respective discrete set of one or more parameter domains for one or more shape parameters. Synthesizing the 3D model may be performed based on such initial data. In particular, synthesizing the dataset may include generating integers representing the section types and, based on the numbers, generating a list of position parameters and values ​​for extrusion. Thus, the 3D model is fully defined.

[0076] In an efficient embodiment of the dataset formation method according to the third aspect, the method may select a value for the type of section (e.g., a value corresponding to a triangle), one or more values ​​for a position parameter for the selected type of section (e.g., 3D coordinates of a triangular section corner), and a value for an extrusion length perpendicular to the section. In some options of this embodiment, the position parameter of the section may correspond to the corner of the section and may be selected relative to a unit circle. In other options of this embodiment, values ​​for the position parameter of the section and the extrusion length may be selected to obtain a maximum 3D model corresponding to that set of position parameters and an extrusion fitting in a unit sphere. In particular, for scaling, the bounding box of the 3D model may be re-centered in the orientation of the coordinate system ((0,0,0)). The 3D model is then rescaled by a scaling ratio to obtain a maximum 3D model fitting in the unit sphere. To rescale the model, simply multiply the norm of each point of the 3D model by the scaling ratio. Scaling the dataset in this way improves learning.

[0077] In an embodiment, the dataset formation method may populate a dataset with several 2D sketches from a single 3D model by applying random rotations. In another embodiment, the 3D model may instead remain fixed, and the method may acquire several 2D sketches from random viewing directions, represented by vectors from a selected point on a unit sphere to its center. In both embodiments, the 2D sketches may then be calculated (e.g., projected) onto a plane perpendicular to the viewing direction. In an embodiment, the calculated 2D sketch may be rendered in black and white, and the rendered edges may be black (e.g., for a photograph of size 256 pixels) and inserted into the dataset. In an embodiment, the color of the inserted 2D sketch may be inverted to obtain a black background and white lines. This maximizes the number of zero values ​​in the neural network's input 2D sketch, thereby improving learning. The values ​​of the rendered sketch may be rescaled from the standard [0,255] to [0,1].

[0078] In an embodiment, the method may add some random noise to the lines of the sketch. The method may add noise to one or more parameters defining each line of the sketch and / or to the endpoints of each line. Such an embodiment improves the robustness of neural networks trained on such datasets against inaccuracies in the input 2D sketch.

[0079] An example of a learning method will now be discussed.

[0080] The learning method forms an improved solution for CAD design. In particular, the trained neural network can be used to convert 2D sketches into 3D models. In the context of CAD, this is particularly useful as discussed above. The learning may include supervised training based at least in part on a dataset. In such a case, at least a portion of the dataset includes, for each 2D sketch, a respective 3D model that forms a training sample (labeled) by the 2D sketch. At least a portion of the dataset may be provided according to any method, for example, formed according to the dataset formation method.

[0081] In a particularly efficient embodiment, the learning method may include supervised training based on a dataset formed by a dataset formation method. As is known from the field of machine learning, each training may involve repeatedly processing the respective dataset, for example, mini-batch by mini-batch, and modifying the weight values ​​of the neural network according to the iterative process. This may be performed according to a stochastic gradient descent method. The weight values ​​may be initialized in any manner for each training. The weight values ​​may be initialized in any manner, for example, randomly or each may be initialized to a value of zero.

[0082] In an embodiment according to the first aspect of the machine learning method, a neural network is configured to take as input a 2D sketch and a random variable. The learning method may include generating values ​​for the input random variable according to a probability distribution (e.g., N(0,1)). The learning method may generate random numbers "on the fly" during the learning process for each one of the training samples.

[0083] The training may include minimizing a loss function, which represents the disparity between each of the 3D models in the training sample set and the 3D model output by the neural network from each 2D sketch of the training sample.

[0084] In embodiments, according to the second and third aspects of the machine learning method, the loss may penalize discrepancies between the parameters defining the 3D model of each of the training samples and the respective parameters calculated by the neural network. In a particularly precise embodiment, according to the third aspect of the machine learning method, the discrepancy may include the mean squared error between the position parameters (e.g., coordinates of points) defining the sections of each of the 3D models of the training samples and their respective predicted values, and / or the mean squared error between the extrusion parameters (e.g., extrusion length) of each of the 3D models of the training samples and their respective predicted values. In another particularly precise embodiment, according to the third aspect of the machine learning method, the discrepancy may include a metric of the difference between the types of sections of each of the 3D models of the training samples calculated by the method, or a probability value for each type.

[0085] In an embodiment, according to a first aspect of the machine learning method, neural network, the loss function may include a regularization loss having a term that penalizes discrepancy between a probability distribution (e.g., N(0,1)) of a random variable (ε) and a probability distribution (e.g., N(μ,δ)) defined by a mean (μ) and a standard deviation (δ). The training method may initialize values ​​for the mean and standard deviation at the start of training. The training method may calculate values ​​for the mean and standard deviation during the training process. The loss function may measure discrepancy using a distance and / or divergence between the two probability distributions. In a particularly efficient embodiment, the discrepancy between the probability distributions is the Kullback-Leibler diffusion loss.

[0086] Returning to FIG. 3, the regularizing effect of the Kullback-Leibler diffusion loss can be observed. In the unnormalized latent space 320, different regions corresponding to different calculated means and / or standard deviations may be completely separated. Thus, the method according to the first aspect may lose the possibility of proposing a 3D model from a region other than the region with the smallest reconstruction loss. The reconstruction loss may represent the error in the proposed 3D model, e.g., the parameters defining the 3D model. Including the Kullback-Leibler diffusion loss in the objective function can force the neural network to consider a balance between the reconstruction loss and the randomness in the input data. Furthermore, including the Kullback-Leibler diffusion loss in the objective function encourages the predicted mean to be regrouped around zero and the standard deviation to be around one. These effects may lead to different regions being close to each other in the normalized latent space 310.

[0087] The first aspect of the method will now be discussed.

[0088] According to a first aspect, the neural network may include a first portion and a second portion. The first portion may be configured to take a 2D sketch and a random variable as inputs and provide respective outputs. The second portion may be configured to take respective outputs of the first portion as inputs and output a 3D model. In this specification, a portion of a neural network refers to a virtual grouping of the neural network architecture and corresponding weights.

[0089] Configuring the first part, including the encoder, to take random variables as input can provide an improved solution for output compared to introducing random variables before the decoding stage by one or more decoders, because it maintains the consistency of the input sketch in the decoding stage so that all decoders decode the same latent vector, and it also fixes the final latent vector decoded for each decoder.

[0090] The first portion may include a first sub-portion and a second sub-portion. The first sub-portion may be configured to take a 2D sketch as input and output a first latent vector. The second sub-portion may be configured to take a first latent vector and a random variable ε as input and reparameterize the first latent vector into a second latent vector using the random variable. A latent vector represents a block of neural network data that serves as an intermediate result in the process of calculating the neural network's output from each input. The latent vector may not be accessible or directly output. In a particularly efficient implementation of the first aspect, the first sub-portion may be a convolutional neural network (CNN).

[0091] The second sub-part may be configured to calculate a "conditional mean" μ and a "conditional standard deviation" δ of the second latent vector. The conditional mean and the conditional standard deviation refer to a vector for the mean and a vector for the standard deviation, respectively. In this specification, re-parameterization refers to performing the so-called "re-parameterization trick," and the output of the second sub-part, i.e., the second latent vector, may be calculated by randomly sampling according to Equation (1):

[0092]

number

[0093] The output represents a second latent vector. Thus, any of the elements of the standard deviation may be multiplied by a random variable as an offset and added to the mean to obtain the second latent vector. Such a formula for obtaining the second latent vector is differentiable with respect to μ and δ. This differentiability allows for training a neural network, for example, by steepest descent.

[0094] According to this first aspect, the random variable ε can conform to a probability distribution (N(.,.)) according to a law defined by a mean and a standard deviation, where the random variable has a respective value for the mean (e.g., 0) and a respective value for the standard deviation (e.g., 1). In an embodiment, the law may be the law of normal distribution. In an ideally trained neural network, i.e., when there is no error in each trained weight of the neural network, the conditional mean and the conditional standard deviation should represent a distribution identical to the distribution of the random variable ε, and in an embodiment, N(μ,δ)=N(0,1). Thus, according to this first aspect, the training may include optimizing an objective function that includes a loss between these two probability distributions. Including such a loss term has a regularizing effect in optimizing the objective function, helping to regularize the latent space. This regularizing effect may be compared to the effect of an “L2-regularization” loss, which is widely used in the machine learning field to penalize large weights in the network. To avoid all normalized weights becoming zero, the associated coefficient of the loss term due to the regularization effect may be chosen to be relatively small. An example of training according to this aspect is discussed later.

[0095] A second aspect of the method will now be discussed.

[0096] According to a second aspect, an architecture may include a piece of data. The neural network may be configured to generate a first value for the piece of data, determine a value for a first subset based on the first value for the piece of data, and selectively output a value for the set to further determine a value for a second subset based on the first value for the piece of data. The neural network may be configured to generate a second value for the piece of data based on the value for the first subset, and output a value for the second subset based on the second value for the piece of data, selectively taking as input a value for the first subset from a user and outputting a value for the second subset.

[0097] The pieces of data may represent blocks of data in a neural network or one or more layers of a neural network, including one or more operations and / or one or more blocks of data. The first and second values ​​for the pieces of data represent how the blocks of data are populated, and the neural network may automatically populate the first value for the pieces of data by computation during its execution, and the second value is populated via user interaction independent of the execution of the neural network.

[0098] Further, according to a second aspect, the neural network may include an operation, i.e., a function. The function may be, for example, predetermined, e.g., constant, during execution of the neural network. To selectively output values ​​for the set, values ​​for a first subset are generated based on the first values, where the values ​​for the first subset correspond to an image of the function of the first values ​​for the data piece. To selectively take values ​​for the first subset from a user as input and output values ​​for a second subset, the second value is a preimage of the values ​​for the first subset according to the function. Since there may generally be more than one preimage of values ​​according to the function, the second value is selected as one of the acceptable preimages according to a criterion. The criterion may be to select a preimage with a largest vector infinity-norm. Thus, a function (e.g., an argmax function, as described later) may be used at inference time to convert input values ​​from a user for the first subset to second values. The second value is used by the neural network to calculate a second subset.

[0099] According to a second aspect, the piece of data may be a vector representing a probability distribution for the first subset, and the function may be an argmax function. Further, the second value of the piece of data may be a vector having a value of 1 for coordinates corresponding to values ​​for the first subset and a value of 0 for all other coordinates.

[0100] In an embodiment of the method for 3D design according to the second aspect, before a user provides values ​​for the first subset, a neural network may be applied to the 2D sketch to output one or more respective values ​​for each of the set, and one or more 3D models may be displayed to the user. The user may provide values ​​for the first subset while the one or more 3D models are displayed.

[0101] A third aspect of the method will now be discussed.

[0102] In this embodiment, the 3D model is a parameterized model defined by extruding or sweeping sections along an extrusion line. The geometric types of sections are represented by numbers. The neural network is configured to output vectors representing probability distributions over the numbers.

[0103] A section may be 2D, i.e., planar, and consist of two or more sides, each of which is either a straight line (segment) or a curved line (arc). Thus, the representative number of a section type can represent the number of sides (segments or arcs) that form the section (nbSides). Curved lines may be represented in various forms, for example, as splines or non-uniform rational B-splines (NURBS).

[0104] A section may be further defined by a list of position parameters. The number of position parameters may be equal to the number of sides of the section. Each position parameter may optionally include a pair of 3D points. Alternatively, one or more of the position parameters may include a single 3D point. A section may be completely defined by a number representing the geometric type of the section and a list of position parameters. A section may be further defined by a flag associated with each predicted pair of points to indicate whether each side is a segment or an arc.

[0105] In accordance with this aspect, the neural network architecture may include a recurrent neural network (RNN) configured to output values ​​for the list. RNNs are highly efficient at sequentially computing values ​​for lists because any of the elements of the list can be computed in conjunction with already computed elements of the list.

[0106] The RNN may include one or more long-short-term memory (LSTM) cells. The number of LSTM cells may be equal to the number of elements in the list. Alternatively, the architecture may be static, with the number of LSTM cells fixed for different types of sections. The number of LSTM cells may be equal to the maximum number of position parameters required to define different types of sections. Having a static architecture can improve training computation time. Each LSTM cell may be configured to belong to one of the elements in the list and output the value of that element. In an embodiment, the RNN may be replaced with a fully connected (FC) layer.

[0107] The neural network may be further configured to output a value for the extrusion based on the final state of the RNN. The extrusion line may be represented by a straight line or a curved line. The curved line may be represented in various forms, for example, a spline or a non-uniform rational B-spline (NURBS). In an embodiment, the curved line may be defined by a second list of position parameters, each position parameter optionally including a pair of 3D points. The neural network may include a second RNN configured to output a value for the second list. In an embodiment, the extrusion line may be a straight line perpendicular to the section, and thus it may be represented by a scalar parameter of the extrusion length (h). In an embodiment, each position parameter is either a single 3D point or a pair of 3D points representing a point located on the boundary of the section.

[0108] In a preferred embodiment of this aspect, the 3D model is either a cylinder or a prism with extrusions perpendicular to the sections.

[0109] Here, the combination of different aspects will be discussed.

[0110] In an embodiment, the machine learning methods according to the first and second aspects may be combined. Such a combination may include a neural network including two portions. The first portion may be configured to take a 2D sketch and random variables as inputs and provide respective latent vectors as outputs, and the second portion may be configured to take the respective latent vectors of the first portion as inputs. The first portion may be identical to the first portion according to the first aspect. The latent vectors may be identical to the second latent vectors according to the first aspect. The second portion may include a first sub-portion configured to output a vector representing a probability distribution, and a second sub-portion configured to take a concatenation of the respective latent vectors and the vector representing the probability distribution as inputs and output a second subset. The first sub-portion may be a fully connected (FC) layer. According to a feature of the second aspect, the first sub-portion may optionally take values ​​for the first subset from a user as inputs. In an embodiment, the neural network may be configured to generate a second value for the vector having a value of 1 for the coordinate corresponding to the value for the first subset and a value of 0 for all other coordinates.

[0111] In an embodiment, the machine learning methods according to the first and third aspects may be combined. In such an embodiment, there may be a portion of the neural network configured to take as input a vector that connects the final state of the RNN and at least a vector representing a probability distribution, and to output a value for extrusion. In an embodiment, the vector representing the probability distribution is a second latent vector. The portion may be an FC layer.

[0112] In a combination of the second and third aspects, the first subset of one or more parameters may consist of a number representing the type of section. This number may be attributed to an argmax function applied to a vector of probability distributions of the data pieces. Furthermore, the inverse image of the values ​​according to the function is selected to correspond to a vector having a value of 1 for the coordinate corresponding to the value for the number representing the type and a value of 0 for all other coordinates. Furthermore, the second subset may include a list of parameters defining the extrusion and / or positional parameters further defining the section.

[0113] Implementations of the method will now be discussed with reference to Figures 4-12. These implementations focus on simple, parametric 3D primitives: parameterized cylinders, boxes, and regular prisms. Figure 4 shows examples of such primitives. The 3D model of any of the 3D primitives is represented by a sweep representation, so that each 3D model can be defined by a 3D planar section and a 3D straight extrusion line perpendicular to the section.

[0114] Possible variations of these implementations may allow for covering more complex 3D shapes represented by the sweep representation, where the sections may be irregular and / or a combination of segments and arcs, and / or the extruded lines may be splines, i.e., curves represented by a set of 3D points.

[0115] To predict a 3D model from a user's 2D sketch, an implementation of the method uses a neural network deep learning algorithm. The algorithm allows for the generation of multiple parameterized 3D models from a single 2D sketch. The neural network can also receive user feedback at its input to constrain the type of model predicted.

[0116] In those implementations, the neural network is a vanilla model, returning one single 3D model from a 2D sketch. However, variations can allow the neural network to return several 3D models from a 2D sketch, for example, if the neural network includes an input parameter representing the number of 3D models to be output.

[0117] In these implementations, the neural network is a VAE model configured to take one or more random variables as inputs to generate one or more propositions from a single input of a 2D sketch. Each input random variable corresponds to a respective proposed 3D model. The random variables are sampled from a normal Gaussian distribution N(0,1).

[0118] In some implementations, the neural network may be a CVAE model that includes a condition variable for imposing a type of 3D model. The CVAE model returns one or more 3D models of the desired imposed type from the 2D sketch. In other words, the neural network may be used as either a VAE or a CVAE.

[0119] In those implementations, each 3D model may be fully described by: - The number of sides (nbSides), which corresponds to the number of points defining the section. It can also be considered as a type of primitive. In an embodiment, the maximum number of sides is limited to 5. Furthermore, numbers of sides between 2 and 5 are attributed to cylinders, triangular prisms, boxes or cubes, and pentagonal prisms, respectively. - A set of 3D points as a list of up to 5 pairs of 3D points defining the points of the section. For example, points = [{(X1, Y1, Z1), (X'1, Y'1, Z'1)}, ..., {(X5, Y5, Z5), (X'5, Y'5, Z'5)}]. In an embodiment, two pairs of points are used to describe the curved sides of the section, e.g., a cylinder. The notations point[i][0] and point[i][1] are used for the first and second elements in each pair of 3D points. - A flag associated with each pair of points to indicate whether the respective side is a segment or an arc. A flag equal to 0 indicates a segment, a flag equal to 1 indicates an arc. The flag is 1 for all pairs of points for a cylinder, and 0 for all pairs of points for a box and a regular prism.

[0120] The construction of the 3D model is explained with reference to Figures 5A and 5B.

[0121] First, the points defining the section, i.e., point[i] for i in [0,nbSides], are considered in 3D space. In general, it may happen that the points do not lie in the same 2D plane, so a mean plane of the points is calculated. The mean plane corresponding to a set of points means the plane that minimizes the sum of the distances to each of the points. The mean plane may be calculated by any method known in the art. The points are sorted in ascending order of angle. The angle attributed to each point[i] may be defined relative to the mean plane as the angle between the segment connecting point[i] and the reference point on the mean plane and a reference line that lies on the mean plane and passes through the reference point. Each of the points is projected orthogonally onto this plane. Each projected point may be represented by the same notation as the original point: point[i]. Then, if the corresponding flag of point[i] represents a segment, i.e., equal to 0, a closed planar section is obtained by drawing a line between each of points [i][0] for i in [0,nbSides] and each of points [(i+1)%nbSides][0]. The notation a%b for two integers a and b (when b is non-zero) represents the remainder of the division of number a by number b. If the corresponding flag of point[i] represents a circular arc, i.e., equal to 1, a curved line is drawn between each of points [i][0] and point [(i+1)%nbSides][0] so as to pass through point[i][1] for i in [0,nbSides]. The resulting curve for the section is circular or elliptical. Furthermore, the extrusion is calculated by the extrusion length (h) in the direction perpendicular (n) to the mean plane.

[0122] With reference to Figure 6, a variant of these implementations will be discussed in which sections are extruded along curved extrusion lines. Sections include both arcs and segments and are represented by a list of four position parameters [{(X1, Y1, Z1), (X'1, Y'1, Z'1)}, (X2, Y2, Z2), (X3, Y3, Z3), (X4, Y4, Z4)], where the first of the position parameters consists of a pair of parameters, each parameter in the list representing the Cartesian coordinate of a point on the boundary of the section in 3D. An extrusion line is a curved extrusion line that is represented by a list of parameters [(X h,1 ,Y h,1 ,Z h,1 ),(X h,2 ,Y h,2 ,Z h,2 ),(X h,3 ,Y h,3 ,Z h,3 ),(X h,4 ,Y h,4 ,Z h,4 )], optionally represented by a spline of less than 3 degrees.

[0123] In another variation of these implementations, the flags can be generalized to take integer values ​​between 0 and n greater than 1. Each value of the flag indicates either a segment or a type of curved line (arc, spline, or NURBS).

[0124] From the examples, it is clear that the set of parameters nbSides, points, and h are sufficient to parameterize and construct a 3D model of a 3D primitive.

[0125] Now, with reference to FIG. 7, an example of forming a data set for this particular selection of parameters will be discussed.

[0126] In an example of a data set formation method, random sampling is performed on the number of sides of the section, and nbSides is sampled according to a uniform probability distribution from integers in the interval [2,5]. In a variation of this example, nbSides is sampled according to a non-uniform probability distribution from integers in the interval [2,5]. Because cylinders and boxes occur more frequently in real 3D designs, the non-uniform probability distribution has larger values ​​for cylinders (nbSides=2) and boxes (nbSides=4) compared to other values ​​for nbSides. Uniform sampling is performed on the interval [h min ,h max The extrusion length (h) is between the maximum and minimum values ​​of []. min and h max is set by the user to, for example, 1 and 10, respectively, or is automatically set to default by the data set formation method. Further, parameter points are calculated to obtain regular sections for the prism when nbSides=3 or 5, for example, by selecting the number of nbSides of points on the circle at uniform distance. The selected points are then sorted in ascending order of their corresponding angles in the polar coordinate system. After obtaining regular sections for the other prisms, for nbSides=4, the interval [r min ,r max A new random parameter (r) is uniformly sampled corresponding to the ratio of the lengths between the two sides between the maximum and minimum values ​​of []. min and r max are set by the user, e.g., to 1 and 10, respectively, or automatically defaulted by the dataset creation method. In the dataset creation method options, the method generates irregular sections for the 3D model when nbSides=3, 4, or 5, for example, by selecting the nbSides number of points inside the unit disc. The selected points are then sorted in ascending order of their corresponding angles in the polar coordinate system.

[0127] The primitive 3D model is sampled from the cross product of the mentioned samples after a random rotation relative to the origin is applied to the primitive 3D model to mimic a random rotation. In an embodiment, the random rotation is applied to the X and Z axes, and the values ​​are uniformly sampled from the interval [10°, 80°] for the X axis and the interval [0°, 180°] for the Z axis. The primitive 3D model is then rescaled by selecting rescaling parameters and scaling the 3D model in all directions to fit within a unit sphere. This step improves learning. The 3D model is then projected from a fixed perspective onto a 2D plane to obtain a 256x256 photo of the model. Random noise is applied to the sketch lines, and any randomly selected vertices are moved in random or predefined directions by random or predefined values. This step improves 3D design methods applied to imprecise input 2D sketches.

[0128] The bounding square of the resulting sketch is then calculated and resized to 256x256. This sketch is black and white, with rendered edges in black. The image colors are inverted to obtain a black background and white lines. This step increases the value of zeros in the neural network's input 2D sketch, improving learning. The values ​​of the rendered sketch may be rescaled from the standard interval [0,255] to ~[0,1].

[0129] In an embodiment, training of the neural network is performed on a portion of the dataset formed by the dataset formation method discussed. In an embodiment, the 3D model may be one of the primitives, and training of the neural network may include supervised training, including minimizing a loss (L). The loss may penalize the sum of one or more of the following terms: -Type term that represents the discrepancy between the predicted extrusion height and the extrusion height of the 3D model

[0130]

number

[0131] Extrusion loss with h n specifies the respective extrusion length,

[0132]

number

[0133] is the predicted h n Specify. - a term of the type that represents the discrepancy between the predicted coordinates of points of a section of the 3D model and their corresponding predicted values

[0134]

number

[0135] where i designates a point and each p n,i specifies the coordinate n of each ground truth point [i][0] or point [i][1] of each 3D model,

[0136]

number

[0137] is the predicted p n,i In addition, specify SHIFT s (array) specifies a function to shift each element of array to the right s times. This function allows to not penalize circular permutations of the predicted set of points. - a term of type that represents the discrepancy between the predicted coordinates of the 3D model of the points of the section of the 3D model and their corresponding predicted values

[0138]

number

[0139] The number of sides with nbSides n is the nbSides ground truth for the nth sample.

[0140]

number

[0141] specifies the predicted probability that each 3D model is a 3D model of type k. n ==k is the number of sides n Or specify a function that gives 1 when it is equal to 0.

[0142]

number

[0143] as

[0144]

number

[0145] and

[0146]

number

[0147] A term of the type that represents the discrepancy between the normal probability distribution defined by and the standard normal distribution N(0,1)

[0148]

number

[0149] have Kullback-Leibler diffusion loss, where:

[0150]

number

[0151] specifies the predicted conditional mean, and

[0152]

number

[0153] specifies the predicted conditional standard deviation.

[0154] Here, N specifies the number of training samples, and n refers to each of the 3D models of the training samples. Furthermore, λ1, λ2, and λ3 specify the weights necessary to set the balance between variability and target reconstruction reliability. Selecting the magnitude of λ3 (compared to λ1 and λ2) privileges the variability of the generated 3D model and authorizes a larger prediction error in terms of side error, point error, and loss error in extrusion error. In an embodiment, (λ1, λ2, λ3) may be set as (4.0, 1.0, 001).

[0155] Now, with reference to FIG. 8, an example of machine learning using a VAE architecture will be discussed. A 2D sketch and random variables are input to a neural network, which outputs a set of parameters including three subsets of parameters: a list of points, an extrusion length, and a vector of probabilities for the predicted number of sides. The input sketch is processed by a CNN encoder to generate a first latent vector. The first latent vector is transformed into a second latent vector via a reparameterization trick using the mean and standard deviation and the input random variables. The notation +* for the operator represents the operation output = μ + δ * ε. An FC layer is used to output a vector probability for the predicted number of sides. The vector of probabilities for the predicted number of sides is concatenated with the second latent vector, which will be input to an RNN decoder, where each LSTM cell predicts a pair of points. The final state of the RNN decoder is used in conjunction with the vector input to the RNN decoder to output the extrusion length using an FC layer.

[0156] Now, referring to FIG. 9, an example of a machine learning method that combines the first and second aspects (i.e., uses a CVAE architecture) will be discussed. Compared to the VAE architecture discussed above, the prediction layer of nbSides is removed. In this example, a vector of probabilities is generated based on a user's selection. For example, being a cylinder is attributed to the third coordinate in the vector of probabilities, so to impose a cylinder, the input may be [0, 0, 1, 0, 0, 0]. This input is consistent with the vector predicted in the VAE architecture. In this example, the CVAE architecture is not trained separately, and the same learned weights from the VAE architecture are used.

[0157] In an example of a machine learning method related to a variation of the implementation, the neural network may use a second RNN to calculate a set of points defining a spline of the extrusion line. The neural network may predict a flag for each pair of points defining a section that refers to either a portion of a segment or a portion of a circular arc. Furthermore, instead of predicting an nbSides probability vector, the neural network may predict a STOP flag associated with each predicted pair of points. The flag represents the probability that the associated pair of points is the last couple of points defining the section. nbSides may be determined by the occurrence of the first STOP flag with a probability >0.5.

[0158] An example of a 3D design method is discussed with reference to FIG. 10. A 2D sketch is provided by a user using a haptic device without inputting random variables. The method calculates a bounding square and resizes it to 256 × 256. The sketch's color is inverted to obtain a white outline and a black background. Furthermore, the sketch's values ​​are rescaled from [0, 255] to [0, 1]. Several most probable 3D models are then calculated using the vanilla architecture and presented to the user. The user can select one of the provided models according to the user's design intent. If the user wants to generate variations on the proposed 3D model for the intended design purpose, the user can use the VAE architecture to generate a new 3D model from the same sketch and random variables automatically sampled from the distribution N(0, 1). To increase the variations on the proposed 3D model, the user can increase the standard deviation of the distribution, and the random variables may be sampled from the distribution N(0, 2). The user can also impose the type of the 3D model to be a box, given the same sketch and automatically sampled random variables, and generate a new 3D model of the imposed type.

[0159] The user can also generate new 3D models by imposing a type on the 3D model given the same sketch and, optionally, random variables. This is discussed with reference to FIG. 11. The user can impose a type on a cube or a cylinder by either entering their corresponding NbSides in an input window in the GUI or by selecting the intended type in the GUI via a haptic device. Several most probable 3D models of the imposed type are then computed and visually presented to the user using the CVAE architecture.

[0160] Another embodiment of a 3D design method is discussed with reference to FIG. 12 . In this embodiment, the method does not wait for a completed sketch to infer a 3D model. A CNN encoder uses partial information (from an incomplete sketch) and learned patterns to encode the 2D sketch into data that defines a 3D model. In this embodiment, different 3D models are suggested to the user while the user is drawing. Furthermore, the user can use a VAE architecture to generate variations on the same sketch and the proposed 3D model from random variables automatically sampled from N(0,1). Additionally or alternatively, the user can also use a CVAE architecture to generate new 3D models by imposing a 3D model type given the same sketch.

Claims

1. 1. A computer-implemented method of machine learning, comprising: - providing an architecture for a neural network configured to take as input a 2D sketch and to output a 3D model represented by said 2D sketch, said 3D model being a parameterized 3D model defined by a set of parameters composed of a first subset of one or more parameters and a second subset of one or more parameters, said neural network optionally being - outputting the values ​​for said set; - configured to take as input values ​​for said first subset from a user and to output values ​​for said second subset; - training said neural network; Equipped with The architecture includes a data fragment, and the neural network includes: - generating first values ​​for the pieces of data, determining the values ​​for the first subset based on the first values ​​for the pieces of data, and determining the values ​​for the second subset based on the first values ​​for the pieces of data, to selectively output values ​​for the set; optionally taking as input the values ​​for the first subset from the user and generating second values ​​for the pieces of data based on the values ​​for the first subset, and outputting the values ​​for the second subset based on the second values ​​for the pieces of data, in order to output the values ​​for the second subset; The method is characterized by being configured to:

2. 2. The machine learning method of claim 1, wherein the values ​​for the first subset correspond to an image of a function of the first values ​​for the data piece, and the second values ​​are a preimage of the values ​​for the first subset according to the function.

3. 3. The machine learning method of claim 2, wherein the piece of data is a vector representing a probability distribution over the first subset, and the function is an argmax function.

4. 4. The machine learning method of claim 3, wherein the second value of the data piece is a vector having a value of 1 for the coordinate corresponding to the value for the first subset and a value of 0 for all other coordinates.

5. 5. The machine learning method of claim 1, wherein the 3D model is defined by sections and extrusions, and the first subset consists of numbers representing types of the sections.

6. 6. The machine learning method of claim 5, wherein the number represents the number of sides that form the section.

7. 7. The machine learning method according to claim 5 or 6, wherein the second subset comprises parameters defining the extrusion and / or parameters further defining the section.

8. 8. The machine learning method of claim 7, wherein the parameters further defining the section include a list of position parameters, each of which optionally includes a pair of 3D points.

9. 9. The machine learning method of claim 8, wherein the neural network includes a recurrent neural network (RNN) configured to output a value for the list, the neural network optionally further configured to output a value for the extrusion based on a final state of the RNN.

10. 1. A computer-implemented method of 3D design, comprising: - providing a neural network trained according to the method of any one of claims 1 to 9; - providing a 2D sketch; - providing, by a user, values ​​for said first subset; - applying said neural network to said 2D sketch to output values ​​for said second subset, taking as input said values ​​for said first subset from said user; A method of 3D design comprising:

11. before the user provides the values ​​for the first subset; - applying said neural network to said 2D sketch to output one or more respective values ​​for each of said sets; displaying one or more 3D models represented by the 2D sketch, each defined by a respective value output for the set, and the user providing the values ​​for the first subset while the one or more 3D models are displayed; 11. The method of 3D design of claim 10, further comprising:

12. A computer program comprising instructions which, when executed on a computer system, cause the computer system to carry out the method of any one of claims 1 to 11.

13. A device comprising a memory storing a computer program according to claim 12.

Images