Learning a neural network for inferring CAD features of solid objects

Abstract

In particular, the present invention relates to a computer-implemented method for learning a neural network configured to infer solid CAD features representing a 3D shape from a hand-drawn drawing representing the 3D shape. The method comprises providing a dataset comprising hand-drawn drawings, each hand-drawn drawing representing a respective 3D shape, and learning a neural network based on the dataset. The method results in an improved solution for inferring 3D modeled objects representing the 3D shape from hand-drawn drawings representing the 3D shape.

Description

Technical Field

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly, to solutions related to learning a neural network configured to infer a 3D modeled object representing a 3D shape from a hand drawing representing the 3D shape. Background Art

[0002] Numerous systems and programs are available on the market for the design, engineering, and manufacturing of objects. CAD stands for computer-aided design and refers to software solutions for designing objects. CAE stands for computer-aided engineering and refers to software solutions for simulating the physical behavior of future products. CAM stands for computer-aided manufacturing and refers to software solutions for defining manufacturing processes and operations. Graphical user interfaces (GUIs) play a key role in the efficiency of these CAD systems. These technologies can be embedded in product lifecycle management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge to develop products from concept to end-of-life across the entire extended enterprise. PLM solutions offered by Dassault Systèmes (trademarks CATIA, ENOVIA, and DELMIA) offer an Engineering Center (which organizes product engineering knowledge), a Manufacturing Center (which manages manufacturing engineering knowledge), and an Enterprise Center (which allows for the integration and connection of engineering and manufacturing centers across the enterprise). Together, the system provides an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, driving optimized product definition, manufacturing preparation, production, and service.

[0003] In this context, researchers have been looking for solutions to infer 3D modeled objects representing 3D shapes from hand drawings representing 3D shapes. Notably, the following references are relevant to this problem:

[0004] [1] T. Igarashi, S. Matsuoka and H. Tanaka, “Teddy: a sketching interface for 3D freeform design”, in Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 1999, pp. 409-416.

[0005] -[2] Z. Lun, M. Gadelha, E. Kalogerakis, S. Maji and R. Wang, “3D shape reconstruction from sketches via multi-view convolutional networks”, in 2017 International Conference on 3D Vision (3DV), 2017, pp. 67-77.

[0006] [3] J. Delanoy, M. Aubry, P. Isola, A. Efros, and A. Bousseau, “3D Sketching using Multi-View Deep Volumetric Prediction”, Proceedings of the ACM on Computer Graphics and Interactive Techniques, vol. 1, no. 21, May 2018.

[0007] -[4] B.Xu, W.Chang, A.Sheffer, A.Bousseau, J.McCrae, and K.Singh, “True2Form: 3D curve networks from 2D sketches via selective regularization”, ACM Transactions on Graphics, vol. 33, no. 4, 2014.

[0008] [5] C. Li, H. Pan, Y. Liu, X. Tong, A. Sheffer, and W. Wang, “BendSketch: modeling freeform surfaces through 2D sketching,” ACM Transactions on Graphics, vol. 36, no. 4, pp. 1-14, July 2017.

[0009] -[6] Y. Zheng, H. Liu, J. Dorsey, and N. J. Mitra, “Smart Canvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies”, Computer Graphics Forum, 2016, vol. 35, pp. 37-48.

[0010] -[7] Y. Li, X. Luo, Y. Zheng, P. Xu, and H. Fu, “SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image,” in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017, pp. 387-399.

[0011] Papers [1] to [3] provide methods for generating 3D meshes from sketches.

[0012] The main drawbacks of meshes are that they are rarely editable and lack semantics. A 3D mesh discretizes a surface into a collection of primitive polygons (usually triangles). If the surface is watertight and manifold, it can represent a solid. However, this representation is unsuitable for 3D modeling, as triangles should be edited individually.

[0013] In addition, the method in [1] generates circular meshes. In addition, the method in [1] requires the sketch strokes to be annotated with associated actions (create, stretch, bend).

[0014] Papers [2] and [3] avoid this circle-like prior and avoid the need for annotated strokes due to recent advances in deep learning. However, both methods still output meshes. In addition, paper [2] requires one or more sketches of the object (i.e., front, side, top) from a normalization perspective, which is a significant limitation for users.

[0015] Papers [4] to [7] provide methods for generating parametric surfaces from sketches.

[0016] The paper [4] generates curves by imposing some regularity (orthogonality, parallelism, coplanarity) on the sketch strokes. Then, the surface can be fitted to the curve. This approach requires the user to be very accurate when drawing the sketch: the strokes must represent the geometry of the object (following its surface, but not necessarily its outline), and closed strokes may be required. In addition, the user must specify the actual 3D intersections between the strokes.

[0017] The paper [5] fits surfaces directly to sketches. However, the sketch strokes must be labeled. The labels provide some information about the surface on which the sketch strokes should lie: the limits of the surface, concavity / convexity, and sharp edges. Due to the nature of these labels, the user must be an expert in geometry, as a simple perspective view of the object will not be sufficient.

[0018] Therefore, the solutions provided in papers [4] and [5] fit parametric surfaces to the input drawing, but the generated surfaces are not necessarily closed and, therefore, they may not represent solid bodies. Even if they do represent solid bodies, they cannot be edited into meaningful solid CAD features. In addition, both methods require additional user input (intersection points or stroke markers).

[0019] The paper [6] generates 3D curves from sketch strokes. The method assumes that each stroke represents a planar 3D curve. Once the plane is determined, the 3D curve can be obtained by back-projecting the strokes from the 2D sketch onto the plane where the strokes should be located. Strokes located on the same plane can intersect each other and define the boundaries of a planar patch. The obtained planar patch can then be stretched in the perpendicular direction. The main disadvantage of this method is that the plane directions are limited to the three canonical directions of an orthogonal reference frame. To define this reference frame, the user is required to explicitly display two sets of parallel lines, the first set of lines being orthogonal to the second set of lines. The reference frame is then inferred from the vanishing point.

[0020] A very similar approach is proposed in [7]. However, in addition to the classic RGB image, the proposed method requires a depth image. The stroke plane is estimated directly from the 3D representation of the scene provided by the depth map.

[0021] Therefore, papers [6] and [7] provide techniques for producing results that include surfaces that may be extruded. However, both methods assume that the underlying 2D sketch lies on a set of constrained planes. In addition, both methods require additional input: images, user definitions of parallel lines, and depth images.

[0022] In this context, there remains a need for improved solutions for inferring a 3D modeled object representing a 3D shape from a hand-drawn drawing representing the 3D shape. Summary of the Invention

[0023] A computer-implemented method for learning a neural network configured to infer solid CAD features representing a 3D shape from a hand-drawn drawing representing the 3D shape is provided. The method comprises providing a dataset comprising hand-drawn drawings, each hand-drawn drawing representing a respective 3D shape, and learning a neural network based on the dataset.

[0024] The method may include one or more of the following:

[0025] - the solid CAD feature includes a corresponding curve sequence, the method further comprising: providing a candidate curve set, the learning also being based on the candidate curve set;

[0026] - the neural network includes recurrent neural network (RNN) units, each RNN unit outputting corresponding data of a corresponding curve for inferring a CAD feature of the solid at a corresponding time step;

[0027] - the candidate curve set comprises a set product of: a discrete set of curve types, and, for each curve type, a corresponding discrete set of one or more parameter domains for each corresponding parameter, each parameter domain having a corresponding parameter value for the corresponding parameter, each curve type forming a corresponding element of the candidate curve set using the corresponding parameter value of each parameter domain in the corresponding discrete set of one or more parameter domains;

[0028] - the corresponding data for inferring the corresponding curve of the solid CAD feature comprises: corresponding first data for inferring the type of the corresponding curve, and corresponding second data for inferring the corresponding parameter value for each parameter domain in the corresponding discrete set of one or more parameter domains;

[0029] - Each RNN unit having one or more previous RNN units takes as input information information that can be inferred from corresponding data output by the one or more previous RNN units;

[0030] - the solid CAD feature includes a scan operation, the method further comprising: providing a candidate set of scan operations, the learning also being based on the candidate set of scan operations;

[0031] - the neural network includes a neural network component that outputs data for inferring a scanning operation of a solid CAD feature;

[0032] - the candidate set of the scanning operation comprises a set product of: a discrete set of scanning curve types, and, for each scanning curve type, a corresponding discrete set of one or more parameter domains for each corresponding parameter, each parameter domain having a corresponding parameter value for the corresponding parameter, each scanning curve type utilizing the corresponding parameter value of each parameter domain in the respective discrete set of one or more parameter domains to form a corresponding element of the candidate set of the scanning operation;

[0033] The data for inferring the scanning operation of the solid CAD feature comprises: first data for inferring a type of a scanning curve; and second data for inferring a parameter value for each parameter domain in a corresponding discrete set of one or more parameter domains;

[0034] - the neural network comprises a convolutional neural network (CNN) that encodes an image representing the hand-drawn image;

[0035] - the dataset further comprises, for each hand drawing in the one or more hand drawings, a corresponding solid CAD feature representing the 3D shape relative to the hand drawing, and the learning comprises supervised training comprising setting the loss L sup Minimize the loss L sup Optionally invariant to: a cyclic permutation of a sequence, each solid CAD feature comprising a corresponding sequence of curves, and / or a reference-based sketch, each solid CAD feature comprising a scan operation; and / or

[0036] - Learning consists of unsupervised training, which involves making the loss L unsup Minimize, for hand-drawn images, the loss L unsup Penalizing differences from renderings derived from corresponding inferred solid CAD features, the neural network optionally outputs one or more discrete distributions of probabilities, the minimization comprising: a loss L with respect to one or more variables unsup The back propagation of the gradient of , which variables represent the corresponding discrete distribution of probabilities, and the gradient is obtained using the enhancement algorithm.

[0037] Also provided are data structures representing neural networks that can learn according to the learning method.

[0038] A computer-implemented method for using the data structure is also provided. The method comprises providing a hand-drawn image representing a 3D shape, applying the neural network to the hand-drawn image, and inferring a solid CAD feature representing the 3D shape based on the results of the application.

[0039] A computer program is also provided, comprising instructions for executing the learning method and / or the using method.

[0040] Also provided is a device comprising a data storage medium having the data structure and / or the program recorded thereon. The device can form or be used as a non-transitory computer-readable medium, for example, on a SaaS (Software as a Service) or other server, or on a cloud-based platform, etc. The device can alternatively include a processor coupled to the data storage medium. Thus, the device can form a computer system in whole or in part (for example, the device is a subsystem of the entire system). The system can also include a graphical user interface coupled to the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments will now be described by way of non-limiting examples and with reference to the accompanying drawings, in which:

[0042] - Figure 1 An example of a graphical user interface for the system is shown;

[0043] - Figure 2 An example of a system is shown; and

[0044] - Figure 3-Figure 12 The method is shown. DETAILED DESCRIPTION

[0045] Therefore, a computer-implemented method is presented for learning a neural network. In this context, a neural network is a function configured to take as input any hand-drawn image, each representing a corresponding 3D shape, process the input, and output data as a result of the processing, the data being configured to infer physical CAD features representing the 3D shape corresponding to the input. Thus, such a neural network can be used to infer physical CAD features corresponding to hand-drawn images representing 3D shapes.

[0046] Therefore, a computer-implemented method for performing such speculation is proposed, which may be referred to as a "speculation method".

[0047] The inference method may include providing a hand-drawn drawing representing a 3D shape. The inference method may also include applying a neural network to the hand-drawn drawing. The inference method may also include inferring (e.g., substantially in real time when applied) at least one solid CAD feature representing the 3D shape based on a result of the application (i.e., data output by the neural network when the hand-drawn drawing is input). The output data may (directly) include the solid CAD feature and / or be processed to (indirectly) determine the solid CAD feature, e.g., automatically and / or according to a predetermined algorithm.

[0048] In the case of indirect determination, the output data may, for example, include one or more probability distributions for selecting values ​​defining the at least one solid CAD feature, and the predetermined algorithm may, for example, include determining the at least one solid CAD feature based on the one or more probability distributions. The determination may, for example, disfavor the selection of values ​​with low probabilities, for example, by selecting one or more most likely values ​​based on the one or more probability distributions. The determination may then derive one or more elements (e.g., geometric shapes) of the solid CAD feature, each element corresponding to a corresponding one or more selected values.

[0049] In an example, determining can also include modifying one or more of the elements. For example, the inferred solid CAD feature can include a sequence of corresponding geometric shapes (e.g., curves). In such an example, modifying can include modifying the geometry of one or more geometric shapes (e.g., curves) in the corresponding sequence. In a specific example, the inference can be limited to closed (e.g., curves forming a loop) corresponding sequences of geometric shapes (e.g., curves). In such a specific example, modifying the geometry of one or more geometric shapes (e.g., curves) in the sequence can include closing the corresponding sequence.

[0050] As is known from the field of machine learning, processing an input by a neural network involves applying operations to the input, where the operations are defined by data including weight values. Therefore, learning a neural network involves determining weight values ​​based on a dataset configured for such learning. To this end, the dataset includes data segments, each of which forms a corresponding training sample. The training samples represent a variety of situations in which the neural network will be used after learning. Any dataset referenced herein may include a number of training samples exceeding 1,000, 10,000, 100,000, or 1,000,000.

[0051] Therefore, in particular, a computer-implemented method for forming such a data set is proposed, which may be referred to as a “data formation method.” A computer-implemented method for performing such learning is also proposed, which may be referred to as a “learning method.”

[0052] The machine learning process may include dataset formation methods and / or learning methods, and optionally inference methods.

[0053] The machine learning process may specifically include both a dataset formation method and a learning method. The learning method may, for example, be performed at least in part based on a dataset formed by the dataset formation method, for example, after the dataset formation method. Such a machine learning process is particularly efficient and provides improved accuracy.

[0054] Alternatively, the machine learning process may include a dataset formation method and performing any other computer-implemented method for learning a neural network based on the dataset (other than the proposed learning method). Alternatively, the machine learning process may include performing the learning method on a dataset provided by any other computer-implemented method (other than the proposed dataset formation method), such as another method for forming a dataset or retrieving such a dataset.

[0055] Before presenting the dataset formation method and the learning method, the data structures involved are now discussed. It will be appreciated that the data structure definitions and examples provided herein can be applied to at least a portion (e.g., all) of a dataset formed by the dataset formation method and / or at least a portion (e.g., all) of a dataset provided in a learning method, and / or to at least one input and / or at least one output of an inference method.

[0056] A solid CAD feature is a 3D modeled object that represents the 3D shape of a solid. Any 3D shape herein represents the 3D spatial distribution of the material that makes up the solid. A 3D modeled object can represent the skin (i.e., the outer surface) of a solid. A 3D modeled object can be provided to a CAD system for display to a user and / or for the user to perform one or more CAD design operations on the 3D modeled object.

[0057] A modeling object is any object defined by stored data (e.g., in a database). By extension, the expression "modeling object" refers to the data itself. Depending on the type of system used to design the modeling object, the modeling object can be defined by different kinds of data. Indeed, the system can be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In those different systems, the modeling object is defined by the corresponding data. Thus, one can refer to a CAD object, a PLM object, a PDM object, a CAE object, a CAM object, CAD data, PLM data, PDM data, CAM data, CAE data. However, these systems are not mutually exclusive, as a modeling object can be defined by data corresponding to any combination of these systems. Thus, a system may very well be both a CAD system and a PLM system.

[0058] By CAD system, it is also meant any system that is at least suitable for designing modeled objects based on a graphical representation of the modeled objects, such as CATIA. In this case, the data defining the modeled objects include data that allow the modeled objects to be represented. The CAD system can, for example, use edges or lines (in some cases using faces or surfaces) to provide a representation of the CAD modeled objects. Lines, edges or surfaces can be represented in various ways, for example, non-uniform rational B-splines (NURBS). In particular, CAD files contain specifications from which geometry can be generated, thereby allowing a representation to be generated. The specifications for the modeled objects can be stored in a single CAD file or in multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per component. A modeled object can typically be a combination of thousands of components.

[0059] In the context of CAD, a modeled object may typically be a 3D modeled object, for example, representing a product such as a part or a combination of parts, or possibly a combination of products. By "3D modeled object" is meant any object that is modeled from data that allows its 3D representation. The 3D representation allows the component to be viewed from various angles. For example, when represented in 3D, a 3D modeled object can be manipulated and rotated about any of its axes or about any axis in the screen on which the representation is displayed. In particular, this excludes 2D icons that are not modeled in 3D. The display of a 3D representation facilitates design (i.e., increases the speed at which a designer can statistically complete a task). Since the design of a product is part of the manufacturing process, this can speed up industrial manufacturing processes.

[0060] Any 3D shape herein may represent the geometry of a product to be manufactured in the real world, such as a component (e.g., of a machine) or an assembly of components (or equivalently, an assembly of components, since from the method's perspective a combination of components can be considered a component in itself, or the method can be applied independently to each component in the assembly), or more generally, any rigid assembly (e.g., a movable machine). The product may be one of a wide variety of industries, including aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation. Thus, a 3D shape can represent an industrial product, which can be any mechanical part, such as parts of land vehicles (including, for example, automobile and light truck equipment, racing cars, motorcycles, van and motor equipment, trucks and buses, trains), parts of aircraft (including fuselage equipment, aerospace equipment, propulsion equipment, defense products, aviation equipment, space equipment), parts of marine transportation vehicles (including naval equipment, commercial vessels, offshore equipment, yachts and work boats, ship equipment), general mechanical parts (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installed equipment, industrial equipment products, metal products, tire manufacturing products), electromechanical or electronic components (including, for example, consumer electronics products, safety and / or control and / or instrument products, computing and communication equipment, semiconductor products, medical devices and equipment), consumer products (including, for example, furniture, home and garden products, leisure goods, fashion products, products of hard goods retailers, products of soft goods retailers), packaging (including, for example, food, beverage and tobacco, beauty and personal care, household product packaging).

[0061] As is known from the CAD field, a solid CAD feature representing a 3D shape is herein a list of one or more instructions that allow a CAD system to generate a 3D geometry of a solid, and, for example, render and display a 3D graphical representation of the solid. The 3D graphical representation may include a 3D subdivision of the outer surface of the solid, such as a mesh subdivision (e.g., polygons, optionally triangles). In an example, the mesh subdivision may include a number of faces (e.g., polygons, optionally triangles) greater than 100 (e.g., greater than 1000).

[0062] As is known from the CAD art, solid CAD features can be selected and / or edited by a user of a CAD system. A list of one or more instructions can refer to one or more elements, each of which is selectable and / or modifiable. The user can edit a solid CAD feature using any method known from the CAD art, including, for example, selecting at least one element thereof and / or deleting or modifying that element and / or adding at least one new element. This type of editing offers ergonomic advantages compared to editing discrete geometric representations such as meshes.

[0063] Specifically, the instruction list can represent one or more constraints involving one or more parameterized geometric shapes to create a solid. Each parameterized geometric shape includes one or more corresponding parameters, for example, including a type parameter and / or one or more shape parameters. One or more parameters can be optional and / or editable (for example, including at least one (for example, each) shape parameter). Editing the parameters can include modifying the values ​​of the parameters. In such a case, the number of geometric shapes involved in the inferred solid CAD feature may be lower than the number of faces of its subdivided mesh by a factor of 5 or more (for example, more than 10), and / or the number of geometric shapes involved may be less than 50 (for example, less than 20). Each geometric shape can be defined by multiple parameters less than 20 (for example, less than 10). Therefore, compared to meshes, solid CAD features represent 3D shapes in a more compact and organized manner. Moreover, as is known from the CAD field, acting on a single element of a solid CAD feature may have a global impact, while in the case of a mesh, acting on a single geometric entity will only cause a local impact. For example, as opposed to moving / adding a large number of mesh triangles, any solid CAD feature herein can be configured to enlarge the 3D shape it represents (e.g., to make it curved, such as a cylinder) by modifying a parameter value below 5 (e.g., below 3). Furthermore, solid CAD functionality is more semantically meaningful to the user than discrete geometric shape representations.

[0064] For any solid CAD feature herein, the instruction list may represent one or more real-time / declarative constraints and / or one or more process constraints. For any solid CAD feature herein, the geometric shape may include a geometric 3D shape, i.e., a geometric shape defined in 3D space. The 3D shape may, for example, include a 3D curve and / or any other 3D shape (e.g., a 3D point and / or a 3D entity). Any 3D curve herein may be planar (i.e., defined in 3D space, but contained in a plane of the 3D space) or non-planar (i.e., not contained in any plane). In an example, for any solid CAD feature herein, the geometric shape may include one or more geometric shape sequences, such as one or more (e.g., closed) sequences of (e.g., planar) 3D curves (e.g., sequence curves). A sequence of planar 3D curves that results in a closed contour (which must be in a plane containing each of these curves) is referred to as a "closed 2D planar sketch." In an example, for any solid CAD feature herein, the geometry may alternatively or additionally include one or more sweep operations, each sweep operation taking one or more geometric objects (e.g., a single closed 2D planar sketch) as input and / or based on another geometric object (e.g., a 3D curve). Any sweep operation herein may, for example, be defined by a 3D curve (e.g., a sweep curve), optionally non-planar.

[0065] Any 3D curve herein may belong to a category (referring to object-oriented programming) defined by one or more fields, each field corresponding to a corresponding parameter defining the shape of the 3D curve in the context of that category. Any 3D curve herein may be defined in particular by one or more shape parameters, for example including one or more positioning parameters (e.g., 3D coordinates) for one or more boundary (e.g., start and / or end) points. Any 3D curve herein may alternatively or additionally be defined by a type parameter (e.g., among curve types including line, arc, and / or cubic Hermite interpolation segment types). Depending on its type, any 3D curve herein may alternatively or additionally be defined by additional shape parameters, optionally including: one or more positioning parameters (e.g., 3D coordinates) for each of one or more intermediate points of the curve and / or one or more tangent vector parameters (e.g., 3D vector definitions) for each of one or more points of the curve (e.g., including the start and / or end points). This parameterization of 3D curves is particularly ergonomic for users to edit.

[0066] In terms of accuracy and robustness of learning and the diversity of situations represented by the intended applications for hand-drawn drawings, in a particularly effective example, any solid CAD feature in this context can be defined specifically (and only) by the following, relative to a given 3D space:

[0067] - a closed 2D planar sketch consisting of a series of planar curves (e.g., a sequence of curves) that form a closed profile,

[0068] - A sweep operation (eg, defined by a sweep curve) is then performed to generate a solid by sweeping a closed surface defined by the 2D planar sketch in 3D space, ie, the solid is defined by the volume passed through by the closed surface.

[0069] In a particularly effective example, solid CAD features have meaningful parameters and can be easily edited. For example, a base 2D planar sketch can be modified, and the resulting geometry can be automatically updated by applying a sweep operation to the new base. Editing can include selecting and editing the parameters of individual 3D curves (e.g., 3D curves that make up a closed 2D planar sketch or 3D curves that define a sweep operation). Editing can be limited by the requirement that the plane and / or sequence not be modified and / or closure be maintained. This type of solid CAD feature is particularly ergonomic for the user to edit.

[0070] In an example, the inference method may include: displaying a 3D graphical representation of a 3D shape represented by a solid CAD feature (i.e., a 3D shape obtained by executing a list of instructions for the solid CAD feature), such as a 3D subdivision thereof. The inference method may further include displaying a graphical representation of the structure of the solid CAD feature, such as a tree arrangement of graphical items representing each corresponding geometric shape or geometric operation, or highlighting of the 3D graphical representation of the geometric shapes involved in the solid CAD feature. Editing of the solid CAD feature may include, for example, selecting a graphical item through graphical interaction. Graphic interaction in this article refers to operating a tactile device to activate pixels at a position corresponding to the position of a graphical item on the display. For example, a mouse cursor may be placed over the displayed graphical item, or the displayed graphical item may be touched directly via a touch screen or indirectly via a touchpad. The selection may trigger the opening of a dialog box, etc., and the input of one or more parameter values ​​by the user, for example via a keyboard. In a particularly effective example of solid CAD features, graphical interaction can include drag-and-drop actions on graphical representations of points or vectors linked to a 3D curve to move the position of the point and / or modify the direction and / or magnitude of the vector, such as when the 3D curve belongs to a closed 2D planar sketch constrained in the plane. This type of ergonomic editing is well known in the CAD field.

[0071] In an example, as an alternative or supplement to the user editing the inferred solid CAD features, the inference method may include, for example, combining the solid CAD function to one or more other solid CAD features via user interaction. For example, the inference method may include adding the solid CAD feature to an existing feature tree that includes one or more other solid CAD features, for example, by connecting the solid CAD feature to any node of the feature tree. Optionally, the inference method may also include editing one or more solid CAD features of the feature tree. In a specific example, the inference method may be repeated, and the inference method includes providing several hand drawings, and inferring several solid CAD features respectively by applying a neural network to each hand drawing. In such a specific example, the inference method may then include combining the inferred solid CAD features in a single feature tree. These specific examples allow the creation of feature trees based on hand drawings in an ergonomic manner, thereby performing complex CAD designs. Any feature tree herein may be a constructive solid geometry (CSG) tree.

[0072] CAD systems can be history-based. In this case, modeled objects are further defined by data including a history of geometric features. Modeled objects can be designed by a human (i.e., a designer / user) using standard modeling features (e.g., extrude, revolve, cut, and / or wrap) and / or standard surfacing features (e.g., sweep, blend, loft, fill, deform, and / or smooth). Many CAD systems that support this type of modeling functionality are history-based. This means that the creation history of design features is typically stored via an acyclic data flow that links these geometric features together via input and output links. The history-based modeling paradigm has been widely known since the early 1980s. Modeled objects can be described using two persistent data representations: history and B-rep (i.e., boundary representation). The B-rep is the result of the calculations defined in the history. When representing a modeled object, the shape of the component displayed on the computer screen is the B-rep (e.g., a subdivision of the B-rep). The component's history represents the design intent. Essentially, the history collects information about the operations that the modeled object has undergone. The B-rep can be stored along with the history to facilitate the display of complex components. The history can be saved with the B-rep to allow design changes to be made to the part according to the design intent. Due to the hand-drawing method, the speculative method of the specific example allows the history of such geometric features to be retrieved via the feature tree in an ergonomic manner.

[0073] Mechanical simulations can be performed based on solid CAD features and / or feature trees. Indeed, since solid CAD features and / or feature trees form a compact and parameterized representation of the 3D shape in a compact form (relative to a discrete geometric representation), mechanical calculations can be performed more accurately. In particular, it is easier and more direct to calculate finite element meshes from CSG (as taught by Edwin Boender, Willem F.Bronsvoort, Frits H.Post, "Finite-elementmesh generation from constructive-solid-geometry models", Computer-Aided Design, Vol. 26, No. 5, May 1994, pp. 379-392, or K Ho-Le, "Finite element mesh generation methods: a review and classification", Computer-Aided Design 38, Vol. 20, No. 1, January / February 1988). In addition, CSG also ensures the water resistance of the surface. Thus, a mechanical simulation can include computing one or more finite element meshes from a solid CAD feature and / or a feature tree (e.g., CSG) and / or one or more modifications thereof, and then performing a finite element analysis based on each finite element mesh. Furthermore, because the solid CAD features are parameterized, the solid CAD features can be easily optimized based on the simulation results. Thus, one or more modifications can be used to optimize the solid CAD features based on each previous finite element analysis result.

[0074] Any hand-drawn image representing a 3D shape herein is a 2D image that a (human) user can design by hand to represent the 3D shape. "Freehand designable" refers to the characteristic that the image represents a 2D image drawn by a user. Thus, the hand-drawn image may have been created by the user, but alternatively, it may have been synthesized, i.e., manually and automatically synthesized by a computer system. In particular, the hand-drawn image in the dataset formation method can be synthesized. On the other hand, it is desired that the hand-drawn image in the inference method is effectively drawn by hand by the user, although synthesized hand-drawn images can also be provided to the inference method.

[0075] Thus, a hand-drawn image is a 2D image data structure that includes one or more (2D planar) strokes that represent a 3D shape from the corresponding viewpoints from which the 3D shape is visible. The one or more strokes are defined on the same plane and represent perspective views of the 3D shape from various viewpoints. Each stroke is a continuous or substantially continuous curve defined in a plane.

[0076] In an example, for any hand-drawn image herein, the one or more strokes may include a number of strokes less than 20 (e.g., less than 10) strokes, optionally greater than or equal to 2. In an example, for any hand-drawn image herein, the one or more strokes may each represent a corresponding characteristic edge of a 3D shape as viewed from a viewpoint, such as a contour edge or a visible edge of an entity represented by the 3D shape. A "visible edge" refers to an edge of an entity that is noticeable to a user through the effect of light reflection. The visible edges of any entity herein may include (e.g., only include) one or more edges, each edge forming a C of the entity representation. k Discontinuity, k is an integer greater than or equal to 1 (eg, and optionally less than 10 or 4).

[0077] In an example, for any hand-drawn drawing herein, one or more (e.g., all) brush strokes may each have an end point at a distance from another brush stroke (e.g., from the end point of the other brush stroke) that is below a predetermined threshold, such as 10 mm or 5 mm. This reflects the fact that when drawing a hand-drawn drawing, the user may intend to connect the end point to the other brush stroke (e.g., the end point of the other brush stroke) in this manner to represent 3D in 2D. In an example, for any hand-drawn drawing herein (e.g., one or more hand-drawn drawings and / or one or more hand-drawn drawings in a dataset input to the inference method), one or more (e.g., all) brush strokes may each have an end point at a non-zero distance (i.e., a positive gap) from another brush stroke (e.g., from the end point of the other brush stroke). This reflects the fact that when drawing a hand-drawn drawing, the user may often fail to perfectly connect the end point to the other brush stroke (e.g., the end point of the other brush stroke), resulting in a gap. This is because the user is sketching freely, and therefore quickly / ergonomically. Despite this imprecision, the inference method can still accurately infer solid CAD features. And the dataset can represent this imprecision to make the learning accurate.

[0078] In the examples, for any hand-drawn drawing herein, the brushstrokes freely represent 3D shapes. This means that they can be defined in any order, without predefined constraints (e.g., having to have parallel and / or orthogonal brushstrokes), and they are not organized into clusters (e.g., based on distance), where each cluster represents a plane of a solid. In the examples, for any hand-drawn drawing herein, alternatively or additionally, one or more brushstrokes can each be irregular. This means that the brushstrokes cannot be represented by parameterized 2D curves (at least one of the types described herein) without approximating the brushstrokes and thereby introducing errors. Since the brushstrokes are expected to be hastily drawn by the user, they generally do not form a regular geometric shape. In the examples, irregular brushstrokes are allowed to be particularly / obviously irregular. In the examples, any hand-drawn drawing herein can alternatively or additionally be provided in a raw format, e.g., without any tags associated with the brushstrokes (e.g., action annotations such as create, stretch, bend) and / or without any additional data (e.g., images, user-defined parallel lines, depth images). The inference method is therefore ergonomic because the user is free to draw strokes without any constraints (e.g., the inference method makes no assumptions on the 2D sketch plane) and no additional input is required beyond the freehand drawing.

[0079] According to speculation, each stroke is unparameterized, meaning that the data structure representing the stroke includes data segments that each represent a portion of the stroke (e.g., pixels), and the data structure represents a raw aggregation of these portions. The speculation method can include a user drawing each stroke of the input hand-drawn drawing. Sketching can be performed by graphical interaction with a computer screen, by sequentially (e.g., substantially continuously) activating a series of pixels of the screen that define the strokes. Activation can be performed via a tactile device (e.g., a mouse) acting on the pixels (e.g., cursor movement that defines the strokes) or via user interaction with a touchscreen or touchpad (e.g., interaction includes drawing the strokes on the touchscreen or touchpad by physical touch). This is known in the CAD field as sketching or stroke design. Alternatively, a scan of a paper sketch or any other image can be input to the speculation method. Alternatively, a hand-drawn drawing (e.g., already generated by another user and / or any of the above-mentioned techniques) can be retrieved from a memory or received from a remote computer system.

[0080] In an example, the inference method can include displaying a graphical representation of the hand-drawn drawing as the user draws the strokes (e.g., in real time). The inference method can trigger the application of the neural network. For example, when the user is finished, the user can trigger the application of the neural network and infer the solid CAD feature. As a result, the method can display a 3D graphical representation of the solid CAD feature, optionally while still displaying the graphical representation of the hand-drawn drawing. The user can then hide the graphical representation of the hand-drawn drawing to process (e.g., edit, integrate into a tree, simulate) the solid CAD feature. The inference method can be performed while displaying a complex 3D modeled object designed by the user, optionally including repeating the above-described inference method. Thus, the inference method provides sketch-based design capabilities to history-based CAD systems.

[0081] Thus, any of the methods herein may be part of a process, in particular a speculative method, for designing a 3D modeled object comprising solid CAD features. "Designing a 3D modeled object" refers to any action or series of actions that is at least part of a process for producing a 3D modeled object. Thus, the method may include creating a 3D modeled object from scratch. Alternatively, the method may include providing a previously created 3D modeled object and then modifying the 3D modeled object.

[0082] Any of the methods herein may be included in a manufacturing process, which may include, after executing the method, producing a physical product corresponding to the modeled object. In any case, the modeled object designed by the method may represent a production object. Thus, the modeled object may be a modeled entity (i.e., a modeled object representing an entity). The production object may be a product, such as a component or a combination of components. Because the method improves the design of the modeled object, the method also improves the production of the product, thereby increasing the productivity of the manufacturing process.

[0083] Now we briefly introduce the dataset formation method and learning method.

[0084] The dataset formation method includes generating (e.g., synthesizing) one or more solid CAD features, the one or more solid CAD features including each solid CAD feature representing a respective 3D shape. The dataset formation method also includes, for each solid CAD feature, determining (e.g., synthesizing) one or more corresponding hand-drawn images, each hand-drawn image representing a corresponding 3D shape (e.g., based on the generated solid CAD features), and inserting one or more training examples into the dataset, each example including a (e.g., synthesized) solid CAD feature and a corresponding (e.g., synthesized) hand-drawn image.

[0085] “Providing / generating / determining / synthesizing a corresponding data segment” means that, at least at a certain moment, the data segment is recorded on a memory of the computer system and is accessible to a processor.

[0086] A "dataset" refers to any data structure that includes training samples and is configured for machine learning, the data structure being recorded in a computer system's memory and accessible to a processor. A method for forming a data set may include recording the data set in a non-volatile memory.

[0087] "Inserting" or "not inserting" data segments into a dataset as training samples means that, at least at a certain point in the dataset formation method (e.g., at the end), the dataset is recorded on a (e.g., non-volatile) memory of a computer system and is in the following state: the dataset includes all "inserted" data segments (i.e., the data segments are therefore recorded on the memory and in a data structure representing the dataset), while the dataset excludes (i.e., does not include) all "uninserted" data segments (i.e., the data segments are therefore not recorded on the memory, at least not in the data structure representing the dataset). Then, in the example after further processing the dataset, learning can be based on the dataset. Therefore, the learned neural network will be consistent with the data segments inserted into the dataset.

[0088] The dataset formation method provides an improved solution for CAD design. The formed dataset includes training samples, each of which includes a hand-drawn drawing of a corresponding 3D shape and a corresponding solid CAD feature representing the 3D shape. The dataset is thus configured to learn a neural network that can be used to convert the hand-drawn drawing into solid CAD features, employing the proposed learning method in this example. Furthermore, the dataset is labeled for intended learning. In fact, the potential inputs of the neural network (hand-drawn drawings) are each labeled in the dataset with the ground truth outputs (corresponding solid CAD features). Therefore, learning can be supervised and is therefore particularly effective.

[0089] The learning method includes providing a data set including hand drawings, each hand drawing representing a corresponding 3D shape, and learning a neural network based on the data set.

[0090] The learning method provides an improved solution for CAD design. Notably, the learned neural network can be used to convert hand drawings into solid CAD features. As already discussed, this is particularly useful in the context of CAD. Furthermore, utilizing a learned neural network to perform this conversion can achieve the benefits offered by the field of machine learning, such as fast or real-time execution, the absence of tedious prior manual programming / writing code, and accuracy of the results (e.g., accuracy here refers to the value of the 3D similarity assessed between the result of performing the solid CAD feature and the 3D shape desired by the user when drawing the hand drawing, for example, optionally measured as the value of the 2D similarity assessed between the 2D projection of the result of performing the solid CAD feature (e.g., the feature edge) and the hand drawing). Thus, the learning method improves ergonomics in the context of 3D design.

[0091] The learning may include supervised training based on at least a portion of the dataset. In this case, the at least a portion of the dataset includes, for each hand-drawn image, corresponding solid CAD features that are associated with the hand-drawn image. Figure 1 The at least one portion of the dataset may be provided according to any method, for example, formed according to a dataset forming method.

[0092] Alternatively or additionally, the learning may comprise unsupervised training based on at least another portion of the dataset. In this case, the hand-drawn images of the at least another portion of the dataset each form a corresponding (unlabeled) training sample. The at least another portion of the dataset may be provided according to any method, for example by searching a commercial database of hand-drawn images.

[0093] In particularly effective examples, the learning method may include performing supervised training based on a dataset formed by the dataset formation method. In reliable examples of such examples, the learning may also include performing unsupervised training based on another dataset, for example, after the supervised training. As is known in the field of machine learning, each training session may include iteratively processing the corresponding dataset, for example, in mini-batches, and modifying the weight values ​​of the neural network along the way. This may be performed using stochastic gradient descent. The weight values ​​may be initialized in any manner for each training session. In one example, the weight values ​​for unsupervised training may be initialized using the weight values ​​obtained at the end of supervised training. The weight values ​​for supervised training may be initialized in any manner, for example, randomly or to zero. Thus, the dataset formation method and supervised training form a method for initializing weight values ​​for further training, which may be unsupervised training, and which may be based on another dataset. The other dataset may, for example, be an unlabeled dataset that is more representative of reality than the labeled dataset formed by the dataset formation method. Therefore, unsupervised training can produce more accurate results than stopping training after supervised training. However, as is known from the field of machine learning, unsupervised training can be very long and provide inaccurate results if initialized arbitrarily. In the proposed method, supervised learning provides a relatively good initialization.

[0094] These methods are computer implemented.

[0095] This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method may be performed fully automatically or semi-automatically by the computer. In an example, the triggering of at least some steps of the method may be performed by user-computer interaction. The required level of user-computer interaction may depend on the foreseen level of automation and be balanced against the need to implement the user's wishes. In an example, the level may be user-defined and / or predefined.

[0096] This also means that the neural network, the data set, one or more (e.g., all) inputs, and / or one or more (e.g., all) outputs can be represented by corresponding data structures, respectively. One or more (e.g., all) data structures can be non-transitory recorded on corresponding data storage media, respectively.

[0097] A typical example of a computer-implemented method is to perform the method using a system suitable for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program including instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for such storage and may include several physically distinct parts (e.g., one for the program and one for the database). The system may be a CAD system.

[0098] Figure 1 An example of a GUI for any system herein is shown, where the system is a CAD system.

[0099] GUI 2100 can be a typical CAD-like interface, with standard menu bars 2110, 2120 and bottom and side toolbars 2140, 2150. As is known in the art, such menus and toolbars contain a collection of user-selectable icons, each associated with one or more operations or functions. Some of these icons are associated with software tools suitable for editing and / or manipulating the 3D modeled object 2000 displayed in GUI 2100. Software tools can be grouped into workbenches. Each workbench includes a subset of software tools. Specifically, one of the workbenches is an editing workbench, which is suitable for editing the geometric features of the modeled product 2000. In operation, a designer can, for example, pre-select a portion of the object 2000 and then initiate an operation (e.g., change size, color, etc.) or edit geometric constraints by selecting the appropriate icon. For example, a typical CAD operation is modeling the punching or folding of a 3D modeled object displayed on the screen. The GUI can, for example, display data 2500 related to the displayed product 2000. In the example of the figure, data 2500 displayed as a "feature tree" and its 3D representation 2000 relate to a brake assembly including a brake caliper and a disc. The GUI may also display various types of graphical tools 2130, 2070, 2080, such as for facilitating 3D orientation of an object, for triggering simulation of operations on the edited product, or for rendering various properties of the displayed product 2000. A cursor 2060 may be controlled by a haptic device to allow the user to interact with the graphical tools.

[0100] The 3D modeled object 2000 may be a graphical representation of a discrete geometric representation of a 3D shape, and the inference method may allow for the determination of any solid CAD features of the editable feature tree 2500. The inference method thus allows for editing the editable feature tree 2500 (e.g., adding solid CAD features) based on the 2D sketching functionality of the CAD system while simultaneously updating the discrete geometric representation 2000 of the 3D shape. This represents a high degree of ergonomics.

[0101] Figure 2An example of any system herein is shown where the system is a client computer system, such as a user's workstation.

[0102] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and random access memory (RAM) 1070, also connected to the bus. The client computer also has a graphics processing unit (GPU) 1110, which is associated with video RAM 1100, also connected to the bus. Video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices, such as hard disk drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated into specially designed ASICs (application-specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, keyboard, etc. A cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse having buttons for generating signals. Alternatively or additionally, the client computer system may include a sensor pad and / or a sensor screen.

[0103] Any computer program herein may include computer-executable instructions, including means for causing the aforementioned system to perform the method. The program may be recorded on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry or computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, for example, a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing the program of instructions to perform the functions of the method by operating on input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. If desired, the application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installer or updater. In any case, application of the program to the system results in the execution of the instructions of the method.

[0104] The examples given above of solid CAD features comprising corresponding (eg, closed) curve sequences and / or sweep operations (eg, defined by sweep curves) are now discussed further.

[0105] One or more (e.g. all) entity CAD features generated in the dataset formation method, one or more (e.g. all) (e.g. candidate) entity CAD features provided in the learning method and / or one or more (e.g. all) entity CAD features provided by the inference method can be consistent with these examples, respectively.

[0106] In these examples, the learning method can include providing a candidate set of sequence curves and / or a candidate set of scanning operations, such as a candidate set of scanning curves (e.g., a common candidate set of curves), and the learning is also based on the candidate set of curves and / or the candidate set of scanning operations. By "providing a candidate set", it is meant that the learning method, in particular the learning step, is able to consider each element of the candidate set as a candidate for speculation (at least at some times). As is known from the field of machine learning, the learning step can include freely setting the weight values ​​of the neural network so that the speculation method tends to perform speculation consistent with the data set fed to the learning method. And, "providing a candidate set of curves / scanning operations" means that the neural network architecture is predetermined in some way, and the learning step is performed in some way so that each element of the candidate set can be used to potentially speculate in the speculation process. In the example, each element of the candidate set is a candidate for each inferred solid CAD feature. In particular, each element of the candidate set of curves can be a candidate for each inferred curve of the curve sequence.

[0107] The candidate set of curves and / or the candidate set of sweep operations can each include at least one continuous subset of curves (respectively, sweep operations). "Continuous" means that each element in the subset can be obtained by a specific continuous deformation of another element of the subset (for example, parameterized with a fixed number of parameters), so that each intermediate state of the continuous deformation is itself a candidate element. In the example, each candidate set includes several continuous subsets (for example, one for each curve type). This improves the accuracy of learning. In fact, since at least one continuum of each element is accessible to become an element of the inferred solid CAD feature, inference can be performed in a good way in such a continuum.

[0108] Now discuss the example of curves in a curve sequence (eg, planar 2D curves, ie, sequence curves) and / or curves defining a scanning operation (eg, scanning curves). For each sequence curve and / or scanning curve, the data structure involved can optionally be the same.

[0109] Given a discrete set of curve types, and for each curve type, given a corresponding discrete set of one or more parameter domains, each discrete set having a corresponding (e.g., continuous) parameter, a curve can be formed from the corresponding curve type given that each parameter domain has a corresponding parameter value for the corresponding continuous parameter, each of the corresponding discrete set of one or more parameter domains having a corresponding parameter value.

[0110] Thus, the dataset formation method and / or the learning method can be based on such a discrete set of curve types, and for each curve type, a corresponding discrete set of one or more parameter domains. These sets can be the same or different for the dataset formation method and the learning method. When the sets are the same, the machine learning process is particularly accurate because the curves of the training examples are consistent with the curves used as inference candidates.

[0111] Now, the candidate curve set in the learning method may comprise a set product between a discrete set of curve types and a corresponding discrete set of one or more parameter domains for each curve type. Each element of the set product represents a corresponding candidate sequence / scan curve.

[0112] In the case of a dataset formation method, each sequence / scan curve of a training sample may belong to such a set product. In an example, the dataset formation method may include providing initial data, the initial data including a discrete set of curve types (data representing the same), and for each curve type, a corresponding discrete set of one or more parameter domains (data representing the same). In such an example, sequence curves and / or scan curves may be generated (e.g., synthesized) based on such initial data.

[0113] A "curve type" refers to a class of curves (e.g., a sequence curve is a 2D plane, a sweep curve is a 3D curve), where each curve exhibits a common general form (e.g., recognizable to the user). A curve type is associated with parameters. Each parameter takes values ​​in a set called a "parameter domain" (which can be an interval, i.e., a continuous range of real numbers), and providing specific parameter values ​​for each parameter domain defines an instance of the curve type. A curve type can be defined by a corresponding set of one or more parameters and a predetermined algorithm that is configured to generate such curve instances of that curve type when fed with the corresponding set of parameter values. The instances form a sequence curve or a sweep curve that can intervene in one or more solid CAD features.

[0114] Each curve type can be associated with one or more continuous parameters. A parameter's "continuity" means that the function of the output curve is at least piecewise continuous with respect to the parameter. Alternatively, the parameter domain is said to be at least piecewise continuous. This allows for compact representation of a wide variety of shapes.

[0115] In an example, for each of one or more curve types, the one or more corresponding continuous parameters include one or more shape parameters as described above, for example, including a positioning parameter for the curve start point and a positioning parameter for the curve end point, and optionally including one or more additional parameters depending on the curve type (for example, including one or more intermediate point positioning parameters and / or one or more tangent vectors, each constraining the curve at a corresponding (e.g., boundary) point of the curve). The positioning parameters can be the same for all curve types, such as (x, y, z) coordinates defined relative to a (0, X, Y, Z) reference.

[0116] In an example that is particularly efficient in terms of computational time and / or convergence for learning, compared to being representative of the diversity of real-world situations, the discrete set of curve types may include line, arc, and / or cubic Hermitian interpolation segment types, and optionally include line, arc, and cubic Hermitian interpolation segment types. The discrete set of curve types may include any other curve types.

[0117] Any curve provided herein (e.g., in a memory) can be recorded (e.g., in a provided data segment, training sample, and / or inferred output) as a data structure, the data structure including: data representing the type of the curve (e.g., text information), and, for each parameter, corresponding data representing the parameter value (e.g., a value, such as a floating point number). Any curve provided herein can be post-processed, for example, represented, rendered, and / or displayed in a vectorized manner (e.g., by dividing the curve into sub-curves and / or approximating the curve using one or more Bézier curves and / or spline curves, for example, for corresponding portions of the curve, for example, in svg format for 2D curves).

[0118] Examples of dataset formation methods are now discussed.

[0119] In an example, providing the corresponding training sample may include synthesizing the data segment. "Synthesizing" the solid CAD feature and / or hand-drawn image (e.g., the training sample) refers to an automated process that includes generating the solid CAD feature and / or determining at least one corresponding hand-drawn image (e.g., representing the 3D image represented by the solid CAD feature).

[0120] In an example, a data set formation method may include providing initial data comprising a discrete set of curve types and, for each curve type, a corresponding discrete set of one or more parameter fields. This means that for each curve type, the initial data comprises a corresponding data structure corresponding to an object class (referring to object-oriented programming), the data structure having fields corresponding to the parameters of the curve type. In this case, a solid CAD feature may be synthesized based on the initial data. For example, generating at least one corresponding solid CAD feature may include synthesizing the solid CAD feature based on the initial data. Synthesis may include: generating an integer representing the number of curves in a sequence; and generating, based on the initial data, a number of sequence curves corresponding to the integer, for example, the generation being constrained by requirements of the solid CAD feature definition, such as each curve being planar and / or on the same plane, the curves being sequential (i.e., the end point of one curve is the starting point of the next curve), and / or the sequence being closed (i.e., the sequence is cyclic). Generation may also include generating a scan curve based on the initial data.

[0121] In particular, the generation of each sequence / scan curve may include selecting a curve type and selecting parameter values ​​for each parameter of the curve type. This corresponds to creating an instance of a given class.

[0122] Any selection performed when synthesizing solid CAD features can be performed via (e.g., random) sampling. This allows for the generation of data that is relatively well representative of the diversity of real-world situations. Consequently, this leads to ultimately accurate learning. Any (e.g., random) sampling can include uniform (e.g., random) sampling within a set of values ​​of the variable to be selected (e.g., random uniform sampling is random sampling according to a uniform probability distribution).

[0123] For example:

[0124] - The lengths of the curve sequences may be sampled according to a uniform distribution over the set of all integers greater than N1 and less than N2, where N1 is a "minimum depth" integer greater than 0 and / or less than 5, for example equal to 1; and N2 is a "maximum depth" integer greater than N1 and / or less than 100, 50 or 20, for example equal to 10.

[0125] - Sequential / scanning curve types can be sampled according to a uniform distribution over a discrete set of curve types.

[0126] - The parameter values ​​for each selected curve type can be sampled in any manner, for example by providing a parameter domain directly or indirectly (for example in the initial data), each parameter domain having one or more intervals (for example, each interval having an upper limit (for example floating point) and a lower limit (for example floating point)), for example to comply with the requirements of a sequential curve.

[0127] The synthesis may also include determining a solution for each hand-drawn image to determine one or more plausible hand-drawn images.

[0128] The scheme includes providing viewpoints (i.e., 3D locations in 3D space defining solid CAD features) from which corresponding 3D shapes (represented by the synthesized solid CAD features) are visible, and rendering one or more edges in the image based on these viewpoints. The scheme may optionally include providing a perspective, or the perspective may be predetermined (e.g., a default conical perspective).

[0129] The viewpoints may be provided in any manner, such as sampling (eg, randomly and / or uniformly) among a set of candidate viewpoints (eg, the set of all points surrounding a sphere of a 3D shape).

[0130] The edges define the locations of reasonable strokes for a freehand drawing of a 3D shape. For example, each edge can be a silhouette edge or a visible edge of the solid representation, as viewed from a viewpoint, relative to a solid representation of a corresponding 3D shape based on a solid CAD feature (i.e., corresponding to an executing solid CAD feature). The silhouette and / or visible edge can be determined in any manner based on the viewpoint.

[0131] In an example, a visible edge may be considered to be one or more (e.g., all) edges that are accessible from the viewpoint (not occluded by the 3D shape from the viewpoint), lie on the 3D shape (thus potentially including non-planar edges), and that are aligned with the C shape having a solid representation on the solid representation. k k is an integer greater than or equal to 1. k may optionally be less than 10 or 4. In an example, the scheme may include providing or based on the maximum discontinuity value k max (For example, k max =2), and calculate all discontinuities C that form the entity representation k The edge of the discontinuity, where k is greater than or equal to 1 and less than or equal to k max This allows extracting visible tangent discontinuities (i.e. sharp edges) and / or visible curvature discontinuities that are naturally represented by hand-drawn sketches. This therefore allows populating the dataset with real data.

[0132] In this example, determining each hand-drawn sketch also includes perturbing at least a portion of one or more rendered edges. In other words, after rendering from a viewpoint, the rendered edges can be modified. This allows for the reproduction of irregularities in hand-drawn sketches, whereas rendering from solid CAD features would produce overly regular geometry. Consequently, this results in more accurate and reliable learning.

[0133] As previously mentioned, rendering of each edge can be performed in a vectorized manner. In other words, each edge can be represented by one or more corresponding parametric curves, for example, with fewer than 50, 20, or 10 control points. These Bézier curves are represented in a 2D image using SVG format. Other formats may also be used. In this case, perturbation of at least one corresponding parametric curve can include adding noise to one or more parameters of the at least one corresponding parametric curve. For example, the noise can shift the control points. This makes learning particularly robust.

[0134] Now let's discuss examples of learning methods.

[0135] In an example, the neural network may include recurrent neural network (RNN) units (i.e., units of the same RNN). Such an example allows learning an accurate neural network that exhibits a sequential structure (via the RNN) corresponding to sequential features of a sequence of geometric shapes (e.g., curves) of a solid CAD feature to be inferred, thereby performing accurate inference.

[0136] Specifically, each RNN unit can output corresponding data for inferring the corresponding geometric shape (e.g., a sequence curve) of the solid CAD feature at a corresponding time step. Therefore, the output data allows the corresponding curve to be inferred. The length of the curve sequence can be inferred in any predetermined manner (e.g., the inferred curve sequence can be limited to a curve sequence with a given length). Alternatively, each RNN unit can also output corresponding data at a corresponding time step to infer the end or continuation decision of the curve sequence (i.e., the end flag).

[0137] We now discuss particularly effective examples of RNN outputs.

[0138] The corresponding data for inferring the corresponding curve of the solid CAD feature may include: corresponding first data for inferring the corresponding curve type, and corresponding second data for inferring the corresponding parameter value for each parameter domain in the corresponding discrete set of one or more parameter domains.

[0139] In an example, the corresponding first data may include a corresponding discrete probability distribution, each probability being attributed to a corresponding one of a discrete set of curve types. In other words, each RNN unit outputs a probability distribution over curve types, so that a possible curve type can be inferred for the sequence curve corresponding to the corresponding time step.

[0140] Alternatively or additionally, in an example, the corresponding second data may include (for each curve type) corresponding parameter values ​​for each parameter domain in a corresponding discrete set of one or more parameter domains. In other words, the RNN unit directly outputs the inferred parameter values ​​of the curve type. This simplifies learning and improves convergence and accuracy, especially when the parameters are continuous and it is difficult to learn a neural network with an output density distribution.

[0141] Similarly, in an example, the neural network may include a standard neural network (NN) component that outputs data for inferring a scan operation of a solid CAD feature. In an example, the data for inferring a scan operation of the solid CAD feature may include: first data for inferring a type of scan curve, and second data for inferring a parameter value for each parameter domain in a corresponding discrete set of one or more parameter domains.

[0142] In an example, the corresponding first data may include a corresponding discrete probability distribution, each probability being attributed to a corresponding one of a discrete set of curve types. In other words, the NN component outputs a probability distribution of curve types so that a possible curve type can be inferred for the scan curve.

[0143] Alternatively or additionally, in an example, the corresponding second data may include (for each curve type) corresponding parameter values ​​for each parameter domain in a corresponding discrete set of one or more parameter domains. In other words, the NN component directly outputs the inferred parameter values ​​for the curve type. This simplifies learning and improves convergence and accuracy, especially when the parameters are continuous and it is difficult to learn a neural network with an output density distribution.

[0144] We now discuss the architecture of the neural network that leads to particularly accurate results.

[0145] In one embodiment, the neural network may include an initial encoder that takes the hand-drawn image (which is a 2D image) as input. The encoder may be a convolutional neural network (CNN). The CNN may be a 2D CNN that takes the 2D image data as input.

[0146] We now discuss example features of the loss for supervised training that lead to accurate results.

[0147] In some examples, the loss can be invariant to cyclic permutations of the sequence and / or to reference-based sketches. In other words, for a candidate solid CAD feature used for inference, the loss can be invariant to cyclic permutations of its curve sequence, and / or the loss can be invariant to replacing a closed 2D planar sketch with its image at the end of a scanning operation and reversing the scanning operation. This loss handles cases where hand-drawn drawings of different solid CAD features can potentially and correctly represent different solid CAD features if the solid CAD features produce the same 3D shape. Learning accuracy is improved by focusing learning on important parameters, thereby preventing the learning method from having to learn unnecessary information.

[0148] The loss can penalize one or more quantities, for example, each quantity is penalized via a corresponding loss term. For each time step of the sequence curve and the step size relative to the scan curve (each hand drawing is associated with a corresponding ground truth solid CAD feature), the penalty quantity can include any one of the following quantities or any combination thereof:

[0149] - the probability of the corresponding data of the respective curve type being attributed to the corresponding ground truth curve is low,

[0150] - the difference between one or more corresponding parameter values ​​of the ground truth curve and one or more corresponding parameter values ​​of its corresponding respective data, and / or

[0151] - (for RNN cell timesteps only) attributed to the corresponding ground truth ending having a lower probability relative to the corresponding probability of the persistent state (output by the RNN cell).

[0152] In other words, supervised training can act on the weights of the neural network by minimizing such a loss, thereby tending to make the corresponding probability of each RNN unit and NN part output for the corresponding ground truth curve type close to 1 (i.e., not low), and the corresponding probability of output for other curve types close to 0 (i.e., low), and / or make the various continuous parameter values ​​output by each RNN unit and the component close to their ground truth values. Such a loss accurately handles the discrete / continuous mixture of problems. Moreover, the loss penalizes the difference between the length of the inferred curve sequence and its ground truth value.

[0153] An example of unsupervised training leading to accurate results is now discussed, where the respective first data comprises a discrete distribution of respective probabilities, each being attributed to a respective one of a discrete set of curve types.

[0154] It is well known that unsupervised training can minimize the loss. Minimization includes exploring the distribution of discrete probabilities of the candidates. In other words, minimization iteratively modifies the neural network weights to explore the discrete probability distributions of the outputs and make them consistent with the unlabeled data set (where each hand-drawn drawing is not associated with any ground solid CAD feature). In a sense, the exploration of the discrete probability distributions of the candidates is indirect and is the result of directly exploring the weights. In the selection, minimization also includes: exploring the discrete probability distributions of the candidates for the end mark. In the example, minimization also includes exploring the corresponding parameter values ​​of the candidates for each parameter domain in each discrete set of one or more parameter domains.

[0155] Now, for each hand-drawn image in the dataset and for each candidate, the loss penalizes the difference from the corresponding hand-drawn image. The corresponding hand-drawn image is derived from the corresponding solid CAD features, which themselves are inferred based on the discrete probability distributions of the explored candidates. In other words, given the weight values ​​of the neural network (at any iteration of unsupervised training), the hand-drawn image from the unlabeled dataset can be input into the neural network to perform inference on the solid CAD features. A hand-drawn image can be derived based on each inferred solid CAD feature itself (by building a tree and determining the resulting hand-drawn image). The resulting hand-drawn image can be compared with the original hand-drawn image of the corresponding dataset.

[0156] In an example, unsupervised training may include a sampling step for at least one (e.g., each) hand-drawn image. Training may include providing mini-batches, and sampling may include, for each mini-batch, sampling solid CAD features for each hand-drawn image in the mini-batch by following data output by the neural network (including probability distributions and continuous parameter values ​​output by the RNN). Sampling may then compute a loss that compares the input hand-drawn image (or a representation thereof, such as a rendering) with the hand-drawn image corresponding to the sampled solid CAD features (i.e., penalizes the difference). Because the sampling is random and discrete, the loss is non-differentiable with respect to the sampling. Therefore, since there is no true gradient, classical backpropagation with respect to the true gradient of the variables cannot be performed.

[0157] The proposed unsupervised training example proposes using an augmentation method to provide pseudo gradients of the loss with respect to variables representing candidate discrete distributions of the corresponding first data, and optionally provide pseudo gradients of the loss with respect to variables representing candidate discrete distributions on geometric operations, and / or pseudo gradients of the loss with respect to variables representing end flag distributions. Minimization can then include backpropagation (e.g., stochastic descent) of such pseudo gradients of the loss to learn the neural network. Since the pseudo gradients are provided by the augmentation method, the learning is accurate even if the backpropagation is not one of the true gradients. Minimization can also include backpropagation of the (true) gradients of the loss with respect to the respective parameter values ​​for each parameter domain in a discrete set of one or more parameter domains (because the loss is differentiable with respect to these variables). Therefore, the augmentation method solves the differentiability problem caused by the introduction of discrete probability distributions, and only solves this problem.

[0158] We now discuss an example machine learning process, including examples of the methodology.

[0159] Recurrent Neural Networks

[0160] The example machine learning process is based on a recurrent neural network (RNN). A recurrent neural network is a deep neural network with an internal memory state. Therefore, unlike feedforward neural networks, recurrent neural networks can exhibit temporal behavior. They implicitly consider past entries to predict the next result. They can be written as Where y is the output, x is the input, and h is the internal hidden state, also known as the memory. w is the weight of the RNN. The memory h is not a hard memory that stacks all past inputs, but a fixed-size vector that can be obtained by Learning is performed so that the RNN learns to retain only useful information from past steps.

[0161] Goals of the example machine learning process

[0162] Given a hand-drawn drawing of a solid, the example machine learning process aims to obtain reasonable solid CAD features representing the hand-drawn drawing. The example machine learning process focuses on the solid CAD features defined for 3D space by (and only by) the following:

[0163] - A closed 2D planar sketch, consisting of a series of planar curves that form a closed profile,

[0164] -Then comes the sweep operation, which generates a solid by sweeping a closed surface defined by a 2D planar sketch in 3D space, i.e. the solid is defined by the volume passed by the closed surface.

[0165] Principles of the example machine learning process

[0166] The example machine learning process takes a hand-drawn rendering of a solid as input and generates solid CAD features through an end-to-end deep learning model.

[0167] 1. First, a compact latent representation of the drawing is generated through a convolutional neural network (CNN).

[0168] 2. Then, solid CAD features are inferred from this latent representation in two steps:

[0169] a. First, a closed 2D planar sketch is inferred using a recurrent neural network (RNN). The sketch is defined by a sequence of curves. Each RNN time step corresponds to a curve in the sequence. At each time step, the corresponding RNN unit estimates the curve type (line, spline, arc), its parameters, and the end / continue flag (whether it is the last element in the sequence).

[0170] b. Second, a neural network (NN) predicts the scan.

[0171] When ground truth solid CAD features are available, the entire model (CNN encoder, RNN, NN) is trained end-to-end in a supervised manner. Otherwise, for unsupervised training, rendered images of the inferred geometry are generated and the entire model is optimized to fit the rendered images to the input hand drawing.

[0172] Implementation options for the example machine learning process

[0173] The example machine learning process can be generalized to any type of 2D planar sketch consisting of a series of parametric curve and parametric sweep operations.

[0174] However, this discussion only considers samples indexed by i, where the 2D planar sketch consists of a series of length T i The parameterized curve (c x (u), With parameter x), each curve consists of Separately, the example machine learning process only considers scanning operations, including scanning along the curve (c x (u), Sweep a 2D planar sketch with parameter x). For both 2D planar sketch and sweep operations, the example machine learning process considers the following curve types:

[0175] Line: with parameters x = (p start ,p end ) of the straight line segment, where p start and p end are the two boundary points of the segment (i.e. the starting point and the end point):

[0176]

[0177] Arc: with parameter x=(p start ,p inter ,p end ) that passes through p start , p inter and p end .p start and p end are the two boundary points of the segment, and p inter is the middle point. For completeness, the appendix provides arc formulas from three points.

[0178] · Cubic Hermitian interpolation fragments The fragments defined by the cubic Hermitian interpolation have parameters x = (p start ,t start ,p end ,t end ), where p start and p end are the two boundary points of the segment, and t start and t end Their tangents are:

[0179]

[0180] Different reasons for choosing these curve types include:

[0181] - The parameters of these curves are meaningful to any user. Therefore, the obtained solid CAD features can be easily edited. In particular, cubic Hermite interpolation is widely used in design (such as Catmull-Rom splines).

[0182] - They can represent a wide variety of shapes. Moreover, if a shape cannot be represented by a sequence of these curves, such a sequence can easily approximate the shape (each curve in the sequence locally approximates a region of the shape).

[0183] - This representation also allows easy implementation of the most common scanning operations: stretching and rotation.

[0184] Pipeline options for the example machine learning process

[0185] The main steps of the example machine learning process pipeline are as follows:

[0186] 1. Generate synthetic labeled dataset (if needed)

[0187] 2. Train the model end-to-end (supervised, using labeled data)

[0188] 3. Train the model end-to-end (unsupervised, using unlabeled data)

[0189] 4. Speculation

[0190] The example machine learning process considers two different ways to train the model. When labeled data is available (or synthetically generated in step 1), the model can be trained in a supervised manner. In an example of this case, the model can be trained in a supervised manner only and / or step 3 can be skipped. In an alternative example of this case, the model can be trained in an unsupervised manner after pre-training with synthetic data. As an alternative to this case, the model can be trained in an unsupervised manner only.

[0191] Description of an example machine learning process

[0192] 1. Generate synthetic labeled dataset (if needed)

[0193] The example machine learning process considers a labeled dataset as a collection of hand-drawn drawings of solids associated with their CAD features, from which the actual geometry of the solids can be generated. Such datasets are rarely available. If they are missing, they can be generated synthetically using a random generator.

[0194] a. Solid CAD feature generation

[0195] First, solid CAD features are sampled from a given distribution. This distribution can be designed by the user with any future application in mind. The example machine learning process will be specifically designed to learn to retrieve solid CAD features from this distribution.

[0196] The following now describes an example of a general distribution that processes a 2D planar sketch consisting of a sequence of curves of length at most N = 10 (N can take any strictly positive integer value), and a sweep operation that involves sweeping the 2D planar sketch along the curves. Each sampled solid CAD feature is indexed by i.

[0197] Sample a 2D planar sketch as follows:

[0198] ·exist Middle pair sequence T i The length of is uniformly sampled.

[0199] · Two real-valued α t and r t are uniformly sampled in [0,2π[ and]0,1] respectively, and the curve type is chosen uniformly among the possible curve types. Then, α is adjusted by increasing the value of t Sort by value Figure 3 An example of such random sampling is shown, where T i =3, expressed in one yuan.

[0200] For each Use convention T i +1=0, for parameters Type Parameters are selected as follows:

[0201] Regardless of the curve type How to set the parameter p start and p end So that p start (respectively, p end ) is of length r t (respectively, r t+1 ) vector with a relative t (Respectively, α t+1 ) radian axis , and lies on the "ground plane" (through the origin and perpendicular to ).

[0202] ο in (arc) case, in the fragment Center point μ inter Perform uniform sampling. From Gaussian distribution For the additional parameter p inter Sampling is performed, where Σ is a diagonal matrix whose non-zero elements are equal to ‖p end -p start ‖2, p inter The z coordinate of is set to 0 so that it is also at the "ground plane".

[0203] ο in In the case of the fragment For two points μ start and μ end Perform uniform sampling. From Gaussian distribution and Point p start ′ and p end ′ is sampled, where Σ is a diagonal matrix whose non-zero elements are equal to ‖p end -p start ‖2. start ′ and p end ' is set to 0 so that it is also in the "ground plane". start and t end Set to t start =p start ′-pstart , and t end =p end ′-p end ..

[0204] If the instantiated curve intersects itself or one of the previously instantiated curves, the process is repeated until there is no longer any intersection. If after many trials (e.g., 10), there are still some intersections, the entire sample will be regenerated from the beginning.

[0205] Figure 4 Shown Figure 3 An example where and

[0206] The scan operation is sampled as follows:

[0207] • Select the curve type s uniformly from the possible curve types.

[0208] ·For parameters Types i Parameters are selected as follows:

[0209] Regardless of the curve type i How to set the parameter p start Set to the origin. In [-0.5,0.5]×[-0.5,0.5]×[0.1,1.0] end The parameters are uniformly sampled.

[0210] οDepends on curve type s i , the remaining parameters are sampled the same way as curves in 2D planar sketches, but are not forced to lie in the "ground plane".

[0211] Figure 5 Shown with Figure 4 The same example, where s i =Cubic Hermitian interpolation fragment.

[0212] This general distribution can be specialized to generate specific types of solid CAD features. The following provides some restrictions that can be applied to generate different kinds of solids:

[0213] Right prism

[0214] οThe curve type is limited to Line curve type ( and s i =Line type).

[0215] ο in in rather than in Middle pair sequence T i The length of is uniformly sampled.

[0216] The boundary points of the linear segment are regularly distributed in a one-dimensional circle ( We set r t =1 and ).

[0217] ο Set the scan line to be vertical (p of the scan curve in {0}×{0}×[0.1,1.0] end parameters are uniformly sampled).

[0218] Cylinder

[0219] ο 2D planar sketches consist of arcs that form a circle. Limit the curve type to 2D planar sketches ( ) and choose its parameters so that its control points (defined by the arc parameters) lie in this one-dimensional circle.

[0220] Limit the type of scanning operation curve to linear curve type (s i =Line type).

[0221] ο Set the scan line to be vertical (p of the scan curve in {0}×{0}×[0.1,1.0] end parameters are uniformly sampled).

[0222] Solid CAD Feature SCF i The example machine learning process can be completely defined by different curve types and their parameters:

[0223] To make all solid CAD features of similar size (which makes the learning process easier), they are sized so that the largest side of their bounding box is equal to 1 and centered (the center of the bounding box is at the origin).

[0224] b. Training sample generation

[0225] Then, for each solid CAD feature, one or more viewpoints are uniformly sampled in a sphere centered at the origin. The sphere radius can be selected so that the entire solid CAD feature is visible in an image rendered from that position and looking toward the origin.

[0226] We define an edge as the C of entities generated k Discontinuities. For each sample, we randomly choose k = 1 (tangent discontinuities, sharp edges) or k = 2 (curvature discontinuities, such as the edge between a planar and cylindrical surface).

[0227] Then, the outlines and visible edges of the solid CAD features can be obtained from the sampled and viewed image I i Point-of-view rendering of the origin in .

[0228] Finally, by adding the rendered image I i Its corresponding solid CAD feature SCF i The association is performed to generate the training sample i, whose parameters have been previously expressed in the viewpoint coordinate system.

[0229] Figure 6 Shows the use of C 1 Rendered image of a solid CAD feature with discontinuities as edges. Figure 7 Shows the use of C 2 Rendered image of the same solid CAD feature with discontinuities as edges. Figure 8 Shown Figure 5 Rendered image of a solid CAD feature represented in . In this case, each C 2 Discontinuity is also C 1 Discontinuity, i.e., using C 1 or C 2 Rendering discontinuities as edges will provide the same results.

[0230] c. Sample perturbation

[0231] Finally, the edges and outlines of the rendering can be disturbed to simulate hand-painted brushstrokes.

[0232] The example machine learning process follows the following optional strategies:

[0233] • Contours and edges are previously rendered in a vector manner, such as in svg format (ie, each rendered segment corresponds to a parameterized curve).

[0234] Divide each curve into an arbitrary number of curves (e.g. Bezier curves).

[0235] Add Gaussian noise to each parameter of the curve.

[0236] Figure 9 Shown Figure 8 Rendered perturbation image shown. Figure 10 Another example of a solid CAD feature rendering is shown (representing a regular prism). Figure 11 Shown Figure 10 Rendered perturbed image of .

[0237] 2. Train the model end-to-end (supervised, using labeled data)

[0238] To train the model, the example machine learning process considers a dataset of size N. Each sample of the dataset indexed by i consists of a hand-drawn image 2D basic sketch curve sequence, scanning operation and its associated parameters Assume that the length of the i-th 2D basic sketch curve sequence is Write each curve sequence as a sequence of curve types For each curve Let its parameters be

[0239] Model Definition

[0240] The model takes as input a single-channel image representing a hand drawing (synthetic or non-synthetic) CNN encodes images into feature vectors Where w is the weight of the model.

[0241] RNN estimation 2D basis sketch. Each RNN unit outputs a discrete distribution p over the possible types of curves at each time step t. t , and the Bernoulli distribution q on the end / continue flag indicating whether the sketch is finished t Each unit also generates a time step t for each curve type c and state vector h t Output predicted continuous parameters

[0242] Each RNN unit takes the following elements as input:

[0243] Eigenvector

[0244] The state vector h output by the previous unit t-1 , where h0=O.

[0245] Optionally, from the estimated distribution predicted by the previous RNN unit Sample one or several curve types and / or their estimated parameters

[0246] More specifically, each RNN unit formula can be written as:

[0247]

[0248]

[0249] We remember Input is optional.

[0250] The standard NN then outputs the possible scan operation types and the continuous parameters y for each scan operation type s s The discrete distribution r over . It takes the following as input:

[0251] Eigenvector

[0252] The state vector output by the last unit

[0253] Optionally, from the estimated distribution predicted by the previous RNN unit Sample one or several curve types and / or their estimated parameters

[0254] Figure 12 An example pipeline of a model presented in an example machine learning process is shown. As described above, the RNN unit and / or NN can take the different inputs described above.

[0255] Definition of loss

[0256] For simplicity, we will from now on take the estimated distribution and parameters as input images , although they may depend on other variables as explained above.

[0257] We define the loss, its penalty curve, and the prediction error for the sweep operation type:

[0258]

[0259] Similarly, we define the following loss, which enforces the curve sequence to have the correct length:

[0260]

[0261] We also define the following loss, which penalizes the squared error on the prediction curve and the parameters of the sweep operation:

[0262]

[0263] Finally, we minimize the loss as follows to train the weights of the network (using standard deep learning techniques such as the ADAM solver on mini-batches):

[0264] L sup (w) = L type (w)+L length (w)+λL parameters (w) (Formula 4)

[0265] where λ is a weighting parameter, which can be initialized as:

[0266] Here, w0 is the randomly initialized weight of the neural network.

[0267] Any low-degree penalty function other than the logarithm (cross entropy term) can be used for the term L type (w) and L length Any of (w), and / or any distance d other than the L2 distance and / or any increasing and / or positive incr function other than the square function may be used for L parameters (w).

[0268] Loss modification: making it invariant to sequence cyclic permutations

[0269] Two curve sequences are equivalent (generate the same 2D basis sketch) if one curve sequence is a cyclic permutation of another, i.e., the same sequence but starting at different points. As a result, during training, the model of the example machine learning process may predict a curve sequence distribution that generates an accurate 2D basis sketch, but incur a high loss because the prediction is a cyclic permutation of the ground truth. To avoid this problem, we change the formula and The loss in is modified to be invariant to loop permutations:

[0270]

[0271]

[0272] Loss correction: make it invariant to reference-based sketches

[0273] Equivalently, two solid CAD features can be generated on the same geometry. In effect, a second 2D plane will be formed at the end of the scan operation. As a result, the solid CAD feature defined by the 2D plane sketch of this second surface and a scan operation performed after the original scan operation (but in the opposite direction) will produce the same solid. Given a defined solid CAD feature, we indicate its equivalent solid CAD feature We modify the loss defined by formula (Equation 6) to be invariant to the feature representation of a single entity:

[0274]

[0275] 3. Train the model end-to-end (unsupervised, using unlabeled data)

[0276] The model can be fine-tuned or trained from scratch using real data (i.e. skipping step 2).

[0277] The model architecture remains almost unchanged (see step 2).

[0278] In each iteration, solid CAD features are sampled from the estimated distribution by input drawing The model is adjusted and instantiated with the parameters predicted by the model. Some of the predicted parameters are modified to ensure a closed 2D base sketch (i.e. the end point of a curve in the sequence must coincide with the start point of the next curve). In the example machine learning process, we force equal In the special case of the start point of the first curve of the sequence and the end point of the last curve, in which case we force equal

[0279] We denote T i is the sampled sketch sequence (minimum t, where the end mark is from q t Sampling) length, is the sampling curve, and s i This is a sampling scan operation.

[0280] With a fixed arbitrary camera pose in image I i The visible edges of the geometry obtained by rendering in . The rendering process can be seen as the process of connecting the solid CAD feature with the generated rendered view I i The mapping between . Compute the differential of this mapping using finite differences.

[0281] Instead of using the loss defined in step 2, we define a loss for comparing the input drawing to the obtained rendered image. An example of such a loss could be:

[0282]

[0283] Where d is I i and For example, we can use I i The rendered edges in The Chamfer distance of non-background pixels (or vice versa).

[0284] Minimizing this loss allows us to optimize the network's weights, except for the discrete distributions p, q, and r, which are only used for sampling. The example machine learning process can backpropagate the gradients of the loss with respect to these distributions. The example machine learning process can use a boosting algorithm to obtain these gradients for the discrete distributions p, q, and r. The boosting algorithm is limited to these discrete distributions, so the unsupervised learning phase cannot obtain the true gradients for other variables. This limits the approximation, which in turn affects accuracy.

[0285] The enhancement algorithm can be the enhancement algorithm described in the following paper:

[0286] - R.J. Williams. A simple statistical gradient following algorithm for connectionist reinforcement learning. Machine Learning, 229–256, 1992.

[0287] This provides the following gradient formula:

[0288]

[0289]

[0290]

[0291] where ρ is an additional sparsity bonus to encourage sketch sequences to be maximally expressive at minimal size, and b is a baseline for reducing the variance of the estimated gradients and is computed as a moving average of past losses.

[0292] 4. Speculation

[0293] Once the model has been learned, inference is straightforward. The user can create a new freehand sketch and feed it to the model. The most likely (according to the inferred distribution) solid CAD features can then be generated. Otherwise, several plausible solutions can be computed by sampling from the inferred distribution. Some of the predicted parameters are modified to ensure a closed 2D base sketch (i.e., the end point of a curve in a sequence must coincide with the start point of the next curve). In one implementation, we enforce equal In the special case of the start point of the first curve of the sequence and the end point of the last curve, in which case the procedure can force equal

[0294] The example machine learning process discussed can allow for the proposal of several plausible solid CAD features, or geometric shapes resulting from a scanned 2D sketch (CAD features can be hidden). The predictions of the example machine learning process are robust to noise.

[0295] Appendix: Arc Equations from 3 Points

[0296] Let p start 、p inter and p end are three points in space. We define:

[0297] o plane Π passes through point p start 、p inter and p end ,

[0298] ο Application f Π : Convert the 3D points belonging to π to 2D points in the local 2D coordinate system of plane π,

[0299] οApplication Convert a 2D point in the local coordinate system of plane π to the corresponding 3D point,

[0300] o

[0301] o and

[0302] o

[0303] Through p start 、p inter and p end The arc segments are then defined by:

[0304]

[0305] in,

[0306] and

[0307]

[0308]

[0309] Finally, we note that θ start ,θ inter and θ end They are vectors and Relative to the axis Angle:

[0310] If θ start <θ inter <θ end Then α=θ start And β=θ end

[0311] If θ start <θ end <θ inter or θ inter <θ star <θ end

[0312] Then α=θ end And β=θ start +2π

[0313] If θinter <θ end <θ star or θ end <θ star <θ inter

[0314] Then α=θ start And β=θ end +2π

[0315] If θ end <θ inter <θ start Then α=θ end And β=θ start

Claims

1. A computer-implemented method for learning a neural network, the method comprising: - providing a data set comprising hand-drawn drawings each representing a corresponding 3D shape; as well as - learning the neural network based on the dataset; - wherein the neural network is configured to infer a physical CAD feature representing the 3D shape from an input hand-drawn image representing the 3D shape, the physical CAD feature comprising a corresponding curve sequence; -Wherein, the input hand drawing is an unparameterized 2D image data structure, including one or more brush strokes representing a given 3D shape viewed from a corresponding viewpoint, wherein the data structure includes data segments, each data segment represents a corresponding part of the brush stroke, and the data structure represents a raw aggregation of these parts, and the neural network is configured to take as input a hand drawing obtained by the user by activating a series of pixel drawing strokes, and the hand drawing is directly input into the neural network without transformation.

2. The method according to claim 1, wherein The method further includes providing a candidate curve set, and the learning is further based on the candidate curve set.

3. The method according to claim 2, wherein: The neural network includes recurrent neural network (RNN) units, and each RNN unit outputs corresponding data for inferring a corresponding curve of the solid CAD feature at a corresponding time step.

4. The method according to claim 3, wherein: - The candidate curve set includes the set product between: oA discrete set of curve types, and o for each curve type, a respective discrete set of one or more parameter domains for each respective parameter, each parameter domain having a respective parameter value for the respective parameter, Each curve type forms a respective element of the candidate curve set using respective parameter values ​​for each parameter domain in a respective discrete set of the one or more parameter domains; as well as -The corresponding data for inferring the corresponding curve of the solid CAD feature includes: corresponding first data for inferring the corresponding curve type, and corresponding second data for inferring the corresponding parameter value for each parameter domain in the corresponding discrete set of the one or more parameter domains.

5. The method according to claim 3 or 4, wherein: Each RNN unit having one or more previous RNN units takes as input information information inferable from corresponding data output by the one or more previous RNN units.

6. The method according to any one of claims 1 to 4, wherein: The solid CAD feature includes a scanning operation, and the method further includes: providing a candidate set of scanning operations, and the learning is also based on the candidate set of scanning operations.

7. The method according to claim 6, wherein: The neural network includes a neural network portion that outputs data for an inference of the scanning operation on the solid CAD feature.

8. The method according to claim 7, wherein: The candidate set for the scan operation includes the set product between: oA discrete set of sweep curve types, and o for each scan curve type, a respective discrete set of one or more parameter domains for each respective parameter, each parameter domain having a respective parameter value for the respective parameter, Each scanning curve type forms a corresponding element of the candidate set for the scanning operation using a corresponding parameter value of each parameter domain in a corresponding discrete set of the one or more parameter domains; as well as - the data for inferring the scanning operation of the solid CAD feature comprises: first data for inferring the type of the scanning curve, and second data for inferring parameter values ​​for each parameter domain in a corresponding discrete set of the one or more parameter domains.

9. The method according to any one of claims 1 to 4, wherein: The neural network includes a convolutional neural network (CNN) that encodes an image representing the hand drawing.

10. The method according to any one of claims 1 to 4, wherein: - the dataset further includes, for each hand-drawn drawing in the one or more hand-drawn drawings: Representing corresponding solid CAD features of the 3D shape relative to the hand drawing, and the learning comprises supervised training, wherein the supervised training comprises making the loss L sup minimize, The loss L sup The following remain unchanged: o a cyclic arrangement of sequences, where each solid CAD feature includes a corresponding sequence of curves, and / or o Reference-based sketches, each solid CAD feature including a sweep operation; and / or - the learning includes unsupervised training, the unsupervised training includes making the loss L unsup Minimize, for hand drawing, the loss L unsup penalizing differences from renderings derived from corresponding inferred solid CAD features, the neural network outputting one or more discrete probability distributions, the minimization comprising the loss L with respect to one or more variables respectively representing the corresponding discrete probability distributions unsup Back propagation of the gradient of , which is obtained using the boosting algorithm.

11. A computer-implemented method using a data structure representing a neural network learnable according to the method of any one of claims 1 to 10, the method comprising: - Provide a freehand drawing representing the 3D shape; - applying the neural network to the hand drawing; as well as - Based on the results of said applying, inferring solid CAD features representing said 3D shape.

12. A computer program product comprising instructions for executing the method according to any one of claims 1 to 10 and / or the method according to claim 11.

13. A device comprising a data storage medium having recorded thereon a data structure representing a neural network learnable by the method according to any one of claims 1 to 10 and / or a program product according to claim 12.

14. The apparatus according to claim 13, wherein The apparatus also includes a processor coupled to the data storage medium.

Images