Method for modifying 3D models by using partial shapes

Abstract

The invention relates to a computer-implemented method for designing a 3D model, the method comprising the following steps: a) providing an initial 3D model (1) in a 3D scene, the initial 3D model (1) comprising at least one extrusion cross-section (2), the extrusion cross-section (2) being defined by a set of parameters; b) receiving the user sketch (3) on a plane perpendicular to the direction of sight; c) in each of a plurality of iterations:-c1) modifying at least one of the parameters, thereby obtaining a modified 3D model (4); c2) performing a perspective projection of the modified 3D model (4) on a plane perpendicular to the line-of-sight direction, thereby obtaining a 2D visible line-of-sight frame (5), the 2D visible line-of-sight frame (5) comprising a visible inner edge (6) and a visible outer edge (7) of the modified 3D model (4); -c3) calculating an energy comprising a first term and a second term, where the first term penalizes an inconsistency between the modified 3D model (4) and the initial 3D model (1) and the second term penalizes a mismatch between the 2D visibility frame (5) and the user sketch (3); the parameter is modified so as to minimize the energy; and d) outputting the modified 3D model (4).

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more particularly to the field of computer-implemented methods for modifying 3D modeled objects in a three-dimensional (3D) scene by using a user sketch. In particular, the present invention belongs to the field of sketching. The present invention can be used in any 3D-based CAD software. Background Art

[0002] 2D sketching and 3D modeling are two major steps in industrial design. Sketching is typically done first because it allows designers to quickly and roughly express their vision. Then, using CAD tools that provide high accuracy and editability, the design sketches are converted into 3D models for downstream engineering and manufacturing.

[0003] Existing software suggests creating 3D models from hand-drawn sketches. Users draw in 2D, which is natural for users, rather than having to sculpt 3D shapes.

[0004] However, the user's design intent is not fully realized, and it may be useful to edit the 3D model in order to refine it.

[0005] In "Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches" (Guillard et al., ICCV 2021), a deep learning-based encoder / decoder architecture is trained to regress a surface mesh from a synthetic sketch. The user draws an initial sketch to obtain its 3D reconstruction. He can then manipulate the 3D object using the complete sketch or a partial sketch and draw one or more desired modifications. The 3D surface is then optimized to match each constraint by solving an optimization problem based on minimizing the 2D Chamfer Distance using an energy minimization strategy.

[0006] For example, in the article Figure 8 In , the user draws a car from a three-quarter view as an input sketch. From this view, the 3D model of the car is reconstructed using the same camera parameters, rendering it in the same image plane. The user then changes the viewpoint so that the car is in a side view, from which he can shorten the length of the trunk and move the rear wheels forward.

[0007] In Sketch2Mesh, users can only edit the outer silhouette of the mesh, not the clear visible inner edges. In fact, in Sketch2Mesh, minimizing the 2D chamfer distance consists of finding the 3D mesh points that project onto the contour of the foreground image, and then minimizing the chamfer distance between the contour and the sketch. As a result, users cannot modify the inner details of the 3D model, for example, the windows or the outer rear windshield (see the article). Figure 8 ).

[0008] Furthermore, editing a mesh has too many degrees of freedom (often thousands of vertices are optimized). Besides being computationally expensive, it can also result in an unreasonable mesh or one with many defects in the edited area (default). Therefore, to avoid these problems, reasonableness constraints are added to the optimization algorithm, which adds more processing to the optimization algorithm.

[0009] Therefore, there is a need to provide a computer-implemented method for designing a 3D model that enables editing of the inner edges of the 3D model without directly editing the mesh. Summary of the invention

[0010] The object of the invention is a computer-implemented method for designing a 3D model, the method comprising the following steps:

[0011] a) providing an initial 3D model in a 3D scene, the initial 3D model comprising at least one extruded section, wherein the extruded section is defined by a set of parameters;

[0012] b) receiving a user sketch on a plane perpendicular to the line of sight;

[0013] c) In each of the multiple iterations:

[0014] -c1) modifying at least one of said parameters, thereby obtaining a modified 3D model;

[0015] -c2) performing perspective projection on the modified 3D model on a plane perpendicular to the line of sight, thereby obtaining a 2D visible wireframe, wherein the 2D visible wireframe includes a visible inner edge and a visible outer edge of the modified 3D model;

[0016] -c3) calculating an energy including a first term and a second term, wherein the first term penalizes inconsistencies between the modified 3D model and the initial 3D model, and the second term penalizes mismatches between the 2D visual frame and the user sketch;

[0017] The parameters are modified so as to minimize the energy;

[0018] d) Output the modified 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Additional features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings:

[0020] Figure 1 A flow chart of the invented method is shown.

[0021] Figure 2 and Figure 3 Two examples of 3D primitive inference from user sketches are shown.

[0022] Figure 4-Figure 6 Several examples of editing 3D models by using the invented method are shown.

[0023] Figure 7 The vertices and extrusion vectors of a 3D primitive are shown.

[0024] Figure 8-Figure 11 The different steps of the rasterization process are shown.

[0025] Figure 12-14 The different steps of the coloring process are shown.

[0026] Figure 15-17 Calculation of custom chamfer energy is shown.

[0027] Fig.18 The calculation of the symmetry energy is shown.

[0028] Fig.19 The inventive method is shown from different angles.

[0029] Fig. 20 An embodiment of the present invention for editing a surface of revolution is shown.

[0030] Fig.21 A computer for implementing the inventive method is shown.

[0031] Fig. 22 A block diagram of a computer system suitable for performing the method is shown. DETAILED DESCRIPTION

[0032] Reference Figure 1 The invented method comprises a first step a): providing an initial 3D model 1 comprising at least one linear extruded section 2 in a 3D scene. An example of the initial 3D model 1 is shown in Figure 2 and Figure 3 Available in.

[0033] Such an initial 3D model 1 can be inferred by using a convolutional neural network (CNN) encoder disclosed in patent application EP3958162A1. The disclosed encoder uses part of the user's sketch. Fig.18 and learned patterns to encode 2D sketches into data that defines a 3D model.

[0034] The Convolutional Neural Network (CNN) encoder disclosed in patent application EP3958162A1 infers real-time 3D primitives; thus the user receives real-time feedback of the 3D primitives. Thus, the design process is not interrupted and the user can continue sketching while being guided by the proposed reasoning.

[0035] For example, in Figure 2 In , the user draws a rectangle with a semicircle on top of it (left part of the figure). The encoder infers the semi-cylinder as a 3D primitive. Figure 3 In , the user draws a rectangle and a stroke starting from the top vertex of the rectangle. The encoder infers a parallelepiped as a 3D primitive.

[0036] To describe 3D primitives, the following representation is used in patent application EP3958162A1:

[0037] - 3D point p of the plane section i a list of vertices (typically vertices of a plane section);

[0038] - Line type l in the form of {0,1} i A list of lines that describe whether two consecutive points of the section are connected by a line segment or an arc;

[0040] - A 3D vector h representing the direction and length of the extrusion.

[0041] It can be noted that it is not necessary to infer the 3D model using the above encoder. The 3D model can be simply obtained in the CAD software or from any other method, for example, downloaded from the Internet. The 3D model parameterization of such a 3D model can be easily and automatically converted to the above parameterization format (list of 3D points, list of line types, 3D vectors) to represent the CAD model.

[0042] With the invented method, the parameters corresponding to the line type do not change (the connection between two vertices will always be of the same type, the invented method affects only the position and the extrusion vector of the vertices).

[0043] The method comprises a second step b), which comprises: receiving a user's grass on a plane perpendicular to the line of sight Figure 3 . Example User Grass Figure 3 exist Figure 4-Figure 6 The left part is shown.

[0044] Given the camera settings (intrinsic and extrinsic parameters), the initial 3D model 1 is rendered onto the camera image plane. Figure 3 is drawn (or projected) onto this same camera image plane.

[0045] User Grass Figure 3 Input into the CAD tool may be via a mouse, an appendage (finger or stylus), or any suitable tactile device.

[0046] By using a mouse, the user drags the pointer, i.e., a combination of displacement of the mouse and a long press on one of the mouse buttons. In touch mode, by using a touch screen, the user moves an appendage (stylus or finger) on the touch screen, wherein contact is maintained between the touch screen and the moving appendage.

[0047] exist Figure 4 In the example of , the user draws a stroke 3 along two adjacent vertices of the parallelepiped from the projection of the initial 3D model 1 on the camera plane (a plane perpendicular to the viewing direction).

[0048] Thus, after several iterations (step c) of the method), a modified 3D model 4 is displayed (step d) of the method), such as Figure 4 The right part of Figure 4 In the example, the modified parallelepiped is larger than the initial parallelepiped. The modified 3D model can be displayed continuously after each iteration or at the end of all iterations.

[0049] Figure 5 Another example is shown. The initial 3D model 1 is a rotating cylinder characterized by a basis and a height. Figure 3 is an elongated circle next to the base of the cylinder. As a reaction to the user's sketch, after several iterations (step c) of the method), a modified 3D model 4 is displayed (step d) of the method), such as Figure 5 as shown in the right part of .

[0050] Figure 6 Another example is shown. The initial 3D model 1 is a rotating cylinder. Figure 3 In response, the height of the cylinder is shortened and fits the user sketch.

[0051] exist Figure 4-Figure 6 As can be seen in the example, the modified 3D model 4 is consistent with the user's rough Figure 3 The modified 3D model 4 matches the initial 3D model 1 and the consistency between the modified 3D model 4 and the initial 3D model 1 is maintained.

[0052] The idea underlying the present invention is to calculate and optimize the parameters defining the 3D model. In one embodiment of the present invention, the initial 3D model is defined by a linearly extruded cross section that can be moved in the 3D scene by parameters (i.e., in the world coordinate space R w The 3D point p of the cross section in the 3D scene is represented by i The position and extrusion vector h) are represented.

[0053] make is the point of the edited primitive (modified 3D model) and is the extrusion direction of the edited primitive, in world coordinate space R w Indicated in.

[0054] Figure 7 An exemplary cross section of a parallelepiped is shown. The cross section consists of 3D points p1, p2, p3, p4, which are the vertices of the cross section.

[0055] In this application, for the sake of clarity and brevity, line types (i.e., line segments or arcs) are not optimized, because line types are discrete parameters: line types can be straight line segments or circular arcs. During the optimization process, straight line segments remain straight line segments and circular arcs remain circular arcs. However, optimization of line types can be achieved by using continuous parameters for line types (parameters in [0,1], linear transformation from line segments to semicircular arcs) or curve parameterization.

[0056] A straightforward possibility would be to directly optimize the 3D point p i The position and extrusion vector h of , therefore, have three degrees of freedom for each point and extrusion vector. However, this may lead to unreasonable results (the edited 3D points are not on a common plane), and due to the complex optimization space, it will make the optimization process more difficult to converge to the optimal solution.

[0057] The invention advantageously defines in a common orthogonal local reference system R L = all points of the edited primitive in (u,v,n) where u and v correspond to arbitrary vectors defining the plane of the section, and n is the normal to the plane (see Figure 7 ).

[0058] Points of edited primitives The coplanarity can be expressed as follows:

[0059]

[0060] o i,u Corresponding to point p i The first offset along vector u.

[0061] o i,vCorresponding to point p i The second offset along vector v.

[0062] o n Corresponding to point p i A third offset along vector n, the third offset o n For all points p of the cross section i are all the same.

[0063] All points of the edited primitive share the same offset o in the direction of the plane's normal n , so that the point Thus, the coplanarity between the points is maintained and the invented method helps to achieve reasonable results.

[0064] Extrusion vector of the edited primitive (modified 3D model) It is expressed as follows:

[0065]

[0066] o h The fourth offset corresponds to the scale of the extrusion vector. It can be noted that the extrusion vector h of the edited primitive is not calculated using the formula Calculated, because when o n When it is different from zero, the section plane is translated. Therefore, it is necessary to balance the extrusion length so as not to also translate the extruded section. h = 0, the extruded section is translated -o n , so the extruded section is at the same location in 3D world space. Therefore, using the extrusion vector of the edited primitive With the above definition, only the length of the extrusion vector is optimized taking into account the third offset.

[0067] Therefore, at least one of the parameters is modified (sub-step c1, Figure 1 ) comprises modifying at least one of the first offset, the second offset, the third offset or the fourth offset. Thus, a modified 3D model 4 comprising the modified parameters is generated.

[0068] The number of parameters to be optimized is equal to 1+nbSides*2+1, where nbSides corresponds to the number of vertices of the cross section of the initial 3D model. For the first iteration step, each offset is initialized to zero.

[0069] The second sub-step c2) comprises performing a perspective projection of the modified 3D model 4 on a plane perpendicular to the line of sight, thereby obtaining a 2D visible line frame 5, the 2D visible line frame 5 comprising a visible inner edge 6 and a visible outer edge 7 of the modified 3D model 4 (see Fig.14 ).

[0070] The 3D model defined only by the parameters can be displayed in the 3D scene due to the generation of a 2D visual line frame 5. Otherwise, only the points of the cross section and the extrusion vectors will be displayed.

[0071] Thus, the user can edit both the outer silhouette and the visible inner edges of the mesh.

[0072] Figure 8 An example of a modified 3D model 4 is shown before conversion into a 2D visual line frame.

[0073] By first performing tessellation on the modified 3D model 4, a 3D mesh 8 (such as Fig. 9 As shown), the modified 3D model 4 can advantageously be converted into a 2D visible visual frame 5.

[0074] Subdivision involves dividing each polygon corresponding to a face of the modified 3D model 4, which is composed of points of the edited primitive, into a suitable structure for rendering. and by the extrusion vector of the edited primitive More specifically, for real-time rendering, each polygon is subdivided into triangles, such as Fig. 9 As shown in Figure 2. State-of-the-art methods can be used for subdivision operations. 3D models are defined by linearly extruded cross sections, which are usually convex. Therefore, the subdivision task is very simple and has a limited computational cost. Cross sections containing arcs or curves need to be discretized into multiple vertices.

[0075] Then, sub-step c2) comprises performing a differentiable rasterization of the 3D mesh, which returns image fragments based on the 3D mesh and based on the camera parameters. It should be noted that a fragment is the data required to generate a single pixel. Differentiable rasterization is performed by Fig.10 and Fig.11 Shown.

[0076] Each vertex of the 3D mesh can be represented by a linear combination of the primitives’ parameters. Therefore, the offset can be optimized by optimizing the mesh vertices.

[0077] Differentiable rasterization of 3D meshes can be performed by using PyTorch3D.

[0078] PyTorch3D, as a differentiable model, not only allows the linking of 2D pixels to 3D points (a necessary link for optimization), but also provides a solution that allows stable and feasible optimization. In fact, Pytorch3D slightly blurs / smears the triangles of the mesh. Therefore, one pixel belongs to several triangles of the mesh (the adjacent triangles and the triangles behind). Thus, discontinuities are avoided during optimization.

[0079] Any alternative differentiable rasterizer may be used, for example Nvdiffrast. Any solution that allows for each 2D pixel to retrieve the 3D point corresponding to the face to which the 2D pixel belongs may be used.

[0080] Fig.10 and Fig.11 It shows how a differentiable renderer can enable retrieval of the link between a pixel on the screen and the 3D vertex used to generate that pixel.

[0081] As in Fig.10 As can be seen above, differentiable rasterization first includes (sub-step ss1): rendering each triangle of the 3D mesh using a unique color (10, 11). Each unique color is converted into an RGB (red, green, blue) index. Each parameter (red, green and blue) defines the intensity of the color using a value between 0 and 255.

[0082] Therefore, it can be determined that all pixels of a particular color belong to the same triangle generated from the same three vertices. Each pixel color corresponds to a triangle identifier, which corresponds to the three 3D vertices of the triangle.

[0083] Then, as in Fig.11 As can be seen above, the differentiable rasterization then comprises (sub-step ss2): in the basis formed by the vertices (V0, V1, V2) of each triangle, representing each pixel of the corresponding triangle with barycentric coordinates.

[0084] For each triangle, one of the vertices (referred to as the first vertex V0) is rendered in a first color (e.g., black). Another vertex of the triangle (referred to as the second vertex V1) is rendered in a second color (e.g., red). The third vertex of the triangle (referred to as the third vertex V2) is rendered in a third color (e.g., green).

[0085] Due to the interpolation that naturally occurs during the rasterization process, each pixel of the triangle has a unique color and is represented in barycentric coordinates in a basis formed by the first vertex V0 (used as the basis origin) and the axis defined by the second vertex V1 (the vector is formed by the first vertex V0 and the second vertex V1) and the third vertex V2 (the vector is formed by the first vertex V0 and the third vertex V2). In another embodiment, the basis can also be defined with the second vertex V1 or the third vertex V2 as the origin.

[0086] Thus, in a given triangle, each pixel color corresponds to the position of the pixel relative to the vertices of the triangle.

[0087] Therefore, by combining sub-steps ss1) and ss2), the parameters of the 3D model can be to represent pixels, and each parameter is represented by an adjustable offset.

[0088] In summary, given the camera parameters and the mesh, the differentiable rasterization of the 3D mesh returns image fragments according to the rasterization calculations. In other words, for each image pixel displayed on the screen (e.g., Fig.11 12), provides the identifier of the visible face of the 3D mesh at that pixel (or a constant negative value, e.g., -1, if the mesh is not visible at the corresponding pixel). The centroid weights of the three vertices of the face are also provided: Using these three weights and the three face vertex coordinates, a 3D point can be obtained. This 3D point corresponds to the visible 3D point belonging to the mesh and projected to this pixel.

[0089] Therefore, due to differentiable rasterization, each image fragment associated with a pixel consists of:

[0090] - The data necessary to calculate the RGB intensity of a pixel;

[0091] - identifiers of triangles of the 3D mesh;

[0092] - The barycentric coordinates of the pixel in the triangle to which it belongs;

[0093] - The normal component of the triangle.

[0094] Then, the shading of the 3D mesh can be performed, including obtaining the visible outer edge 7 and the visible inner edge 6 of the 3D mesh without hidden lines (see Fig.14 ). In fact, hidden lines are not visible to the user and therefore should not be included in the optimization process.

[0095] In fact, it is easy to obtain the silhouette mask (background / foreground) based on the image segments: every pixel is in the foreground if its associated segment face is not -1. Using an edge detector (e.g., Sobel 2D convolution), one can obtain a 2D silhouette contour13, as Fig.12 However, wireframe rendering of the 3D mesh is not sufficient, since the visible inner contours of the 3D mesh have not yet been calculated at this stage.

[0096] To obtain each visible contour (inner and outer edges), the gradient between the normal components of each pixel is calculated to determine the variability of the direction of the normal components at each image fragment. It should be noted that the normal components are stored for each image fragment.

[0097] To this end, one method is to associate a color with each pixel according to its normal (where the color components red, green and blue correspond to the world coordinate space R w The normal components nx, ny, and nz in Fig.13 As shown. Then, the standard gradient-based strategy can be used to detect contours. Finally, the detected contours are actually visible contours.

[0098] Once the 3D mesh is rendered, an edge detection algorithm can be applied to detect inner edges6, such as Fig.14 shown.

[0099] Thus, a list of the following elements is obtained:

[0100] - List of triangle identifiers [t1,t2,…,tn];

[0101] - List of barycentric coordinates [(p11,p12,p13),…,(pn1,pn2,pn3)];

[0102] - List of normal components [(n11,n12,n13),…,(nn1,nn2,nn3)].

[0103] The 2D wireframe rendering is a 2D binary mask which is the result of the merging of two edge detected images, ie, the image of outer edges and the image of inner edges, which provides the 2D coordinates of 2D points of the foreground belonging to the 2D wireframe.

[0104] It is worth noting that the discrete 2D coordinates of the foreground pixels of the 2D wireframe can be obtained by directly obtaining the index of the foreground pixels. However, the link between these 2D coordinates and the vertices of the 3D mesh will be lost, making it impossible to optimize the mesh by using these 2D coordinates.

[0105] According to the above method, each 2D point is calculated from the projection (matrix operation) of a linear combination of the 3D mesh vertices, where each mesh vertex is a linear combination of the CAD parameters of the primitives. Therefore, it is possible to optimize the offset by optimizing the position of the 2D points. It is worth noting that, in contrast to the prior art methods, the 3D mesh is not optimized; rather, the 3D mesh is used to generate a 2D wireframe, and the optimization is done only by modifying the parameters of the cross-section and the extrusion vectors.

[0106] like Figure 1 As shown, the invented method comprises a step c3) in each iteration, which comprises: calculating an energy composed of a first term and a second term, wherein the first term penalizes the inconsistency between the modified 3D model 4 and the initial 3D model 1, and the second term penalizes the inconsistency between the 2D visible line frame 5 and the user's sketch. Figure 3 After each iteration, the parameters are modified to minimize the energy.

[0107] The first term encourages the above offsets to be close to zero. In fact, in order to maintain consistency between the original primitives and the edited primitives, most offsets may be equal to zero (which means that there is no change in the original primitives and the 3D models are consistent with each other). "Consistency" refers to the fact that the geometry remains the same (e.g., a cube is still a cube, a cylinder is still a cylinder, and a pyramid is still a pyramid), but the size may change.

[0108] According to an embodiment of the present invention, the first term is called the offset regularization energy and is calculated as follows:

[0109]

[0110] in, e3=(o h -o n ) 2

[0111]

[0112] nbSides corresponds to the number of edges of the cross section of the initial 3D model, %nbSides corresponds to the “modulo” number, ε is determined to prevent square root non-differentiability for the first iteration, p i Corresponds to the position of the point.

[0113] o i,u Corresponding to point p i The first offset along vector u.

[0114] o i,v Corresponding to point p i The second offset along vector v.

[0115] o n Corresponding to point p i A third offset along vector n, the third offset o n For all points p of the cross section i are all the same.

[0116] o h A fourth offset corresponding to the scale of the extrusion vector.

[0117] Minimizing the first term means finding a solution that minimizes the offset, noting that the best solution is usually the simplest one in terms of offset change.

[0118] The relative section plane translation in the direction corresponding to its normal. e1 is squared so as to be positive at each iteration as the energy is minimized. h Corresponds to the length of the extrusion vector.

[0119] The relative modification of the point in the plane corresponding to the cross section. side Corresponding to the two points p of the cross section i For example, for a four-point section (p1, p2, p3, p4), l side The calculation is as follows:

[0120]

[0121] Corresponds to a relative modification of the scale of the extrusion vector.

[0122] The first item offsets is the sum of quadratic energies, and minimizing the sum of quadratic elements tends to factor the error into each distinct element. This is not the desired behavior, as it often leads to unreasonable results with respect to user intent.

[0123] In fact, it is unlikely that the user will edit both the extrusion direction length and the section plane / section point with a single stroke. It is also unlikely that he will edit both the section point and the section plane at the same time. Due to the interactive nature of the invented method, the 3D model is iteratively edited after each user stroke, and a single stroke usually means minor modifications.

[0124] That's why the item was added This term highly penalizes simultaneous modifications. In fact, this term is equal to zero when at least two of the energies e1, e2 and e3 are equal to zero.

[0125] In a preferred embodiment, 1≤K≤10 6 and ε≤10 -6 .

[0126] Too high a K factor (e.g., 10 9 , or even 10 6 ) will make the optimization unstable (the term becomes too overwhelming compared to the other energies, especially because we deal with floating values: there is no value strictly equal to zero). Therefore, lowering K is acceptable in order to accept small errors in floating point numbers (both terms are very close to zero but not exactly zero).

[0127] Therefore, K = 10 3 It's a good trade-off.

[0128] Advantageously, ε≤10 -6 (For example, ε≤10 -8 ) to prevent non-differentiability (for the first iteration, the square root is not differentiable at zero).

[0129] The energy to be minimized includes a second term, called the custom chamfer energy, which penalizes the 2D visible line frame 5 from the user's grass Figure 3 The mismatch between .

[0130] In the following, the foreground points (pixels) of the 2D wireframe 5 are referred to as "rendered 2D points"; and the user sketch Figure 3 The point (pixel) is called the "target 2D point".

[0131] The idea is to iterate over each 2D target point 3 and minimize the distance between that point 3 and the closest 2D point from the rendered 2D point list 5. All target points should "attract" (i.e. reduce the distance to) the closest points from other list points, such as Fig.15 shown.

[0132] The notion of a "close" point depends on a distance metric, which in the prior art is usually the L2 norm. However, in the present case, it has been pointed out that the L2 norm (or L2 distance) is not suitable in all cases for finding the right rendering point to attract, e.g. Fig.16 shown.

[0133] exist Fig.16 In the left part of , a 2D rendering point 5 is displayed together with a 2D target point 3 , which should “attract” the 2D rendering point 5 .

[0134] exist Fig.16 In the middle part of , using the L2 norm, the closest 2D rendering points for each 2D target point are concentrated in regions 14 and 15, which is inconsistent with the user's intention.

[0135] In fact, in Fig.16, line segments 16 correspond to the correct rendering points that should be "attracted" and should match the user target stroke 3. There is no ambiguity here about the user's intention, because attracting rendering points belonging to orthogonal edges to the target stroke line would be meaningless (and would not respect the idea of ​​the simplest solution described above).

[0136] Since the well-known L2 distance is not satisfactory for calculating the distance between a rendered 2D point and a target 2D point, the inventive method advantageously comprises calculating a distance that is particularly suitable for the current situation.

[0137] In fact, the user stroke is viewed as a collection of lines that must attract close parallel contours. The calculated distance makes the input more meaningful than a collection of 2D target points that are all isolated and independent from each other.

[0138] Fig.17 and Fig.18 This embodiment is shown. For each 2D target point pix i , its 2D local stroke direction u i is to use its target 2D point neighbors (user grass Figure 3 Two points that pass by the point pix of the user's sketch i ) to approximate. i In the point pix i Place and u i Orthogonal, thus forming a local 2D orthogonal system (pix i ;u i ,n i ).

[0139] For each target point pix i , defines the local 2D coordinate system (pix i ;u i ,n i ), where n i with u i Orthographic (in a plane normal to the viewpoint).

[0140] Extend through two points of the user sketch (3) (which pass through the point pix of the user sketch i ) defines the direction u of the 2D local stroke i .n i In the point pix i Place and u i Orthogonal, thus forming a local 2D orthogonal system (pix i ;u i ,n i ).

[0141] Then, for each point pix of the user's sketch i , calculate at this point pix i The distances used with the invented method are defined in such a way that the rendered 2D points of the visible 2D wireframe that are not part of the user's sketch are located at the same distance from the rendered 2D points of the visible 2D wireframe that are also part of the user's sketch (see Fig.16 Points 14 and 15 in the middle portion are closer.

[0142] d(pix i ,pix k ) 2 =λ*(pix k .u i ) 2 +(pix k .n i ) 2

[0143] Therefore, the local direction u of the stroke i The distance between points on increases when λ>1. The target stroke line will attract parallel primitive lines.

[0144] Then, customize the chamfer energy e chamfer The calculation is as follows:

[0145]

[0146] n corresponds to the user grass Figure 3 Dot pix i The number of

[0147] pix i Corresponding to user grass Figure 3 point, and L target Corresponding to user grass Figure 3 n points pix i The entire collection.

[0148] pix k corresponds to the point of the 2D visible line frame 5, and L rendered Point pix corresponding to 2D visible line frame 5 k The entire collection.

[0149] λ is a scalar such that λ>1. In fact, in the case of λ>1, points parallel to the user sketch are preferentially used to calculate the custom chamfer energy e chamfer .

[0150] Advantageously, 2≤λ≤20, and preferably, λ=4.

[0151] If λ is too low (e.g., 1≤λ≤2), the 2D local stroke direction u i will not be penalized sufficiently. If λ is too high, there may be local normals n i instabilities in terms of noise, which may slow down real-time calculations.

[0152] Due to the normalization factor n 1, so the custom chamfer energy e chamfer does not depend on the size of the sketch. Moreover, the energies are easy to compare with each other since they are all of the same order of magnitude, which facilitates the calculation of gradients during optimization.

[0153] It has been seen that energy is calculated in each iteration of the invented method. The energy includes a first term (offset regularization energy) which penalizes the inconsistency between the modified 3D model (4) and the initial 3D model (1) and a second term (custom chamfer energy) which penalizes the mismatch between the 2D visual line box (5) and the user sketch (3);

[0154] Optionally, the energy can also include a third term, which is called the projection regularization energy, which is calculated as follows:

[0155]

[0156] Mask1 is a first binary mask of the projection of the initial 3D model on a plane perpendicular to the line of sight. Mask2 is a second binary mask of the projection of the modified 3D model on a plane perpendicular to the line of sight. The operators "*" and "+" are pixel-by-pixel operators. "mean" corresponds to the mean pixel value of the image. It can be noted that Mask1 and Mask2 can be interchanged because the operators "*" and "+" are symmetric functions.

[0157] The projection regularization energy represents the Intersection Over Union loss between two images. It consists in comparing the projected foreground / background masks of the initial 3D model with the modified 3D model.

[0158] The regularization energy penalizes pixel differences. As with the offset regularization energy, the idea is to maintain consistency between the two 3D models. Unlike numerical offsets that do not take into account camera parameters, using 2D projections of primitives can give realistic visual aspect feedback. In fact, the same non-zero offset may have a small or large effect on the primitive aspect behind the current viewpoint.

[0159] Optionally, the energy may also include a fourth term called symmetry energy, which penalizes the lack of symmetry of the modified 3D model and improves the plausibility of the modified 3D model.

[0160] Artificial objects are often composed of symmetrical and regular primitives. In fact, cylindrical artificial objects are more common than elliptical ones; and rectangular shapes are more common than arbitrary quadrilaterals.

[0161] The symmetry energy can be calculated as follows:

[0162]

[0163] P is a set of points including the vertices of the cross section and the midpoints between two consecutive vertices.

[0164] Q is a set of symmetry plane candidates, which are all planes that contain two different points in P and have normals that are orthogonal to the normal of the cross section.

[0165] are all the points of the cross section containing the initial 3D model (p i ) and all squeeze points A collection of points.

[0166] At a given point set and the plane q that defines the symmetry, the set of symmetric points is calculated using the symmetry plane q in yes symmetric point.

[0167] exist Fig.18 In the example of (left part), the outline 22 (here a quadrilateral) of the cross section of the 3D model is to be optimized. The point set P includes the point 23. The line 24 represents the axis of the best symmetry plane Q. The point 25 is the point set Fig.18 The right part of shows the 3D model optimized by symmetry energy. The point set P and The points are all merged, which means that the symmetry energy has been minimized, which means good symmetry with respect to axis 24.

[0168] Finally, the energy to be minimized is equal to the sum of at least the first and second terms (and potentially the third and fourth terms as well):

[0169] e=k chamfer *e chamfer +k offsets *e offsets +k projection *e projection +k symmetry *e symmetry

[0170] k chamfer , k offsets , k projection and ksymmetry is a weight, which is determined so that in the regularization energy (e offsets 、e projection and e symmetry ) and target energy (e chamfer ) to strike a balance between.

[0171] According to a specific embodiment, the following weights may be used:

[0172] k chamfer =10 3

[0173] k offsets =10 -1

[0174] k projection =1

[0175] k symmetry =10 -1

[0176] As mentioned above, each energy can be expressed using inputs (original CAD parameters of the extrusion, camera parameters, and user sketch) and offsets. Furthermore, each operation performed is differentiable (rasterization, shading, matrix operations, etc.). Therefore, at each iteration, the energy and offset are determined and can be calculated by performing a gradient descent optimization. to minimize the energy e, where o is any offset.

[0177] The descent learning rate parameter (or descent rate) is adjusted to match the scale of the primitive. For this purpose, the following descent rate DR can be used:

[0178]

[0179] l side corresponds to the average length of the side of the cross section of the initial 3D model, and l h The length of the extrusion vector corresponding to the initial 3D model. α is a scalar.

[0180] In a preferred embodiment, 1≤α≤10 6 , preferably, 5≤α≤10 4 , 10≤α≤10 3 , 20≤α≤100, for example, α=50.

[0181] During optimization, the learning rate parameter is preferably low so as not to change the bias too abruptly.

[0182] The invented method comprises a number of iterations of steps c1), c2) and c3). After each iteration the parameters are modified so as to minimize the energy. The offsets are thus updated using the gradient descent step results. The number of iterations can be predetermined (e.g. 200, which has been empirically verified to be very satisfactory), or the iterations can be stopped if one or several predetermined criteria have been met.

[0183] Finally, the edited CAD parameters are stored so that the modified 3D model can be further used in CAD software.

[0184] Due to the interactive nature of the invented method, the 3D model is iteratively edited after each user stroke, and a single stroke usually means minor modifications.

[0185] The method comprises a final step d), which comprises: displaying the modified 3D model (see Figure 1 ). The modified 3D model can be displayed continuously after each iteration or at the end of all iterations.

[0186] It can be noted that shape modifications are not limited to a single view. As the user draws, the original sketch as well as the inferred 3D model can be projected to any other arbitrary view or selected view, thus helping the user refine the final design by starting the drawing from one viewpoint and ending it in another.

[0187] For example, in Fig.19 In the example, the user draws a circle 19 according to a first view, and thus infers a cylinder 20. However, according to this first view, it is difficult for the user to accurately modify the height of the cylinder. The user then changes the view; a user sketch may be provided according to this second view in order to refine the shape. The inferred shape is shortened (see the user sketch). Fig.21 ) to better fit the stroke.

[0188] The method has been disclosed in particular for linear extrusion of a planar cross section, wherein the extrusion vector is always orthogonal to the planar cross section.

[0189] However, the method can be extended to sections consisting of multiple curves, where the linetype list contains curves of different degrees. Non-planar sections and extrusion vectors that are not orthogonal to the section plane can also be considered.

[0190] In particular, the invented method is also particularly suitable for surfaces of revolution, which are considered as curved extrusions (extrusion by rotation). Fig.19 The left part of shows a plane section including a list of 3D points (p0-p6), and the rotation axis is represented by the vector h= Definition, where and Belongs to the section plane on the axis of rotation.

[0191] It can be seen that connections p0-p1, p2-p3, p3-p4, p4-p5 and p6-p0 are line segments, and connections p1-p2 and p5-p6 are arcs. It is assumed that the connection type (line segment or arc) does not change during the design process.

[0192] Fig. 20 The right part of shows the solid of revolution (initial 3D model) obtained by rotating the planar sections p0-p6 around the axis of rotation defined by the vector h.

[0193] The modified 3D model can be composed of a modified set of points Defined with respect to the initial 3D model, these points correspond to modified planar sections of the modified solid of revolution (modified 3D model):

[0194]

[0195] u, v correspond to the vectors of the plane that defines the planar section, o i,u and i,v corresponds to the offset to be optimized;

[0196] In addition, the point corresponding to the modified rotation axis and The modified position of can be defined as follows:

[0197]

[0198]

[0199] Corresponds to the offset to be optimized.

[0200] In each iteration of the method, modifying at least one of the parameters of the 3D model comprises: modifying the offset o i,u , o i,v , One of the items.

[0201] According to this embodiment, steps a), b), c1) and c2) are identical for linear extrusion and will not be described in detail for the sake of brevity.

[0202] In step c3), the first term (bias regularization energy) is calculated as follows:

[0203]

[0204]

[0205]

[0206] l h =||p h1 -p h0 || 2

[0207] nbPoints corresponds to the number of points in the cross section of the initial 3D model, and K is determined to penalize the user's rough Figure 3 While modifying the penalty weight in , ε is determined to prevent the non-differentiability of the square root for the first iteration.

[0208] For linear extrusion, the values ​​of K and ε are determined similarly.

[0209] Furthermore, in step c3), the second term (custom chamfer energy) and the third term (projection regularization energy) are the same as the second term for linear extrusion. In the case of a revolved surface, the fourth term (symmetry energy) is irrelevant.

[0210] The invented method may be performed by a suitably programmed general purpose computer or computer system, which may include a computer network, storing an appropriate program, optionally in non-volatile (i.e., non-transitory) form, on a computer readable medium (e.g., a hard disk, a solid state disk, or a CD-ROM), and executing said program using its microprocessor and memory.

[0211] refer to Fig.21 A computer - more precisely a computer-aided design station - suitable for carrying out the method according to the exemplary embodiment of the present invention is described. Fig.21 In the embodiment, the computer includes a central processing unit CP that performs the above-mentioned process. The process can be stored in a memory (e.g., RAM MEM1 or ROM MEM2) as an executable program (i.e., a set of computer-readable instructions), or stored on a hard disk drive (HDD) MEM3, a DVD / CD drive MEM4, or can be stored remotely. The 3D model database in a form suitable for being processed by the executable program according to the method of the present invention can also be stored on one or more of the memory devices MEM1 to MEM4, or can be stored remotely.

[0212] The claimed invention is not limited by the form of the computer-readable medium on which the computer-readable instructions of the inventive process are stored. For example, the instructions and database may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other information processing device (e.g., a server or computer) with which the computer-aided design station communicates. The program and database may be stored on the same memory device or on different memory devices.

[0213] In addition, computer programs suitable for executing the inventive methods can be provided as utility applications, background daemons, or components of an operating system, or a combination thereof, and executed in conjunction with a CPU PR and an operating system (e.g., Microsoft VISTA, Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art).

[0214] The central processing unit CP may be a Xenon processor from Intel, USA or an Opteron processor from AMD, USA, or may be other processor types, for example, Freescale ColdFire, IMX or ARM processors from Freescale, USA. Alternatively, as will be appreciated by those skilled in the art, the central processing unit CP may be a processor such as Core2 Duo from Intel, USA, or may be implemented on FPGA, ASIC, PLD or using discrete logic circuits. In addition, the CPU may be implemented as a plurality of processors that work in coordination to perform computer-readable instructions of the above-mentioned inventive process.

[0215] Fig.21 The computer-aided design station in the embodiment also includes a network interface NI for connecting with a network (e.g., a local area network (LAN), a wide area network (WAN), the Internet, etc.), for example, an Intel Ethernet PRO network interface card from Intel Corporation of the United States. The computer-aided design station also includes a display controller DC for connecting with a display DY (e.g., a Hewlett Packard HPL2445w LCD monitor or a virtual reality head-mounted display), for example, an NVIDIAGeForce GTX graphics adapter from NVIDIA Corporation of the United States. The general I / O interface IF is connected with a keyboard KB and a pointing device PD (e.g., a rolling ball, a mouse, a touch pad, a control device of a virtual reality system, etc.). The display, keyboard, and pointing device together with the display controller and the I / O interface form a graphical user interface.

[0216] The invention may also be implemented in touch mode, where the computer system includes a touch-sensitive display for displaying the 3D scene and detecting interaction with one of the appendages.

[0217] The user sketch is provided by a pointing device PD, which may also be an appendage such as a stylus or a user's finger. The user sketch may be viewed on a display DY.

[0218] The disk controller DKC connects the HDD MEM3 and the DVD / CD MEM4 using a communication bus CB which may be ISA, EISA, VESA, PCI or the like for interconnecting all components of a computer-aided design station.

[0219] The computer also includes a memory having a data structure recorded thereon, the data structure including a convolutional neural network (CNN) encoder that infers 3D primitives based on the 2D sketch and based on the learned pattern. Those skilled in the art may refer to patent application EP3958162A1 for an exemplary description of the encoder.

[0220] The modified 3D model can be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other server or computer for further use in computer-aided design. A physical (mechanical) object or a part of said object can be manufactured based on the file containing the modified 3D model. The file can be converted into a readable format for the manufacturing process. Therefore, the present invention also relates to a method for manufacturing a mechanical part, the method comprising the following steps:

[0221] -Design mechanical components by the above design method;

[0222] -Physically manufacture mechanical parts.

[0223] For the sake of brevity, descriptions of the general features and functions of displays, keyboards, pointing devices, and display controllers, disk controllers, network interfaces, and I / O interfaces are omitted herein because these features are known.

[0224] Fig. 22 is a block diagram of a computer system suitable for executing methods according to different exemplary embodiments of the present invention.

[0225] exist Fig. 22 In the embodiment, the server SC is connected to the administrator system ADS and the end-user computers EUC via a network NW.

[0226] It is understood that the network NW can be a public network (e.g., the Internet), or a private network (e.g., a LAN or WAN network), or any combination thereof, and can also include a PSTN or ISDN subnet. The network NW can also be wired (e.g., Ethernet), or can be wireless (e.g., a cellular network including EDGE, 3G and 4G wireless cellular systems). The wireless network can also be Wi-Fi, Bluetooth, or any other known wireless communication form. Therefore, the network NW is merely exemplary and does not limit the scope of the present invention.

[0227] A client program stored in a memory device of an end-user computer and executed by a CPU of the end-user computer accesses a 3D model database on a server via a network NW.

[0228] Although only one administrator system ADS and one end-user system EUX are shown, the system can support any number of administrator systems and / or end-user systems without limitation.

[0229] Any process, description or block in the flowchart described herein should be understood to represent a module, segment or portion of code, which includes one or more executable instructions for implementing specific logical functions or steps in the process, and alternative implementations are included within the scope of exemplary embodiments of the present invention.

Claims

1. A computer-implemented method for designing a 3D model, the method comprising the following steps: a) providing an initial 3D model (1) in a 3D scene, the initial 3D model (1) comprising at least one extruded section (2), the extruded section (2) is defined by a set of parameters; b) receiving a user sketch (3) on a plane perpendicular to the line of sight; c) In each of the multiple iterations: -c1) modifying at least one of the parameters, thereby obtaining a modified 3D model (4); -c2) performing a perspective projection of the modified 3D model (4) on a plane perpendicular to the viewing direction; The modified 3D model is then obtained by performing a 2D visual frame (5). A visible inner edge (6) and a visible outer edge (7) of type (4); -c3) calculating an energy including a first term and a second term, wherein the first term penalizes the modified 3D model (4) is inconsistent with the initial 3D model (1), the second term penalizes the 2D visible frame (5) mismatch with the user sketch (3); The parameters are modified so as to minimize the energy; d) Outputting the modified 3D model (4).

2. The method according to claim 1, wherein: The initial 3D model (1) comprises at least one linear extruded section (2), wherein the linear extruded section (2) is defined by the set of parameters, wherein the set of parameters comprises a 3D point (p i ) position and extrusion vector (h), the modified 3D model (4) is composed of the modified point set and by the modified extrusion vector The modified point set is defined with respect to the initial 3D model (1). and the modified extrusion vector It is expressed as follows: Among them, p i corresponds to the set of points of the cross section of the initial 3D model (1), and h corresponds to the coordinate space R of the 3D scene. w The extrusion vector of the initial 3D model (1) represented in; u, v correspond to vectors defining the plane of said section, and n is the normal to said vectors; in: -o i,u Corresponding to the point p i The first offset along vector u, -o i,v Corresponding to the point p i The second offset along vector v, -o n Corresponding to the point p i A third offset along vector n, the third offset o n For all points p of the cross section i They are all the same; -o h a fourth offset corresponding to a scale of the extrusion vector; Wherein, modifying at least one of the parameters includes: modifying at least one of the first offset, the second offset, the third offset or the fourth offset.

3. The method according to claim 2, wherein: The first term is called the bias regularization energy and is calculated as follows: in, e3=(o h -o n ) 2 , Wherein, nbSides corresponds to the number of edges of the cross section of the initial 3D model, K is a penalty weight determined to penalize simultaneous modifications in the user sketch (3), and ε is determined to prevent square root non-differentiability for the first iteration.

4. The method according to claim 1, wherein: The initial 3D model (1) describes a 3D surface of revolution (16) defined by a set of parameters and by a rotation axis, wherein the set of parameters includes a 3D point (p i ), the rotation axis is represented as follows: h=p h1 -p h0 , Among them, p h0 and p h1 The plane of the section belonging to the rotation axis, The modified 3D model is a modified set of points of the plane section. and the modified rotation axis vector With respect to the initial 3D model, in, and u, v correspond to the vectors of the plane defining the planar section, o i,u and i,v Corresponding to the fifth and sixth offsets to be optimized, respectively, as well as and corresponds to the offset to be optimized, Wherein, modifying at least one of the parameters includes: modifying at least one of the offsets.

5. The method according to claim 4, wherein: The first term is called the bias regularization energy and is calculated as follows: in, l h =||p h1 -p h0 || 2 , Wherein, nbPoints corresponds to the number of points of the cross section of the initial 3D model, K is a penalty weight determined to penalize simultaneous modifications in the user sketch (3), and ε is determined to prevent square root non-differentiability for the first iteration.

6. A method according to any one of the preceding claims, wherein: Sub-step c2) comprises: - performing a subdivision of said modified 3D model (4), thereby obtaining a 3D mesh (8); - performing a differentiable rasterization on the 3D mesh, the differentiable rasterization returning image fragments based on the 3D mesh and based on camera parameters (9); - performing rendering of the 3D mesh, comprising: obtaining a visible outer edge (7) of the 3D mesh and a visible inner edge (6) of the 3D mesh, the visible inner edge (6) being obtained based on data stored in the image segment.

7. The method according to claim 6, wherein: The tessellation includes converting each face of the modified 3D model into a plurality of triangles (10, 11), and the differentiable rasterization includes the following sub-steps: -ss1) Render each triangle (10, 11) using a unique color; -ss2) in the basis formed by the vertices (V0, V1, V2) of each triangle, represent each pixel of the corresponding triangle with barycentric coordinates; - Combining sub-steps ss1) and ss2).

8. The method according to any one of claims 6 or 7, wherein: For each image segment of the 3D grid (8), the visible inner edge (6) is calculated by calculating the normals of the image segment and by calculating the gradient between the normals.

9. The method according to claim 8, wherein: A unique color is associated with the normal direction of each image segment, and the inner edge is calculated by calculating the gradient between the unique colors.

10. A method according to any one of the preceding claims, wherein: The second term is called custom bevel energy and is calculated as follows: d(pix i ,pix k ) 2 =λ*(pix k .u i ) 2 +(pix k .n i ) 2 , Among them, pix i Corresponding to the point in the user sketch (3), L target The point pix corresponding to the user sketch (3) i The entire collection, Among them, pix k Corresponding to the point of the 2D visible line frame (5), L rendered The point pix corresponding to the 2D visible line frame (5) k The entire collection, For each point pix of the user sketch i ,u i Corresponding to a 2D local stroke direction, the 2D local stroke direction extends through two points of the user sketch (3), and the two points of the user sketch (3) pass through the point pix of the user sketch i , n i At the point pix i Place and u i Orthogonal, thus forming a local 2D orthogonal system (pix i ;u i ,n i ); n is the number of points of the user sketch (3); Here, λ is a scalar such that λ>1.

11. A method according to any one of the preceding claims, wherein: The energy includes a third term called the projection regularization energy, which is calculated as follows: Among them, Mask1 is the first mask of the projection of the initial 3D model on the plane perpendicular to the line of sight direction; Mask2 is the second mask of the projection of the modified 3D model on the plane perpendicular to the line of sight direction; the operators "*" and "+" are pixel-by-pixel operators; "mean" corresponds to the average pixel value of the image.

12. The method according to any one of claims 2, 3 or 6 to 11, wherein The energy includes a fourth term called the symmetry energy, which is calculated as follows: Wherein, P is a point set, and the point set includes the vertex of the cross section and the midpoint between two consecutive vertices; Q is a set of symmetry plane candidates, which are all planes containing two different points in P and having normals orthogonal to the normal of the cross section; are all the points (p i ) and all squeeze points The set of points of Among them, given the point set and the plane q that defines the symmetry, the set of symmetric points is calculated using the symmetry plane q in, yes symmetric point.

13. The method according to any one of claims 2, 3 and 6 to 12, wherein: The energy is minimized by performing a gradient descent optimization with the following descent rate DR: Among them, l side corresponds to the average length of the side of the cross section of the initial 3D model, and l h The length of the extrusion vector corresponding to the initial 3D model, α is a scalar.

14. A computer program product comprising computer executable instructions for causing a computer system to perform the method according to any one of the preceding claims.

15. A computer-readable data storage medium (MEM1, MEM2, MEM3, MEM4) comprising computer-executable instructions for causing a computer system to perform the method according to any one of claims 1 to 13.

16. A computer system comprising a processor (CP) coupled to a memory (MEM1, MEM2, MEM3, MEM4), the memory storing computer executable instructions for causing the computer system to perform the method according to any one of claims 1 to 13.

Images