Computer-implemented method for assisting positioning of 3D objects in a 3D scene

Abstract

The invention relates to a computer-implemented method for assisting positioning of a digitally modeled 3D object, comprising the steps of: S1: providing a first digitally modeled 3D object (OBJ1) having a 3D position in a 3D scene; S2: rendering a projection on a screen of said first digitally modeled 3D object (OBJ1) according to a first axis (AX1) and a first point of view; S3: while modifying the 3D position of the first digitally modeled 3D object (OBJ1) along the first axis (AX1) according to a user action, automatically scaling the first 3D object (OBJ1) so that the projection on the screen of the moved object remains constant.

Description

TECHNICAL FIELD

[0001] The present invention relates to a computer-implemented method for assisting positioning of 3D objects in digital modeling in a 3D scene. For example, the invention enables taking a photo of an object such as furniture, creating a 3D model of the object, or starting from existing images of the object, storing and / or sharing the images for further CAD operations. BACKGROUND

[0002] Creating a digital model of a complex 3D physical object (e.g. a table consisting of a flat top and one or more legs) based on a single image is a complex task. Indeed, the image has an angle determined by the camera’s point of view and position, and this angle must be taken into account when modeling all the parts of the object. Otherwise, the 3D model will not express the real arrangement of the physical object. In the following, any reference to a 3D object will refer to a digitally modeled 3D object, and not to a physical object, unless explicitly stated to the contrary. Similarly, any reference to a 3D scene will refer to a computer-generated digital 3D scene, and not to a scene in the physical world, unless explicitly stated to the contrary.

[0003] In a computer-aided design (CAD) system, such as the one provided by Dassault Systèmes under the trademark CATIA, a user can create a complex 3D object comprising several 3D parts based on a single image. To do so, he creates 3D parts by drawing them on the 2D image (e.g. by drawing a first parallelepiped on the flat top of a table, and another parallelepiped on the leg). Alternatively, he can start from existing 3D parts. Due to the fact that the 3D parts are assembled from only one point of view of the virtual camera of the CAD system, the relative positions of the 3D parts can not correspond to the expected relative positions from this point of view. For example, from a first point of view, the user sees that the 3D part representing the flat top of the table and the 3D part representing the table leg are aligned. From another point of view, the two 3D parts can be misaligned.

[0004] To correct the wrong relative positioning of the 3D parts, the user can manipulate a graphical manipulator, such as the 3D compass in CATIA. The 3D compass embodies a compact representation of multiple functions that can be associated to the object on which the function (e.g. rotation, translation, scaling) will be performed.

[0005] The user drags the 3D compass toward one of the 3D components (for example, the one representing the flat top) and then releases the pointer on the 3D component, which anchors the 3D compass on this 3D component. When changing the viewpoint of the virtual camera, the user translates the 3D component on which the 3D compass is anchored by dragging one of the arrows of the 3D compass.

[0006] However, once the user changes the viewpoint again and returns to the initial viewpoint, the size of the projection on the screen of the 3D component that has been translated appears to have changed due to the translation. The 3D component after translation is larger or smaller compared to its initial size. Therefore, the user drags the 3D compass on the 3D component after translation; he then holds the zoom button of the 3D compass without releasing it and moves the pointer until he considers that the 3D component has been correctly re-scaled.

[0007] Therefore, the user performs in two steps, one for translation and the other for re-scaling, which is time-consuming. Moreover, these two steps can be repeated to iteratively obtain the desired relative positioning, which further increases the design time.

[0008] A solution using photogrammetry techniques can achieve a very precise 3D reconstruction, for example as implemented in the application "Catia Photo To Shape" (sold by Dassault Systemes). However, this technique requires starting from at least two images with two different viewpoints. Therefore, this is not a satisfactory solution for modeling a physical 3D object starting from a single image of this physical 3D object.

[0009] Academic research has also recently focused on 3D pose estimation, for example as disclosed in the article "3D Human Pose Machines with Self-supervised Learning" (Keze Wang et al., IEEE Transactions on Pattern, Analysis and Machine Intelligence, 2019). In this case, starting from a 2D pose of a human body, a 3D pose can be reconstructed by using deep learning. However, this article is specific to a particular class, namely the human body, and cannot be extended to other classes of objects. More generally, each neural network is adapted to a particular class of objects.

[0010] Therefore, there is a need to help the user to quickly position a 3D object in a 3D scene based on a single image of the object and without requiring prior knowledge of the components. SUMMARY

[0011] The aim of the application is therefore a computer-implemented method for assisting the positioning of a 3D object of digital modeling, comprising the following steps:

[0012] - S1 : providing a first 3D object of digital modeling having a 3D position in a 3D scene;

[0013] - S2: rendering a projection on a screen of said first 3D object of digital modeling according to a first axis and a first point of view;

[0014] - S3: while modifying the 3D position of the first 3D object of digital modeling along the first axis according to a user action, automatically scaling the first 3D object so that the projection on the screen of the moved object remains constant.

[0015] In a preferred embodiment, step S3 comprises displaying said first 3D object of digital modeling according to a second axis and from a second point of view, said second point of view and said second axis being different from the first point of view and the first axis, respectively.

[0016] In a preferred embodiment, the first axis and the second axis are orthogonal to each other.

[0017] In a preferred embodiment, the first axis is represented by a dotted or solid line, said dotted or solid line being coincident with the first axis.

[0018] In a preferred embodiment:

[0019] - step S1 comprises providing a second 3D object of digital modeling having a predetermined size and having a 3D position in the 3D scene;

[0020] - step S3 comprises keeping the 3D position and the size of the second 3D object of digital modeling fixed.

[0021] In a preferred embodiment, step S3 comprises displaying at least one capture point relative to the second 3D object of digital modeling on the first axis to enable quick positioning of the first 3D object of digital modeling relative to the second 3D object of digital modeling.

[0022] In a preferred embodiment, the capture point corresponds to:

[0023] - the projection on the first axis of the center of the smallest bounding box enclosing the second 3D object of digital modeling, or

[0024] - the projection on the first axis of the edge of the smallest bounding box enclosing the second 3D object of digital modeling.

[0025] In a preferred embodiment, step S3 comprises highlighting the first 3D object of digital modeling.

[0026] In a preferred embodiment, step S1 includes receiving user input for fitting the 3D object modeled by the first digital model at least partially to a portion of a 2D image on the screen.

[0027] The present invention also relates to a computer program product stored on a computer-readable data storage medium, comprising computer-executable instructions for causing a computer system to perform the methods described above.

[0028] The present invention also relates to a computer-readable data storage medium comprising computer-executable instructions for causing a computer system to perform the methods described above.

[0029] The present invention also relates to a computer system including a processor coupled to a memory and a screen, the memory storing computer-executable instructions for causing the computer system to perform the methods described above. Attached Figure Description

[0030] Additional features and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings, which illustrate:

[0031] - Figure 1 A flowchart of the method according to the present invention;

[0032] - Figure 2 Based on the first viewpoint, including images of the components to be modeled in three dimensions, in this case the flat top of the table and the table legs, the choice of 3D shape;

[0033] - Figure 3 A diagram illustrating the rescaling of an object;

[0034] - Figure 4 The 3D scene of the first and second 3D objects is displayed from a second viewpoint;

[0035] - Figure 5 The image is dragged along the first axis of the virtual camera;

[0036] - Figure 6 A map capturing points;

[0037] - Figure 7 This includes 3D components that show the corrected position of 3D objects.

[0038] - Figure 8 A block diagram suitable for performing the method according to the invention in a computer system. Detailed Implementation

[0039] In the first step of this method ( Figure 1In step S1) of the flowchart, a first (digitally modeled) 3D object OBJ1 is provided. According to a 3D position, the first object OBJ1 is located in a (computer-generated) 3D scene. It is created by the user or can already be located in the 3D scene before the user starts to manipulate.

[0040] In a second step (S2) of the flowchart, a projection of the first 3D object OBJ1 is rendered on the screen of the user's electronic computing device (computer, laptop, tablet, smartphone), as shown in Figure 1 Figure 2

[0041] The image IM contains the components to be modeled in 3D. In the current case, the 2D image comprises a table consisting of three parts: a flat top, a footboard, and a leg supporting the flat top and located on the footboard. In the following, the correct positioning of only two 3D objects is disclosed. The positioning of the other 3D objects constituting the component is performed in the same way as the positioning of the two objects.

[0042] If the user starts from a 2D image, the drawing angle has to be detected. The method of the invention is particularly suitable for images complying with the Manhattan hypothesis, because the drawing angle can be easily detected. The Manhattan hypothesis is disclosed in the article "Manhattan Word: Compass Direction from a Single Image by Bayesian Inference" (Coughla et al.). An image complies with the Manhattan hypothesis if there are three orthogonal dominant directions in the image, i.e. three sets of predominantly parallel lines.

[0043] Then, once the correct drawing angle is obtained, the user can draw 3D objects (also called 3D shapes) on the 2D image based on the three detected orthogonal dominant directions for fitting the Manhattan hypothesis to the image. A user input is received fitting the first 3D object OBJ1 at least partially to a part of the 2D image on the screen.

[0044] The drawing angle can be detected on the 2D image by identifying at least one vanishing point. A vanishing point corresponds to a point of convergence of two-dimensional perspective projections of lines that are parallel to each other in three-dimensional space. Thus, the drawing angle takes the form of a vanishing point. In Figure 2 One vanishing point P1 is shown in the right part of Figure 2 Another vanishing point can be identified in the left part of

[0045] ​​The first 3D object OBJ1 is drawn based on the drawing angle. In the current case, the user draws a rectangle by dragging a pointer (mouse, finger, or stylus) from one point on the screen to another point, where the first and other points are defined as opposite angles of the rectangle. The user then extrudes the rectangle to create a parallelepiped. The first object OBJ1 is created; it is a 3D object drawn on a 2D image. It corresponds to the flat top of a table. The same steps are performed to draw the second object OBJ2. However, the method according to the invention can be implemented even for only one 3D object.

[0046] Provide a 3D object OBJ1 in the 3D scene rendered by the virtual camera CAM based on the first axis AX1 and the first viewpoint VP1. Figure 3 (Characters in the text). The first viewpoint VP1 corresponds to the camera's 3D position. The first axis AX1 connects the first viewpoint VP1 to the centroid of the bounding boxes of all 3D objects in the 3D scene. Figure 3 The above assumes that a 3D scene contains only one 3D object, namely 3D object OBJ1.

[0047] Then, in the third step ( Figure 1 In step S3 of the flowchart, the user drags (presses down without releasing) the first object OBJ1 along the first axis AX1. While dragging the first object OBJ1 along the first axis AX1, the first object OBJ1 is rescaled so that the projection of the first object OBJ1 on the screen SCR remains constant.

[0048] First, calculate the distance D between the first viewpoint and the center of the minimum bounding box CBB surrounding the first object OBJ1. OldPosition To achieve rescaling. Figure 3 Before manipulating the first object OBJ1, the distance between the first viewpoint VP1 (i.e., the position of the virtual camera) and the first object OBJ1 is denoted as D. OldPosition When the user drags the first object OBJ1 along the first axis AX1, the distance between the first viewpoint VP1 and the moved first object OBJ1 is denoted as D. NewPosition When the distance between the first object OBJ1 and the first viewpoint VP1 is D... OldPosition At that time, a dimension of the first object OBJ1, such as its height, is denoted as H. OldPosition ; and when the first object OBJ1 is at a distance D from the first viewpoint VP1 NewPosition When the same dimension is denoted as H, NewPosition Size H NewPosition It is calculated based on the following relationship:

[0049]

[0050] Then, the first object OBJ1 is re-scaled.

[0051] When the user drags the first object OBJ1 away from the first viewpoint VP1, the first object OBJ1 is enlarged, and when the user drags the first object OBJ1 closer to the first viewpoint VP1, the first object OBJ1 is smaller. Thus, the projection of the first object OBJ1 remains constant and the user does not need to manually re-scale the first object OBJ1 by using a compass.

[0052] In a preferred embodiment, the first 3D object OBJ1 is displayed according to a second viewpoint VP2 and a second axis AX2 of a virtual camera. To this end, as Figure 2 illustrated, when the first object OBJ1 is rendered on the screen according to the first viewpoint VP1 and the first axis AX1, the user selects (e.g. by clicking on a dedicated button or a contextual button in a menu) a command to display the first object OBJ1 according to a second viewpoint VP2 and a second axis AX2 of a virtual camera. Thus, the user can see the re-scaling of the first object OBJ1 and he can also finely drag the first object OBJ1.

[0053] Advantageously, the first axis AX1 and the second axis AX2 are orthogonal to each other. Thus, the user can easily see the position of the first 3D object OBJ1 on the axis AX1.

[0054] Figure 4 The first 3D object OBJ1 is shown according to the second viewpoint VP2. In Figure 4 it can be seen that the first object OBJ1 and the second object OB2 are not correctly aligned. Thus, the user drags the first object OBJ1 along the first axis AX1 until he estimates that the relative position of the two objects is correct.

[0055] In a preferred embodiment, when the 3D scene is rendered according to the first viewpoint, the user selects the object to be moved by clicking on it and the moved object is highlighted. For example, once selected, the outline of the selected object is blinking or the color of the first object OBJ1 object is changed. If other 3D objects have been imported or drawn in the 3D scene, these objects are not highlighted. Thus, even if the object to be moved is partially hidden by other objects in the 3D scene, the user can easily identify it in the 3D scene.

[0056] The first axis AX1 is advantageously represented by a dashed or solid line coinciding with the first axis AX1. Thus, when the user modifies the position of the first 3D object OBJ1, the displacement of the first 3D object OBJ1 can be easily predicted. Without the dashed or solid line, the user would have difficulties to assess the position of the first viewpoint compared to the second viewpoint, especially when the first axis AX1 and the second axis AX2 are not orthogonal. It would also be difficult to assess the exact orientation of the first axis AX1 with respect to the second axis AX2. This embodiment is also useful when the second axis AX2 cannot be orthogonal to the first axis AX1 because the 3D object hides the second viewpoint VP2. This can happen when the 3D scene contains many 3D objects.

[0057] Figure 4 The displacement of the first object OB1 along the first axis AX1 is illustrated. Now, the first object OBJ1 is closer to the first viewpoint VP1 and thus the first object OBJ1 becomes smaller. Thus, the projection of the first object OBJ1 on the screen remains constant when the camera is located at the first viewpoint V1.

[0058] In a preferred embodiment, when at least the first 3D object and the second 3D object have to be positioned with respect to each other, the step S3 comprises displaying at least one snapping point SP on the first axis with respect to the second 3D object OBJ2. In the following, a snapping point is a graphical icon located on the first axis AX1 at a position corresponding to the projection on the first axis AX1 of a feature point of the second 3D object OBJ2. Thus, the movement of the first 3D object OBJ1 on the snapping point SP is not as fast as on other parts of the first axis AX1 and thus the user can align the first 3D object OBJ1 with respect to the second 3D very finely and accurately. For example, when dragging the first 3D object OBJ1, hovering the pointer on the snapping point can be slower than on other parts of the first axis AX1. Alternatively, the first object can be anchored on the snapping point SP when hovering on it (however, this does not prevent at all the user to move the first 3D object OBJ1 ).

[0059] Advantageously, the snapping point can correspond to:

[0060] - the projection on the first axis AX1 of the center of the smallest bounding box encompassing the second 3D object OBJ2, or

[0061] - the projection on the first axis of an edge of the smallest bounding box encompassing the second 3D object OBJ2.

[0062] As Figure 6 illustrated, several snapping points of the same object can be displayed on the first axis AX1. Thus, the user can choose to align the first 3D object with the center of the smallest bounding box encompassing the second 3D object OBJ2 or with one of the edges of this bounding box. The type of snapping point is configurable.

[0063] The user can navigate in the 3D scene by using for example the combination of pressing the middle button of the mouse while moving the mouse in order to check if the first 3D object OBJ1 is positioned as expected. For example, he can reposition the virtual camera according to the first viewpoint as shown in Figure 7 .

[0064] At any time, he can also switch to the 2D image by clicking (pressing and releasing) outside the 3D objects in the 3D scene as shown in Figure 2 . Thus, he can check if the dimensions of the 3D objects remain constant.

[0065] In another embodiment of the application, when the 2D image is displayed in the main part of the screen, the 3D scene according to another viewpoint of the camera can be displayed in a thumbnail view. Thus, the user can see that it is necessary to move and realign one of the 3D objects.

[0066] The method of the application can be performed by a general purpose computer or computer system, possibly including a computer network, which is suitably programmed on a computer readable medium, for example a hard disk, a solid state disk or a CD-ROM, in a non-volatile form, and which executes the program using its microprocessor and memory.

[0067] Reference is made to Figure 8 a computer CPT adapted to perform the method according to the exemplary embodiment of the application. In Figure 8 , the computer CPT comprises a central processing unit (CPU) P which performs the method steps described above when running an executable program, i.e. a set of computer readable instructions, which is stored in a memory device such as a RAM M1 or a ROM M2 or a hard disk drive (HDD) M3, a DVD / CD drive M4, or remotely. Moreover, the 3D objects created in step S1 of the method can also be stored in one or more of the memory devices M1 to M4, or remotely.

[0068] The claimed invention is not limited by the form of the computer readable medium on which the computer readable instructions and / or data structures of the present application processes are stored. For example, instructions and files can be stored on a CD, a DVD, a flash memory, a RAM, a ROM, a PROM, an EPROM, an EEPROM, a hard disk, or any other information processing device in communication with a computer, such as a server or a computer. The programs and files can be stored on the same storage device or on different storage devices.

[0069] Furthermore, the computer program adapted to perform the method of the present application can be provided as an utility program, a background daemon or a component of the operating system, or a combination thereof, executed with the CPU P and an operating system such as Microsoft VISTA, Microsoft Windows 10, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.

[0070] The CPU P can be an Intel Xenon processor or an AMD Opteron processor, or can be of other processor types such as Freescale ColdFire, IMX or ARM processors. Alternatively, the CPU can be a processor such as Core2 Duo by Intel, or can be implemented on an FPGA, ASIC, PLD or using discrete logic circuitry, as will be appreciated by those skilled in the art. Furthermore, the CPU can be implemented as multiple processors working in concert to execute the computer readable instructions of the above described inventive processes.

[0071] Figure 8 The computer CPT in the computer system of Fig. 1 further comprises a network interface NI, such as an Intel Ethernet PRO network interface card by Intel Corporation of America, for interfacing to networks such as a local area network (LAN), a wide area network (WAN), the Internet, etc. The computer further comprises a display controller DC, such as an NVIDIA GeForce GTX graphics adapter by NVIDIA Corporation of America, for interfacing to a display DY, such as a HP L2445w LCD monitor. A general purpose I / O interface IF interfaces to a keyboard KB and a pointing device PD, such as a roller ball, mouse, or touchpad. The display, keyboard, and pointing device, together with the display controller and the I / O interface, form a graphical user interface through which the user can provide input commands to the computer and the computer can display 3D objects.

[0072] The disk controller DKC interfaces the HDD M3 and the DVD / CD M4 to a communication bus CBS, which can be an ISA, EISA, VESA, PCI or similar, for interconnecting the components of the computer.

[0073] For the sake of brevity, the description of the general features and functionality of the display, keyboard, pointing device, and the display controller, disk controller, network interface, and I / O interface, is omitted here, as these features are known.

[0074] Any method steps described herein should be understood as representing a module, segment, or portion of code which includes one or more executable instructions for implementing specific logical functions or steps in the process, and alternative implementations are included within the scope of the exemplary embodiments of the present application.

Claims

1. A computer-implemented method for assisting positioning of a 3D object for digital modeling, comprising the steps of: S1 : providing a first digitally modeled 3D object (OBJ1 ) having a 3D position in a 3D scene; S2: rendering a projection on a screen of said first digitally modeled 3D object (OBJ1 ) according to a first axis (AX1 ) and a first viewpoint; triggering, according to a user command, a display of said first digitally modeled 3D object (OBJ1 ) according to a second axis and a second viewpoint for rescaling said first digitally modeled 3D object (OBJ1 ) when rendering said first digitally modeled 3D object (OBJ1 ) on said screen according to said first viewpoint and said first axis, wherein said second viewpoint and said second axis are different from said first viewpoint and said first axis respectively, and said first axis and said second axis are orthogonal to each other; S3: automatically scaling said first digitally modeled 3D object (OBJ1 ) while modifying said 3D position of said first digitally modeled 3D object (OBJ1 ) along said first axis (AX1 ) according to a user action, so that the projection on said screen of the moved object remains constant.

2. The computer-implemented method of claim 1, wherein, Said first axis (AX1 ) is represented by a dashed or solid line, said dashed or solid line being coincident with said first axis.

3. The computer-implemented method of any one of the preceding claims, wherein: Step S1 comprises providing a second digitally modeled 3D object (OBJ2) having a predetermined size and a 3D position in said 3D scene; step S3 comprises keeping said 3D position and said size of said second digitally modeled 3D object (OBJ2) fixed.

4. The computer-implemented method of claim 3, wherein, Step S3 comprises displaying at least one snapping point (SP) relative to said second digitally modeled 3D object (OBJ2) on said first axis for enabling a quick positioning of said first digitally modeled 3D object (OBJ1 ) relative to said second digitally modeled 3D object (OBJ2); Said snapping point is a graphical icon located on the first axis (AX1 ) at a position corresponding to a projection on said first axis (AX1 ) of a feature point of said second digitally modeled 3D object (OBJ2).

5. The computer-implemented method of claim 4, wherein, Said snapping point corresponds to a projection on said first axis (AX1 ) of a center of a smallest bounding box enclosing said second digitally modeled 3D object (OBJ2), or a projection on said first axis of an edge of a smallest bounding box enclosing said second digitally modeled 3D object (OBJ2).

6. The computer-implemented method of claim 1 or 2, wherein, Step S3 comprises highlighting said first digitally modeled 3D object (OBJ1 ).

7. The computer-implemented method of claim 1 or 2, wherein, Step S1 comprises receiving a user input for fitting said first digitally modeled 3D object (OBJ1 ) at least partially with a portion of a 2D image on said screen.

8. A computer program product stored on a computer readable data storage medium, comprising computer executable instructions for causing a computer system to perform the method according to any one of the preceding claims.

9. A computer readable data storage medium containing computer executable instructions for causing a computer system to perform the method according to any one of claims 1 to 7.

10. A computer system comprising a processor coupled to a memory, a screen, the memory storing computer-executable instructions for causing the computer system to perform the method of any one of claims 1 to 7.

Images