COMPUTER-IMPLEMENTED METHOD FOR ASSISTING LOCATING A 3D OBJECT IN A 3D SCENE - Patent application
The method addresses the inefficiencies in creating 3D models from single images by using orthogonal axes and snap points for precise alignment, enhancing the speed and accuracy of 3D object positioning in CAD systems.
Patent Information
- Application Number
- JP2020155649
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-23
- Filing Date
- 2020-09-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-09-16
AI Technical Summary
Existing methods for creating a digital model of a complex 3D object from a single image are time-consuming and require multiple iterative steps due to inconsistent relative positioning and scaling of 3D parts, and existing photogrammetric techniques and neural networks are limited in applicability.
A computer-implemented method that allows users to position and scale 3D objects in a 3D scene by rendering the object from multiple perspectives, using orthogonal axes and snap points to maintain projection constancy, enabling automatic scaling and precise alignment.
Facilitates quick and accurate positioning of 3D objects in a 3D scene from a single image without requiring prior knowledge of assembly, reducing design time and complexity.
Smart Images

Figure 0007584262000002 
Figure 0007584262000003 
Figure 0007584262000004
Abstract
Description
[Technical field]
[0001] The present invention relates to a computer-implemented method for assisting in the positioning of digitally modeled 3D objects in a 3D scene. For example, the present invention allows one to photograph an object, such as a piece of furniture, and create a 3D model of the object, or to start from an existing image of the object, and store and / or share it for further CAD manipulation. [Background technology]
[0002] Creating a digital model of a complex 3D physical object (such as a table made of a flat top and one or more legs) based on a single image is a complex task. Indeed, an image has a viewpoint determined by the camera's angle of view and position, which must be taken into account when modeling all parts of the object, otherwise the 3D model would not represent a realistic configuration of the physical object. Hereinafter, any reference to a 3D object refers to a digitally modeled 3D object, not a physical one, unless explicitly stated to the contrary. Similarly, any reference to a 3D scene refers to a computer-generated digital 3D scene, not a scene in the physical world, unless explicitly stated to the contrary.
[0003] In computer-aided design (CAD) systems, such as those offered by Dassault Systemes under the trademark CATIA, a user can create complex 3D objects with several 3D parts based on a single image. To do so, the user creates the 3D parts by drawing them relative to the 2D image (e.g., by drawing a first parallelepiped on the flat top surface of the table and another one for the foot plate). Alternatively, the user can start from existing 3D parts. Since the 3D parts are assembled according to only one viewpoint of the virtual camera of the CAD system, the relative positioning of the 3D parts from that viewpoint may not correspond to the desired relative positioning. For example, from a first viewpoint, the user sees that the 3D part representing the flat top surface of the table and the 3D parts representing the legs of the table are aligned. From another viewpoint, the 3D parts may not be aligned together.
[0004] To correct incorrect relative positioning of 3D parts, the user can manipulate a graphical manipulator, such as the 3D compass in CATIA, which implements a compact representation of multiple functions that can be associated with an object on which functions such as rotation, translation, scaling, etc. can be performed.
[0005] The user drags the 3D compass towards one of the 3D parts (such as one representing a flat top surface) and releases the pointer over the 3D part to anchor the 3D compass on the 3D part. When changing the viewpoint of the virtual camera, the user moves the 3D part to which the 3D compass is anchored by dragging one of the 3D compass arrows.
[0006] However, after the user changes the viewpoint again and back to the initial one, the on-screen projection dimensions of the moved 3D part appear to have changed due to the movement: the moved 3D part becomes larger or smaller compared to its initial dimensions. Thus, the user drags the 3D compass over the moved 3D part, then, without releasing, presses the scale button of the 3D compass and moves the pointer until the 3D part appears to be correctly rescaled.
[0007] Thus, using the user proceeds in two steps, one for translation and the other for rescaling, which is time consuming. Moreover, these two steps may be repeated to iteratively obtain the desired relative positioning, further increasing the design time.
[0008] Solutions using photogrammetry techniques, as implemented for example in the "Catia Photo To Shape" application (marketed by Dassault Systèmes), allow very accurate 3D reconstructions. However, this technique requires starting from at least two images with two different viewpoints. It is therefore not a satisfactory solution for modeling a physical 3D object starting from a single image.
[0009] Academic research has also recently focused on 3D pose estimation (see, for example, non-patent document 1). In that context, starting from a 2D pose of a human, it is possible to reconstruct the 3D pose by using deep learning. However, non-patent document 1 is specific to a certain class, namely the human body, and cannot be extended to other classes of objects. More generally, each neural network is adapted to a specific class of object.
[0010] Therefore, there is a need to assist a user in quickly positioning a 3D object in a 3D scene based on a single image of the object, without requiring prior knowledge of the assembly. [Prior art documents] [Non-patent literature]
[0011] [Non-Patent Document 1] Paper "3D Human Pose Machines with Self-supervised Learnong" (Keze Wang et al., IEEE Transaction on patter, analysis and machine intelligence, 2019) [Non-Patent Document 2] The paper "Manhattan Word: Compass Direction from a Single Image by Bayesian Inference" (Coughla et al.) Summary of the Invention
[0012] An object of the present invention is then a computer-implemented method for assisting in the positioning of a digitally modeled 3D object, the method comprising: - S1: providing a first digitally modeled 3D object having a 3D position in a 3D scene; - S2: rendering on a screen a projection of said first digitally modeled 3D object according to a first axis and a first viewpoint; - S3: upon a user action, automatically scaling the first 3D object while changing a 3D position of the first digitally modeled 3D object along a first axis in order to keep the projection of the moved object constant on the screen; Includes.
[0013] In a preferred embodiment, step S3 includes displaying the first digitally modeled 3D object from a second perspective along a second axis, the second perspective and the second axis being different from the first perspective and first axis, respectively.
[0014] In a preferred embodiment, the first axis and the second axis are mutually orthogonal.
[0015] In a preferred embodiment, the first axis is represented by a dashed or solid line, said dashed or solid line coinciding with the first axis.
[0016] In a preferred embodiment, - step S1 comprises providing, in the 3D scene, a second digitally modelled 3D object having predetermined dimensions and a 3D position; - Step S3 comprises holding fixed the 3D position and dimensions of the second digitally modelled 3D object.
[0017] In a preferred embodiment, step S3 includes displaying at least one snap point on the first axis relative to the second digitally modeled 3D object to enable rapid positioning of the first digitally modeled 3D object relative to the second digitally modeled 3D object.
[0018] In a preferred embodiment, the snap points are: - the projection on the first axis of the center of the smallest bounding box that encloses a second digitally modeled 3D object, or - a projection on the first axis of the edge of a minimal bounding box that encloses the second digitally modeled 3D object; Corresponds to.
[0019] In a preferred embodiment, step S3 includes enhancing the first digitally modeled 3D object.
[0020] In a preferred embodiment, step S1 includes receiving user input for at least partially matching said first digitally modeled 3D object with a portion of an on-screen 2D image.
[0021] The present invention also relates to a computer program product stored on a computer readable data storage medium comprising computer executable instructions for causing a computer system to perform the above-mentioned method.
[0022] The invention also relates to a computer readable data storage medium containing computer executable instructions for causing a computer system to perform the above-mentioned method.
[0023] The invention also relates to a computer system comprising a memory, a processor coupled to a screen, the memory storing computer executable instructions for causing the computer system to carry out the above-mentioned method. [Brief description of the drawings]
[0024] Further features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. [Figure 1] 2 is a flow diagram of a method according to the present invention; [Diagram 2] A first perspective view of an image including a selection of 3D shapes, in this case a flat top surface of a table, and an assembly modeled in three dimensions, the foot plate of the table. [Diagram 3] FIG. 2 is a schematic diagram of rescaling an object. [Figure 4] 1 is a diagram of a 3D scene displaying a first and a second 3D object from a second perspective. [Diagram 5] FIG. 1 illustrates dragging along a first axis of the virtual camera. [Figure 6] FIG. 1 is a diagram of a snap point. [Figure 7] FIG. 13 is a diagram of the 3D assembly including the corrected positions of the 3D objects. [Figure 8] FIG. 1 is a block diagram of a computer system suitable for carrying out methods according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] In a first step of the method (step S1 of the flow chart of FIG. 1), a first (digitally modeled) 3D object OBJ1 is provided. The first object OBJ1 exists in a (computer-generated) 3D scene according to a 3D position. It can be created by the user or it can already exist in the 3D scene before the user starts the manipulation.
[0026] In a second step (step S2 of the flowchart in FIG. 1), a projection of the first 3D object OBJ1 is rendered on the screen of the user's electronic computing device (computer, laptop, tablet, smartphone), as shown in FIG. The image IM contains an assembly to be modeled in 3D. In this case, the 2D image comprises a table made of three parts: a flat top, a bottom plate and legs supporting the flat top, which rests on the bottom plate. In the following, the correct positioning of only two 3D objects is disclosed. The positioning of the other 3D objects that make up the assembly is performed in a similar way to the two objects.
[0027] When a user starts with a 2D image, the drawing viewpoint needs to be detected. The method of the present invention is particularly suitable for images that conform to the Manhattan hypothesis, since the drawing viewpoint can be easily detected. The Manhattan hypothesis is disclosed in Non-Patent Document 2. An image conforms to the Manhattan hypothesis if there are three orthogonal main directions in the image, i.e., three main sets of parallel lines.
[0028] Then, after the right drawing viewpoint is obtained, the user can draw a 3D object (also called a 3D shape) on the 2D image based on the three orthogonal principal directions detected for the image that fit the Manhattan hypothesis. A user input of a first 3D object OBJ1 that at least partially fits with a portion of the on-screen 2D image is received.
[0029] A drawing viewpoint can be detected on a 2D image by identifying at least one vanishing point. A vanishing point corresponds to a convergence point of two-dimensional viewpoint projections of mutually parallel lines in a three-dimensional space. Thus, the drawing viewpoint takes the form of a vanishing point. One vanishing point P1 is shown in the right part of Fig. 2. Another one would be identified in the left part of Fig. 2, but is not shown for clarity.
[0030] A first 3D object OBJ1 is drawn according to the drawing viewpoint. In this case, the user draws a rectangle by dragging a pointer (mouse, finger or stylus) from a first point on the screen to another point defined as the opposite corner of the rectangle. The user then performs an extrusion operation on the rectangle to create a parallelepiped. A first object OBJ1 is created, which is a 3D object drawn on top of a 2D image. It corresponds to the flat top surface of the table. The same steps are performed to draw a second object OBJ2. However, the method according to the invention can be performed even for only one 3D object.
[0031] A 3D object OBJ1 (a person in Fig. 3) is provided in the 3D scene, which is rendered by a virtual camera CAM according to a first axis AX1 and a first viewpoint VP1. The first viewpoint VP1 corresponds to the 3D position of the camera. The first axis AX1 connects the first viewpoint VP1 with the centroids of the bounding boxes of all 3D objects located in the 3D scene. In Fig. 3, the 3D scene is considered to contain only one 3D object, namely the 3D object OBJ1.
[0032] Then, in a third step (step S3 of the flow chart of FIG. 1), the user drags (presses without releasing) the first object OBJ1 along the first axis AX1. While dragging the first object OBJ1 along the first axis AX1, in order to keep the projection of the first object OBJ1 constant on the screen SCR, the first object OBJ1 is rescaled.
[0033] The rescaling is done by first rescaling the distance D between the first viewpoint and the center of the smallest bounding box CBB that encloses the first object OBJ1. OldPosition In FIG. 3, before the first object OBJ1 is operated, the distance between the first viewpoint VP1, i.e., the position of the virtual camera, and the first object OBJ1 is calculated as D OldPosition When the user drags the first object OBJ1 along the first axis AX1, the distance between the first viewpoint VP1 and the moved first object OBJ1 is denoted as D NewPosition Furthermore, the first object OBJ1 is located at a distance D OldPosition When the object is far from the first viewpoint VP1, one dimension of the first object OBJ1, for example its height, is H. OldPosition The first object OBJ1 is located at a distance D from the first viewpoint VP1. NewPosition When they are separated by H, the same dimension is NewPosition Dimension H NewPosition is calculated according to the following relationship:
[0034]
number
[0035] The first object OBJ1 is then rescaled.
[0036] When the user drags the first object OBJ1 away from the first viewpoint VP1, the first object OBJ1 becomes larger, and when the user drags the first object OBJ1 closer to the first viewpoint VP1, the first object OBJ1 becomes smaller, so that the projection of the first object OBJ1 remains constant and the user does not need to manually rescale the first object OBJ1 by using the compass.
[0037] In a preferred embodiment, the first 3D object OBJ1 is displayed according to the second viewpoint VP2 and the second axis AX2 of the virtual camera. For this, when the first object OBJ1 is rendered on the screen according to the first viewpoint VP1 and the first axis AX1, as shown in Fig. 2, the user selects a command (for example by clicking a dedicated button from a menu or a contextual button) that triggers the display of the first object OBJ1 according to the second viewpoint VP2 and the second axis AX2 of the virtual camera. Thus, the user can see the rescaling of the first object OBJ1 and can also drag the first object OBJ1 in fine detail.
[0038] Advantageously, the first axis AX1 and the second axis AX2 are mutually perpendicular, in this way the user can easily see the position of the first 3D object OBJ1 on the axis AX1.
[0039] Figure 4 shows the first 3D object OBJ1 displayed according to the second viewpoint VP2. In Figure 4, it can be seen that the first object OBJ1 and the second object OB2 are not correctly aligned. Therefore, the user drags the first object OBJ1 along the first axis AX1 until he assesses that the relative positions of both objects are correct.
[0040] In a preferred embodiment, when the 3D scene is rendered according to the first viewpoint, the user selects the object to be moved by clicking on it, and the moved object is highlighted. For example, the outline of the selected object flashes or the color of the first object OBJ1 object changes after it is selected. If other 3D objects are imported or drawn in the 3D scene, these objects are not highlighted. Thus, even if the object to be moved is partially hidden by other objects in the 3D scene, the user can easily identify it in the 3D scene.
[0041] The first axis AX1 is advantageously represented by a dashed or solid line coinciding with the first axis AX1. Thus, when the user changes the position of the first 3D object OBJ1, the displacement of the first 3D object OBJ1 can be easily predicted. Without the dashed or solid line, it would be difficult for the user to evaluate the position of the first viewpoint compared to the second viewpoint, especially if the first axis AX1 and the second axis AX2 are not orthogonal. It would also be difficult to evaluate the correct orientation of the first axis AX1 relative to the second axis AX2. This embodiment is also useful when the second axis AX2 cannot be orthogonal to the first axis AX1 due to a 3D object hiding the second viewpoint VP2. This can occur when the 3D scene includes a large number of 3D objects.
[0042] 4 shows the displacement of the first object OB1 along the first axis AX1. The first object OBJ1 then becomes closer to the first viewpoint VP1, and the first object OBJ1 becomes smaller. Thus, when the camera is positioned at the first viewpoint V1, the projection of the first object OBJ1 on the screen remains constant.
[0043] In a preferred embodiment, if at least the first and second 3D objects need to be positioned relative to each other, step S3 includes displaying at least one snap point for the second 3D object OBJ2 on the first axis. Hereinafter, a snap point is a graphical icon positioned on the first axis AX1 at a position corresponding to the projection of a characteristic point of the second 3D object OBJ2 on the first axis AX1. Thus, the movement of the first 3D object OBJ1 on the snap point SP is not as fast as on other parts of the first axis AX1, allowing the user to make very fine and precise adjustments of the first 3D object OBJ1 with respect to the second 3D object. For example, when dragging the first 3D object OBJ1, advancing the pointer over the snap point may be slower than on other parts of the first axis AX1. Alternatively, the first object may be fixed there when advancing over the snap point SP (but not completely preventing the user from moving the first 3D object OBJ1).
[0044] The snap point is - the projection on the first axis AX1 of the center of the smallest bounding box that encloses the second 3D object OBJ2, or - the projection on the first axis of the edge of the smallest bounding box that encloses the second 3D object OBJ2 It may be advantageous to correspond
[0045] As shown in Fig. 6, several snap points of the same object may be displayed on the first axis AX1. Thus, the user can choose to align the first 3D object with the center of the smallest bounding box surrounding the second 3D object OBJ2, or with one of the bounding box edges. The type of snap point is configurable.
[0046] The user can navigate in the 3D scene to verify that the first 3D object OBJ1 is positioned as expected, for example by using the combination of pressing the middle mouse button and moving the mouse. For example, the user can reposition the virtual camera according to the first viewpoint, as shown in FIG.
[0047] The user can also switch at any time to the 2D image shown in Figure 2 by clicking (pressing and releasing) in the 3D scene outside the 3D object, thus allowing the user to see that the dimensions of the 3D object remain constant.
[0048] In another embodiment of the invention, the 2D image may be displayed in the main part of the screen while a thumbnail image of the 3D scene from another camera perspective may be displayed so that the user knows when one of the 3D objects needs to be moved and realigned.
[0049] The methods of the present invention may be carried out by a computer system, including a suitably programmed general-purpose computer, or perhaps a computer network, which stores a suitable program in non-volatile form on a computer-readable medium, such as a hard disk, solid state disk, or CD-ROM, and executes the program using its microprocessor and memory.
[0050] A computer CPT suitable for carrying out a method according to an exemplary embodiment of the invention is described with reference to Fig. 8. In Fig. 8, the computer CPT comprises a central processing unit (CPU) P which performs the above-described method steps while operating an executable program, i.e. a set of computer-readable instructions, stored in a memory device such as a RAM M1 or a ROM M2, or a hard disk drive (HDD) M3, a DVD / CD drive M4, or stored remotely. Furthermore, the 3D object created in step S1 of the method may also be stored in one or more memory devices M1 to M4, or remotely.
[0051] The claimed invention is not limited by the form of computer readable media on which the computer readable instructions and / or data structures of the processes of the invention are stored. For example, the instructions and files may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other information processing device with which the computer communicates, such as a server or computer. The programs and files may be stored in the same memory device or in different memory devices.
[0052] Furthermore, computer programs suitable for carrying out the methods of the present invention may be provided as a utility application, a background daemon, or a component of an operating system, or a combination thereof, and run in conjunction with a CPU P and an operating system such as Microsoft VISTA, Microsoft Windows 10, UNIX, Solaris, LINUX, Apple MAC-OS, and other systems known to those skilled in the art.
[0053] The CPU P may be a Xenon processor from Intel Corporation of America, or an Opteron processor from AMD Corporation of America, or other processor types such as a Freescale ColdFire, IMX, or ARM processor from Freescale Corporation of America. Alternatively, the CPU may be a processor such as a Core2 Duo from Intel Corporation of America, or may be implemented in an FPGA, ASIC, PLD, or with discrete logic circuitry as will be appreciated by those skilled in the art. Additionally, the CPU may be implemented as multiple processors operating in cooperation to perform the computer readable instructions of the process of the present invention described above.
[0054] The computer CPT of FIG. 8 also includes a network interface NI, such as an Intel Ethernet PRO network interface card from Intel Corporation, USA, for interfacing with a network, such as a local area network (LAN), a wide area network (WAN), the Internet, and the like. The computer further includes a display controller DC, such as an NVIDIA GeForce GTX graphics adapter from NVIDIA Corporation, USA, for interfacing with a display DY, such as a Hewlett-Packard HPL2445w LCD monitor. The general purpose I / O interface IF interfaces with a keyboard KB, and a pointing device PD, such as a roller ball, mouse, touchpad, and the like. The display, keyboard, and pointing device, together with the display controller and the I / O interface, form a graphical user interface, used by a user to provide input commands, and used by the computer to display 3D objects.
[0055] A disk controller DKC connects the HDD M3 and DVD / CD M4 with a communication bus CBS, which can be ISA, EISA, VESA, PCI, or the like, for interconnecting all of the components of the computer. Descriptions of the general features and functionality of displays, keyboards, pointing devices, and display controllers, disk controllers, network interfaces, and I / O interfaces are omitted from this specification for the sake of brevity, as these features are known.
[0056] Any method steps described herein should be understood to represent a module, segment, or portion of code that comprises one or more executable instructions for performing a particular logical function or step in a process, and alternative implementations are within the scope of the exemplary embodiments of the present invention.
Claims
1. 1. A computer-implemented method for assisting in positioning of a digitally modeled 3D object, the method comprising: S1: providing a first digitally modeled 3D object (OBJ1) having a 3D position in a 3D scene; S2: rendering on a screen a projection of said first digitally modeled 3D object according to a first axis (AX1) and a first viewpoint; S3: upon a user action, automatically scaling the first digitally modeled 3D object (OBJ1) in order to keep the projection of the moved first digitally modeled 3D object constant on the screen while changing the 3D position of the first digitally modeled 3D object (OBJ1) along the first axis (AX1); The method includes:
2. 2. The computer-implemented method of claim 1, wherein step S3 includes displaying the first digitally modeled 3D object (OBJ1) from a second perspective along a second axis, the second perspective and the second axis being different from the first perspective and the first axis, respectively.
3. The method of claim 2 , wherein the first axis and the second axis are mutually orthogonal.
4. 4. The method according to claim 1, wherein the first axis (AX1) is represented by a dashed or solid line, the dashed or solid line coinciding with the first axis.
5. - step S1 comprises providing a second digitally modelled 3D object (OBJ2) having predetermined dimensions and a 3D position in said 3D scene; A computer-implemented method according to any one of claims 1 to 4, wherein step S3 comprises keeping the 3D position and the dimensions of the second digitally modelled 3D object (OBJ2) fixed.
6. 6. The method of claim 5, wherein step S3 includes displaying at least one snap point (SP) relative to the second digitally modeled 3D object (OBJ2) on the first axis to enable rapid positioning of the first digitally modeled 3D object (OBJ1) relative to the second digitally modeled 3D object (OBJ2).
7. The snap point is a projection onto said first axis (AX1) of the center of a minimal bounding box enclosing said second digitally modeled 3D object (OBJ2), or a projection on the first axis of an edge of the smallest bounding box surrounding the second digitally modeled 3D object (OBJ2); The method of claim 6, corresponding to:
8. The method according to any one of claims 5 to 7, wherein step S3 comprises enhancing the first digitally modelled 3D object (OBJ1).
9. 9. The method of claim 1, wherein step S1 comprises receiving a user input for at least partially matching the first digitally modeled 3D object (OBJ1) with a portion of the on-screen 2D image.
10. A computer program stored on a computer readable data storage medium (M1, M2, M3, M4) comprising computer executable instructions for causing a computer system to carry out the method of any one of claims 1 to 9.
11. A computer readable data storage medium (M1, M2, M3, M4) comprising computer executable instructions for causing a computer system to carry out the method according to any one of claims 1 to 9.
12. A computer system comprising a processor (P) coupled to memories (M1, M2, M3, M4) and a screen (DY), the memories storing computer executable instructions for causing the computer system to perform a method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Image processing system, visual axis measuring system, image processing method, visual axis measuring method and program
JP2016143321A
Systems and methods for a modeless 3-d graphics manipulator
US20140040832A1
