Method for generating 3D modeling data with reduced operation time

Abstract

The present specification discloses a method that can generate 3D modeling data with improved speed compared to conventional methods. The method for generating 3D modeling data according to the present specification may comprise the steps in which: (a) a processor stores an image (hereinafter referred to as "3D image data") having a depth value for an object (hereinafter referred to as "3D modeling object," for which a 3D model is to be constructed; (b) the processor configures 3D type data by using the 3D image data (c) the processor initializes a neural rendering-based machine learning model by using the 3D type data; and (d) the processor inputs image data of the 3D modeling object to the initialized neural rendering-based machine learning model.

Description

A method for generating 3D modeling data with reduced computation time

[0001] The present invention relates to 3D modeling, and more particularly, to a method for generating 3D modeling data.

[0002] The material described in this section merely provides background information on the embodiments described herein and does not necessarily constitute prior art.

[0003] NeRF (Neural Radiance Fields) receives a discontinuous 2D image as input, predicts the shape in a 3D coordinate space (X, Y, Z) so that the image can be composed continuously, and creates a new image at an arbitrary point in time using the predicted 3D information. For example, if it receives several images taken from certain directions, such as the front, back, left, and right of an object, NeRF can predict the color and transparency in the 3D coordinate space (X, Y, Z) and use this to create an image at a new point in time. In addition, the predicted 3D information can be used as 3D modeling data.

[0004] Neural rendering techniques such as NeRF cast a ray from each pixel of the image frames into 3D space, sample the coordinates of the 3D space along the ray, and input these into a coordinate-based network to obtain 3D information such as radiance fields and signed distance functions, and then render them, and train using the difference between the actual value of each pixel and the rendered value.

[0005] Figure 1 is an example of the initial values ​​of 3D modeling data used in the neural rendering technique.

[0006] Referring to Figure 1, since the data input to neural rendering is unknown, an initial value, such as a spherical shape (a) or a random shape (b), is provided. Then, based on this initial value, the rendering process proceeds using the aforementioned difference value. However, this machine learning technique suffers from the problem of long processing times due to the computational complexity of the neural rendering process.

[0007] The purpose of this specification is to provide a method for generating 3D modeling data with improved speed compared to conventional methods.

[0008] This specification is not limited to the above-mentioned tasks, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] A method for generating 3D modeling data according to the present specification for solving the above-described problem may include: (a) a step in which a processor stores an image (hereinafter referred to as "3D image data") having a depth value for an object to be constructed as a 3D model (hereinafter referred to as "3D modeling object"); (b) a step in which the processor constructs 3D shape data using the 3D image data; (c) a step in which the processor initializes a neural rendering-based machine learning model using the 3D shape data; and (d) a step in which the processor inputs image data of the 3D modeling object into the initialized neural rendering-based machine learning model.

[0010] According to one embodiment of the present specification, the 3D image data may be generated from a 3D camera that photographs a 3D modeling target.

[0011] According to one embodiment of the present specification, the neural rendering-based machine learning model may be Neural Radiance Fields (NeRF).

[0012] According to one embodiment of the present specification, step (b) may be a step in which the processor constructs 3D shape data from 3D image data using at least one algorithm among real-time 3D reconstruction, Volumetric TSDF fusion, and BundleFusion.

[0013] According to one embodiment of the present specification, the step (c) may be a step in which the processor sets the 3D shape data as the correct answer and initializes it through a loss function that compares the difference between the value output from the neural rendering-based machine learning model and the correct answer.

[0014] The method for generating 3D modeling data according to the present specification can be implemented in the form of a computer program written to perform each step on a computer and recorded on a computer-readable recording medium.

[0015] Other specific details of the present invention are included in the detailed description and drawings.

[0016] According to this specification, 3D modeling data can be generated at a faster speed than conventional methods.

[0017] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0018] Figure 1 is an example of the initial values ​​of 3D modeling data used in the neural rendering technique.

[0019] Figure 2 is a schematic flowchart of a 3D modeling method according to the present specification.

[0020] Figure 3 is an example of 3D shape data.

[0021] Figure 4 is an example of an initialized neural rendering-based machine learning model.

[0022] Figure 5 is an example of the final generated 3D modeling data.

[0023] Figure 6 is a comparative example of a product produced by a conventional method and a method according to the present specification.

[0024] The advantages and features of the invention disclosed in this specification, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, this specification is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of this specification is complete and to fully inform those of ordinary skill in the art (hereinafter referred to as "skilled workers") of the scope of this specification, and the scope of rights of this specification is defined only by the scope of the claims.

[0025] The terminology used herein is for the purpose of describing embodiments and is not intended to limit the scope of the present disclosure. In this specification, singular forms also include plural forms, unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0026] Throughout the specification, the same reference numerals refer to the same elements, and the term "and / or" includes each and every combination of the elements mentioned. Although terms such as "first," "second," etc. are used to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. Therefore, it should be understood that a first element mentioned below may also be a second element within the technical scope of the present invention.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those skilled in the art to which this specification pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0028] The terms used in this specification are defined as follows.

[0029] As used herein, "3D modeling target" refers to an object to be constructed as a 3D model. The 3D modeling target may be a specific object, person, animal, or space, and the target is not particularly limited.

[0030] In this specification, a "3D camera" is a device capable of outputting a depth value for each pixel. While a typical camera outputs RGB values ​​for each pixel, a 3D camera can further output a depth value for each pixel, which is the distance value between the actual object and the camera. 3D cameras can be of various types, such as stereo, LiDAR, pattern projection, and ToF, and are not specifically limited by the type or method of the 3D camera in this specification.

[0031] In this specification, "3D image data" refers to image data having a depth value. In the case of images captured by the 3D camera described above, image data having a depth value can be generated. However, 3D image data in this specification is not necessarily limited to those captured by a 3D camera. Image data generated by capturing 3D modeling created in a virtual space with a virtual camera can also be 3D image data in this specification.

[0032] In this specification, "machine learning model" refers to a system or network in which nodes are connected to form a neural network. Nodes form layers, and multiple layers can form a neural network. Each node may include various parameters. The network may undergo a learning process to perform specific operations, during which the parameter values ​​of each node may be adjusted. A machine learning model is one that can perform this learning process on its own.

[0033] In this specification, "neural rendering-based machine learning model" refers to a machine learning model that generates 3D modeling data using a neural network. An example of a neural rendering-based machine learning model is NeRF (Neural Radiance Fields). NeRF has been previously described, so a repetitive description will be omitted.

[0034] In this specification, "3D modeling data" refers to data expressed as positional information in a three-dimensional coordinate space (X, Y, Z). The 3D image data described above can be distinguished from data having depth information based on a single point (e.g., a camera).

[0035] Hereinafter, a method for generating 3D modeling data according to the present specification will be described with reference to the attached drawings. The method for generating 3D modeling data according to the present specification may include a processor, an application-specific integrated circuit (ASIC), another chipset, a logic circuit, a register, a communication modem, a data processing device, etc. known in the technical field to which the present invention pertains, in order to execute various control logics to be described below. In addition, when the above-described control logic is implemented in software, the method for generating 3D modeling data according to the present specification may be implemented as a set of program modules. In this case, the program modules may be stored in a memory device and executed by the processor. Therefore, the following description will focus on an example in which the method for generating 3D modeling data according to the present specification is executed by a processor. However, the method for generating 3D modeling data according to the present specification is not limited to the examples described below.

[0036] Figure 2 is a schematic flowchart of a 3D modeling method according to the present specification.

[0037] Referring to FIG. 2, first, in step S100, the processor can store 3D image data for a 3D modeling target. The 3D image data can be generated by capturing images with a 3D camera. Furthermore, the 3D image data can be acquired from one or more cameras, and can be a single frame from images captured continuously over time. It is also possible to retrieve previously stored 3D image data.

[0038] In the next step S200, the processor can construct 3D shape data using the 3D image data. The 3D shape data may have an approximate shape of the 3D modeling target. For example, the processor can construct 3D shape data from the 3D image data using an algorithm such as real-time 3D reconstruction, Volumetric TSDF fusion, or BundleFusion. The method by which the processor constructs the 3D shape data is not limited to a specific algorithm, and various algorithms known to those skilled in the art can be used.

[0039] Figure 3 is an example of 3D shape data.

[0040] Referring to FIG. 3, an example of 3D shape data according to the present specification can be seen. The example is 3D shape data constructed using RGB-D data photographed from an office space, and can identify the approximate shapes of spaces and objects. For convenience of explanation in this specification, the 3D shape data is referred to as "G," and G may be in the form of a signed distance function, voxel, etc.

[0041] In the next step S300, the processor can initialize a neural rendering-based machine learning model using the 3D shape data. In this specification, "initialization" means a process of setting the 3D shape data as the correct answer and training the neural rendering-based machine learning model. The processor can initialize each point (P) of the 3D shape data by inputting the output value NN(P) of the neural rendering-based machine learning model (hereinafter referred to as "NN"), which is a coordinate-based network, and comparing the difference between the value G(P) of G at the corresponding P location. At this time, the loss function (loss) can be expressed as in Equation 1 below.

[0042] [Formula 1]

[0043]

[0044] Figure 4 is an example of an initialized neural rendering-based machine learning model.

[0045] In the next step S400, the processor can input image data of the 3D modeling target into the initialized neural rendering-based machine learning model. The input image data may be at least one 2D image data or 3D image data. Preferably, the input image data may be 3D image data used when configuring 3D shape data. As a result of step S400, the neural rendering-based machine learning model can generate 3D modeling data.

[0046] Figure 5 is an example of the final generated 3D modeling data.

[0047] Conventional neural rendering-based machine learning models begin training with initial values ​​unrelated to the 3D modeling target, requiring numerous point sampling operations to generate the final 3D modeling data. However, the 3D modeling data generation method described herein sets the approximate 3D shape data of the 3D modeling target as the initial value, thereby reducing the overall training time compared to conventional methods.

[0048] Figure 6 is a comparative example of a product produced by a conventional method and a method according to the present specification.

[0049] Fig. 6(a) shows 3D modeling data generated by a conventional neural rendering-based machine learning model. It took approximately 17 minutes to generate the result of Fig. 6(a). Fig. 6(b) shows 3D modeling data generated by a 3D modeling data generation method according to the present specification. It took approximately 15 minutes to generate the result of Fig. 6(b). The 3D modeling data generation method according to the present specification is approximately 12% faster than the conventional method, and naturally, the amount of computation is also reduced.

[0050] The method for generating 3D modeling data according to the present specification can be implemented in the form of a computer program written to perform each step on a computer and recorded on a computer-readable recording medium.

[0051] The above computer program may include codes coded in a computer language, such as C / C++, C#, JAVA, Python, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary to execute the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to send and receive during communication.

[0052] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0053] While the embodiments of this specification have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. (a) A step of storing an image (hereinafter referred to as “3D image data”) having a depth value for an object that the processor wants to build as a 3D model (hereinafter referred to as “3D modeling object”); (b) a step in which the processor constructs 3D shape data using the 3D image data; (c) a step in which the processor initializes a neural rendering-based machine learning model using the 3D shape data; and (d) A method for generating 3D modeling data, comprising: a step of inputting image data of the 3D modeling target into a machine learning model based on neural rendering in which the processor is initialized; 2. In claim 1, A method for generating 3D modeling data, characterized in that the above 3D image data is generated from a 3D camera that photographs a 3D modeling target.

3. In claim 1, A method for generating 3D modeling data, characterized in that the above neural rendering-based machine learning model is Neural Radiance Fields (NeRF).

4. In claim 1, A method for generating 3D modeling data, wherein the step (b) is a step in which a processor constructs 3D shape data from 3D image data using at least one algorithm among real-time 3D reconstruction, Volumetric TSDF fusion, and BundleFusion.

5. In claim 1, Step (c) above, A method for generating 3D modeling data, wherein the processor sets the 3D shape data as the correct answer and initializes it through a loss function that compares the difference between the value output from the neural rendering-based machine learning model and the correct answer.

6. A computer program written to perform each step of the method for generating 3D modeling data according to any one of claims 1 to 5 on a computer and recorded on a computer-readable recording medium.

Images