3D mesh denoising method and system thereof

Abstract

A three-dimensional mesh noise removal method according to an embodiment of the present disclosure is performed by a computing device and comprises the steps of: generating a first depth map corresponding to each of the plurality of directions by projecting first three-dimensional image data that does not contain noise in a plurality of directions; generating a first normal map corresponding to each of the first depth maps; generating a second depth map corresponding to each of the plurality of directions by projecting second three-dimensional image data, in which a preset amount of noise is added to the first three-dimensional image data, in the plurality of directions; generating a second normal map corresponding to each of the plurality of directions; calculating the first loss function based on a first noise reduction amount for the second depth map predicted by inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model; and a second normal map for the second normal map predicted by inputting the first normal map, the second normal map, and the second normal map into the artificial intelligence model. It may include the step of calculating a second loss function based on the amount of noise reduction, and the step of training the artificial intelligence model so that the sum of the first loss function and the second loss function is minimized.

Description

Technology Field

[0001] The present disclosure relates to a three-dimensional mesh noise removal method and a system thereof, and more specifically, to a method and a system thereof for removing noise using a projection technique on a three-dimensional mesh. Background Technology

[0002] With the advancement of deep learning technology, learning-based 3D mesh noise removal methods have been extensively studied. Typically, learning-based noise removal methods utilize convolutional networks to learn regression functions between noisy and noise-free meshes; these methods have demonstrated higher generalization performance than previous approaches without making specific assumptions about noise or the surface. However, because these methods are designed to learn regression functions directly without considering the probabilistic nature of noise, they had inherent limitations in modeling noise distributions. In other words, these methods tended to produce poor results when noise significantly eroded invisible noise distributions or mesh surface structures. Recent diffusion models have demonstrated superior performance compared to other deep network structures for modeling data distributions and have shown good results not only in image generation but also in 3D point cloud generation. Accordingly, methods for designing loss functions using diffusion models to learn the noise distribution of distorted 3D meshes are being researched. Prior art literature

[0003] Registered Patent Publication No. 10-2534748 (Date of publication: May 22, 2023) The problem to be solved

[0004] The technical problem to be solved through the embodiments of the present disclosure is to provide a method for removing noise from a 3D image by performing noise removal on the depth map and normal map of the 3D image.

[0005] In addition, the technical problem to be solved through the embodiments of the present disclosure is to provide a method for training an artificial intelligence model to predict the amount of noise reduction for a depth map and a normal map. means of solving the problem

[0006] A three-dimensional mesh noise removal method according to an embodiment of the present disclosure is performed by a computing device and comprises the steps of: generating a first depth map corresponding to each of the plurality of directions by projecting first three-dimensional image data that does not contain noise in a plurality of directions; generating a first normal map corresponding to each of the first depth maps; generating a second depth map corresponding to each of the plurality of directions by projecting second three-dimensional image data, in which a preset amount of noise is added to the first three-dimensional image data, in the plurality of directions; generating a second normal map corresponding to each of the plurality of directions; calculating the first loss function based on a first noise reduction amount for the second depth map predicted by inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model; and a second normal map for the second normal map predicted by inputting the first normal map, the second normal map, and the second normal map into the artificial intelligence model. It may include the step of calculating a second loss function based on the amount of noise reduction, and the step of training the artificial intelligence model so that the sum of the first loss function and the second loss function is minimized.

[0007] In one embodiment, the step of calculating the first loss function may include the step of calculating the difference between the first depth map and the second depth map, and the step of calculating the difference between the calculated difference and the predicted first noise reduction amount as the first loss function.

[0008] In one embodiment, the step of calculating the second loss function may include the step of calculating the difference between the first normal map and the second normal map, and the step of calculating the difference between the calculated difference and the predicted second noise reduction amount as the second loss function.

[0009] In one embodiment, the plurality of second vertices constituting the second three-dimensional image data are calculated by adding the preset amount of noise to the plurality of first vertices constituting the first three-dimensional image data, and the preset amount of noise follows a normal distribution.

[0010] In one embodiment, the step of training the artificial intelligence model may include the step of minimizing the sum of the first loss function and the second loss function using the gradient descent method.

[0011] A three-dimensional mesh noise removal method according to another embodiment of the present disclosure is performed by a computing device and may include the steps of: projecting first three-dimensional image data containing noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; generating a first normal map corresponding to the first depth map; inputting the first depth map and the first normal map into an artificial intelligence model to predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map; and applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively, to remove noise from the first three-dimensional image data.

[0012] In one embodiment, the step of removing noise may include the step of generating a second depth map by applying the first noise reduction amount to the first depth map, the step of generating a second normal map by applying the second noise reduction amount to the first normal map, and the step of generating second three-dimensional image data with noise removed based on the second depth map and the second normal map.

[0013] In one embodiment, the step of generating the second three-dimensional image data may include: generating a third depth map and a third normal map corresponding to the third three-dimensional image data generated by reflecting a pre-set vertex change amount in the first three-dimensional image data; adjusting the vertex change amount so that the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map are minimized; and generating the second three-dimensional image data by reflecting the adjusted vertex change amount in the first three-dimensional image data.

[0014] A three-dimensional mesh noise removal system according to an embodiment of the present disclosure comprises a processor and a memory storing instructions, wherein when the instructions are executed by the processor, the processor performs the following operations: projecting first three-dimensional image data that does not contain noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; generating a first normal map corresponding to each of the first depth maps; projecting second three-dimensional image data, in which a preset amount of noise is added to the first three-dimensional image data, in the plurality of directions to generate a second depth map corresponding to each of the plurality of directions; generating a second normal map corresponding to each of the second depth maps; inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model to calculate the first loss function based on a first noise reduction amount for the predicted second depth map; and the first normal map, the second The normal map and the second normal map can be input into the artificial intelligence model to calculate a second loss function based on a second noise reduction amount for the predicted second normal map, and the artificial intelligence model can be trained such that the sum of the first loss function and the second loss function is minimized.

[0015] In one embodiment, the operation of calculating the first loss function may include the operation of calculating the difference between the first depth map and the second depth map, and the operation of calculating the difference between the calculated difference and the predicted first noise reduction amount as the first loss function.

[0016] In one embodiment, the operation of calculating the second loss function may include the operation of calculating the difference between the first normal map and the second normal map, and the operation of calculating the difference between the calculated difference and the predicted second noise reduction amount as the second loss function.

[0017] In one embodiment, the plurality of second vertices constituting the second three-dimensional image data are calculated by adding the preset amount of noise to the plurality of first vertices constituting the first three-dimensional image data, and the preset amount of noise follows a normal distribution.

[0018] In one embodiment, the operation of training the artificial intelligence model may include minimizing the sum of the first loss function and the second loss function using the gradient descent method.

[0019] A three-dimensional mesh noise removal system according to another embodiment of the present disclosure includes a processor and a memory for storing instructions, and when the instructions are executed by the processor, the processor may perform the following operations: projecting first three-dimensional image data containing noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; generating a first normal map corresponding to each of the first depth maps; inputting the first depth map and the first normal map into an artificial intelligence model to predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map; and applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively, to remove noise from the first three-dimensional image data.

[0020] In one embodiment, the noise removal operation may include the operation of generating a second depth map by applying the first noise reduction amount to the first depth map, the operation of generating a second normal map by applying the second noise reduction amount to the first normal map, and the operation of generating a second three-dimensional image data with noise removed based on the second depth map and the second normal map.

[0021] In one embodiment, the operation of generating the second three-dimensional image data may include: generating a third depth map and a third normal map corresponding to the third three-dimensional image data generated by reflecting a pre-set vertex change amount in the first three-dimensional image data; adjusting the vertex change amount so as to minimize the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map; and generating the second three-dimensional image data by reflecting the adjusted vertex change amount in the first three-dimensional image data. Effects of the invention

[0022] The present invention can remove noise from a 3D image by performing noise removal on the depth map and normal map of the 3D image.

[0023] In addition, an artificial intelligence model can be trained to predict the amount of noise reduction for depth maps and normal maps. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram showing an exemplary configuration of a three-dimensional mesh noise removal system according to an embodiment of the present disclosure. FIG. 2 exemplarily shows a depth map and a normal map according to an embodiment of the present disclosure. FIG. 3 illustrates a three-dimensional mesh noise removal method based on a diffusion model according to an embodiment of the present disclosure. FIG. 4 conceptually illustrates a three-dimensional mesh noise removal operation according to an embodiment of the present disclosure. FIG. 5 is a flowchart exemplarily illustrating a three-dimensional mesh noise removal method according to an embodiment of the present disclosure. Figure 6 is a flowchart specifically illustrating the steps for calculating the first loss function of Figure 5. Figure 7 is a flowchart specifically illustrating the steps for calculating the second loss function of Figure 5. FIG. 8 is a flowchart exemplarily illustrating a three-dimensional mesh noise removal method according to another embodiment of the present disclosure. Figure 9 is a flowchart specifically illustrating the steps for removing noise in Figure 8. FIG. 10 is a flowchart specifically illustrating the steps for generating the second three-dimensional image data of FIG. 9. FIG. 11 is a block diagram showing the hardware configuration of a computing device that performs three-dimensional mesh noise removal according to an embodiment of the present disclosure. Specific details for implementing the invention

[0025] Preferred embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments described below but may be implemented in various different forms. The embodiments are provided merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention, and the present disclosure is defined only by the scope of the claims.

[0026] It should be noted that when assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0027] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0028] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are intended only to distinguish the components from other components, and the nature, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "coupled," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be "connected," "coupled," or "joined" between each component.

[0029] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0030] FIG. 1 is a block diagram showing an exemplary configuration of a three-dimensional mesh noise removal system (10) according to an embodiment of the present disclosure. Referring to FIG. 1, the three-dimensional mesh noise removal system (10) may include a depth map generation module (100), a normal map generation module (200), and a noise removal module (300). For convenience, the three-dimensional mesh noise removal system (10) will be referred to as the system (10) below.

[0031] A system (10) according to an embodiment of the present disclosure can receive three-dimensional image data representing a three-dimensional object and can remove noise. The three-dimensional image data is modeled using a three-dimensional mesh, and removing noise from the three-dimensional image data means removing noise from the three-dimensional mesh. Therefore, in this specification, three-dimensional image data and three-dimensional mesh may be used as the same concept. The three-dimensional image data may include a plurality of vertices, and the three-dimensional image data can also be adjusted through the adjustment of each vertex.

[0032] In particular, the system (10) according to the embodiment of the present disclosure can, instead of applying noise removal directly to the 3D image data, divide the 3D image data into a 2D depth map and a normal map to remove noise, and then use the noise-removed depth map and normal map to generate noise-removed 3D image data.

[0033] Here, a depth map refers to a 2D image obtained for each of multiple directions by projecting 3D image data in multiple directions (i.e., projecting 3D image data onto a plane existing in each of the multiple directions). In other words, a depth map can correspond to a 2D image obtained by capturing 3D image data from multiple directions. A depth map can represent the distance between the corresponding plane and the 3D image data. A normal map is a 2D image obtained by mapping a normal corresponding to each pixel of the depth map, and can correspond to the curvature of the depth map. The depth map and the normal map will be explained with reference to FIG. 2.

[0034] FIG. 2 illustrates an exemplary depth map and a normal map according to an embodiment of the present disclosure. Referring to FIG. 2, various depth maps and normal maps for three-dimensional image data corresponding to a rabbit are shown. Specifically, on the left side of FIG. 2, three-dimensional image data and a plurality of planes existing in a plurality of directions from said three-dimensional image data are shown. The depth map may correspond to the result of projecting the three-dimensional image data onto the said plane, or the result of capturing the three-dimensional image data on the said plane, and the normal map may correspond to the result of mapping a normal corresponding to each pixel of the depth map. For example, reference numeral 24 may represent some of the depth maps and corresponding normal maps projected onto a plurality of planes located at reference numeral 21, reference numeral 25 may represent some of the depth maps and corresponding normal maps projected onto a plurality of planes located at reference numeral 22, and reference numeral 26 may represent some of the depth maps and corresponding normal maps projected onto a plurality of planes located at reference numeral 23.

[0035] Returning to FIG. 1, the depth map generation module (100) can generate a depth map corresponding to three-dimensional image data, the normal map generation module (200) can generate a normal map corresponding to the depth map, and the noise removal module (300) can apply noise removal to the depth map and the normal map. Specifically, the noise removal module (300) may include an artificial intelligence model (310) for predicting the amount of noise reduction for the depth map and the normal map. The artificial intelligence model (310) can receive the depth map and the normal map as input and output the amount of noise reduction, and the noise removal module (300) can generate a depth map and a normal map with noise removed by applying the predicted amount of noise reduction to the depth map and the normal map.

[0036] The noise removal described above may be performed in stages rather than all at once. That is, the noise reduction amount described above may be applied in stages, and as each stage is performed, the noise contained in the 3D image data may be gradually reduced. The noise removal method for depth maps and normal maps according to an embodiment of the present disclosure is based on a diffusion model, which is widely used as a noise removal method for 2D images, and each of the described stages may correspond to a timestep of the diffusion model.

[0037] Below, we examine an example of training an artificial intelligence model (300) and an example of predicting the amount of noise reduction for a depth map and a normal map using the artificial intelligence model (300).

[0038] First, an embodiment for training an artificial intelligence model (300) will be described. First, the depth map generation module (100) may receive 3D image data that does not contain any noise for training the artificial intelligence model (300). Then, the received 3D image data may be projected in multiple directions to generate a first depth map corresponding to each of the multiple directions. Then, the normal map generation module (200) may generate a first normal map corresponding to each first depth map. Here, the 3D image data that does not contain noise If so, the first depth map is It can be expressed as such, and the first normal map is It can be expressed as follows. is an operator for projecting 3D image data onto the i-th direction among multiple directions, and is an operator for calculating the normal vector for a depth map.

[0039] After that, the depth map generation module (100) has 3D image data that does not contain noise The amount of noise set in 3D image data with added The second depth map corresponding to It can generate. Here, t corresponds to the time step of the diffusion model, and can be a hyperparameter representing the diffusion rate at each time step. And, a preset amount of noise It can follow a normal distribution. That is, 3D image data The multiple vertices constituting are 3D image data A pre-set amount of noise at multiple vertices constituting It can be calculated by adding them. Subsequently, the normal map generation module (200) generates a second normal map corresponding to each second depth map. It can generate.

[0040] Next, the noise reduction module (300) is the first depth map described above , second depth map Using this, a first loss function for training the artificial intelligence model (310) can be calculated. At this time, the artificial intelligence model (310) is a second depth map It can predict the amount of noise reduction to be applied by taking it as input. 3D image data with a pre-set amount of noise added is 3D image data corresponding to time step t It can be represented as follows. The artificial intelligence model (310) If so, the second depth map at time step t The amount of noise reduction to be applied to is It can be expressed as such. Therefore, a first loss function for training an artificial intelligence model (310) in relation to noise removal for a depth map It can be calculated as shown in mathematical formula 1 below.

[0041]

[0042] That is, the first loss function represents the difference between a preset amount of noise included in arbitrary 3D image data and the amount of noise reduction predicted for the depth map, and the smaller the first loss function, the more accurately the amount of noise reduction for the depth map is predicted.

[0043] And, similar to the first loss function, a second loss function for training an artificial intelligence model (310) with respect to noise removal for a normal map It can be calculated as shown in mathematical formula 2 below.

[0044]

[0045] That is, the second loss function represents the difference between a preset amount of noise included in arbitrary 3D image data and the amount of noise reduction predicted for the normal map, and the smaller the second loss function, the more accurately the amount of noise reduction for the normal map is predicted.

[0046] In conclusion, the noise removal module (300) can train the artificial intelligence model (310) so that the sum of the first loss function and the second loss function described above is minimized. For example, the noise removal module (300) can minimize the sum of the first loss function and the second loss function by using the gradient descent method as shown in Equation 3 below.

[0047]

[0048] For the artificial intelligence model (310) trained in this way, if a depth map and a normal map corresponding to time step t are input, a noise reduction amount that becomes a depth map and a normal map corresponding to time step t-1 can be output.

[0049] Secondly, an example of predicting a noise reduction amount using an artificial intelligence model (300) is described. The depth map generation module (100) can generate a first depth map corresponding to each of the multiple directions by projecting the input 3D image data in multiple directions, and the normal map generation module (200) can generate a first normal map corresponding to the generated first depth map. Then, the artificial intelligence model (310) can receive the generated first depth map and the first normal map and predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map. After that, the noise removal module (300) can remove noise from the input 3D image data by applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively.

[0050] More specifically, the noise removal module (300) can generate a second depth map by applying a first noise reduction amount to a first depth map, and can generate a second normal map by applying a second noise reduction amount to a first normal map. For example, the first depth map and the first normal map may be a depth map and a normal map corresponding to time step t, and the second depth map and the second normal map may be a depth map and a normal map corresponding to time step t-1. After that, the noise removal module (300) can generate noise-removed 3D image data based on the second depth map and the second normal map.

[0051] For example, the noise removal module (300) can generate arbitrary 3D image data by reflecting a preset vertex change amount for the initially input 3D image data, and can generate a third depth map and a third normal map corresponding to the arbitrarily generated 3D image data. After that, the noise removal module (300) can adjust the vertex change amount so that the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map are minimized, and finally, the adjusted vertex change amount can be reflected in the input 3D image data to generate noise-removed 3D image data.

[0052] The system (10) may be implemented with at least one computing device. For example, all functions of the system (10) may be implemented in a single computing device, or a first function of the system (10) may be implemented in a first computing device and a second function may be implemented in a second computing device. Alternatively, specific functions of the system (10) may be implemented in multiple computing devices. For example, computing devices may include all types of devices equipped with computing means and communication means, such as smartphones, desktops, laptops, and tablets. Furthermore, the system (10) may be configured using one or more physical servers included in a server farm based on cloud technology, such as a virtual machine. An embodiment in which the system (10) is implemented as a computing device is described with reference to FIG. 11.

[0053] FIG. 3 illustrates a three-dimensional mesh noise removal method based on a diffusion model according to an embodiment of the present disclosure. For reference, FIG. 3 is illustrated only with respect to a depth map, but it should be noted that the same removal method can be applied to a normal map as well. FIG. 3 is three-dimensional image data Depth map corresponding to Generates and performs denoising on the depth map stepwise (i.e., up to time steps T, …, t, t-1, … 0) to obtain 3D image data with completely removed noise. This illustrates the process of generating the depth map. In the depth map shown in Fig. 3, each row represents a depth map corresponding to a different direction, and each column represents a depth map corresponding to a different time step. Finally, the noise-removed depth map. When it is generated, the final 3D image data is obtained by adjusting the vertex change amount as described with reference to Fig. 1. can be generated. In Fig. 3, represents reverse diffusion in the diffusion model, and represents forward diffusion.

[0054] FIG. 4 conceptually illustrates a three-dimensional mesh noise removal operation according to an embodiment of the present disclosure. Referring to FIG. 4, a normal map corresponding to various directions of three-dimensional image data from which noise is to be removed is shown. Additionally, it is shown that the noise of the normal map decreases as the time step approaches 0. That is, FIG. 4 corresponds to the result of performing noise reduction 100 times from time step 100 to 0.

[0055] FIG. 5 is a flowchart exemplarily illustrating a three-dimensional mesh noise removal method according to an embodiment of the present disclosure. FIG. 5 and the subsequent FIGS. 6 through 10 illustrate steps / operations of a three-dimensional mesh noise removal method performed in the system (10) of FIG. 1. Accordingly, in the following descriptions, if the subject of a specific step / operation is omitted, it can be understood as being performed in the system (10). For example, FIG. 5 may correspond to an embodiment for training an artificial intelligence model to predict the amount of noise reduction for a depth map and a normal map.

[0056] In step S110, a first depth map corresponding to the first three-dimensional image data may be generated. Then, in step S120, a first normal map corresponding to the first depth map may be generated. In step S130, a second depth map corresponding to the second three-dimensional image data, to which a preset amount of noise has been added to the first three-dimensional image data, may be generated. Then, in step S140, a second normal map corresponding to the second depth map may be generated.

[0057] In step S150, a first loss function can be calculated based on a first noise reduction amount predicted by inputting a first depth map, a second depth map, and a second depth map into an artificial intelligence model. Similarly, in step S160, a second loss function can be calculated based on a second noise reduction amount predicted by inputting a first normal map, a second normal map, and a second normal map into an artificial intelligence model.

[0058] Steps S150 to S160 below will be explained with reference to FIGS. 6 to 7.

[0059] FIG. 6 is a flowchart specifically illustrating the step (S150) of calculating the first loss function of FIG. 5. Referring to FIG. 6, in step S151, the difference between the first depth map and the second depth map can be calculated. Then, in step S152, the difference between the calculated difference and the predicted first noise reduction amount can be calculated as the first loss function. For example, steps S151 to S152 may correspond to the above-described mathematical formula 1.

[0060] FIG. 7 is a flowchart specifically illustrating the step (S160) of calculating the second loss function of FIG. 5. Referring to FIG. 7, in step S161, the difference between the first normal map and the second normal map can be calculated. Then, in step S162, the difference between the calculated difference and the predicted second noise reduction amount can be calculated as the second loss function. For example, steps S161 to S162 may correspond to Equation 2 described above.

[0061] Returning to Fig. 5, in step S170, the artificial intelligence model can be trained so that the sum of the first loss function and the second loss function is minimized. For example, the sum of the first loss function and the second loss function can be minimized using gradient descent.

[0062] FIG. 8 is a flowchart exemplarily illustrating a three-dimensional mesh noise removal method according to another embodiment of the present disclosure. For example, FIG. 8 may correspond to an embodiment in which a depth map and a normal map are input into an artificial intelligence model to predict the amount of noise reduction.

[0063] In step S210, a first depth map corresponding to first three-dimensional image data containing noise may be generated. Then, in step S220, a first normal map corresponding to the first depth map may be generated. In step S230, the first depth map may be input into an artificial intelligence model to predict a first noise reduction amount for the first depth map, and the first normal map may be input into an artificial intelligence model to predict a second noise reduction amount for the first normal map. Then, in step S240, the first noise reduction amount and the second noise reduction amount are applied to the first depth map and the first normal map, respectively, so that noise can be removed from the first three-dimensional image data. Step S240 will be explained below with reference to FIG. 9.

[0064] FIG. 9 is a flowchart specifically illustrating the noise removal step (S240) of FIG. 8. Referring to FIG. 9, in step S241, a first noise reduction amount may be applied to a first depth map to generate a second depth map. In step S242, a second noise reduction amount may be applied to a first normal map to generate a second normal map. Subsequently, in step S243, noise-removed second 3D image data may be generated based on the second depth map and the second normal map. Step S243 will be explained below with reference to FIG. 10.

[0065] FIG. 10 is a flowchart specifically illustrating the step (S243) of generating the second three-dimensional image data of FIG. 9. Referring to FIG. 10, in step S243-1, a third depth map and a third normal map corresponding to the third three-dimensional image data generated by reflecting a pre-set vertex change amount in the first three-dimensional image data can be generated. Next, in step S243-2, the pre-set vertex change amount can be adjusted differently so that the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map are minimized. Finally, in step S243-3, the adjusted vertex change amount can be reflected in the first three-dimensional image data to generate the second three-dimensional image data.

[0066] FIG. 11 is a block diagram showing the hardware configuration of a computing device (500) that performs three-dimensional mesh noise removal according to an embodiment of the present disclosure.

[0067] A computing device (500) may include one or more processors (510), a bus (530), a communication interface (540), a memory (520) for loading a computer program executed by the processor (510), and a storage (550) for storing a computer program (560). However, only components related to the embodiments of the present disclosure are illustrated in FIG. 11. Therefore, a person skilled in the art to which the present disclosure belongs will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 11. That is, the computing device (500) may include various additional components in addition to the components illustrated in FIG. 11. Furthermore, depending on the case, the computing device (500) may be configured in a form in which some of the components illustrated in FIG. 11 are omitted. Each component of the computing device (500) will be described below.

[0068] The processor (510) can control the overall operation of each component of the computing device (500). The processor (510) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the art of the present disclosure. Additionally, the processor (510) may perform operations for at least one application or program for executing operations / methods according to embodiments of the present disclosure. The computing device (500) may have one or more processors.

[0069] Next, the memory (520) may store various data, commands and / or information. The memory (520) may load a computer program (560) from the storage (550) to execute an operation / method according to the embodiments of the present disclosure. The memory (520) may be implemented as a volatile memory such as RAM, but the present disclosure is not limited thereto.

[0070] Next, the bus (530) can provide communication functions between components of the computing device (500). The bus (530) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0071] Next, the communication interface (540) may support wired and wireless internet communication of the computing device (500). Additionally, the communication interface (540) may support various communication methods other than internet communication. To this end, the communication interface (540) may be configured to include a communication module well known in the art of the present disclosure.

[0072] Next, the storage (550) may store one or more computer programs (560) non-temporarily. The storage (550) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.

[0073] Next, the computer program (560) may include one or more instructions that cause the processor (510) to perform an operation / method according to various embodiments of the present disclosure when loaded into memory (520). That is, the processor (510) may perform an operation / method according to various embodiments of the present disclosure by executing one or more loaded instructions.

[0074] For example, the computer program (560) performs the operation of generating a first depth map corresponding to each of the plurality of directions by projecting first three-dimensional image data that does not contain noise in a plurality of directions, the operation of generating a first normal map corresponding to each of the first depth maps, the operation of generating a second depth map corresponding to each of the plurality of directions by projecting second three-dimensional image data in which a preset amount of noise has been added to the first three-dimensional image data in a plurality of directions, the operation of generating a second normal map corresponding to each of the second depth maps, the operation of calculating a first loss function based on a first noise reduction amount for the predicted second depth map by inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model, the operation of calculating a second loss function based on a second noise reduction amount for the predicted second normal map by inputting the first normal map, the second normal map, and the second normal map into an artificial intelligence model, and the first loss function and It may include instructions that perform an operation to train an artificial intelligence model so that the sum of the second loss function is minimized.

[0075] Additionally, the computer program (560) may include instructions for performing operations such as projecting first three-dimensional image data containing noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions, generating a first normal map corresponding to each first depth map, inputting the first depth map and the first normal map into an artificial intelligence model to predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map, and applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively, to remove noise from the first three-dimensional image data.

[0076] According to an embodiment of the present disclosure, noise can be removed without deforming the structure of a three-dimensional mesh. That is, an artificial intelligence model according to an embodiment of the present disclosure can efficiently remove noise contained in a three-dimensional mesh by learning to predict the amount of noise reduction for a two-dimensional image regardless of the mesh topology.

[0077] Various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 11. The effects according to the technical concept of the present disclosure are not limited to those described above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0078] In the foregoing, although all components constituting an embodiment of the present disclosure have been described as being combined or operating together, the present disclosure is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present disclosure, all components may be selectively combined and operated in one or more ways.

[0079] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0080] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present disclosure shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present disclosure.

Claims

Claim 1 A method for removing noise from a three-dimensional mesh performed by a computing device, comprising: a step of generating a first depth map corresponding to each of the plurality of directions by projecting a first three-dimensional image data that does not contain noise in a plurality of directions; a step of generating a first normal map corresponding to each of the first depth maps; a step of generating a second depth map corresponding to each of the plurality of directions by projecting a second three-dimensional image data, in which a preset amount of noise is added to the first three-dimensional image data, in the plurality of directions; a step of generating a second normal map corresponding to each of the second depth maps; a step of calculating a first loss function based on a first noise reduction amount for the second depth map predicted by inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model; and a second noise reduction amount for the second normal map predicted by inputting the first normal map, the second normal map, and the second normal map into the artificial intelligence model. A three-dimensional mesh noise removal method comprising: a step of calculating a second loss function based on; and a step of training an artificial intelligence model such that the sum of the first loss function and the second loss function is minimized. Claim 2 delete Claim 3 delete Claim 4 A 3D mesh noise removal method according to claim 1, wherein a plurality of second vertices constituting the second 3D image data are calculated by adding a preset amount of noise to a plurality of first vertices constituting the first 3D image data, and the preset amount of noise follows a normal distribution. Claim 5 A three-dimensional mesh noise removal method according to claim 1, wherein the step of training the artificial intelligence model includes the step of minimizing the sum of the first loss function and the second loss function using a gradient descent method. Claim 6 A three-dimensional mesh noise removal method performed by a computing device, comprising: a step of projecting first three-dimensional image data containing noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; a step of generating a first normal map corresponding to the first depth map; a step of inputting the first depth map and the first normal map into an artificial intelligence model to predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map; and a step of removing noise from the first three-dimensional image data by applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively. Claim 7 A three-dimensional mesh noise removal method according to claim 6, wherein the step of removing noise comprises: a step of generating a second depth map by applying the first noise reduction amount to the first depth map; a step of generating a second normal map by applying the second noise reduction amount to the first normal map; and a step of generating noise-removed second three-dimensional image data based on the second depth map and the second normal map. Claim 8 In claim 7, the step of generating the second three-dimensional image data comprises: generating a third depth map and a third normal map corresponding to the third three-dimensional image data generated by reflecting a pre-set vertex change amount in the first three-dimensional image data; adjusting the vertex change amount so as to minimize the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map; and generating the second three-dimensional image data by reflecting the adjusted vertex change amount in the first three-dimensional image data, a three-dimensional mesh noise removal method. Claim 9 processor; The method includes a memory for storing instructions, and when the instructions are executed by the processor, the processor performs the following operations: projecting first three-dimensional image data, which is free of noise, in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; generating a first normal map corresponding to each of the first depth maps; projecting second three-dimensional image data, in which a preset amount of noise is added to the first three-dimensional image data, in the plurality of directions to generate a second depth map corresponding to each of the plurality of directions; generating a second normal map corresponding to each of the second depth maps; inputting the first depth map, the second depth map, and the second depth map into an artificial intelligence model to calculate a first loss function based on a first noise reduction amount for the predicted second depth map; and inputting the first normal map, the second normal map, and the second normal map into the artificial intelligence model A three-dimensional mesh noise removal system comprising: an operation of calculating a second loss function based on a second noise reduction amount for the second normal map predicted by input; and an operation of training the artificial intelligence model such that the sum of the first loss function and the second loss function is minimized. Claim 10 delete Claim 11 delete Claim 12 A 3D mesh noise removal system according to claim 9, wherein a plurality of second vertices constituting the second 3D image data are calculated by adding a preset amount of noise to a plurality of first vertices constituting the first 3D image data, and the preset amount of noise follows a normal distribution. Claim 13 A three-dimensional mesh noise removal system according to claim 9, wherein the operation of training the artificial intelligence model includes the operation of minimizing the sum of the first loss function and the second loss function using a gradient descent method. Claim 14 A 3D mesh noise removal system comprising: a processor; and a memory for storing instructions, wherein when the instructions are executed by the processor, the processor performs the operation of: projecting first 3D image data containing noise in a plurality of directions to generate a first depth map corresponding to each of the plurality of directions; generating a first normal map corresponding to each of the first depth maps; inputting the first depth map and the first normal map into an artificial intelligence model to predict a first noise reduction amount for the first depth map and a second noise reduction amount for the first normal map; and applying the first noise reduction amount and the second noise reduction amount to the first depth map and the first normal map, respectively, to perform the operation of removing noise from the first 3D image data. Claim 15 In claim 14, the noise removal operation comprises: an operation of generating a second depth map by applying the first noise reduction amount to the first depth map; an operation of generating a second normal map by applying the second noise reduction amount to the first normal map; and an operation of generating noise-removed second three-dimensional image data based on the second depth map and the second normal map, a three-dimensional mesh noise removal system. Claim 16 In claim 15, the operation of generating the second three-dimensional image data comprises: generating a third depth map and a third normal map corresponding to the third three-dimensional image data generated by reflecting a pre-set vertex change amount in the first three-dimensional image data; adjusting the vertex change amount so as to minimize the difference between the second depth map and the third depth map and the difference between the second normal map and the third normal map; and generating the second three-dimensional image data by reflecting the adjusted vertex change amount in the first three-dimensional image data, a three-dimensional mesh noise removal system.

Images