Method for constructing noise removal model, and apparatus and method for removing image noise by using noise removal model

Abstract

A noise removal apparatus for removing noise from an image, according to the present invention, comprises: a memory that stores a noise removal program; and a processor that executes the noise removal program, wherein the noise removal program receives image data including noise, and removes the noise included in the image data by inputting the image data into a noise removal model, the noise removal model removes the noise included in the input image data, and is machine-trained so as to generate a first noise-removed image for a training image including noise, generate a second noise-removed image for a hard image that is generated on the basis of the training image and the first noise-removed image by an adversarial frequency mixing model described below, and enable the first noise-removed image and the second noise-removed image to be the same as a clean image on the basis of a first loss between the first noise-removed image and the clean image corresponding to the training image and a second loss between the second noise-removed image and the clean image, and the adversarial frequency mixing model generates a mask that applies noise by using the training image and the first noise-removed image, and respective frequency domain size values for the training image and the first noise-removed image, and generates the hard image having a low possibility of noise being removed by the noise removal model on the basis of the mask, the training image, and the first noise-removed image.

Description

Method for constructing a noise removal model and an image noise removal device and method using the noise removal model

[0001] The present invention relates to a method for constructing a noise removal model and an image noise removal device and method using the noise removal model.

[0002] Conventional noise removal models, which effectively remove noise from images, effectively remove noise learned through training data. However, they often overfit to the training data and fail to remove new noise. For example, if a noise removal model is trained using training images with Gaussian noise (standard deviation 10) applied, and then tested on image data containing Gaussian noise (standard deviation 15) or naturally occurring noise, the model fails to remove noise encountered for the first time.

[0003] In other words, conventional noise removal models are limited to specific domains and lack generalization. Therefore, a technology to address these issues is needed.

[0004] The present invention provides a noise removal model construction method for training a noise removal model using a hard image generated so that the noise removal model cannot remove noise based on a learning image and a noise removal image for the learning image, and a noise removal device and method using the noise removal model, in order to solve the above-mentioned problem.

[0005] However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

[0006] As a technical means for solving the above-described technical problem, an image noise removal device for removing noise from an image according to one embodiment of the present invention comprises: a memory in which a noise removal program is stored; And a processor executing the noise removal program, wherein the noise removal program receives image data containing noise, inputs the image data into a noise removal model to remove noise contained in the image data, wherein the noise removal model removes noise contained in the input image data, thereby generating a first noise-removed image for a learning image containing noise, and an adversarial frequency mixing model below generates a second noise-removed image for a hard image generated based on the learning image and the first noise-removed image, wherein the first noise-removed image and the second noise-removed image are machine-learned to be the same as the clean image based on a first loss between the first noise-removed image and a clean image corresponding to the learning image and a second loss between the second noise-removed image and the clean image, and the adversarial frequency mixing model generates a mask for applying noise using the frequency domain size values ​​for the learning image and the first noise-removed image and the learning image and the first noise-removed image, and generates a mask for applying noise based on the mask, the learning image, and the first noise-removed image, wherein the noise is less likely to be removed by the noise removal model. It is to create a hard image.

[0007] In addition, a method for removing noise from an image using an image noise removing device according to another embodiment of the present invention comprises the steps of: receiving image data containing noise; and inputting the image data into a noise removing model to remove noise contained in the image data, wherein the noise removing model is machine-learned to remove noise contained in the input image data, the step of generating a first noise removed image for a learning image containing noise; and the step of generating a second noise removed image for a hard image generated based on the learning image and the first noise removed image by an adversarial frequency mixing model. And based on a first loss between the first noise-removed image and the clean image corresponding to the training image and a second loss between the second noise-removed image and the clean image, the first noise-removed image and the second noise-removed image are trained to be the same as the clean image, and the adversarial frequency mixing model generates a mask for applying noise using the frequency domain size values ​​for the training image and the first noise-removed image and the training image and the first noise-removed image, and generates the hard image having a low possibility of noise being removed by the noise-removal model based on the mask, the training image, and the first noise-removed image.

[0008] In addition, a method for constructing a noise removal model using a noise removal model construction device according to another embodiment of the present invention includes the steps of: inputting a learning image including noise into a noise removal model to generate a first noise removed image; applying the learning image and the first noise removed image to an adversarial frequency mixing model to generate a hard image; inputting the hard image into the noise removal model to generate a second noise removed image; and updating the noise removal model so that the first noise removed image and the second noise removed image become the same as the clean image based on a first loss between the first noise removed image and a clean image corresponding to the learning image and a second loss between the second noise removed image and the clean image, wherein the adversarial frequency mixing model generates a mask for applying noise using frequency images for the learning image and the first noise removed image, respectively, and generates the hard image to which noise is applied, to which noise is unlikely to be removed by the noise removal model, through an operation using the mask, the learning image, and the frequency images of the first noise removed image.

[0009] According to the problem solving means of the present invention described above, the learning data can be augmented by generating a hard image based on the learning data and the first noise-removed image for the learning data through an adversarial frequency mixing model.

[0010] The adversarial frequency mixing model is updated to generate masks that are not removed by the noise removal model, and the noise removal model is updated to remove masks generated by the adversarial frequency mixing model, so that it can learn noise other than the noise included in the training data, thereby improving the performance of the noise removal model.

[0011] Additionally, during the update process of the adversarial frequency mixing model, the efficiency of generating a mask that is not removed by the noise removal model can be improved by removing noise that can be removed by the noise removal model through an easy image.

[0012] Fig. 1 is a conceptual diagram schematically illustrating a noise removal model construction device according to one embodiment of the present invention.

[0013] Figure 2 is a conceptual diagram for explaining the operation of a noise removal model construction device.

[0014] Figure 3 is a conceptual diagram explaining the operation of the adversarial frequency mixing model.

[0015] Figures 4 and 5 are exemplary diagrams showing the structure of a mask generation model.

[0016] Figure 6 is a flowchart illustrating a noise model construction method according to one embodiment of the present invention.

[0017] Figure 7 is a flowchart illustrating a noise model construction method according to one embodiment of the present invention.

[0018] Figure 8 is a conceptual diagram schematically illustrating a noise removal device according to one embodiment of the present invention.

[0019] Figure 9 is a flowchart for explaining a noise removal method according to one embodiment of the present invention.

[0020] Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in the present specification, and the technical ideas disclosed in the present specification are not limited by the attached drawings. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and the size, shape, and shape of each component shown in the drawings can be variously modified. The same / similar drawing reference numerals are assigned to the same / similar parts throughout the specification.

[0021] The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. Furthermore, in describing the embodiments disclosed herein, detailed descriptions of related known technologies have been omitted if they are deemed to obscure the gist of the embodiments disclosed herein.

[0022] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0023] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship between the components. For example, the first component of the present invention may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component."

[0024] Fig. 1 is a conceptual diagram schematically illustrating a noise removal model construction device according to one embodiment of the present invention.

[0025] Referring to FIG. 1, a noise removal model construction device (100) according to one embodiment of the present invention is described. The noise removal model construction device (100) constructs a noise removal model that removes noise contained in image data using learning data containing noise and a clean image corresponding to the learning data. To perform such an operation, the noise removal model construction device (100) includes a memory (110) and a processor (120).

[0026] The memory (110) stores a construction program for constructing a noise removal model. The construction program inputs a training image including noise into the noise removal model to generate a first noise removal image, applies the training image and the first noise removal image to an adversarial frequency mixing model to generate a hard image, inputs the hard image into the noise removal model to generate a second noise removal image, and trains the noise removal model so that the first noise removal image and the second noise removal image become the same as the clean image based on a first loss between the first noise removal image and a clean image corresponding to the training image and a second loss between the second noise removal image and the clean image.

[0027] Here, the adversarial frequency mixing model generates a mask to which noise is applied using the frequency magnitude values ​​of the learning image and the first noise-removed image, and the learning image and the first noise-removed image, and generates a hard image to which noise is applied through an operation using the mask, the learning image, and the first noise-removed image.

[0028] Meanwhile, the memory (110) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain the stored information. The memory (110) may perform a function of temporarily or permanently storing data processed by the processor (120). The memory (110) may include a magnetic storage media or a flash storage media in addition to a volatile storage device that requires power to maintain the stored information, but the scope of the present invention is not limited thereto.

[0029] And, the processor (120) executes the construction program stored in the memory (110) to train a noise removal model that removes noise included in the input image data. In the present embodiment, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0030] In addition, the processor (120) provides a function of executing a noise removal model construction program stored in the memory (110) and controlling the hardware of the noise removal model construction device (100) according to the execution of the program. That is, the processor (120) can perform hardware control functions such as a necessary file system, memory allocation, network, basic library, timer, device control (display, media, input device, 3D, etc.), and other utilities according to the execution of the program.

[0031] Figure 2 is a conceptual diagram illustrating the process by which a noise removal model is learned. Referring to Figure 2, the operation of the noise model construction program to construct a noise removal model is described.

[0032] The construction program inputs a learning image (10) containing noise into a noise removal model (200) to generate a first noise-removed image (20), applies the learning image (10) and the first noise-removed image (20) to an adversarial frequency mixing model (300) to generate a hard image (40), and inputs the hard image (40) into the noise removal model (200) to generate a second noise-removed image (50). Then, a first loss ( ) and the second loss between the second noise-removed image (50) and the clean image (30). ) is used to update the noise removal model (200) so that the first noise removal image (20) and the second noise removal image (50) correspond to the clean image (30). Here, the clean image (30) may be generated by taking many pictures of the target object with the same settings and taking an average.

[0033] Here, the adversarial frequency mixing model (300) receives a learning image (10) and a first noise-removed image (20) as input, generates a mask to which noise is applied using the frequency magnitude values ​​of each of the learning image (10) and the first noise-removed image (20), and generates a hard image (40) to which noise is applied through an operation using the mask, the learning image (10), and the first noise-removed image (20). The hard image (40) is an image to which difficult noise is applied that is unlikely to be removed by the noise-removal model (200). In other words, the hard image (40) may be an image to which noise is applied that the noise-removal model (200) has not learned.

[0034] Afterwards, the construction program calculates the second loss ( ) based on which the probability of noise being removed by the noise removal model (300) is low, or the adversarial frequency model (300) is updated to generate a hard image (40) that the noise removal model (200) has not previously learned.

[0035] Figure 3 is a conceptual diagram illustrating the operation of an adversarial frequency mixing model. Referring to Figure 3, the configuration of the adversarial frequency mixing model (300) generating a hard image (40) is specifically described.

[0036] When a learning image (10) and a first noise-removed image (20) are input, the adversarial frequency mixing model (300) applies a fast Fourier transform (FFT) to the learning image (10) and the first noise-removed image (20), respectively, to generate frequency images (11, 21) for the learning image (10) and the first noise-removed image (20), respectively. Then, the learning image (10) and the frequency image (11) for the learning image (10), the first noise-removed image (20) and the frequency image (21) for the first noise-removed image (20) are applied to a mask generation model (310) to generate a mask (80).

[0037] The mask generation model (310) is machine-learned to generate a mask (80) using a learning image (10), a frequency image (11) for the learning image (10), a first noise-removed image (20), and a frequency image (21) for the first noise-removed image (20). The mask (80) may be either a first mask (80-1) which is an image in which a mixing ratio is set for each of all pixels, or a second mask (80-2) which is an image in which a mixing ratio is set for each of a plurality of bands divided into predetermined areas. The mask (80) may be an image having a resolution corresponding to the resolution of the learning image (10), and the mixing ratio set for each pixel or band may have a value between 0 and 1.

[0038] The mask generation model (310) can generate a first mask (80-1) through a structure like FIG. 4, and can generate a second mask (80-2) through a structure like FIG. 5.

[0039] And, the adversarial frequency mixing model (300) generates a hard image (40) through an operation using a mask (80), a frequency image (11) for a learning image (10), and a frequency image (21) for a first noise-removed image (20).

[0040] Specifically, the adversarial frequency mixing model (300) can generate a hard image (40) by multiplying a frequency image (11) and a mask (80) for a learning image (10), multiplying a frequency image (21) and an interpolation mask for a first noise-removed image (20), adding the multiplied results, and then applying an inverse fast Fourier transform (IFFT) thereto. Here, the interpolation mask can be a mask whose ratio is set to 1 minus the mixing ratio set in the mask (80).

[0041] The adversarial frequency mixing model (300) can generate a hard image (40) using mathematical expression 1.

[0042]

[0043] In mathematical expression 1 is the frequency image (11) of the wet image, is the multiplication operator, is the frequency image (21) of the first noise-removed image, is a mask (80), is an interpolation mask.

[0044] Next, the construction program is the first loss ( ) and the second loss ( ) is used to describe the operation of updating the noise removal model (200). The construction program is expressed as the first loss ( ) and the second loss ( ) can be used to update the noise removal model (200).

[0045]

[0046] In mathematical expression 2 is the loss for updating the noise removal model (200), and the first noise removal image (20) ( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the first loss ( ) represents the second noise-removed image (50)( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the second loss ( ) represents, represents a hyperparameter.

[0047] Additionally, the construction program has a first loss ( ) and the second loss ( ) can also be used individually to update the noise removal model (200). The first loss ( ) may be used to update the noise removal model (200) so that the first noise removal image (20) corresponds to the clean image (30), and the second loss ( ) may be used to update the noise removal model (200) so that the second noise removal image (50) corresponds to the clean image (30). This may be performed simultaneously or sequentially.

[0048] And, the construction program has a second loss ( ) describes the operation of updating the adversarial frequency mixing model (300). The construction program is a second

[0049] Second loss between the image with removed edges (50) and the clean image (30) ) is less likely to be removed by the noise removal model (200) or the adversarial frequency mixing model (300) is updated to generate a hard image (40) that the noise removal model (200) has not previously learned.

[0050] Specifically, the construction program is a second loss ( ) updates the mask generation model (310). The construction program updates the second loss ( used in updating the noise removal model (200). ) is used to update the mask generation model (310). The noise removal model (200) is used to update the second loss ( ) is updated to remove noise well based on the second loss ( ) is updated in the opposite direction to the noise removal model (200) to create a mask (80) that applies noise that is unlikely to be removed by the noise removal model (200) or that has not been learned by the noise removal model (200).

[0051] In other words, the mask generation model (310) is a second loss between the second noise-removed image (50) and the clean image (30). ) is updated to generate a mask (80) that makes it difficult for the noise removal model (200) to remove the mask (80). The noise removal model (200) can improve noise removal performance by repeating the process of generating a second noise removal image (50) for the hard image (40) to which the mask (80) generated in this manner is applied.

[0052] In another embodiment, referring again to FIGS. 2 and 3, the adversarial frequency mixing model (300) can generate an easy image (60) together with a hard image (40). The easy image (60) is an image to which relatively easy noise has been applied so that, unlike the hard image (40), the noise has a high probability of being removed by the noise removal model (200), and may be an image to which noise has already been applied that the noise removal model (200) has learned. The easy image (60) can be generated through an operation using a mask (80), a frequency image (11) for the learning image (10), and a frequency image (21) for the first noise removal image (20).

[0053] The adversarial frequency mixing model (300) can generate an easy image (60) by multiplying a frequency image (11) for a learning image (10) and an interpolation mask, multiplying a frequency image (21) for a first noise-removed image (20) and a mask (80), adding the multiplied results, and then performing an inverse fast Fourier transform (IFFT).

[0054] The adversarial frequency mixing model (300) can generate an easy image (60) using mathematical expression 3.

[0055]

[0056] The easy image (60) can generate an image with noise applied that is easy to remove in the noise removal model (200) by multiplying the frequency image (11) for the learning image (10) by an interpolation mask, as opposed to the hard image (40), and multiplying the frequency image (21) for the first noise removal image (20) by a mask (80).

[0057] And, the construction program inputs the easy image (60) into the noise removal model (200) to generate a third noise-removed image (70), and generates a second loss ( ) and the third loss between the third noise-removed image (70) and the clean image (30). ) to update the adversarial frequency model (300) to generate a hard image (40) from which noise is not removed by the noise removal model (300).

[0058] Specifically, the construction program is a second loss ( ) and the third loss ( ) is used to update the mask generation model (310) to generate a mask (80) that prevents noise from being removed by the noise removal model (200). The construction program can update the mask generation model (310) using mathematical expression 4.

[0059]

[0060] In mathematical formula 4 is the loss for updating the mask generation model (310), and the second noise-removed image (50) ( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the second loss ( ) represents the third noise removal image (70)( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the third loss ( ) represents, represents the hyperparameter. The second loss ( ) in the third loss ( ) can prevent the creation of a mask (80) that is easy to remove.

[0061] The communication module (130) may include a device including hardware and software required to transmit and receive signals, such as control signals or data signals, through wired or wireless connections with other network devices in order to perform data communication with external devices and signal data.

[0062] The database (140) may store various data required for the operation of the noise removal model construction program. For example, data required for the operation of the noise removal model construction program, such as learning data required for the noise removal model learning process and noise-removed images generated during the learning process, may be stored.

[0063] Figure 6 is a flowchart for explaining a noise removal model construction method according to one embodiment of the present invention.

[0064] Referring to FIGS. 1, 2, and 6, a noise removal model construction method (S100) using a noise removal model construction device (100) according to an embodiment of the present invention will be described. The noise removal model construction device (100) inputs a learning image (10) containing noise into a noise removal model (200) to generate a first noise removal image (20) (step S110), applies the learning image (10) and the first noise removal image (20) to an adversarial frequency mixing model (300) to generate a hard image (40) (step S120), and inputs the hard image (40) into the noise removal model (200) to generate a second noise removal image (50) (step S130).

[0065] And, the first loss ( between the first noise-removed image (20) and the clean image (30) corresponding to the learning image (10) ) and the second loss between the second noise-removed image (50) and the clean image (30). ) is updated (step S140) so that the first noise-removed image (20) and the second noise-removed image (50) correspond to the clean image (30). Here, the clean image (30) is an image obtained by taking many pictures of the target object with the same settings and averaging them.

[0066] In addition, the noise removal model building device (100) is configured to construct a second loss ( ) is updated (step S150) to generate a hard image (40) that is unlikely to be removed by the noise removal model (200) or that has not been learned by the noise removal model (200).

[0067] Here, the adversarial frequency mixing model (300) receives a learning image (10) and a first noise-removed image (20) as input, generates a mask to which noise is applied using the frequency magnitude values ​​of each of the learning image (10) and the first noise-removed image (20), and generates a hard image (40) to which noise is applied through an operation using the mask, the learning image (10), and the first noise-removed image (20). Here, the hard image (40) is an image to which difficult noise is applied, which is unlikely to have noise removed by the noise-removal model (200), or an image to which noise is applied that the noise-removal model (200) has not learned.

[0068] Afterwards, each step is explained in detail.

[0069] Referring to FIG. 3, the process (step S120) in which the adversarial frequency mixing model (300) generates a hard image (40) is specifically described.

[0070] When a learning image (10) and a first noise-removed image (20) are input, the adversarial frequency mixing model (300) applies a fast Fourier transform (FFT) to the learning image (10) and the first noise-removed image (20) to generate frequency images (11, 21) for each of the learning image (10) and the first noise-removed image (20). Then, the learning image (10) and the frequency image (11) for the learning image (10), the first noise-removed image (20) and the frequency image (21) for the first noise-removed image (20) are applied to a mask generation model (310) to generate a mask (80).

[0071] The mask generation model (310) is machine-learned to generate a mask (80) using a learning image (10), a frequency image (11) for the learning image (10), a first noise-removed image (20), and a frequency image (21) for the first noise-removed image (20). The mask (80) may be either a first mask (80-1) which is an image in which a mixing ratio is set for each of all pixels, or a second mask (80-2) which is an image in which a mixing ratio is set for each of a plurality of bands divided into predetermined areas. The mask (80) may be an image having a resolution corresponding to the resolution of the learning image (10), and the mixing ratio set for each pixel or band may have a value between 0 and 1.

[0072] And, the adversarial frequency mixing model (300) generates a hard image (40) through an operation using a mask (80), a frequency image (11) for a learning image (10), and a frequency image (21) for a first noise-removed image (20).

[0073] Specifically, the adversarial frequency mixing model (300) multiplies a frequency image (11) and a mask (80) for a learning image (10), multiplies a frequency image (21) and an interpolation mask for a first noise-removed image (20), adds the multiplied results, and then applies an inverse fast Fourier transform (IFFT) to the multiplied results to generate a hard image (40). Here, the interpolation mask may be a mask having a ratio set to 1 minus the mixing ratio set in the mask (80).

[0074] The adversarial frequency mixing model (300) can generate a hard image (40) using mathematical expression 5.

[0075]

[0076] Mathematical expression 5 is substantially identical to Mathematical expression 1 above. In Mathematical expression 5, is the frequency image (11) of the learning image, is the multiplication operator, is the frequency image (21) of the first noise-removed image, is a mask (80), is an interpolation mask.

[0077] Next, the noise removal model building device (100) generates the first loss ( ) and the second loss ( ) is described as a process (step S140) of updating a noise removal model (200). The noise removal model construction device (100) is configured to construct a first loss ( as in mathematical expression 6. ) and the second loss ( ) can be used to update the noise removal model (200).

[0078]

[0079] Mathematical expression 6 is substantially identical to Mathematical expression 2 above. In Mathematical expression 6, is the loss for updating the noise removal model (200), and the first noise removal image (20) ( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the first loss ( ) represents the second noise-removed image (50)( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the second loss ( ) represents, represents a hyperparameter.

[0080] Additionally, the noise removal model building device (100) has a first loss ( ) and the second loss ( ) can also be used individually to update the noise removal model (200). The first loss ( ) may be used to update the noise removal model (200) so that the first noise removal image (20) corresponds to the clean image (30), and the second loss ( ) may be used to update the noise removal model (200) so that the second noise removal image (50) corresponds to the clean image (30). This may be performed simultaneously or sequentially.

[0081] And, the noise removal model building device (100) has a second loss ( ) is described as a process (step S150) of updating an adversarial frequency mixing model (300). The noise removal model building device (100) calculates a second loss ( between a second noise removal image (50) and a clean image (30). ) can be updated to generate a hard image (40) that is difficult to remove noise from through the noise removal model (200) or that the noise removal model (200) has not learned.

[0082] Specifically, the noise removal model building device (100) has a second loss ( ) to update the mask generation model (310), and the second loss ( used to update the noise removal model (200) ) is used to update the mask generation model (310). The noise removal model (200) is used to update the second loss ( ) is updated to remove noise well based on the second loss ( ) is updated in the opposite direction to the noise removal model (200) to create a mask (80) that applies noise that is difficult to remove by the noise removal model (200) or that has not been learned by the noise removal model (200).

[0083] In other words, the mask generation model (310) is a second loss between the second noise-removed image (50) and the clean image (30). ) is updated to generate a mask (80) that makes it difficult for the noise removal model (200) to remove the mask (80). The noise removal model (200) can improve noise removal performance by repeating the process of generating a second noise removal image (50) for the hard image (40) to which the mask (80) generated in this manner is applied.

[0084] Figure 7 is a flowchart for explaining a noise removal model construction method according to one embodiment of the present invention.

[0085] Referring to FIGS. 1, 2, and 7, a noise removal model construction method (S200) using a noise removal model construction device (100) according to an embodiment of the present invention is described. The noise removal model construction device (100) inputs a learning image (10) containing noise into a noise removal model (200) to generate a first noise removal image (20) (step S210), applies the learning image (10) and the first noise removal image (20) to an adversarial frequency mixing model (300) to generate a hard image (40) and an easy image (60) (step S220), and inputs the hard image (40) and the easy image (60) into the noise removal model (200) to generate a second noise removal image (50) (step S230).

[0086] And, the first loss ( between the first noise-removed image (20) and the clean image (30) corresponding to the learning image (10) ) and the second loss between the second noise-removed image (50) and the clean image (30). ) is updated (step S240) to update the noise removal model (200) so that the first noise removal image (20) and the second noise removal image (50) correspond to the clean image (30).

[0087] In addition, the noise removal model building device (100) is configured to construct a second loss ( ) and the third loss between the third noise-removed image (70) and the clean image (30). ) is updated (step S250) to generate a hard image (40) that is unlikely to be removed by the noise removal model (200) or that the noise removal model (200) has not learned.

[0088] Afterwards, each step is explained in detail.

[0089] Referring to FIG. 3, the process (step S220) in which the adversarial frequency mixing model (300) generates a hard image (40) and an easy image (60) is specifically described.

[0090] The process of generating a hard image (40) in step S220 corresponds to step S120 of the noise removal model construction method (S100) described above, so a detailed description is omitted, and the process of generating an easy image (60) is described.

[0091] The easy image (60) is an image generated so that noise can be easily removed by a noise removal model (200), as opposed to a hard image (40), and can be generated through an operation using a mask (80), a frequency image (11) for a learning image (10), and a frequency image (21) for a first noise removal image (20).

[0092] The adversarial frequency mixing model (300) can generate an easy image (60) by multiplying a frequency image (11) for a learning image (10) and an interpolation mask, multiplying a frequency image (21) for a first noise-removed image (20) and a mask (80), adding the multiplied results, and then performing an inverse fast Fourier transform (IFFT). The adversarial frequency mixing model (300) can generate an easy image (60) using mathematical expression 7.

[0093]

[0094] Mathematical expression 7 is substantially identical to Mathematical expression 3 above.

[0095] The easy image (60) can generate an image with noise applied that is easy to remove in the noise removal model (200) by multiplying the frequency image (11) for the learning image (10) by an interpolation mask, as opposed to the hard image (40), and multiplying the frequency image (21) for the first noise removal image (20) by a mask (80).

[0096] And, the process (step S250) of updating the adversarial frequency mixing model (300) by the noise removal model building device (100) is described. The noise removal model building device (100) calculates the second loss ( between the second noise removal image (50) and the clean image (30). ) and the third loss between the third noise-removed image (70) and the clean image (30). ) based on which the probability of noise being removed by the noise removal model (200) is low or the adversarial frequency model (300) is updated to generate a hard image (40) that the noise removal model (200) has not learned.

[0097] Specifically, the noise removal model building device (100) has a second loss ( ) and the third loss ( ) is used to update the mask generation model (310) to generate a mask (80) that applies noise that has a low probability of being removed by the noise removal model (200) or that has not been learned by the noise removal model (200). The noise removal model construction device (100) can update the mask generation model (310) using mathematical expression 8.

[0098]

[0099] Mathematical expression 8 is substantially identical to Mathematical expression 4 above. In Mathematical expression 8, is the loss for updating the mask generation model (310), and the second noise-removed image (50) ( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the second loss ( ) represents the third noise removal image (70)( ) and clean image (30)( ) is the signal-to-noise ratio (PSNR) between the third loss ( ) represents, represents the hyperparameter. The second loss ( ) in the third loss ( ) can prevent the creation of a mask (80) that is easy to remove.

[0100] Figure 8 is a conceptual diagram schematically illustrating an image noise removal device according to one embodiment of the present invention.

[0101] Referring to FIG. 8, an image noise removal device (400) according to one embodiment of the present invention will be described. The image noise removal device (400) receives image data containing noise and applies the image data to a noise removal model to remove noise contained in the image data. To perform such an operation, the image noise removal device (400) includes a memory (410) and a processor (420).

[0102] The memory (410) stores a noise removal program that removes noise from image data containing noise. The noise removal program receives image data containing noise, applies the image data to a noise removal model, and provides image data with noise removed.

[0103] Meanwhile, the memory (410) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain the stored information. The memory (410) may perform a function of temporarily or permanently storing data processed by the processor (420). In addition to a volatile storage device that requires power to maintain the stored information, the memory (410) may include a magnetic storage media or a flash storage media, but the scope of the present invention is not limited thereto.

[0104] And, the processor (420) executes a noise removal program stored in the memory (410) to remove noise included in the received image data. The noise removal program inputs the received image into a noise removal model to provide an image with noise removed. Here, the noise removal model is machine-learned to remove noise included in the input image data, and is constructed through a process in which the noise removal model is primarily learned based on the first loss between the first noise-removed image for a noise-containing training image and a clean image corresponding to the training image, and secondarily learned based on the second loss between the second noise-removed image and the clean image for a hard image generated by applying the first noise removal model to an adversarial frequency mixing model that generates an image containing noise. Since the specific learning process of the noise removal model is the same as the noise removal model construction method (S100, S200) using the noise removal model construction device (100) described above, a detailed description thereof will be omitted.

[0105] In this embodiment, the processor (420) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0106] In addition, the processor (420) provides a function of executing a noise removal program stored in the memory (410) and controlling the hardware of the image noise removal device (400) according to the execution of the program. That is, the processor (420) can perform hardware control functions such as a necessary file system, memory allocation, network, basic library, timer, device control (display, media, input device, 3D, etc.), and other utilities according to the execution of the program.

[0107] The communication module (430) may include a device including hardware and software required to transmit and receive signals, such as control signals or data signals, through wired or wireless connections with other network devices in order to perform data communication with external devices and signal data.

[0108] The database (440) may store various data required for the operation of a noise removal program. For example, data required for the operation of a noise removal model may be stored.

[0109] Figure 9 is a flowchart for explaining a noise removal method according to one embodiment of the present invention.

[0110] Referring to FIGS. 8 and 9, a noise removal method (S300) using a noise removal device (400) according to an embodiment of the present invention is described. The noise removal device (400) receives image data containing noise (step S310), inputs the image data into a noise removal model constructed through the process described above, and outputs image data from which noise has been removed (step S320).

[0111] The present invention may also be implemented in the form of a non-transitory storage medium containing computer-executable instructions, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Computer-readable media may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0112] Additionally, although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0113] Those skilled in the art will appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention based on the above description. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

[0114] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

[0115] The form for carrying out the invention is substantially the same as the best form for carrying out the invention.

[0116] The present invention can be used in a technology for removing noise from an image, and thus has industrial applicability.

Claims

1. In an image noise removal device for removing noise from an image, Memory where the noise removal program is stored; and Including a processor for executing the above noise removal program, The above noise removal program is, Receiving image data containing noise, inputting the image data into a noise removal model to remove noise contained in the image data, The above noise removal model is, By removing noise contained in the input image data, A first noise-removed image is generated for a learning image containing noise, and an adversarial frequency mixing model below generates a second noise-removed image for a hard image generated based on the learning image and the first noise-removed image, and the first noise-removed image and the second noise-removed image are machine-learned to be the same as the clean image based on a first loss between the first noise-removed image and a clean image corresponding to the learning image and a second loss between the second noise-removed image and the clean image. The above adversarial frequency mixing model is, An image noise removal device, which generates a mask for applying noise using the learning image and the first noise removal image and the respective frequency domain size values ​​for the learning image and the first noise removal image, and generates the hard image having a low probability of noise being removed by the noise removal model based on the mask, the learning image, and the first noise removal image.

2. In paragraph 1, The above adversarial frequency mixing model is, An image noise removal device, which is trained to generate the mask applying noise that is unlikely to be removed by the noise removal model noise based on the second loss.

3. In paragraph 1, The above adversarial frequency mixing model is, Generating an easy image with noise applied based on the above mask, the above learning image, and the first noise-removed image, The above noise removal model is, Generate a third noise-removed image for the above easy image, The above adversarial frequency mixing model is, An image noise removal device, which is trained to generate the mask by applying noise that is less likely to be removed by the noise removal model based on the second loss and the third loss between the third noise-removed image and the clean image.

4. In paragraph 3, The above adversarial frequency mixing model is, An image noise removal device that applies a Fourier transform to each of the learning image and the first noise removal image to generate a frequency image including a size in the frequency domain, and generates the mask using the learning image, the frequency image for the learning image, the first noise removal image, and the frequency image for the first noise removal image.

5. In paragraph 4, The above adversarial frequency mixing model is, An image noise removal device that generates the hard image based on an operation of the frequency image of the learning image and the mask and an operation of the frequency image of the first noise removal image and the interpolation mask.

6. In paragraph 4, The above adversarial frequency mixing model is, An image noise removal device that generates the easy image based on the operation of the frequency image of the learning image and the interpolation mask and the operation of the frequency image of the first noise removal image and the mask.

7. In paragraph 1, The above mask, An image noise removal device in which a blending ratio is set for each pixel of the image.

8. In paragraph 1, The above mask, An image noise removal device in which a mixing ratio is set for each of a plurality of bands divided into predetermined areas for an image.

9. A method for removing noise from an image using an image noise removal device, A step of receiving image data containing noise; and A step of inputting the above image data into a noise removal model to remove noise contained in the image data, The above noise removal model is, It is machine-learned to remove noise contained in the input image data. A step of generating a first noise-removed image for a training image including noise; a step of generating a second noise-removed image for a hard image generated based on the training image and the first noise-removed image by an adversarial frequency mixing model; and a step of learning such that the first noise-removed image and the second noise-removed image become identical to the clean image based on a first loss between the first noise-removed image and a clean image corresponding to the training image and a second loss between the second noise-removed image and the clean image, wherein the learning is performed in this manner. The above adversarial frequency mixing model is, An image noise removal method, comprising: generating a mask for applying noise using the learning image and the first noise-removed image and the respective frequency domain size values ​​for the learning image and the first noise-removed image; and generating the hard image, which is unlikely to have noise removed by the noise removal model, based on the mask, the learning image, and the first noise-removed image.

10. In paragraph 9, The above adversarial frequency mixing model is, An image noise removal method, wherein the mask is learned to apply noise that is unlikely to be removed by the noise removal model based on the second loss.

11. In paragraph 9, The above adversarial frequency mixing model is, Generating an easy image with noise applied based on the above mask, the above learning image, and the first noise-removed image, The above noise removal model is, A third noise-removed image is created by removing noise from the above image, The above adversarial frequency mixing model is, An image noise removal method, wherein the mask is learned to apply noise that is unlikely to be removed by the noise removal model based on the second loss and the third loss between the third noise-removed image and the clean image.

12. In paragraph 11, The above adversarial frequency mixing model is, An image noise removal method comprising: applying a Fourier transform to each of the learning image and the first noise removal image to generate a frequency image including a magnitude in the frequency domain, and generating the mask using the learning image, the first noise removal image, and the frequency image.

13. In paragraph 12, The above adversarial frequency mixing model is, Generate the hard image based on the operation of the frequency image of the learning image and the mask and the operation of the frequency image of the first noise removal image and the interpolation mask, An image noise removal method, wherein the easy image is generated based on an operation of the frequency image of the learning image and the interpolation mask and an operation of the frequency image of the first noise removal image and the mask.

14. A method for constructing a noise removal model using a noise removal model construction device, A step of generating a first noise-removed image by inputting a training image containing noise into a noise removal model; A step of generating a hard image by applying the above learning image and the first noise-removed image to an adversarial frequency mixing model; A step of inputting the hard image into the noise removal model to generate a second noise-removed image; and A step of updating the noise removal model so that the first noise removed image and the second noise removed image become the same as the clean image based on a first loss between the first noise removed image and a clean image corresponding to the training image and a second loss between the second noise removed image and the clean image, The above adversarial frequency mixing model is, A method for constructing a noise removal model, comprising: generating a mask for applying noise using the learning image and the first noise removal image and the frequency images for each of the learning image and the first noise removal image; and generating the hard image having a low probability of noise being removed by the noise removal model through an operation using the frequency images of the mask, the learning image, and the first noise removal image.

15. In paragraph 14, A method for building a noise removal model, further comprising the step of updating the adversarial frequency model to generate the hard image from which noise is not removed by the noise removal model based on the second loss.

16. In paragraph 15, The step of updating the above adversarial frequency model is: A method for building a noise removal model, wherein the adversarial frequency model is updated to generate the mask applying noise that is less likely to be removed by the noise removal model by using the second loss with the opposite sign applied.

17. In paragraph 14, In the step of creating the above hard image, The above adversarial frequency mixing model generates an easy image using the training image and the first noise-removed image, In the step of generating the second noise-removed image, The above easy image is input into the above noise removal model to generate a third noise removal image, A method for building a noise removal model, further comprising the step of updating the adversarial frequency model to generate the hard image having a low probability of noise being removed by the noise removal model based on the second loss and the third loss between the third noise-removed image and the clean image.

18. In paragraph 17, The step of updating the above adversarial frequency model is: A method for constructing a noise removal model, wherein the adversarial frequency model is updated to generate the mask applying noise that is unlikely to be removed by the noise removal model through an operation using the second loss and the third loss.

19. In paragraph 17, The above adversarial frequency mixing model is, Applying a Fourier transform to the above learning image and the first noise-removed image, a frequency image including the frequency magnitude for each is generated, Generate the hard image based on the operation of the frequency image of the learning image and the mask and the operation of the frequency image of the first noise removal image and the interpolation mask, A method for constructing a noise removal model, wherein the easy image is generated based on the operation of the frequency image of the learning image and the interpolation mask and the operation of the frequency image of the first noise removal image and the mask.

Images