Method and electronic device for adaptive frequency filtering

Through the joint training method of frequency filter processing and denoising neural network, the problem of denoising denoising performance of denoising images of different noise distributions in the prior art is solved, effectively removing various noises, and improving the image quality.

CN118967493APending Publication Date: 2024-11-15SAMSUNG ELECTRONICS CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410587013.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-08-24
Filing Date
2024-05-13
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When existing denoising neural networks process input images with different noise distributions from training images, denoising performance deteriorates, making it difficult to effectively remove noise from different noise distributions.

Method used

By transforming the input image into the frequency domain, generating a transformed frequency image and applying a frequency filter, normalizing the noise distribution and inputting the denoising neural network, using the filter to generate a neural network and a denoising neural network to train together to generate an effective frequency filter to remove noise.

Benefits of technology

Effective denoising of images with different noise distributions is achieved, denoising performance is improved, and image quality is ensured under different noise environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967493A_ABST
    Figure CN118967493A_ABST
Patent Text Reader

Abstract

A method and an electronic device for adaptive frequency filtering are provided. The electronic device includes: one or more processors configured to generate a transformed frequency image by transforming an input image to a frequency domain, obtain a frequency filter corresponding to the input image by inputting the generated transformed frequency image to a filter generation neural network, and generating an output image in which the noise distribution of the input image is normalized by applying the obtained frequency filter to the generated transformed frequency image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0062592 filed on May 15, 2023, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2023-0111196 filed on August 24, 2023, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] The following description relates to methods and apparatus with adaptive frequency filtering for robust image reconstruction. Background Art

[0003] An input image captured by a camera may include various types of noise. For example, there may be shot noise that occurs when light is detected by a photodiode in an image sensor, circuit noise that occurs when an analog signal is converted into a digital signal, and quantization noise that occurs when a digital signal is quantized and represented. A method of removing noise included in an image may be a denoising method that uses a denoising neural network based on a convolutional neural network (CNN) to remove noise included in the input image. However, when the denoising neural network is used to remove noise from the input image, the denoising neural network may have a desired denoising performance for an input image having a noise distribution similar to that of an image used for training, but may have a degraded denoising performance for an input image having a noise distribution different from that of an image used for training. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one or more general aspects, an electronic device includes: one or more processors configured to generate a transformed frequency image by transforming an input image into a frequency domain, obtain a frequency filter corresponding to the input image by inputting the generated transformed frequency image into a filter generating neural network, and generate an output image in which a noise distribution of the input image is normalized by applying the obtained frequency filter to the generated transformed frequency image.

[0006] The electronic device may include an image sensor including a plurality of photodiodes, wherein the one or more processors are configured to generate an input image using the plurality of photodiodes of the image sensor.

[0007] The one or more processors may be configured to generate a denoised image in which noise is removed from the input image by inputting the generated output image into a denoising neural network.

[0008] The one or more processors may be configured to train the filter generation neural network and the denoising neural network together based on a loss calculated using the generated denoised image and a true denoised image mapped to the input image.

[0009] The one or more processors may be configured to train a filter generating neural network based on a loss calculated using the generated denoised image and a true denoised image mapped to the input image.

[0010] The one or more processors may be configured to train a filter generating neural network based on a loss calculated using the generated output image and a true output image mapped to the input image.

[0011] To generate the output image, the one or more processors may be configured to: generate an intermediate image in the frequency domain by applying the obtained frequency filter to the generated transformed frequency image; and generate the output image by transforming the generated intermediate image into the spatial domain.

[0012] The filter generation neural network may include one or more convolutional neural networks (CNNs) and a fully connected layer connected to the one or more CNNs.

[0013] The first output data output from the fully connected layer may indicate a frequency range of a frequency domain in the frequency filter, and the second output data output from the fully connected layer may indicate a weight value of a component corresponding to the frequency range indicated by the first output data.

[0014] To obtain the frequency filter, the one or more processors may be configured to maintain the component in the frequency filter in response to the weight value being greater than or equal to a threshold value.

[0015] The filter generation neural network may be configured to output different frequency filters according to the noise distribution of the image input to the filter generation neural network.

[0016] In one or more general aspects, an electronic device includes: one or more processors configured to generate a temporary transformed frequency image by transforming a training input image into a frequency domain, obtain a temporary frequency filter by inputting the generated temporary transformed frequency image into a filter generating neural network, generate a temporary output image by applying the obtained temporary frequency filter to the generated temporary transformed frequency image, and generate a temporary denoised image by inputting the generated temporary output image into a denoising neural network, and train the filter generating neural network and the denoising neural network together based on a loss determined using the generated temporary denoised image and a true denoised image mapped to the training input image.

[0017] In one or more general aspects, a processor-implemented method includes: generating a transformed frequency image by transforming an input image into a frequency domain; obtaining a frequency filter corresponding to the input image by inputting the generated transformed frequency image into a filter generating neural network; and generating an output image in which a noise distribution of the input image is normalized by applying the obtained frequency filter to the generated transformed frequency image.

[0018] The method may include generating a denoised image in which noise is removed from an input image by inputting the generated output image into a denoising neural network.

[0019] The step of generating the output image may include: generating an intermediate image in a frequency domain by applying the obtained frequency filter to the generated transformed frequency image; and generating the output image by transforming the generated intermediate image into a spatial domain.

[0020] The filter generation neural network may include one or more convolutional neural networks (CNNs) and a fully connected layer connected to the one or more CNNs.

[0021] The first output data output from the fully connected layer may indicate a frequency range of a frequency domain in the frequency filter, and the second output data output from the fully connected layer may indicate a weight value of a component corresponding to the frequency range determined by the first output data.

[0022] The filter generation neural network may be configured to output different frequency filters according to the noise distribution of the image input to the filter generation neural network.

[0023] In one or more general aspects, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, configure the processor to perform any one, any combination, or all of the operations and / or methods described herein.

[0024] In one or more general aspects, a processor-implemented method includes: generating a temporary transformed frequency image by transforming a training input image into a frequency domain, obtaining a temporary frequency filter by inputting the generated temporary transformed frequency image into a filter generating neural network, generating a temporary output image by applying the obtained temporary frequency filter to the generated temporary transformed frequency image, and generating a temporary denoised image by inputting the generated temporary output image into a denoising neural network; and training the filter generating neural network and the denoising neural network together based on a loss determined using the generated temporary denoised image and a true denoised image mapped to the training input image.

[0025] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example method of denoising an input image by an electronic device according to one or more example embodiments is illustrated.

[0027] Figure 2 An example of obtaining, by an electronic device, a denoised image in which noise is removed from an input image according to one or more example embodiments is illustrated.

[0028] Figure 3 An example structure of a filter generating neural network according to one or more example embodiments is shown.

[0029] Figure 4 An example of generating a neural network by training a filter by an electronic device according to one or more example embodiments is shown.

[0030] Figure 5 Examples of an input image, a denoised image, a noise distribution of the input image, and a noise distribution of the denoised image are illustrated according to one or more example embodiments.

[0031] Figure 6 An example structure of an electronic device according to one or more example embodiments is illustrated.

[0032] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0033] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of known features after understanding the disclosure of the present application can be omitted.

[0034] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0035] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the articles "a" and "the (or, this, that)" are also intended to include plural forms. The terms "comprise", "include" and "have" specify the presence of narrated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof. The use of the term "may" for examples or embodiments herein (e.g., what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such features, and all examples are not limited thereto.

[0036] Throughout the specification, when an element is described as being "connected to" or "coupled to" another element, the element may be directly "connected to" or "coupled to" the other element, or there may be one or more other elements intervening therebetween. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements intervening therebetween. Similarly, similar expressions (e.g., "between" versus "immediately between" and "adjacent to" versus "immediately adjacent to") should also be interpreted in the same manner. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0037] Although terms such as "first", "second" and "third", or A, B, (a), (b), etc. may be used herein to describe various components, components, regions, layers or parts, these components, components, regions, layers or parts should not be limited by these terms. Each of these terms is not used to define, for example, the essence, order or sequence of the corresponding components, components, regions, layers or parts, but is only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teaching of the example, the first component, first component, first region, first layer or first part referred to in the example described herein may also be referred to as the second component, second component, second region, second layer or second part.

[0038] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs and based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the disclosure of the relevant art and the present application, and should not be interpreted in an idealized or overly formal manner.

[0039] Furthermore, in the description of the exemplary embodiments, when it is considered that a detailed description of a structure or function known therefrom after understanding the disclosure of the present application will lead to an obscure interpretation of the exemplary embodiments, such description will be omitted.

[0040] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings denote like elements throughout.

[0041] Figure 1 An example method of denoising an input image by an electronic device according to one or more example embodiments is illustrated.

[0042] An image generated by a camera may include various types of noise. The noise included in the image may be removed using a denoising neural network based on a convolutional neural network (CNN). The denoising neural network may represent one or more models having a machine learning structure configured to extract a denoised image in which noise is removed from the input image in response to an input image being input.

[0043] The denoising neural network may overfit the noise distribution of the training images used for training. That is, the denoising neural network may effectively perform denoising on images having a noise distribution similar to that of the training images, but may not be able to effectively perform denoising on images having a noise distribution different from that of the training images. For example, when the denoising neural network is trained using training images including Gaussian noise that follows a normal distribution with a standard deviation of 10, the denoising neural network may not be able to effectively denoise images having Gaussian noise that follows a normal distribution with a standard deviation of 15. For another example, when the denoising neural network is trained using training images with real noise, the denoising neural network may not be able to effectively denoise images with synthetic noise.

[0044] The electronic device of one or more embodiments can solve such problems described above by normalizing the noise distribution of the image. When the noise distribution of the input image is normalized, the electronic device can input the image with the normalized noise distribution to a pre-trained denoising neural network. That is, the electronic device of one or more embodiments can transform the noise distribution of the image into a noise distribution that can be effectively denoised by the denoising neural network. The denoising neural network can effectively denoise the image in which the noise distribution is normalized from the input image.

[0045] Reference Figure 1 , in operation 110, the electronic device may generate a transformed frequency image (eg, a frequency transformed image) by transforming an input image into a frequency domain.

[0046] In one example embodiment, an electronic device may obtain an input image. For example, an electronic device (e.g., Figure 6 The electronic device 601 may obtain an input image from a camera disposed inside the electronic device. An image sensor (eg, Figure 6 The image sensor 610 may include a plurality of photodiodes. The plurality of photodiodes included in the image sensor may receive light and generate an input image of an external object. For another example, the electronic device may obtain the input image from an external electronic device. The input image used herein may be data in a spatial domain.

[0047] In an example embodiment, the electronic device may transform the input image from the spatial domain to the frequency domain using Fourier transform. In one example, transforming the input image from the spatial domain to the frequency domain may include applying frequency analysis using changes in the brightness of pixels as a signal while following the input image in the x-axis or y-axis direction. The intensity of each frequency component obtained by Fourier transform may be defined as a spectrum, and the spectrum may be represented as a transformed frequency image.

[0048] In operation 120, the electronic device may obtain (eg, generate or determine) a frequency filter (or a target filter) corresponding to the input image by inputting the generated transformed frequency image to the first filter generation neural network.

[0049] In an example embodiment, the first filter generation neural network may be one or more models having a machine learning structure configured to extract a frequency filter for denoising the input image in response to an input of an image as data in the frequency domain. The first filter generation neural network may generate a frequency filter corresponding to the input image by performing an operation based on the machine learning model on the input image. The frequency filter may allow a specific frequency band to pass or block a specific frequency band. For example, the first filter generation neural network may generate various types of frequency filters (including, as non-limiting examples, a band pass filter (BPF), a band stop filter (BRF), a high pass filter (HPF), and / or a low pass filter (LPF)).

[0050] In operation 130, the electronic device may generate an output image in which the noise distribution of the input image is normalized by applying the obtained frequency filter to the generated transformed frequency image. The electronic device may perform frequency filtering on the transformed frequency image by applying the frequency filter to the transformed frequency image. The electronic device may enhance or attenuate each frequency region in the transformed frequency image. For example, the frequency filter may be represented in the form of an image having the same size as the transformed frequency image. In one example, applying the frequency filter to the transformed frequency image may include multiplying the transformed frequency image by the frequency filter.

[0051] In one example embodiment, the electronic device may input an output image in which the noise distribution of the input image is normalized to a denoising neural network. The denoising neural network may output a denoised image in which noise is removed from the output image by effectively removing noise from the output image.

[0052] As described above, the electronic device may input a transformed frequency image of an input image to a filter generation neural network to normalize a noise distribution of the input image, and may obtain a frequency filter corresponding to the input image from the filter generation neural network.

[0053] In a similar manner, the electronic device may perform deblurring on the input image. Deblurring may refer to a process of removing blur from an image. In an example embodiment, the electronic device may generate a transformed frequency image by transforming the input image into a frequency domain, and obtain a frequency filter corresponding to the input image by inputting the generated transformed frequency image into a second filter generating neural network. The second filter generating neural network may be one or more models having a machine learning structure configured to extract a frequency filter through which blur included in the input image is removed for deblurring the input image.

[0054] In addition, in a similar manner, the electronic device may perform upscaling on the input image. Upscaling may refer to a process of increasing the resolution of the input image. In an example embodiment, the electronic device may generate a transformed frequency image by transforming the input image into a frequency domain, and obtain a frequency filter corresponding to the input image by inputting the generated transformed frequency image into a third filter generating neural network. The third filter generating neural network may be one or more models having a machine learning structure configured to extract a frequency filter through which the resolution of the input image is increased for upscaling the input image.

[0055] The first filter generating neural network for denoising the input image, the second filter generating neural network for deblurring the input image, and the third filter generating neural network for upscaling the input image may be different from each other.

[0056] In one example embodiment, the electronic device may store a plurality of filter generation neural networks (e.g., a first filter generation neural network, a second filter generation neural network, and a third filter generation neural network). The electronic device may select a filter generation neural network from among the plurality of filter generation neural networks according to a task to be performed on an input image. The electronic device may obtain a frequency filter corresponding to the input image by inputting a transformed frequency image corresponding to the input image to the selected filter generation neural network.

[0057] Each of the plurality of filter generation neural networks may correspond to a task. For example, the first filter generation neural network may correspond to a noise distribution normalization task, the second filter generation neural network may correspond to a deblurring task, and the third filter generation neural network may correspond to an amplification task. In this example, when the task to be performed on the input image is amplification, the electronic device may select the third filter generation neural network from among the plurality of filter generation neural networks.

[0058] The following describes how the electronic device performs the task of normalizing the noise distribution of the input image.

[0059] Figure 2 An example of obtaining, by an electronic device, a denoised image in which noise is removed from an input image according to one or more example embodiments is illustrated.

[0060] First, the electronic device may transform the input image 210, which is data in the spatial domain, into the frequency domain. The electronic device may generate a transformed frequency image (e.g., a frequency transformed image) 211 by transforming the input image 210 into the frequency domain. That is, the transformed frequency image 211 may be data in the frequency domain. The electronic device may obtain a frequency filter 221 corresponding to the input image 210 by inputting the transformed frequency image 211 into the filter generation neural network 220. The frequency filter 221 may be a frequency filter for denoising the transformed frequency image 211.

[0061] The frequency filter 221 may be represented in the form of an image having the same size as the transformed frequency image 211. The entire region of the frequency filter 221 may be divided into a preset number of regions. The regions in the frequency filter 221 may correspond to separate frequency bands. The entire frequency band of the frequency filter 221 output from the filter generation neural network 220 may be divided into a preset number of frequency bands, and a weight value may be set separately for each divided frequency band.

[0062] For example, the entire area of ​​the frequency filter 221 may be divided into four areas. In this example, the first area may be a region whose distance from the center point 222 in the frequency filter 221 is x. 1 or a smaller group of points; the second region may be a group of points whose distance from the center point 222 in the frequency filter 221 is greater than x 1 and is less than or equal to x 2 The third region may be a set of points in the frequency filter 221 whose distance from the center point 222 is greater than x 2 and less than or equal to x 3 and the fourth region may be a region outside the third region (eg, a region whose distance from the center point 222 is greater than x 3 In this example, x 2 Can be greater than x 1 , and x 3 Can be greater than x 2 The same weight value may be set for points in the same area included in the frequency filter 221 .

[0063] For another example, the frequency filter 221 corresponding to the transformed frequency image 211 may be an LPF. For example, a transfer function of the LPF may be expressed as Equation 1 below.

[0064] Equation 1:

[0065]

[0066] In Equation 1, u and v may represent the coordinate values ​​of the frequency component, H(u,v) may represent the transfer function at the frequency coordinate value (u,v), and D(u,v) may represent the distance between the (u,v) coordinate and the origin in the frequency domain. According to the above Equation 1, the frequency filter 221 may transform the frequency image 211 with a radius of D. 0 The frequency band corresponding to the inner region of the block transformation frequency image 211 is passed, and the frequency band corresponding to the inner region of the block transformation frequency image 211 is passed, except for the frequency band having a radius D. 0 That is, the entire area of ​​the frequency filter 221 may be divided into two areas. The area within the radius D from the center of the frequency filter 221 may be 0 A weight value of 1 may be set for points in an inner area of ​​the circle, and a weight value of 0 may be set for points included in an outer area of ​​the circle.

[0067] For another example, the frequency filter 221 corresponding to the transformed frequency image 211 may be a Gaussian LPF. For example, the transfer function of the Gaussian LPF may be expressed as Equation 2 below.

[0068] Equation 2:

[0069]

[0070] In the case where the frequency filter 221 is a Gaussian LPF, the frequency filter 221 may pass a frequency band corresponding to a central region in the transformed frequency image 211 and block a frequency band corresponding to a remaining region in the transformed frequency image 211. Using a Gaussian LPF may block edge information that may exist in a high frequency region to obtain a smoothing effect.

[0071] In one example embodiment, the electronic device may apply the frequency filter 221 to the transformed frequency image 211 to generate an output image 213 in which the noise distribution of the input image 210 is normalized. For example, the electronic device may generate an intermediate image as data in the frequency domain by multiplying the transformed frequency image 211 by the frequency filter 221, and the electronic device may generate the output image 213 by transforming the generated intermediate image into the spatial domain. The electronic device may transform the intermediate image from the frequency domain to the spatial domain using an inverse Fourier transform. The output image 213 may be an image with a noise distribution normalized from the input image 210.

[0072] The electronic device may input the output image 213 having the normalized noise distribution to the denoising neural network 230. The electronic device may input the output image 213 to the denoising neural network 230 to obtain a denoised image 240 in which noise is removed from the input image 210.

[0073] As described above, the filter generation neural network 220 and the denoising neural network 230 may be implemented independently of each other. However, the example is not limited thereto, and the filter generation neural network 220 and the denoising neural network 230 may be implemented as a single integrated network.

[0074] Figure 3 An example structure of a filter generating neural network according to one or more example embodiments is shown.

[0075] In an example embodiment, the filter generation neural network 320 may include at least one CNN 330 and a fully connected layer 340 connected to the at least one CNN 330. The fully connected layer 340 may include a plurality of layers. The fully connected layer 340 may receive the output of the at least one CNN 330 as an input.

[0076] In an example embodiment, two output data (eg, first output data and second output data) may be output from the fully connected layer 340. The first output data output from the fully connected layer 340 may be combined with a frequency filter (eg, Figure 2 The radius used herein may represent a frequency filter (e.g., Figure 2 A point in the frequency filter 221 (eg, Figure 2 point 223) and the center point of the frequency filter (eg, Figure 2 That is, the first output data may determine the frequency range of the frequency domain in the frequency filter. The second output data output from the fully connected layer 340 may represent the frequency filter (e.g., Figure 2 The weight value w corresponding to the radius r corresponding to the first output data in the frequency filter 221 of FIG. Figure 2 A point in the frequency filter 221 (eg, Figure 2 The weight value corresponding to the point 223 of the second output data may be determined from the second output data. That is, the second output data may determine the importance of the component corresponding to the frequency range determined by the first output data. For example, the weight value may increase in response to an increase in the importance of the component corresponding to a specific frequency range, and the weight value may decrease in response to a decrease in the importance of the component corresponding to the specific frequency range.

[0077] In an example embodiment, the output data (e.g., the first output data and the second output data) of the fully connected layer 340 may be used to calculate weight values ​​corresponding to a plurality of points included in the frequency filter, respectively. For example, using the distance separating a point included in the frequency filter from the origin as the radius, the weight value corresponding to the radius may be calculated as the weight value corresponding to the point. The filter generation neural network 320 may output a frequency filter so that in an image input to the filter generation neural network 320, important frequency components will be maintained. That is, the frequency filter output by the filter generation neural network 320 may maintain important low-frequency components and important high-frequency components in the image. For example, the electronic device may maintain the component in the frequency filter in response to the weight value of the component being greater than or equal to a threshold value.

[0078] In an example embodiment, the filter generation neural network 320 may output different frequency filters according to the noise distribution of the image input to the filter generation neural network 320. For example, considering that most of the noise included in the image is mainly distributed in the high-frequency region, the filter generation neural network 320 may be configured such that the frequency filter output by the filter generation neural network 320 passes more components in the low-frequency band than components in the high-frequency band.

[0079] Figure 4 An example of generating a neural network by training a filter by an electronic device according to one or more example embodiments is shown.

[0080] In one example embodiment, the electronic device may train the filter generation neural network 420. The electronic device may train the filter generation neural network 420 using the training image 410 and the real denoised image 441 mapped to the training image 410 as training data. In one example, the real denoised image 441 may be an image from which noise is removed from the training image 410.

[0081] In one example embodiment, the electronic device may train the filter generation neural network 420 and the denoising neural network 430 together. The electronic device may calculate the loss using the training data including the training image 410, and train the filter generation neural network 420 and the denoising neural network 430 together using the calculated loss. The electronic device may generate the trained filter generation neural network 420 by repeatedly updating the parameters of each of the filter generation neural network 420 and the denoising neural network 430 until the loss converges and / or until the loss becomes less than a threshold loss.

[0082] An example of how to calculate the loss is described below. In an example embodiment, the electronic device may generate a temporary transformed frequency image 411 by transforming a training image 410 included in the training data into the frequency domain. The electronic device may obtain a temporary frequency filter 412 to be applied to the temporary transformed frequency image 411 by inputting the temporary transformed frequency image 411 in the frequency domain into a filter generation neural network 420. The electronic device may generate a temporary output image 413 by applying the obtained temporary frequency filter 412 to the generated temporary transformed frequency image 411. The electronic device may obtain a temporary denoised image 440 by inputting the temporary output image 413 into a denoising neural network 430. The electronic device may calculate the loss using the obtained temporary denoised image 440 and a true denoised image 441 mapped to the training image 410. For example, the loss may represent the error between the temporary denoised image 440 and the true denoised image 441.

[0083] In another example embodiment, the electronic device may independently train the filter generation neural network 420 separately from the denoising neural network 430. The electronic device may obtain a pre-trained denoising neural network 430. That is, the parameters of the obtained denoising neural network 430 may be predetermined parameters, and the electronic device may only update the parameters of the filter generation neural network 420. The electronic device may calculate the loss using the training data including the training image 410, and independently train the filter generation neural network 420 using the calculated loss. The loss may be calculated in a manner similar to that described above. In an example embodiment, the electronic device may generate a temporary transformed frequency image 411 by transforming the training image 410 included in the training data into the frequency domain. The electronic device may obtain a temporary frequency filter 412 to be applied to the temporary transformed frequency image 411 by inputting the temporary transformed frequency image 411 in the frequency domain to the filter generation neural network 420. The electronic device may generate a temporary output image 413 by applying the obtained temporary frequency filter 412 to the generated temporary transformed frequency image 411. The electronic device may obtain a temporary denoised image by inputting the temporary output image 413 into a pre-trained denoising neural network (e.g., a denoising neural network different from the denoising neural network 430 before being trained). The electronic device may calculate the loss using the obtained temporary denoised image and the real denoised image 441 mapped to the training image 410. For example, the loss may represent the error between the temporary denoised image 440 and the real denoised image 441.

[0084] In addition, the electronic device may train the filter generation neural network 420 without using a denoising neural network. First, the electronic device may normalize the noise distribution of the training image 410. For example, normalizing the noise distribution may be interpreted herein as transforming the previous noise distribution of the training image 410 into a standard normal distribution with a mean of zero (0) and a standard deviation of 1, but normalizing the noise distribution is not limited thereto. There may be various methods of normalizing the noise distribution that may vary depending on the setting of the electronic device that performs the training of the filter generation neural network 420. The electronic device may obtain a real output image by normalizing the noise distribution in the training image 410. In one example, the electronic device may calculate a loss using training data including a training image 410 and a real output image corresponding to the training image 410, and may train the filter generation neural network 420 using the calculated loss. For example, the electronic device may generate a temporary transformed frequency image 411 by transforming the training image 410 into the frequency domain, and may input the temporary transformed frequency image 411 of the frequency domain into the filter generation neural network 420 to obtain a temporary frequency filter 412 to be applied to the temporary transformed frequency image 411. The electronic device may generate a temporary output image 413 by applying the obtained temporary frequency filter 412 to the generated temporary transformed frequency image 411. The electronic device may calculate a loss using the obtained temporary output image 413 and a real output image mapped to the training image 410. For example, the loss may represent an error between the temporary output image 413 and the real output image mapped to the training image 410.

[0085] Figure 5 Examples of an input image, a denoised image, a noise distribution of the input image, and a noise distribution of the denoised image are illustrated according to one or more example embodiments.

[0086] Reference Figure 5, input image 501 (e.g., an image including real noise), input image 502 (e.g., an image including Gaussian noise), and input image 503 (e.g., an image including adversarial attack) may be images to which no frequency filter is applied, and noise distribution 511 (e.g., real noise distribution), noise distribution 512 (e.g., Gaussian noise distribution), and noise distribution 513 (e.g., adversarial attack distribution) may be distributions of noise included in input images 501, 502, and 503, respectively. In addition, output image 521 (e.g., an image including real noise using lpf), output image 522 (e.g., an image including Gaussian noise using lpf), and output image 523 (e.g., an image including adversarial attack using lpf) may be images generated as a result of applying frequency filters to input images 501, 502, and 503, respectively, and noise distribution 531 (e.g., real noise distribution), noise distribution 532 (e.g., Gaussian noise distribution), and noise distribution 533 (e.g., adversarial attack distribution) may be distributions of noise included in output images 521, 522, and 523, respectively. For example, output image 521 may be generated by combining the frequency filter with a neural network generated from a filter (e.g., Figure 3 The filter generating neural network 320 of FIG. 3 may be an image generated by applying a frequency filter corresponding to the input image 501 output by the filter generating neural network 320 to the input image 501. Similarly, the output image 522 may be an image generated by applying a frequency filter corresponding to the input image 502 to the input image 502, and the output image 523 may be an image generated by applying a frequency filter corresponding to the input image 503 to the input image 503.

[0087] For example, input image 501 and noise distribution 511 of input image 501 may be obtained from a real noise dataset. Input image 502 and noise distribution 512 of input image 502 may be generated from synthetic noise (e.g., Gaussian noise). Input image 503 and noise distribution 513 of input image 503 may be generated from an adversarial attack. The frequency filter that generates the output of the neural network from the filter may normalize the corresponding noise distributions of input images 501, 502, and 503. Figure 5 , it can be verified that the noise distributions respectively included in the input images 501 , 502 , and 503 are normalized so that the noise distributions 531 , 532 , and 533 of the output images 521 , 522 , and 523 look similar.

[0088] Figure 6 An example structure of an electronic device according to one or more example embodiments is illustrated.

[0089] In an example embodiment, the electronic device 601 may include an image sensor 610, a processor 611 (e.g., one or more processors), a display 613, and a memory 614 (e.g., one or more memories). Figures 1 to 5 The electronic device described above. The image sensor 610 may generate an input image of an external object by receiving light. The image sensor 610 may send the generated input image to the processor 611. The processor 611 may perform denoising on the input image received from the image sensor 610. The processor 611 may perform the above-mentioned reference Figures 1 to 5 The processor 611 may use training data including training images to train a filter generating neural network. The processor 611 may send a denoised image in which noise is removed from an input image to a display 613 and a memory 614. The display 613 may output the denoised image received from the processor 611. The memory 614 may store the denoised image received from the processor 611. The memory 614 may map the denoised image to the input image and store them together. In one example, the memory 614 may be or include a non-transitory computer-readable storage medium storing instructions that, when executed by the processor 611, configure the processor 611 to perform the operations described herein. Figures 1 to 5 Any one, any combination, or all of the operations and methods described.

[0090] about Figures 1 to 6The electronic devices, image sensors, processors, displays, memories, electronic devices 601, image sensors 610, processors 611, displays 613, memories 614 and other devices, devices, units, modules and components disclosed and described herein are implemented by hardware components or represent hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond and run instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or multiple computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components. As described above, or in addition to the description above, the example hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0091] The operations described in this application are performed in Figures 1 to 6 The method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as executing instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0092] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) run directly by one or more processors or computers. In other examples, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be written using any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, and the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein disclose algorithms for performing operations performed by hardware components and methods as described above.

[0093] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, and are therefore not signals themselves. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type memory (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0094] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered to be merely descriptive and not for limiting purposes. The description of features or aspects in each example is to be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0095] Therefore, in addition to the disclosure above and throughout the accompanying drawings, the scope of the disclosure also includes the claims and their equivalents (that is, all changes within the scope of the claims and their equivalents should be construed as being included in the disclosure).

Claims

1. An electronic device, include: One or more processors configured to: Generates a transformed frequency image by transforming the input image into the frequency domain, A frequency filter corresponding to the input image is obtained by inputting the generated transformed frequency image into a filter generation neural network, and An output image in which the noise distribution of the input image is normalized is generated by applying the obtained frequency filter to the generated transformed frequency image.

2. The electronic device according to claim 1, further comprising: include: An image sensor comprising a plurality of photodiodes, Wherein the one or more processors are configured to generate an input image using the plurality of photodiodes of the image sensor.

3. The electronic device according to claim 1, in, The one or more processors are configured to generate a denoised image in which noise is removed from an input image by inputting the generated output image into a denoising neural network.

4. The electronic device according to claim 3, in, The one or more processors are configured to train a filter generation neural network and a denoising neural network together based on a loss calculated using a generated denoised image and a true denoised image mapped to an input image.

5. The electronic device according to claim 3, in, The one or more processors are configured to train a filter generation neural network based on a loss calculated using the generated denoised image and a true denoised image mapped to an input image.

6. The electronic device according to claim 1, in, The one or more processors are configured to train a filter generation neural network based on a loss calculated using a generated output image and a true output image mapped to an input image.

7. The electronic device according to claim 1, in, To generate an output image, the one or more processors are configured to: generating an intermediate image in the frequency domain by applying the obtained frequency filter to the generated transformed frequency image; and The output image is generated by transforming the generated intermediate image into the spatial domain.

8. The electronic device according to any one of claims 1 to 7, in, The filter generation neural network includes one or more convolutional neural networks and a fully connected layer connected to the one or more convolutional neural networks.

9. The electronic device according to claim 8, in, The first output data output from the fully connected layer indicates a frequency range of a frequency domain in a frequency filter, and The second output data output from the fully connected layer indicates a weight value of a component corresponding to the frequency range indicated by the first output data.

10. The electronic device according to claim 9, in, To obtain the frequency filter, the one or more processors are configured to: in response to the weight value being greater than or equal to a threshold, maintain the component in the frequency filter.

11. The electronic device according to any one of claims 1 to 7, in, The filter generation neural network is configured to output different frequency filters according to the noise distribution of the image input to the filter generation neural network.

12. An electronic device, include: One or more processors configured to: generating a temporary transformed frequency image by transforming the training input image into the frequency domain, obtaining a temporary frequency filter by inputting the generated temporary transformed frequency image into a filter generating neural network, generating a temporary output image by applying the obtained temporary frequency filter to the generated temporary transformed frequency image, and generating a temporary denoised image by inputting the generated temporary output image into a denoising neural network, and The filter generation neural network and the denoising neural network are trained together based on a loss determined using the generated temporary denoised image and the true denoised image mapped to the training input image.

13. A method for adaptive frequency filtering, include: generating a transformed frequency image by transforming an input image into the frequency domain; Obtaining a frequency filter corresponding to an input image by inputting the generated transformed frequency image into a filter generation neural network; as well as An output image in which the noise distribution of the input image is normalized is generated by applying the obtained frequency filter to the generated transformed frequency image.

14. The method according to claim 13, further comprising: include: A denoised image in which noise is removed from the input image is generated by inputting the generated output image to a denoising neural network.

15. The method according to claim 13, in, The steps to generate the output image are: generating an intermediate image in the frequency domain by applying the obtained frequency filter to the generated transformed frequency image; and The output image is generated by transforming the generated intermediate image into the spatial domain.

16. The method according to any one of claims 13 to 15, in, The filter generation neural network includes one or more convolutional neural networks and a fully connected layer connected to the one or more convolutional neural networks.

17. The method according to claim 16, in, The first output data output from the fully connected layer indicates a frequency range of a frequency domain in a frequency filter, and The second output data output from the fully connected layer indicates a weight value of a component corresponding to the frequency range determined by the first output data.

18. The method according to any one of claims 13 to 15, further comprising: include: The filter generation neural network outputs different frequency filters according to the noise distribution of the image input to the filter generation neural network.

19. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method according to any one of claims 13 to 18.

20. A method for adaptive frequency filtering, include: generating a temporary transformed frequency image by transforming the training input image into the frequency domain, obtaining a temporary frequency filter by inputting the generated temporary transformed frequency image into a filter generating neural network, generating a temporary output image by applying the obtained temporary frequency filter to the generated temporary transformed frequency image, and generating a temporary denoised image by inputting the generated temporary output image into a denoising neural network; as well as The filter generation neural network and the denoising neural network are trained together based on a loss determined using the generated temporary denoised image and the true denoised image mapped to the training input image.