Method and system for image denoising

Abstract

Methods and systems for image denoising are provided. The methods include obtaining a difference map between a reference frame and a non-reference frame, estimating a noise variance of the obtained difference map, obtaining a merging weight based on the estimated noise variance and the obtained difference map, and merging the non-reference frame with the reference frame using the obtained merging weight.

Description

[0001] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 124,247, filed December 11, 2020, and U.S. Non-Provisional Patent Application No. 17 / 215,529, filed March 29, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to image denoising. Background Technology

[0003] In multi-frame image fusion, it is desirable to reduce noise and for the fusion to be robust to alignment errors caused by incomplete camera compensation and / or motion in the scene. Summary of the Invention

[0004] According to one aspect of this disclosure, a method is provided. The method includes: obtaining a difference map between a reference frame and a non-reference frame; estimating the noise variance of the obtained difference map; obtaining a merging weight based on the estimated noise variance and the obtained difference map; and merging the non-reference frame with the reference frame using the obtained merging weight.

[0005] According to one aspect of this disclosure, a system is provided. The system includes a memory and a processor configured to: obtain a difference map between a reference frame and a non-reference frame; estimate the noise variance of the obtained difference map; obtain a merging weight based on the estimated noise variance and the obtained difference map; and merge the non-reference frame with the reference frame using the obtained merging weight. Attached Figure Description

[0006] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0007] Figure 1 A diagram illustrating a system for multi-frame image fusion according to an embodiment is shown;

[0008] Figure 2 A flowchart illustrating a method for multi-frame image fusion according to an embodiment is shown; and

[0009] Figure 3 A block diagram of an electronic device in a network environment according to one embodiment is shown. Detailed Implementation

[0010] In the following description, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, they will be designated by the same reference numerals. In the following description, specific details such as detailed configurations and components are provided only to aid in a comprehensive understanding of the embodiments of the present disclosure. Therefore, those skilled in the art will understand that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and constructions have been omitted. The terminology described below is defined in consideration of the functions in this disclosure and may vary depending on the user, the user's intent, or habit. Therefore, the definitions of the terms should be determined based on the content throughout this specification.

[0011] This disclosure can have various modifications and embodiments, which are described in detail below with reference to the accompanying drawings. However, it should be understood that this disclosure is not limited to the embodiments, but includes all modifications, equivalents, and substitutions within the scope of this disclosure.

[0012] Although terms including ordinal numbers (such as first, second, etc.) may be used to describe various elements, structural elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, a first structural element may be referred to as a second structural element without departing from the scope of this disclosure. Similarly, a second structural element may also be referred to as a first structural element. As used herein, the term "and / or" includes any and all combinations of one or more related items.

[0013] The terminology used herein is for the purpose of describing various embodiments of this disclosure only and is not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In this disclosure, it should be understood that the terms “comprising” or “having” indicate the presence of features, quantities, steps, operations, structural elements, components, or combinations thereof, and do not exclude the possibility of the presence of one or more other features, quantities, steps, operations, structural elements, components, or combinations thereof, or the addition of one or more other features, quantities, steps, operations, structural elements, components, or combinations thereof.

[0014] Unless otherwise defined, all terms used herein shall have the same meaning as understood by one of skill in the art to which this disclosure pertains. Unless expressly defined in this disclosure, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as in the context of the relevant field, and shall not be interpreted as having an idealized or overly formal meaning.

[0015] The electronic device according to one embodiment can be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to one disclosed embodiment, the electronic device is not limited to those described above.

[0016] The terminology used in this disclosure is not intended to limit the disclosure, but rather to include various modifications, equivalents, or substitutions of the corresponding embodiments. Similar reference numerals may be used to denote similar or related elements in relation to the description of the drawings. Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to an item may include one or more things. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include all possible combinations of the items listed together in the corresponding one of the phrases. As used herein, terms such as “first,” “second,” “first,” and “second” may be used to distinguish a corresponding component from another component, and are not intended to limit the component in other respects (e.g., importance or order). The intention is that if an element (e.g., a first element) is referred to as being "combined," "connected to," "attached to," or "linked" to another element (e.g., a second element) with or without the terms "operably" or "communically," it indicates that the element can be combined with the other element directly (e.g., wired), wirelessly, or via a third element.

[0017] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," and "circuit"). A module may be a single integrated component or its smallest unit or component suitable for performing one or more functions. For example, according to one embodiment, a module may be implemented as an application-specific integrated circuit (ASIC).

[0018] For adaptive temporal denoising, a multi-frame-based noise variance estimation is disclosed, where a noise power map (per pixel) is estimated to control the temporal fusion power. The power of the difference between the reference frame and each non-reference frame is compared with the corresponding estimated noise power to determine whether the difference is due to noise (i.e., applying full averaging) or due to local motion (i.e., rejecting fusion).

[0019] For the system model, consider bursts of N noisy images, where y k (x,y) represents the brightness of the k-th deformed image frame (aligned with the reference frame after image registration), c k(x,y) represents the chroma (U channel or V channel) of the k-th image frame, k∈{0,…,N-1}, and (x,y) represents the pixel position in two-dimensional (2D) space.

[0020] One objective of multi-frame fusion is to apply temporal averaging in non-motion regions (i.e., where reference and non-reference frames are well aligned) and avoid averaging in locally motion regions (i.e., where reference and non-reference frames are misaligned). An example of a fusion strategy is to obtain a difference map between reference and non-reference frames to determine the fusion level, which is determined by comparing it to a preset threshold (i.e., a representative noise power). For adaptive temporal denoising, a multi-frame-based noise variance estimation is disclosed, where a noise power map (e.g., per pixel) is estimated to control the level of temporal fusion. The power of the difference between the reference frame and each non-reference frame is compared to the corresponding estimated noise power to determine whether the difference is due to noise (i.e., applying full averaging) or due to local motion (i.e., rejecting fusion).

[0021] First, the system obtains a difference map between the reference frame and the non-reference frame. The difference map can be the difference map (D) between the reference frame (e.g., y0) and the k-th non-reference frame. k (i,j), where i∈{1,…,W},j∈{1,…,H}). The difference map can be obtained as in equation (1), where W and H are the width and height of the image frame, respectively, Blur() represents the blur operation, and d Blur,k Intermediate result of the first blurring operation:

[0022]

[0023] Blur operations can be applied to reduce the impact of noise in the difference map. Any blur operation can be used (such as 5×5 Gaussian filtering or 5×5 box filtering).

[0024] Next, the system estimates the noise variance of the difference plot. The noise variance can be estimated as in equation (2):

[0025] V k (i,j)=min(D k (i,j),SigmaMaxD)-Blur(d Blur,k (i,j)) 2 (2)

[0026] Equation (2) calculates the local variance V of the difference between the two frames. k(i,j), and the local variance is based on the difference between the difference map and the fuzzy term. For perfectly aligned regions, the variance indicates the noise power. For locally moving regions, the variance of the difference map indicates large values ​​around the boundaries of the moving object, which can be limited by the tuning parameter SigmaMaxD, which acts as a predetermined upper limit threshold. For Blur(d Blur,k (i,j)) 2 >SigmaMaxD, V k (i,j)=0, which is equivalent to the declared motion.

[0027] The estimated noise variance can be further controlled using the tuning parameters SigmaMin, SigmaMax and SigmaScal as in equation (3), where the tuning parameters SigmaMin, SigmaMax and SigmaScal are used as the predetermined lower threshold, predetermined upper threshold and predetermined scaling threshold, respectively.

[0028]

[0029] Then, the system obtains the merge weights. The merge weights can be obtained as shown in equation (4):

[0030]

[0031] in, This represents the estimated noise power map. The merging weights can be obtained using the floor. In equation (4), although the floor is set to 0, other values ​​for the floor can be used. The merging weights can be normalized for the difference map. In equation (4), this is done by dividing by D. k The merging is performed using (i,j). The merging weights in Equation (4) are based on the difference between the difference map and the noise power map. Merging can be performed in the form shown in Equation (5).

[0032]

[0033] Figure 1 A diagram illustrating a system 100 for multi-frame image fusion according to an embodiment is shown. The system receives the k-th input image and performs image registration 102. The system determines the image brightness y. k Or chromaticity, and obtain the difference map d based on luminance or chromaticity. k While this disclosure discusses luminance and chromaticity by way of examples of image, video, or pixel parameters, other such parameters (e.g., red-green-blue (RGB), red-green-blue-green (RGBG), or parameters of other encoding schemes) may be appropriately used (e.g., to determine a difference map). At 104, the system performs noise variance estimation to produce a noise variance estimate. In step 106, the system performs a merge weight calculation to obtain the merge weight A.k (i,j). In 108, the system accumulates and normalizes k images to obtain the final merged result.

[0034] Figure 2 A flowchart 200 illustrates a method for multi-frame image fusion according to an embodiment. At 202, the system obtains a difference map between a reference frame and non-reference frames. At 204, the system estimates the noise variance of the obtained difference map. At 206, the system obtains merging weights based on the estimated noise variance and the obtained difference map. At 208, the system merges the non-reference frames with the reference frames using the merging weights obtained in 206. Merging can be performed independently for each non-reference frame, and the final merging result can be obtained after accumulating and normalizing the merging contributions from each non-reference frame.

[0035] Figure 3 A block diagram of an electronic device 301 in a network environment 300 according to one embodiment is shown. (Refer to...) Figure 3 In network environment 300, electronic device 301 can communicate with electronic device 302 via a first network 398 (e.g., a short-range wireless communication network), or with electronic device 304 or server 308 via a second network 399 (e.g., a long-range wireless communication network). Electronic device 301 can communicate with electronic device 304 via server 308. Electronic device 301 may include processor 320, memory 330, input device 350, sound output device 355, display device 360, audio module 370, sensor module 376, interface 377, haptic module 379, camera module 380, power management module 388, battery 389, communication module 390, connection terminal 378, subscriber identification module (SIM) 396, or antenna module 397. In one embodiment, at least one of the components (e.g., display device 360 ​​or camera module 380) may be omitted from electronic device 301, or one or more other components may be added to electronic device 301. In one embodiment, some of the components may be implemented as a single integrated circuit (IC). For example, sensor module 376 (e.g., fingerprint sensor, iris sensor, or illuminance sensor) may be embedded in display device 360 ​​(e.g., display).

[0036] Processor 320 can execute, for example, software (e.g., program 340) to control at least one other component (e.g., hardware or software component) of electronic device 301 connected to processor 320, and can perform various data processing or calculations. As at least part of the data processing or calculations, processor 320 can load commands or data received from other components (e.g., sensor module 376 or communication module 390) into volatile memory 332, process the commands or data stored in volatile memory 332, and store the resulting data in non-volatile memory 334. Processor 320 may include a main processor 321 (e.g., central processing unit (CPU) or application processor (AP)) and an auxiliary processor 323 (e.g., graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that can operate independently of or in conjunction with the main processor 321. Additionally or optionally, auxiliary processor 323 may be adapted to consume less power than main processor 321 or to perform specific functions. The auxiliary processor 323 can be implemented as separate from the main processor 321 or as part of the main processor 321.

[0037] The auxiliary processor 323 may replace the main processor 321 when the main processor 321 is inactive (e.g., in sleep) or, when the main processor 321 is active (e.g., executing an application), work with the main processor 321 to control at least some of the functions or states associated with at least one component of the electronic device 301 (e.g., display device 360, sensor module 376, or communication module 390). According to one embodiment, the auxiliary processor 323 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 380 or communication module 390) functionally associated with the auxiliary processor 323.

[0038] The memory 330 may store various data used by at least one component of the electronic device 301 (e.g., processor 320 or sensor module 376). The various data may include, for example, software (e.g., program 340) and input or output data for commands associated therewith. The memory 330 may include volatile memory 332 or non-volatile memory 334.

[0039] The program 340 may be stored as software in the memory 330 and may include, for example, an operating system (OS) 342, middleware 344, or application 346.

[0040] Input device 350 can receive commands or data from outside electronic device 301 (e.g., a user) that will be used by other components of electronic device 301 (e.g., processor 320). Input device 350 may include, for example, a microphone, mouse, or keyboard.

[0041] The sound output device 355 can output sound signals to the outside of the electronic device 301. The sound output device 355 may include, for example, a speaker or a receiver. The speaker can be used for general purposes (such as playing multimedia or recording), and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0042] Display device 360 ​​can visually provide information to the outside of electronic device 301 (e.g., to a user). Display device 360 ​​may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to one embodiment, display device 360 ​​may include touch circuitry adapted to detect touch, or sensor circuitry adapted to measure the intensity of the force caused by touch (e.g., a pressure sensor).

[0043] The audio module 370 can convert sound into electrical signals and vice versa. According to one embodiment, the audio module 370 can obtain sound via an input device 350, or output sound via a sound output device 355 or headphones of an external electronic device 302 that is directly (e.g., wired) or wirelessly connected to the electronic device 301.

[0044] Sensor module 376 can detect the operating state of electronic device 301 (e.g., power or temperature) or the environmental state outside electronic device 301 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. Sensor module 376 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0045] Interface 377 may support one or more specified protocols for direct (e.g., wired) or wireless connection of electronic device 301 to external electronic device 302. According to one embodiment, interface 377 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0046] Connection terminal 378 may include a connector via which electronic device 301 can be physically connected to external electronic device 302. According to one embodiment, connection terminal 378 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module 379 can convert electrical signals into mechanical stimulation (e.g., vibration or motion) or electrical stimulation that can be recognized by a user via touch or kinesthesia. According to one embodiment, the haptic module 379 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0048] Camera module 380 can capture still or moving images. According to one embodiment, camera module 380 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0049] The power management module 388 manages the power supplied to the electronic device 301. The power management module 388 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0050] Battery 389 can supply power to at least one component of electronic device 301. According to one embodiment, battery 389 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] Communication module 390 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 301 and external electronic devices (e.g., electronic device 302, electronic device 304, or server 308), and perform communication via the established communication channel. Communication module 390 may include one or more communication processors that can operate independently of processor 320 (e.g., AP) and support direct (e.g., wired) or wireless communication. According to one embodiment, communication module 390 may include wireless communication module 392 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 394 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 398 (e.g., such as Bluetooth). TMThe communication module 392 communicates with external electronic devices via a short-range communication network (such as a Wi-Fi Direct or Infrared Data Association (IrDA) standard network) or a second network 399 (such as a long-range communication network like a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)) . These various types of communication modules can be implemented as a single component (e.g., a single IC) or as multiple components that are separate from each other (e.g., multiple ICs). The wireless communication module 392 can use user information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the user identification module 396 to identify and authenticate electronic device 301 in a communication network (such as a first network 398 or a second network 399).

[0052] Antenna module 397 can transmit or receive signals or power to or from the outside of electronic device 301 (e.g., external electronic device). According to one embodiment, antenna module 397 may include one or more antennas, and at least one antenna suitable for a communication scheme used in a communication network (such as a first network 398 or a second network 399) may be selected, for example, by communication module 390 (e.g., wireless communication module 392). Signals or power can then be transmitted or received between communication module 390 and external electronic device via the selected at least one antenna.

[0053] At least some of the aforementioned components can be interconnected and transmit signals (e.g., commands or data) between them via inter-peripheral communication schemes (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0054] According to one embodiment, commands or data can be sent or received between electronic device 301 and external electronic device 304 via server 308 connected to a second network 399. Each of electronic devices 302 and 304 can be a device of the same or different type as electronic device 301. All or some of the operations to be performed at electronic device 301 can be performed at one or more of the external electronic devices 302, 304, and server 308. For example, if electronic device 301 is required to perform a function or service automatically or in response to a request from a user or another device, electronic device 301 may perform the function or service instead, or may request one or more external electronic devices to perform at least a portion of the function or service in addition to performing the function or service. One or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to electronic device 301. Electronic device 301 may provide the result, with or without further processing, as at least part of a response to the request. For this purpose, technologies such as cloud computing, distributed computing, or client-server computing may be used.

[0055] One embodiment may be implemented as software (e.g., program 340) including one or more instructions stored in a storage medium (e.g., internal memory 336 or external memory 338) readable by a machine (e.g., electronic device 301). For example, a processor of electronic device 301 may invoke at least one of the one or more instructions stored in the storage medium and execute at least one instruction under the control of the processor with or without one or more other components. Thus, the machine may be operable to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between a location where data is semi-permanently stored in the storage medium and a location where data is temporarily stored in the storage medium.

[0056] According to one embodiment, the disclosed method may include being in and providing a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TMThe computer program product may be distributed online (e.g., downloaded or uploaded) or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0057] According to one embodiment, each of the above components (e.g., a module or program) may include a single entity or multiple entities. One or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component can still perform one or more functions of each of the multiple components in the same or similar manner as one or more functions of each of the multiple components were performed by the corresponding components of the multiple components prior to integration. Operations performed by modules, programs, or other components may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

[0058] Although specific embodiments of this disclosure have been described in detail, this disclosure can be modified in various forms without departing from its scope. Therefore, the scope of this disclosure should not be determined solely based on the described embodiments, but rather on the appended claims and their equivalents.

Claims

1. A method for image denoising, comprising: Obtain the difference map between the reference frame and the non-reference frame; Estimate the noise variance of the obtained difference plot; A noise power map is obtained by controlling the noise variance using at least one tuning parameter. The merging weight is obtained by normalizing the difference between the difference map and the noise power map, wherein the difference is normalized by dividing by the difference map; as well as The non-reference frames are merged with the reference frames using the obtained merging weights.

2. The method according to claim 1, wherein, The steps to obtain a difference map include applying a fuzzing operation.

3. The method according to claim 2, wherein, Blurring operations include Gaussian filtering.

4. The method according to claim 2, wherein, Blur operations include box filtering.

5. The method according to claim 1, wherein, The estimated noise variance is based on the difference between the difference map and the fuzzy term.

6. The method according to claim 1, wherein, The steps to obtain a difference map between a reference frame and a non-reference frame include: obtaining a difference map based on brightness.

7. The method according to claim 1, wherein, The steps to obtain a difference map between a reference frame and a non-reference frame include: obtaining a difference map based on chroma.

8. A system for image denoising, comprising: Memory; as well as The processor is configured as follows: Obtain the difference map between the reference frame and the non-reference frame. Estimate the noise variance of the obtained difference plot. A noise power map is obtained by controlling the noise variance using at least one tuning parameter. The merging weights are obtained by normalizing the difference between the difference map and the noise power map, wherein the difference is normalized by dividing by the difference map. The non-reference frames are merged with the reference frames using the obtained merging weights.

9. The system according to claim 8, wherein, The processor is also configured to obtain a difference map by applying a fuzzing operation.

10. The system according to claim 9, wherein, Blurring operations include Gaussian filtering.

11. The system according to claim 9, wherein, Blur operations include box filtering.

12. The system according to claim 8, wherein, The estimated noise variance is based on the difference between the difference map and the fuzzy term.

13. The system according to claim 8, wherein, The processor is also configured to obtain a difference map based on brightness.

14. The system according to claim 8, wherein, The processor is also configured to obtain a difference map based on chroma.

Images