Device and method for denoising continuous multi-frame images

Abstract

This document discloses an apparatus and method for denoising consecutive multi-frame images. According to one embodiment, the apparatus includes a Wiener filter configured to filter frames of an image; and a frame delayer configured to feed the filtered frames back to the Wiener filter, wherein the Wiener filter is further configured to filter subsequent frames of the image based on the filtered frames. The apparatus for denoising consecutive multi-frame images disclosed herein has good quality.

Description

[0001] [priority]

[0002] This application claims priority to U.S. provisional patent application No. 62 / 675,377 filed in the U.S. Patent and Trademark Office on May 23, 2018, and U.S. non-provisional patent application No. 16 / 380,473 filed in the U.S. Patent and Trademark Office on April 10, 2019, which are incorporated by reference in their entirety into this application. Technical Field

[0003] The present disclosure relates generally to image processing, and more particularly to an apparatus and method for successive multi-frame denoising. Background Art

[0004] Digital photography in low-light conditions can be challenging, as long exposure times (to collect more light) can cause motion blur, while short exposure times (to avoid motion blur) can increase noise levels. To overcome these challenges, a typical approach is to use burst photography and fuse multiple frames to produce a clear, low-noise image. However, multi-frame fusion requires two main steps: registration to align the image frames (accounting for both global and local motion), and merging (fusion) multiple frames to enhance quality. Summary of the Invention

[0005] According to one embodiment, an apparatus is provided, comprising: a Wiener filter configured to filter a frame of an image; and a frame delayer configured to feed the filtered frame back to the Wiener filter, wherein the Wiener filter is further configured to filter a subsequent frame of the image based on the filtered frame.

[0006] According to one embodiment, an apparatus is provided. The apparatus includes: a first subtractor including a first input for receiving a frame of an image, a second input for receiving a reference frame, and an output; an absolute value function block including an input connected to the output of the first subtractor and an output; a second subtractor including a first input connected to the output of the absolute value function block, a second input for receiving a first predetermined value, and an output; and a maximum value divider function block including an input connected to the output of the second subtractor and an output for outputting filter weights.

[0007] According to one embodiment, a method is provided, comprising: filtering a frame of an image by a Wiener filter; feeding the filtered frame back to the Wiener filter by a frame delayer; and filtering subsequent frames of the image by the Wiener filter based on the filtered frame.

[0008] According to one embodiment, a method is provided. The method includes: subtracting a reference value from a frame of an image by a first subtractor; determining an absolute value of an output of the first subtractor by an absolute value function block; subtracting a first predetermined value from the output of the absolute value function block by a second subtractor; and determining a maximum value divider of an output of the second subtractor by a maximum value divider function block. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent upon reading the following detailed description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram of an apparatus for providing multi-frame denoising according to one embodiment.

[0011] Figure 2 According to one embodiment Figure 1 Figure 1 shows a block diagram of a combiner for successive robust merging.

[0012] Figure 3 According to one embodiment Figure 1 Figure 1 shows a block diagram of a combiner for difference successive robust merging.

[0013] Figure 4 According to one embodiment Figure 1 FIG. 4 is a block diagram of a combiner for minimum mean square error (MMSE) combining.

[0014] Figure 5 According to one embodiment Figure 1 FIG. 4 is a block diagram of a combiner for unbiased MMSE (U-MMSE) combining.

[0015] Figure 6 is a block diagram of an apparatus for providing multi-frame denoising according to one embodiment.

[0016] Figure 7 According to one embodiment Figure 6 A block diagram of a merger for continuous pair-wise three-dimensional (3D) merging is shown.

[0017] Figure 8 The figure is a flowchart of a multi-frame denoising method according to an embodiment.

[0018] Figure 9 The figure is a flowchart of a multi-frame denoising method according to an embodiment.

[0019] Figure 10 is a block diagram of electronic devices in a network environment to which the apparatus and method of the present disclosure are applied according to one embodiment.

[0020] Figure 11 It is a block diagram of a program for applying the apparatus and method of the present disclosure according to one embodiment. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. Although identical elements are shown in different figures, identical elements are designated by the same reference numerals. In the following description, specific details such as detailed configurations and components are provided solely to facilitate a comprehensive understanding of the embodiments of the present disclosure. Therefore, it should be apparent to those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure.

[0022] For the sake of clarity and brevity, descriptions of well-known functions and structures are omitted. The following terms are defined with reference to the functions of this disclosure and may vary depending on the user, their intent, or their habits. Therefore, the definitions of these terms should be determined based on the content of this specification.

[0023] The present disclosure may have various modifications and various embodiments, some of which are described in detail below with reference to the accompanying drawings. However, the present disclosure is not limited to the embodiments described, but includes all modifications, equivalent forms and non-reference forms within the scope of the present disclosure.

[0024] Although terms including ordinal numbers such as "first" and "second" may be used to describe various elements, the structural elements are not limited by these terms. These terms are only used to distinguish between the various elements. For example, without departing from the scope of the present disclosure, a "first structural element" may be referred to as a "second structural element." Similarly, a "second structural element" may also be referred to as a "first structural element."

[0025] As used herein, the term "and / or" includes any and all combinations of one or more associated items.

[0026] The terms used herein are only used to illustrate various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In the present disclosure, it should be understood that the term "include" or "have" indicates the presence of a feature, number, step, operation, structural element, component or combination thereof, and does not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, structural elements, components or combinations thereof.

[0027] Unless otherwise defined, all terms used herein have the same meaning as understood by those skilled in the art to which the present disclosure pertains. For example, terms defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant technical field, and should not be interpreted as having an idealized or overly formal meaning unless clearly defined in the present disclosure.

[0028] According to one embodiment, an apparatus and method provide multi-frame denoising, also known as successive pairwise 3D merging, in which bursts of noisy images are merged to produce a less noisy image. This reduces noise and is robust to residual alignment errors (due to residual errors when compensating for both camera and scene motion).

[0029] There may already be some level of alignment between the frames, and this can be focused on during the multi-frame merging step. Additionally, the image frames may be aligned with some level of residual alignment error, and artificial alignment error may be introduced to evaluate the merging solution in the presence of alignment error.

[0030] In one embodiment, a device selects a frame as a reference frame (either predetermined or dynamically selected) and divides all frames into small patches. The device sorts the aligned non-reference patches based on their similarity to the corresponding reference patches across the entire burst and continuously merges the ordered patches in a transform domain (e.g., discrete Fourier transform (DFT)). The device performs merging continuously (one patch at a time), merging non-reference patches that are more similar to the reference patch first. The device merges each non-reference patch in the 3D domain with the most recently merged result initialized with the reference patch by applying Wiener filtering to the 3D transform domain components, which improves the peak signal to noise ratio (PSNR) and structural similarity (SSIM) over the synthesized noisy burst. The device and method perform better than other typical methods, including robust merging and video block-matching 4D filtering (V-BM4D).

[0031] In an embodiment, a robust merging method specifies that the exposure is constant across all frames in a burst, which makes the alignment more robust (no noise level differences exist between different frames in a burst). Second, the merging is performed on the original Bayer image rather than demosaiced red, green, and blue (RGB) (or luminance (Y) and chrominance (U and V) (YUV)) frames, which provides the benefits of increased dynamic range and a simpler noise model. Third, a fast Fourier transform (FFT)-based alignment algorithm and a hybrid two-dimensional (2D) / three-dimensional Wiener filter are used to denoise and merge the frames in the burst.

[0032] In an embodiment, both alignment and merging operations are performed on a tile-by-tile basis. The merging method is based on pairwise frequency-domain temporal filters operating on the input tiles. Each tile in the reference frame is merged with one tile from each of the non-reference frames. Typically, 16×16 tiles in the color planes from the Bayer raw input are used, but for very dark scenes where low-frequency noise can be objectionable, 32×32 tiles are used.

[0033] In robust merging, each plane of the Bayer image is merged independently. In the following, the merging method is disclosed in terms of single-channel images, which can be easily extended to the case of Bayer images where each plane is merged independently.

[0034] The robust merging method operates on image patches in the spatial frequency domain. For a given reference patch, a set of corresponding patches is assembled one per frame over the entire burst, and their corresponding two-dimensional-discrete Fourier transform (2D-DFT) is calculated as T z (w), where w=(w x ,w y ) represents the spatial frequency, z is the frame index, and without loss of generality, frame 0 is regarded as the reference.

[0035] Robust merging applies the frequency domain merging in equation (1) below:

[0036]

[0037] A z is the shrinkage operator, which is defined as the variable of the Wiener filter in equation (2) below:

[0038]

[0039] Among them D z (w) = T0(w) - T z (w),σ 2 represents the noise variance, and c is a design parameter that increases noise reduction at the expense of some robustness.

[0040] The above pairwise temporal filter does not perform any spatial filtering, so spatial filtering is used as a separate post-processing step in the 2D DFT domain. By assuming that all N frames are perfectly averaged, and the estimated noise variance is updated as σ 2 / N, spatial filtering is performed by applying a pointwise shrinkage operator of the same form as in equation (2) above to the spatial frequency coefficients. In summary, the robust combined spatial filtering is in the form of the following equation (3):

[0041]

[0042] Robust merging operates on tiles that overlap by half in each spatial dimension. By smoothly blending between the overlapping tiles, visually objectionable discontinuities can be avoided at tile boundaries. Additionally, a window function is applied to the tiles to avoid edge artifacts when operating in the DFT domain. A modified raised cosine window is used, i.e., it is otherwise 0.

[0043] According to one embodiment, the apparatus and method of the present invention provide continuous robust merging as a modified form of the above robust merging method. In robust merging, the goal is to reject the non-reference frame from being merged whenever the difference (D z (w) = T0(w) - T z (w)) between the corresponding non-reference frame and the reference frame is independent of noise.

[0044] In robust merging, the non-reference frames are merged with the reference frame independently. can be defined as the merging result after merging the (M - 1)th non-reference frame. Thus, (as defined in Equation (1) above and Equation (4) below) represents the intermediate merging result of Equation (1) above:

[0045]

[0046] The goal of merging is to reduce the noise level, so it is desirable to observe a lower noise level compared to T0, i.e., assuming perfect alignment is achieved by the Mth frame, all frames are fully averaged, and after merging M frames, the noise level is reduced by If the true noise-free image of scene R can be obtained, a better merging strategy is to obtain D z (w) = R(w) - T z (w) rather than D z (w) = T0(w) - T z (w), and reject the zth non-reference frame from being merged whenever its difference from the true image R is independent of noise. Generally, the true image R cannot be obtained; however, after merging M frames, is obtained as an estimate of R.

[0047] Therefore, the apparatus according to one embodiment can use M when determining the merging level of T assuming experiences For the noise level, indexing the frames starts from 0. This scheme can be referred to as successive robust merging, which can be expressed by the following equation (5):

[0048]

[0049] where Equation (5) can be equivalently written as the following equation (6):

[0050]

[0051] According to one embodiment, an apparatus and method provide differential successive robust merging, where the obtained under the following equation (7) is only used to update A z (w):

[0052]

[0053] where and D z (w) = T0(w) - T z (w), and then can be equivalently written as the following equation (8):

[0054]

[0055] According to one embodiment, a burst of N noisy images, where I k (x, y) represents the k-th image frame k ∈ {0,..., N - 1}, and (x, y) represents the two-dimensional spatial pixel position. Each block overlaps by half in each spatial dimension of size 16×16, and is filtered using a modified raised cosine window (i.e., when x < n and 0 otherwise). The two-dimensional discrete Fourier transform (2D-DFT) of the considered block of the k-th frame (weighted by the above cosine window) is denoted as F k (w), where w = (w[[ID=​​​​​According to one embodiment, an apparatus and method provide multi-frame merging, specifically continuous pair-by-pair 3D merging. There may be residual alignment errors between the frames. In the continuous pair-by-pair 3D merging, the apparatus of the present invention continuously merges non-reference frames, wherein each reference frame is merged with the result of the most recent merge (initialized with a reference block). The aligned non-reference blocks are sorted based on their similarity to the corresponding reference blocks over the entire burst, and then the non-reference blocks are continuously merged in the transform domain. The apparatus performs the merging continuously (one block at a time), wherein the non-reference blocks that are more similar to the reference blocks are merged first. The apparatus performs the merging in the transform domain using a two-step transform.

[0057] In the first transform step, the apparatus of the present invention obtains a two-dimensional transform (e.g., a discrete Fourier transform 2D-DFT) of the reference frame and the non-reference frame. In the second transform step, the apparatus of the present invention performs a simple two-point one-dimensional DFT (1D-DFT) in the time direction on the most recently merged result and a pair of aligned frequency components of an available non-reference frame.

[0058] The device applies Wiener filtering to the 3D transform results, and then converts the filtered 3D transform results back to the 2D domain (simple 2-point inverse DFT) to update the most recently merged results. The device initializes merging using Wiener filtering on a reference frame in the 2D transform domain. For example, the continuous pairwise merging scheme can be formulated as follows:

[0059]

[0060]

[0061] in represents the latest merging result after merging M frames (merging M-1 non-reference frames into the reference frame), and equations (11) and (12) are as follows:

[0062]

[0063]

[0064] in and

[0065]

[0066] In continuous merging, the order of merging may affect the quality of the final merged result. Different metrics can be used when sorting frames, such as clarity (selecting the clearest frame as the reference and merging clearer frames first) or the matching score of non-reference frames with reference frames, where frames with higher matching scores are merged first.

[0067] According to one embodiment, the apparatus provides for merging based on the principle of MMSE estimation in the frequency domain. The motivation for considering MMSE estimation in the frequency domain (rather than the spatial domain) is that the sparse nature of natural images in the frequency domain enables us to obtain MMSE estimates independently based on the frequency components. The apparatus applies MMSE estimation in the transform domain (frequency domain) that considers each frequency component independently.

[0068] As explained above with reference to the robust merging method, overlapping blocks are merged in the frequency domain, where T z (w) is defined as the frequency domain representation of the z-th frame block, and w = (w x ,w y ) represents the spatial frequency.

[0069] R is defined as the true signal (e.g., the original noise-free image of the scene), T0(w) is considered as the noisy reference frame, and I z (w) is defined as the alignment error at the z-th frame relative to the reference frame, that is, I0(w) = 0. For the z-th non-reference frame T z For a noisy and possibly misaligned block of (w), equation (13) is as follows:

[0070] T z (w)=R(w)+I z (w)+N z (w)…(13)

[0071] Where R is the real image, and N z (w) is the variance Independent and identically distributed (iid) zero-mean Gaussian noise; T0(w)=R(w)+N0(w).

[0072] According to one embodiment, the apparatus implements merging as wherein the goal is to z The problem of estimating the image R in (w). When I0(w) = 0, frame T0 serves as the reference frame. By considering a burst of N frames and for a given spatial frequency w, we have T(w) = [T0(w), T1(w), ..., T N-1 (w)] T :

[0073]

[0074] If μ(w)=E{R(w)} and P(w) is the true image power at spatial frequency w, that is, P(w)=E{|R(w)-μ(w)| 2} and I z (w) is where E{I z For the Gaussian variable with R(w)}=0, the MMSE estimate of R(w) is given by equation (15):

[0075]

[0076] According to equation (15), the following equation (16) can be obtained:

[0077]

[0078] To obtain MMSE results, unknown And P(w). For example, the instantaneous power can be used, as shown in the following equation (17):

[0079]

[0080] As for P(w), since multiple frames are available, P(w) can be determined as follows:

[0081]

[0082] because Therefore, the above estimator is biased. Therefore, the unbiased MMSE estimate of R(w) can be obtained as follows:

[0083]

[0084] Only the To obtain an estimate In the above calculation, μ(w)=E{R(w)} is also required. μ(w) can be calculated based on the mean of the blocks in the spatial domain. By defining J(x,y) as the image intensity at the spatial location (x,y) and assuming E{J(x,y)}=u, the following equation (20) is obtained:

[0085]

[0086] Where G represents the windowing matrix, and ".*" represents element wise matrix multiplication.

[0087] In the present disclosure, five sets of multi-frame denoising apparatuses and methods are described, namely, successive robust combination (which is combined with robust combination), differential successive robust combination, successive pairwise 3D combination (which applies pairwise Wiener filtering in the 3D transform domain), MMSE combination, and unbiased MMSE combination).

[0088] Consider a burst of N noisy images, where I k (x, y) represents the k-th image frame, k ∈ {0, …, N−1} and (x, y) represents the two-dimensional spatial pixel position. For ease of notation, the color channel index is omitted. However, the present disclosure may include a color channel.

[0089] Extract M×M image patches, and apply a two-dimensional transform (e.g., 2D-DFT or 2D wavelet analysis (wavelet)) to the patches, and represent the image patches in the transform domain as F k (w x , w y ), where w = (w x , w y ) represents the two-dimensional spatial frequency. For example, the image patches can be selected to overlap by half in each spatial dimension of size M×M, and filtered using a modified raised cosine window (i.e., otherwise 0).

[0090] In an embodiment, there is a certain level of alignment between the frames, where focusing is performed during multi-frame combination with residual alignment errors present.

[0091] Figure 1 is a block diagram of an exemplary apparatus 100 for providing multi-frame image denoising according to one embodiment.

[0092] Referring to Figure 1 , apparatus 100 includes an image registrator 101, a patch extractor 103, a 2D transformer 105, a merger 107, a spatial denoiser 109, and a 2D inverse transformer and patch combiner 111.

[0093] Image registrator 101 includes means for receiving the signal I zInput 113 and output 115. The block extractor 103 includes an input connected to the output 115 of the image register 101 and an output 117. The output 117 of the block extractor 103 can output blocks of size M×M, where M is a rational number. The two-dimensional transformer 105 includes an input connected to the output 117 of the block extractor 103. The output 119 of the two-dimensional transformer 105 can be a frame w of the input signal (e.g., F z (w)). The combiner 107 includes an input connected to the output 119 of the two-dimensional transformer 105 and an output 121. The output 121 of the combiner 107 can provide an intermediate estimate of size M×M The spatial denoiser 109 includes an input connected to the output 121 of the combiner 107 and an output 123. The output 123 of the spatial denoiser 109 can provide a post-processing estimate of size M×M<了 The two-dimensional inverse transformer and block combiner 111 includes an input connected to the output 123 of the spatial denoiser 109 and an output that outputs a signal for multi-frame image denoising Output 125.

[0094] Image blocks can be selected to overlap by half in each spatial dimension of size M×M and filtered using a modified raised cosine window (i.e., 0 otherwise). The 2D-DFT is used as the two-dimensional transform. Frame 0 is selected as the reference frame. Obtained as the following equation (21):

[0095]

[0096] For a given frequency, A z (w) controls the degree to which the z-th non-reference frame is merged into the final result relative to falling back to the reference frame. Equation (21) can be rewritten as the following equation (22):

[0097]

[0098] The above pairwise time filters do not perform any spatial filtering. Therefore, spatial filtering is used as a separate post-processing step in the 2D DFT domain. Spatial filtering is performed by applying a pointwise shrinkage operator. An additional noise shaping filter of the form is also applied. The spatial filtering of robust merging is as the following equation (23):

[0099]

[0100] Figure 2is a block diagram of an exemplary apparatus for providing continuous robust merging according to one embodiment.

[0101] Reference Figure 2 The device of the present invention includes a merger 200, which has a subtractor 201, a Wiener filter 203, a summation function block 205, a multiplier 207 and a frame delayer 209.

[0102] Subtractor 201 includes a circuit for receiving F z The first input 211 of (w) is connected to the output 223 of the frame delay 209 for receiving The second input and output The Wiener filter 203 comprises a first input connected to the output 213 of the subtractor 201, a second input connected to the output 213 of the subtractor 201, and an output 215. The summation function block 205 comprises a first input connected to the output 215 of the Wiener filter 203, a second input for receiving the F z (w) and a second input 211 for outputting The multiplier 207 comprises a first input connected to the output 217 of the summation function block 205, a second input 219 for receiving 1 / z, and an output 221. The frame delay 209 comprises an input connected to the output 221 of the multiplier 207 and a second input 219 for outputting The output is 223.

[0103] The goal of merging is to reject merging non-reference frames whenever the difference between the corresponding non-reference frames and the reference frame is not related to noise.

[0104] However, after merging z frames, we can obtain As an estimate of R. Therefore, assuming experience The noise level can be determined by z Used when merging level This scheme is called continuous robust merging and can be formulated as follows:

[0105]

[0106] in It can be equivalently expressed as the following formula (25):

[0107]

[0108] Figure 3 is a block diagram of an exemplary apparatus for providing differential successive robust combining according to one embodiment.

[0109] Reference Figure 3 The device of the present invention includes a combiner 300, which has a first subtractor 301, a second subtractor 303, a Wiener filter 305, a summation function block 307, a multiplier 309 and a frame delay 311.

[0110] The first subtractor 301 includes a circuit for receiving F z (w), a first input 313 for receiving F0(w), a second input 315 for outputting F0(w)-F z (w) output 317. The second subtractor 303 includes a circuit for receiving F z The first input 313 of (w) is connected to the output 327 of the frame delay 311 for receiving The second input and output The Wiener filter 305 comprises a first input connected to the output 317 of the first subtractor 301, a second input connected to the output 329 of the second subtractor 303, and an output 319. The summation function block 307 comprises a first input connected to the output 319 of the Wiener filter 305, a second input for receiving the F z (w) and a second input 313 for outputting The multiplier 309 comprises a first input connected to the output 321 of the summation function block 307, a second input 323 for receiving 1 / z, and an output 325. The frame delay 311 comprises an input connected to the output 325 of the multiplier 309 and a second input for outputting The output is 327.

[0111] can be used to update the filter weights and can be expressed as the following equation (26):

[0112]

[0113] in and D m (w)=F0(w)-F m (w), and wherein the above equation (26) can be equivalently formulated as the following equation (27):

[0114]

[0115] Figure 4is a block diagram of a combiner 400 for minimum mean square error (MMSE) combining according to one embodiment.

[0116] Reference Figure 4 The combiner 400 includes a first subtractor 401, a first multiplier 403, a first summation function block 405, a second multiplier 407, a divider 409, a first adder 411, a second subtractor 413, an absolute value function block 415, a third subtractor 417, a maximum value divider 419, a second summation function block 421, a third multiplier 423 and a second adder 425.

[0117] The first subtractor 401 includes a circuit for receiving F z (w), a first input 427 for receiving μ(w), a second input 429 for receiving μ(w), and a second input 429 for outputting μ(w)-F z (w). The first multiplier 403 includes a first input connected to the output 431 of the first subtractor 401, a second input connected to the output 457 of the maximum divider 419 for receiving the filter weight, and an output 433. The first summation function block 405 includes an input connected to the output 433 of the first multiplier 403, and an output 435. The second multiplier 407 includes a first input connected to the output 435 of the first summation function block 405, a second input 437 for receiving P(w), and an output 439. The divider 409 includes a first input connected to the output 439 of the second multiplier 407, a second input connected to the output 467 of the second adder 425, and an output 441. The first adder 411 includes a first input connected to the output 441 of the divider 409, a second input 443 for receiving μ(w), and an output 445. The second subtractor 413 includes a first input for receiving F z (w), a first input 427 for receiving F0(w), a second input 447 for outputting F k (w)-F0(w) output 449. The absolute value function block 415 includes an input connected to the output 449 of the second subtractor 413 and an output 451. The third subtractor 417 includes a first input connected to the output 451 of the absolute value function block 415, a The maximum value divider function block 419 includes an input connected to the output 455 of the third subtractor 417 and an output 457 for outputting the filter weight. The second summation function block 421 includes an input connected to the output 457 of the maximum value divider 419 and an output 459. The third multiplier 423 includes a first input connected to the output 459 of the second summation function block 421, a second input 461 for receiving P(w), and an output 463. The second adder 425 includes a first input connected to the output 463 of the third multiplier 423, a second input 465 for receiving the value 1, and an output 467.

[0118] R can be a real signal (e.g., the original noise-free image of the scene), where F0(w) is the noisy reference frame, and e z (w) is the alignment error of the zth frame relative to the reference frame, that is, e0(w) = 0. For the zth non-reference frame F z For a noisy and possibly misaligned block of (w), equation (28) is as follows:

[0119] F z (w)=R(w)+e z (w)+N z (w)…(28)

[0120] Where R is the real image, and N z (w) is the variance Therefore, F0(w)=R(w)+N0(w).

[0121] The merge solution can be viewed as the goal of z The problem of estimating the image R in (w). By assuming e0(w) = 0, frame F0 serves as the reference frame. Consider a burst of N frames and for a given spatial frequency w, the vector F(w) = [F0(w), F1(w), ..., F N-1 (w)] T It can be expressed as the following formula (29):

[0122]

[0123] If μ(w) = E{R(w)} and P(w) is the true image power at spatial frequency w, that is, P(w) = E{|R(w)-μ(w)| 2}, and e z (w) is an iid Gaussian variable with E{ez(w)}=0, then for the MMSE estimate of R(w), equation (30) is as follows:

[0124]

[0125] The above equation (30) can be expressed as the following equation (31):

[0126]

[0127] To obtain MMSE results, unknown And P(w). For , the instantaneous power is as shown in the following equation (32):

[0128]

[0129] Since multiple frames are available, P(w) is calculated according to the following equation (33):

[0130]

[0131] because Therefore, the combined MMSE estimate of R(w) is biased.

[0132] Figure 5 is a block diagram of a combiner 500 for U-MMSE combining according to one embodiment.

[0133] Reference Figure 5 The combiner 500 includes a multiplier 501, a first summation function block 503, a divider 505, a first subtractor 507, an absolute value function block 509, a second subtractor 511, a maximum value divider 513 and a second summation function block 515.

[0134] The multiplier 501 includes a first input 517 for receiving Fz(w), a second input connected to the output 535 of the maximum divider 513 for receiving the filter weight, and an output 519. The first summation function block 503 includes an input connected to the output 519 of the multiplier 501, and an output 521. The divider 505 includes a first input connected to the output 521 of the first summation function block 503, a second input connected to the output 537 of the second summation function block 515, and an output 523. The first subtractor 507 includes a first input 517 for receiving Fz(w), a second input 525 for receiving F0(w), and an output 527 for outputting Fz(w)-F0(w). The absolute value function block 509 includes an input connected to the output 527 of the first subtractor 507, and an output 529. The second subtractor 511 includes a first input connected to the output 529 of the absolute value function block 509, a second input for receiving The maximum divider function block 513 includes an input connected to the output 533 of the second subtractor 511 and an output 535 for outputting the filter weight. The second summation function block 515 includes an input connected to the output 535 of the maximum divider 513 and an output 537.

[0135] The unbiased MMSE estimate of R(w) is expressed as follows:

[0136]

[0137] Only the To obtain an estimate

[0138] Figure 6 is a block diagram of another exemplary apparatus for providing multi-frame image denoising according to one embodiment.

[0139] Reference Figure 6 The apparatus 600 includes an image aligner 601 , a block extractor 603 , a two-dimensional transformer 605 , a merger 607 , and a two-dimensional inverse transformer and block combiner 609 .

[0140] The image register 601 includes a device for receiving a signal I z The block extractor 603 includes an input connected to the output 613 of the image register 601 and an output 615. The two-dimensional transformer 605 includes an input connected to the output 615 of the block extractor 603. The merger 607 includes an input connected to the output 617 of the two-dimensional transformer 605 and an output 619. The two-dimensional inverse transformer and block combiner 609 includes an input connected to the output 619 of the merger 607 and outputs a signal for denoising the multi-frame image. The output is 621.

[0141] Figure 7 According to one embodiment Figure 6 A block diagram of a merger 607 for continuous pair-wise 3D merger is shown.

[0142] Reference Figure 7 The combiner 607 includes a subtractor 701 , a first adder 703 , a first Wiener filter 705 , a second Wiener filter 707 , a second adder 709 , a multiplier 711 and a frame delay 713 .

[0143] Subtractor 701 includes an input 617 connected to combiner 607 for receiving F z (w), connected to the output 727 of the frame delay 713 for receiving The second input and output The first adder 703 comprises an input 617 connected to the combiner 607 for receiving the F z (w), connected to the output 727 of the frame delay 713 for receiving The second input and output 719 of the combiner 607. The second Wiener filter 707 includes a first input connected to the output 719 of the first adder 703, a second input connected to the output 719 of the first adder 703, and an output 721. The second adder 709 includes a first input connected to the output 721 of the second Wiener filter 707, a second input connected to the output 717 of the first Wiener filter 705, and an output connected to the output 619 of the combiner 607. The multiplier 711 includes a first input connected to the output 619 of the second adder 709, a second input 723 for receiving a value of 0.5, and an output 725. The frame delayer 713 includes an input connected to the output 725 of the multiplier 711, and an output 727.

[0144] Non-reference frames are merged continuously, where each non-reference frame is merged with the most recent merge result, and the merging is performed using a two-step transform in the transform domain. In the first step, a two-dimensional transform of the reference frame and the non-reference frame is obtained. The second step transform is a 2-point 1D-DFT performed in the time direction on the frequency components of the most recent merge result and a pair of available non-reference frames that are aligned. A Wiener filter is applied to the 3D transform result, which is then converted back to the 2D domain (a simple 2-point inverse DFT) to update the most recent merge result. The merging is initialized using the reference frame. The continuous pair-wise merging scheme can be formulated as follows:

[0145]

[0146]

[0147]

[0148] in and

[0149] There is no pair This is different from the continuous robust merging and differential continuous robust merging that apply post-spatial denoising separately.

[0150] Figure 8 The present invention is a flowchart of a multi-frame imaging denoising method according to an embodiment.

[0151] Reference Figure 8 , at 801, the system of the present invention determines the registration of the images in the signal.

[0152] At 803, the system of the present invention extracts blocks from the image.

[0153] At 805, the system of the present invention performs a two-dimensional transformation on the extracted blocks.

[0154] At 807, the system of the present invention merges the two-dimensionally transformed blocks. The merging method can be one of the above-mentioned continuous robust merging, differential continuous robust merging, MMSE merging, and U-MMSE merging.

[0155] At 809 , the system of the present invention performs spatial denoising on the merged blocks.

[0156] At 811, the system of the present invention performs a two-dimensional inverse transform and combines the spatially denoised blocks.

[0157] Figure 9 The present invention is a flowchart of a multi-frame imaging denoising method according to an embodiment.

[0158] Reference Figure 9 , at 901, the system of the present invention determines the registration of the images in the signal.

[0159] At 903, the system of the present invention extracts blocks from the image.

[0160] At 905, the system of the present invention performs a two-dimensional transformation on the extracted blocks.

[0161] At 907, the system of the present invention merges the two-dimensionally transformed blocks. The merging method may be successive pairwise 3D merging.

[0162] At 909 , the system of the present invention performs a two-dimensional inverse transform and combines the merged blocks.

[0163] Figure 10 is a block diagram of electronic devices in a network environment to which the apparatus and method of the present disclosure are applied according to one embodiment.

[0164] Reference Figure 10, an electronic device 1001 in a network environment 1000 can communicate with an electronic device 1002 via a first network 1098 (e.g., a short-range wireless communication network), or can communicate with an electronic device 1004 or a server 1008 via a second network 1099 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 1001 can communicate with the electronic device 1004 via the server 1008. According to an embodiment, the electronic device 1001 may include a processor 1020, a memory 1030, an input device 1050, a sound output device 1055, a display device 1060, an audio module 1070, a sensor module 1076, an interface 1077, a haptic module 1079, a camera module 1080, a power management module 1088, a battery 1089, a communication module 1090, a subscriber identification module (SIM) 1096, or an antenna module 1097. In some embodiments, at least one of the components (for example, the display device 1060 or the camera module 1080) may be omitted from the electronic device 1001, or one or more other components may be added to the electronic device 1001. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 1076 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented to be embedded in the display device 1060 (eg, a display).

[0165] The processor 1020 may execute, for example, software (eg, program 1040 ) to control at least one other component (eg, hardware component or software component) of the electronic device 1001 coupled to the processor 1020 , and may perform various data processing or calculations.

[0166] According to one embodiment, as at least part of data processing or calculation, the processor 1020 may load a command or data received from another component (e.g., the sensor module 1076 or the communication module 1090) into the volatile memory 1032, process the command or data stored in the volatile memory 1032, and store the resulting data in the non-volatile memory 1034. According to an embodiment, the processor 1020 may include a main processor 1021 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 1023 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor (SHP), or a communication processor (CP)) that can operate independently of the main processor 1021 or in conjunction with the main processor 1021.

[0167] Additionally or independently, the secondary processor 1023 may be adapted to consume less power than the primary processor 1021 or be dedicated to a dedicated function. The secondary processor 1023 may be implemented separately from the primary processor 1021 or as part of the primary processor 1021.

[0168] When the main processor 1021 is in an inactive state (e.g., a sleep state), the auxiliary processor 1023 may replace the main processor 1021 to control at least some of the functions or states related to at least one of the components of the electronic device 1001 (e.g., the display device 1060, the sensor module 1076, or the communication module 1090); or when the main processor 1021 is in an active state (e.g., when an application is being executed), the auxiliary processor 1023 may control at least some of the above functions or states together with the main processor 1021. Depending on the embodiment, the auxiliary processor 1023 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 1080 or the communication module 1090) that is functionally related to the auxiliary processor 1023.

[0169] The memory 1030 may store various data used by at least one component of the electronic device 1001 (e.g., the processor 1020 or the sensor module 1076). The various data may include, for example, software (e.g., the program 1040) and input data or output data for commands associated with the software. The memory 1030 may include a volatile memory 1032 or a non-volatile memory 1034.

[0170] The program 1040 may be stored in the memory 1030 as software, and may include, for example, an operating system (OS) 1042 , middleware 1044 , or an application 1046 .

[0171] The input device 1050 may receive a command or data from outside the electronic device 1001 (e.g., a user) to be used by another component of the electronic device 1001 (e.g., the processor 1020). The input device 1050 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

[0172] The sound output device 1055 can output sound signals to the outside of the electronic device 1001. The sound output device 1055 may include, for example, a speaker or a receiver. The speaker can be used for general purposes (e.g., playing multimedia or recordings), and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0173] The display device 1060 can visually provide information to the outside of the electronic device 1001 (e.g., a user). The display device 1060 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. Depending on the embodiment, the display device 1060 may include a touch circuit adapted to detect a touch, or a sensor circuit adapted to measure the strength of a force caused by a touch (e.g., a pressure sensor).

[0174] The audio module 1070 can convert sound into electrical signals and convert electrical signals into sound. Depending on the embodiment, the audio module 1070 can obtain sound through the input device 1050, or output sound through the sound output device 1055 or through headphones of an external electronic device (e.g., electronic device 1002) directly (e.g., wired) or wirelessly coupled to the electronic device 1001.

[0175] The sensor module 1076 can detect the operating state (e.g., power or temperature) of the electronic device 1001 or the environmental state (e.g., user state) outside the electronic device 1001, and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 1076 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or a brightness sensor.

[0176] The interface 1077 may support one or more prescribed protocols to be used to couple the electronic device 1001 directly (e.g., by wire) or wirelessly with an external electronic device (e.g., the electronic device 1002). Depending on the embodiment, the interface 1077 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0177] The connection terminal 1078 may include a connector through which the electronic device 1001 can be physically connected to an external electronic device (e.g., the electronic device 1002). Depending on the embodiment, the connection terminal 1078 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0178] The haptic module 1079 may convert the electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be recognized by the user through the user's tactile sensation or kinesthetic sensation. Depending on the embodiment, the haptic module 1079 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0179] The camera module 1080 may capture still images or moving images. Depending on the embodiment, the camera module 1080 may include one or more lenses, image sensors, image signal processors, or flashes.

[0180] The power management module 1088 may manage power supplied to the electronic device 1001. According to one embodiment, the power management module 1088 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0181] The battery 1089 may supply power to at least one component of the electronic device 1001. According to embodiments, the battery 1089 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

[0182] The communication module 1090 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 1001 and an external electronic device (e.g., electronic device 1002, electronic device 1004, or server 1008) and communicating through the established communication channel. The communication module 1090 may include one or more communication processors that can operate independently of the processor 1020 (e.g., an application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 1090 may include a wireless communication module 1092 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1094 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 1098 (e.g., a short-range communication network, such as Bluetooth TM, wireless-fidelity (Wi-Fi) direct or infrared data association (IrDA)) or a second network 1099 (for example, a long-distance communication network such as a cellular network, the Internet, or a computer network (for example, a LAN or a wide area network (WAN)))) to communicate with an external electronic device. These various types of communication modules may be implemented as a single component (for example, a single chip) or may be implemented as multiple components separated from each other (for example, multiple chips). The wireless communication module 1092 may use user information (for example, an international mobile subscriber identity (IMSI)) stored in the user identification module 1096 to identify and authenticate the electronic device 1001 in the communication network (for example, the first network 1098 or the second network 1099).

[0183] The antenna module 1097 can transmit or receive signals or power to or from the outside of the electronic device 1001 (e.g., an external electronic device). Depending on the embodiment, the antenna module 1097 may include an antenna comprising a radiating element formed in or on a substrate (e.g., a printed-circuit board (PCB)) and made of a conductive material or conductive pattern. Depending on the embodiment, the antenna module 1097 may include multiple antennas. In this case, the communication module 1090 (e.g., the wireless communication module 1092) may select, for example, at least one antenna from the multiple antennas that is suitable for the communication scheme used in the communication network (e.g., the first network 1098 or the second network 1099). Signals or power can then be transmitted or received between the communication module 1090 and the external electronic device via the selected at least one antenna. Depending on the embodiment, another component (e.g., a radiofrequency integrated circuit (RFIC)) in addition to the radiating element may be formed as part of the antenna module 1097.

[0184] At least some of the above components may be coupled to each other and transmit signals (e.g., commands or data) between the at least some components via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0185] According to an embodiment, commands or data can be sent or received between electronic device 1001 and external electronic device 1004 via server 1008 coupled to second network 1099. Each of electronic device 1002 and electronic device 1004 can be of the same or different type as electronic device 1001. According to an embodiment, all or some operations that would be performed at electronic device 1001 can be performed at one or more of external electronic device 1002, external electronic device 1004, or external electronic device 1008. For example, if electronic device 1001 is to perform a function or service automatically or in response to a request from a user or another device, electronic device 1001 may request one or more external electronic devices to perform at least a portion of the function or service instead of or in addition to performing the function or service. Upon receiving the request, the one or more external electronic devices may perform the at least a portion of the requested function or service, or perform other functions or other services related to the request, and transmit the results of the execution to electronic device 1001. The electronic device 1001 may provide the result as at least part of a reply to the request with or without further processing the result. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0186] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to embodiments of the present disclosure, the electronic device is not limited to the above-mentioned electronic devices.

[0187] The various embodiments of the present disclosure and the terms used herein are not intended to limit the technical features of the present disclosure to specific embodiments, but rather include various changes, equivalents, or substitutes to the corresponding embodiments.

[0188] With respect to the description of the drawings, like reference numerals may be used to designate like or related elements.

[0189] Unless the relevant context clearly indicates otherwise, singular forms of nouns corresponding to items may include one or more things.

[0190] Each of the phrases used herein, such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B, and C," and "at least one of A, B, or C," may include any one or all possible combinations of the items enumerated with the corresponding one of the phrases.

[0191] As used herein, for example, "first (1 st Terms such as “first”, “second”, and “second” may be used to distinguish a corresponding component from another component without limiting the components in other respects (e.g., importance or order). If an element (e.g., a first element) is referred to as being “coupled”, “coupled to”, “connected to”, or “connected to” another element (e.g., a second element), with or without the term “operably” or “communicatively”, the first element may be coupled to the second element directly (e.g., in a wired manner), wirelessly, or through a third element.

[0192] The term "module" as used herein may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "component," and "circuit." A module may be a single integral component adapted to perform one or more functions, or a minimum unit or component of the single integral component. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0193] The various embodiments described herein may be implemented as software (e.g., program 1040) comprising one or more instructions stored in a storage medium (e.g., internal memory 1036 or external memory 1038) readable by a machine (e.g., electronic device 1001). For example, a processor (e.g., processor 1020) of the machine (e.g., electronic device 1001) may invoke at least one of the one or more instructions stored in the storage medium and execute the at least one instruction, with or without one or more other components controlled by the processor. This allows the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between situations where data is stored in a storage medium in a semi-permanent manner and situations where data is stored in a storage medium temporarily.

[0194] According to an embodiment, the method according to various embodiments of the present disclosure may be included in a computer program product and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or through an application store (e.g., a Play Store). TM (Play Store TM ) online distribution (e.g., download or upload), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., a memory of a manufacturer's server, a server of an app store, or a relay server).

[0195] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Non-referencely or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still implement the one or more functions of each of the multiple components in the same or similar manner as the manner in which the corresponding one of the multiple components implements the one or more functions before integration. According to various embodiments, the operations implemented by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed in a different order or omitted, or one or more other operations may be added.

[0196] Figure 11 is a block diagram of process 1040 according to one embodiment.

[0197] Reference Figure 11 The program 1040 may include an operating system 1042 for controlling one or more resources of the electronic device 1001, middleware 1044, or an application 1046 that can be executed in the operating system 1042. The operating system 1042 may include, for example, Apple operating system or Bada TM (Bada TM For example, at least a portion of the program 1040 may be preloaded on the electronic device 1001 during manufacturing, or may be downloaded from or updated by an external electronic device (e.g., the electronic device 1002 or 1004 or the server 1008) during user use.

[0198] The operating system 1042 may control the management (e.g., allocation or deallocation) of one or more system resources (e.g., processes, memory, or power) of the electronic device 1001. Additionally or independently, the operating system 1042 may include one or more driver programs to drive other hardware devices of the electronic device 1001 (e.g., the input device 1050, the sound output device 1055, the display device 1060, the audio module 1070, the sensor module 1076, the interface 1077, the haptic module 1079, the camera module 1080, the power management module 1088, the battery 1089, the communication module 1090, the user identification module 1096, or the antenna module 1097).

[0199] The middleware 1044 may provide various functions to the applications 1046 so that the applications 1046 can use functions or information provided by one or more resources of the electronic device 1001. The middleware 1044 may include, for example, an application manager 1101, a window manager 1103, a multimedia manager 1105, a resource manager 1107, a power manager 1109, a database manager 1111, a package manager 1113, a connectivity manager 1115, a notification manager 1117, a location manager 1119, a graphics manager 1121, a security manager 1123, a telephony manager 1125, or a speech recognition manager 1127.

[0200] For example, the application manager 1101 may manage the lifecycle of the application 1046. The window manager 1103 may manage one or more graphical user interface (GUI) resources used on the screen. The multimedia manager 1105 may identify one or more formats to be used to play a media file and may encode or decode a corresponding one of the media files using a codec suitable for the corresponding format selected from the one or more formats. The resource manager 1107 may manage, for example, the source code of the application 1046 or the memory space of the memory 1030. The power manager 1109 may manage, for example, the capacity, temperature, or power of the battery 1089 and determine or provide relevant information for the operation of the electronic device 1001 based at least in part on the corresponding information about the capacity, temperature, or power of the battery 1089. According to one embodiment, the power manager 1109 may interoperate with the basic input / output system (BIOS) of the electronic device 1001.

[0201] The database manager 1111 may, for example, generate, search, or change a database to be used by the application 1046. The package manager 1113 may, for example, manage the installation or update of applications distributed in the form of package files. The connectivity manager 1115 may, for example, manage wireless connections or direct connections between the electronic device 1001 and external electronic devices. The notification manager 1117 may, for example, provide a function of notifying the user of the occurrence of a specified event (e.g., an incoming call, message, or alarm). The location manager 1119 may, for example, manage the location information of the electronic device 1001. The graphics manager 1121 may, for example, manage one or more graphic effects to be provided to the user or a user interface related to the one or more graphic effects.

[0202] The security manager 1123 can, for example, provide system security or user authentication. The phone manager 1125 can, for example, manage the voice call function or video call function provided by the electronic device 1001. The voice recognition manager 1127 can, for example, send the user's voice data to the server 1008 and receive from the server 1008 a command corresponding to a function to be executed on the electronic device 1001 based at least in part on the voice data, or receive text data converted at least in part based on the voice data. According to one embodiment, the middleware 1044 can dynamically delete some existing components or add new components. According to one embodiment, at least a portion of the middleware 1044 can be included as part of the operating system 1042 or can be implemented in other software separate from the operating system 1042.

[0203] The applications 1046 may include, for example, a home application 1151, a dialer application 1153, a short message service (SMS) / multimedia messaging service (MMS) application 1155, an instant message (IM) application 1157, a browser application 1159, a camera application 1161, an alarm application 1163, a contact application 1165, a voice recognition application 1167, an email application 1169, a calendar application 1171, a media player application 1173, an album application 1175, a watch application 1177, a health application 1179 (e.g., for measuring exercise level or biometric information (e.g., blood sugar)), or an environmental information application 1181 (e.g., for measuring air pressure, humidity, or temperature information). According to one embodiment, the applications 1046 may also include an information exchange application capable of supporting information exchange between the electronic device 1001 and an external electronic device. The information exchange application may include, for example, a notification relay application adapted to transmit specified information (e.g., a call, message, or alert) to an external electronic device, or a device management application adapted to manage an external electronic device. The notification relay application may transmit notification information corresponding to a specified event (e.g., email reception) occurring at another application (e.g., email application 1169) on the electronic device 1001 to the external electronic device. Alternatively or independently, the notification relay application may receive notification information from the external electronic device and provide the notification information to the user of the electronic device 1001.

[0204] The device management application can control the power (e.g., turning on or off) or function (e.g., adjusting brightness, resolution, or focus) of the external electronic device or some components of the external electronic device (e.g., a display device or camera module of the external electronic device). Additionally or independently, the device management application can support the installation, removal, or update of applications running on the external electronic device.

[0205] Although specific embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined based solely on the embodiments described, but should be determined based on the appended claims and their equivalents.

Claims

1. A device for denoising a continuous multi-frame image, comprising: a first subtractor comprising a first input for receiving a frame of an image, a second input, and an output; a first Wiener filter comprising an input connected to the output of the first subtractor and configured to filter the frame of the image and output the filtered frame; a summation function block comprising a first input connected to the output of the first Wiener filter, a second input connected to the first input of the first subtractor, and an output; a multiplier comprising a first input connected to the output of the summation function block, a second input for receiving a predetermined fractional value less than 1, and an output; as well as a frame delayer comprising a first input connected to the output of the multiplier and an output and configured to feed back an estimate of the image based on the filtered frame to the first Wiener filter, wherein the second input of the first subtractor is the output of the frame delayer or a reference frame, and Wherein the first Wiener filter is further configured to filter subsequent frames of the image based on respective estimates of the image based on previously filtered frames.

2. The apparatus for denoising continuous multi-frame images according to claim 1, wherein: The first subtractor includes the second input connected to the output of the frame delayer and the output connected to the second input and the first input of the first Wiener filter.

3. The apparatus for denoising consecutive multi-frame images according to claim 2, further comprising: Image aligner, including input and output; a patch extractor comprising an input connected to the output of the image register and an output; a two-dimensional transformer comprising an input connected to the output of the block extractor and an output connected to the first input of the first subtractor; a spatial denoiser comprising an input connected to said output of said summation function block and an output; as well as A two-dimensional inverse transformer and block combiner includes an input connected to the output of the spatial denoiser and an output.

4. The apparatus for denoising consecutive multi-frame images according to claim 1, further comprising: a second subtractor comprising a first input connected to the output of the frame delayer, a second input connected to the first input of the first subtractor, and an output connected to the second input of the first Wiener filter, and The first subtractor comprises the second input for receiving the reference frame and the output connected to the first input of the first Wiener filter.

5. The apparatus for denoising consecutive multi-frame images according to claim 4, further comprising: Image aligner, including input and output; a patch extractor comprising an input connected to the output of the image register and an output; a two-dimensional transformer comprising an input connected to the output of the block extractor and an output connected to the first input of the first subtractor; a spatial denoiser comprising an input connected to said output of said summation function block and an output; as well as A two-dimensional inverse transformer and block combiner includes an input connected to the output of the spatial denoiser and an output.

6. A device for denoising a continuous multi-frame image, comprising: a first subtractor comprising a first input for receiving a frame of an image, a second input, and an output; a first Wiener filter comprising an input connected to the output of the first subtractor and configured to filter the frame of the image and output the filtered frame; a first adder comprising a first input connected to the output of the first Wiener filter, a second input connected to the output of the second Wiener filter, and an output; a multiplier comprising a first input connected to the output of the first adder, a second input for receiving a predetermined fractional value less than 1, and an output; as well as a frame delayer comprising a first input connected to the output of the multiplier and an output and configured to feed back an estimate of the image based on the filtered frame to the first Wiener filter, wherein the first Wiener filter is further configured to filter subsequent frames of the image based on respective estimates of the image based on previously filtered frames, and The device further comprises: a second adder comprising a first input connected to the first input of the first subtractor, a second input connected to the output of the frame delayer, and an output; as well as The second Wiener filter comprises a first input, a second input and an output, wherein the first input and the second input of the second Wiener filter are connected to the output of the second adder. The first subtractor includes the second input connected to the output of the frame delayer and the output connected to the first input and the second input of the first Wiener filter.

7. The apparatus for denoising consecutive multi-frame images according to claim 6, further comprising: Image aligner, including input and output; a patch extractor comprising an input connected to the output of the image register and an output; a two-dimensional transformer comprising an input connected to the output of the block extractor and an output connected to the first input of the first subtractor; a spatial denoiser comprising an input connected to the output of the second summer and an output; as well as A two-dimensional inverse transformer and block combiner includes an input connected to the output of the spatial denoiser and an output.

8. A method for denoising a continuous multi-frame image, comprising: subtracting a first input of a frame of an image from a second input by a first subtractor; filtering, by a first Wiener filter, the frame of the image obtained by subtracting the first input of the frame of the image from the second input to output a filtered frame; summing, by a summation function block, an output of the first Wiener filter and the first input of the first subtractor; multiplying the output of the summation function block by a predetermined fractional value less than 1 by a multiplier; inputting the result of the multiplication into the frame delayer; feeding back, by the frame delayer, an estimate of the image based on the filtered frame to the first Wiener filter; as well as filtering, by the first Wiener filter, subsequent frames of the image based on respective estimates of the image based on previously filtered frames, The second input of the first subtractor is the output of the frame delayer or a reference frame.

9. The method for denoising a continuous multi-frame image according to claim 8, wherein: The frame of the image is subtracted from the output of the frame delay by the first subtractor.

10. The method for denoising consecutive multi-frame images according to claim 9, further comprising: determining, by an image registrar, registration of images in the signal; extracting, by a patch extractor, patches from the image for which the registration is determined by the image register; transforming the output of the block extractor by a two-dimensional transformer; providing an output of the two-dimensional transformer to the first subtractor; Performing spatial denoising on the output of the summation function block by a spatial denoiser; as well as transforming the output of the spatial denoiser by a two-dimensional inverse transformer and a block combiner; as well as The outputs from the two-dimensional inverse transformer and the block combiner are combined into blocks by the two-dimensional inverse transformer and the block combiner.

11. The method for denoising consecutive multi-frame images according to claim 8, further comprising: subtracting, by the first subtractor, the frame of the image from the reference frame; subtracting the frame of the image from the output of the frame delayer by a second subtractor; as well as An output of the second subtractor is provided to the first Wiener filter.

12. The method for denoising consecutive multi-frame images according to claim 11, further comprising: determining, by an image registrar, registration of images in the signal; extracting, by a patch extractor, patches from the image for which the registration is determined by the image register; transforming the output of the block extractor by a two-dimensional transformer; providing an output of the two-dimensional transformer to the first subtractor; Performing spatial denoising on the output of the summation function block by a spatial denoiser; transforming the output of the spatial denoiser by a two-dimensional inverse transformer and a block combiner; as well as The outputs from the two-dimensional inverse transformer and the block combiner are combined into blocks by the two-dimensional inverse transformer and the block combiner.

13. A method for denoising a continuous multi-frame image, comprising: subtracting a first input of a frame of an image from a second input by a first subtractor; filtering, by a first Wiener filter, the frame of the image obtained by subtracting the first input of the frame of the image from the second input to output a filtered frame; summing, by a first adder, the output of the first Wiener filter and the output of the second Wiener filter; multiplying the output of the first adder by a predetermined fractional value less than 1 by a multiplier; inputting the result of the multiplication into the frame delayer; feeding back, by the frame delayer, an estimate of the image based on the filtered frame to the first Wiener filter; as well as filtering subsequent frames of the image by the first Wiener filter based on respective estimates of the image based on previously filtered frames, wherein the second input of the first subtractor is the output of the frame delayer, and The method further comprises: subtracting the frame of the image from the output of the frame delayer by the first subtractor; providing the result of the subtraction to the first Wiener filter; adding, by a second adder, the frame of the image and the output of the frame delayer; and The output of the second adder is filtered by the second Wiener filter.

14. The method for denoising consecutive multi-frame images according to claim 13, further comprising: determining, by an image registrar, registration of images in the signal; extracting, by a patch extractor, patches from the image for which the registration is determined by the image register; transforming the output of the block extractor by a two-dimensional transformer; providing an output of the two-dimensional transformer to the first subtractor; performing spatial denoising on the output of the second adder by a spatial denoiser; transforming the output of the spatial denoiser by a two-dimensional inverse transformer and a block combiner; as well as The outputs from the two-dimensional inverse transformer and the block combiner are combined into blocks by the two-dimensional inverse transformer and the block combiner.

Images