Electronic device for noise reduction for video, operating method thereof, and storage medium

Abstract

According to one embodiment, the electronic device may include a camera including an image sensor; at least one processor including a processing circuit; and a memory for storing instructions. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire a first image frame having a first noise level from the camera, identify a second image frame having a second noise level preceding the first image frame, reduce the spatial noise level of a motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame, acquire a third image frame using the motion area of ​​the first image frame where the spatial noise level of the first image frame has been reduced and at least a portion of the first image frame, and acquire a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

Description

Technology Field

[0001] One embodiment disclosed in this document relates to an electronic device for reducing noise in video, a method of operation thereof, and a storage medium. Background Technology

[0002] An image of a subject captured through a camera can be captured as an electrical image signal by an image sensor. During the process of acquiring the image, noise may be added depending on the characteristics of the image sensor or during transmission through a channel, and noise may be added for various reasons other than those mentioned above. Since the electrical image signal inevitably contains noise, an image signal processor that receives and processes the signal from the image sensor may include function blocks that perform noise removal functions.

[0003] If noise contained in the electrical image signal is not properly filtered during image signal processing, image quality degradation may occur, data compression efficiency may decrease, and the degraded image may impair the performance of the post-processing unit connected to the image processing unit. Therefore, it is necessary to efficiently reduce the noise generated during image acquisition.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure. means of solving the problem

[0005] According to one embodiment, the electronic device may include a camera including an image sensor; at least one processor including a processing circuit; and a memory for storing instructions. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire a first image frame having a first noise level from the camera.

[0006] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to identify a second image frame having a second noise level preceding the first image frame.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire a third image frame using the motion area of ​​the first image frame where the spatial noise level is reduced and at least a portion of the first image frame.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

[0010] According to one embodiment, a method for reducing noise in a video in an electronic device may include the operation of acquiring a first image frame having a first noise level from a camera.

[0011] According to one embodiment, the method may include the operation of identifying a second image frame having a second noise level preceding the first image frame.

[0012] According to one embodiment, the method may include an operation to reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame.

[0013] According to one embodiment, the method may include the operation of acquiring a third image frame using the motion area in which the spatial noise level of the first image frame is reduced and at least a portion of the first image frame.

[0014] According to one embodiment, the method may include the operation of acquiring a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

[0015] According to one embodiment, in a storage medium storing at least one computer-readable instruction, the at least one instruction causes the electronic device to perform at least one operation when executed individually or collectively by at least one processor of the electronic device, and the at least one operation may include the operation of acquiring a first image frame having a first noise level from a camera.

[0016] According to one embodiment, the at least one operation may include an operation of identifying a second image frame having a second noise level preceding the first image frame.

[0017] According to one embodiment, the at least one operation may include an operation to reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame.

[0018] According to one embodiment, the at least one operation may include acquiring a third image frame using the motion area of ​​the first image frame where the spatial noise level is reduced and at least a portion of the first image frame.

[0019] According to one embodiment, the at least one operation may include acquiring a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment. FIG. 2a is a block diagram illustrating a camera module according to one embodiment. Figure 2b is a block diagram of an image signal processor for noise reduction. FIG. 3 is an internal block diagram of an electronic device according to one embodiment. FIG. 4 is a flowchart of the operation of an electronic device for noise reduction for video according to one embodiment. FIG. 5 is a detailed configuration diagram of an electronic device for noise reduction for video according to one embodiment. FIG. 6 is a detailed configuration diagram of an electronic device for noise reduction in video when a spatiotemporal filter is applied to a local motion area according to one embodiment. FIG. 7 is a detailed configuration diagram of an electronic device for noise reduction in video when a spatiotemporal filter is applied to a static area according to one embodiment. FIG. 8 is a detailed configuration diagram of an electronic device for noise reduction for video using a coring function unit according to one embodiment. FIG. 9a is a detailed configuration diagram of a spatiotemporal filter according to one embodiment. FIG. 9b is a drawing for explaining the detailed operation of a spacetime filter according to one embodiment. FIG. 10 is an example diagram illustrating a method for setting the window sizes of a previous image frame and a current image frame for spatiotemporal filtering according to one embodiment. FIG. 11 is a drawing showing the window area of ​​a previous image frame and a current image frame according to one embodiment. FIG. 12 is a graph for explaining a linear function by interval according to one embodiment. FIG. 13 is a drawing showing an image with a noise removal function applied by a coring function part according to one embodiment. FIG. 14 is a diagram illustrating the noise reduction effect resulting from the application of spatiotemporal filtering in a low-light image according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0021] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0022] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0023] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0024] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0025] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0026] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0027] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0028] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0029] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0030] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0031] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0032] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0033] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0034] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0035] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0036] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0037] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0038] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0039] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0040] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0041] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0042] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0043] FIG. 2a is a block diagram (200) illustrating a camera module (or camera device) (180) according to one embodiment.

[0044] Referring to FIG. 2a, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (ISP) (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.

[0045] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0046] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing) in response to the movement of the camera module (180) or the electronic device (101) including it. This allows for compensating for at least some of the negative effects caused by the movement on the captured image. According to one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) can be implemented, for example, as an optical image stabilizer. The memory (250) can temporarily store at least some of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in memory (250) can be acquired and processed by, for example, an image signal processor (260). According to one embodiment, memory (250) may be configured as at least a portion of memory (130) or as a separate memory that operates independently thereof.

[0047] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) can perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least part of the processor (120) or as a separate processor operating independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after further image processing by the processor (120).

[0048] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0049] Methods for reducing (or suppressing) or eliminating noise generated during image acquisition may include temporal methods that utilize time-axis data and spatial methods that utilize spatial data.

[0050] Temporal image noise reduction is an image processing method for pixels at the same location across multiple images. In the temporal image noise reduction method, temporal information corresponding to the degree of similarity between a pixel in a previous image frame and a pixel in the current image frame can be utilized. By performing noise reduction based on the degree of pixel similarity, the temporal image noise reduction method can possess adaptive properties. For example, the weights reflecting the corresponding pixel information may vary depending on the degree of similarity. If the degree of similarity increases, the weights are changed to reflect the previous image frame more, thereby enabling greater noise reduction.

[0051] Spatial image noise reduction is an image processing method that utilizes information about each pixel and surrounding pixels of an image frame. In spatial image noise reduction methods, spatial information including the noise level and edge information of an image frame can be utilized.

[0052] In one embodiment, by utilizing the characteristics of continuously acquired video images, noise occurring at corresponding positions or pixels between temporally different video frames can be removed or reduced by using information about pixels included in a previous video frame from which noise has been removed.

[0053] According to one embodiment, when noise is reduced (or suppressed, removed) by utilizing the characteristics of noise occurring at corresponding positions between temporally different video frames, a better noise reduction effect can be obtained than a noise reduction method for still images that uses only spatial information, because the information of previous video frames can be used continuously.

[0054] According to one embodiment, the electronic device (101) may receive a noise-removed previous image frame as feedback and remove or reduce noise in the current image frame. A weight for pixel or region-based synthesis of the two image frames may be determined based on the previous image frame and the current image frame. The electronic device (101) may obtain a current image frame in which noise is reduced (or suppressed, removed) by synthesis of the previous image frame and the current image frame.

[0055] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0056] Refer to FIG. 2b to explain a method for removing or reducing noise in a current video frame by receiving feedback from a previous video frame from which noise has been removed. In the following description, 'unit' may refer to a unit in which video processing processes such as noise removal, motion analysis, motion correction, and / or blending are performed. Additionally, 'unit' may be referred to as a 'block'.

[0057] Figure 2b is a block diagram of an image signal processor for noise reduction.

[0058] Referring to FIG. 2b, the image signal processor (260) may include a spatial noise reduction (SNR) (262), a temporal noise reduction (TNR) (266), and a motion analysis (264) for noise reduction.

[0059] The image signal processor (260) can perform noise processing on one or more image frames acquired through the image sensor (230). The image signal processor (260) may use a noise filter to improve the quality of the image frames by removing noise contained in the image frames, and examples of noise filters may include a spatial noise reduction unit (262) and a temporal noise reduction unit (266).

[0060] The spatial noise reduction unit (262) can reduce (or remove, suppress) noise regarding the spatial axis data of the current image frame (e.g., current input) acquired through the camera, that is, reduce spatial noise of the image information. The spatial noise reduction unit (262) can output the current image frame with reduced noise.

[0061] The motion analysis unit (264) can determine how much shaking exists when a video frame (e.g., one or more video frames) is captured. For example, the motion analysis unit (264) may use collected sensor measurements (e.g., gyroscope, accelerometer, any combination thereof, and / or one or more inertial measurement units (IMUs)) to verify the motion of the camera module (180). The motion analysis unit (264) may calculate a weight for controlling the intensity of noise reduction of the temporal noise reduction unit (266). That is, the motion analysis unit (264) may determine (or verify) by the weight how much the noise of the temporal noise reduction unit (266) will be reduced. For example, if the degree of similarity between the pixels of the current video frame and the previous video frame is high, a weight is calculated to reflect the previous video frame more, so the degree of noise reduction in the temporal noise reduction unit (266) may vary according to the calculated weight.

[0062] The temporal noise reduction unit (266) can output a current video frame with reduced temporal noise by using the current video frame with reduced spatial noise from the spatial noise reduction unit (262) and the previous video frame. For example, the image signal processor (260) can reduce noise by applying a weight according to the degree of movement to the current video frame using the temporal noise reduction unit (266). The image signal processor (260) can output a final video frame (e.g., TNR result) by blending (or combining, synthesizing) the current video frame with reduced temporal noise and the previous video frame.

[0063] In the process of removing noise in the temporal noise reduction unit (266), the processing effect of the temporal noise reduction unit (266) may be greater in areas with less movement than in areas with movement or static areas within the image. Therefore, the image signal processor (260) may apply different weights related to the intensity of noise reduction depending on whether there is movement. For example, in the movement area, the spatial noise reduction result from the spatial noise reduction unit (262) may be used to reduce the noise of the current image frame. On the other hand, in the area without movement (e.g., background area), the temporal noise reduction unit (266) may be used to give a higher weight to the previous image frame with a lower noise level, thereby blending the previous image frame with a lower noise level with the current image frame.

[0064] However, in motion regions where spatial noise reduction results from the current video frame are used, an imbalance in noise reduction results may occur between motion and non-motion regions. For example, an unbalanced final video frame may be output, where the results of applying noise reduction intensity (or weight) differ between moving and non-moving areas. If the intensity of spatial noise reduction is increased to resolve this imbalance (or to balance the results)—for instance, by increasing the strength of the filter for spatial noise reduction—side effects may occur, such as blurring of motion regions within the video and the loss of detail. For instance, when removing noise from video with severe motion, simply increasing the noise level can have the counterproductive effect of leaving afterimages known as ghost artifacts.

[0065] When processing a current video frame containing noise to preserve (or maintain) details within the video, image quality degradation may occur as the current video frame containing noise is blended with a previous video frame without properly filtering the noise, due to limitations in removing the noise contained in the current video frame.

[0066] In one embodiment, an electronic device for reducing noise in video, a method of operation thereof, and a storage medium may be provided so as to efficiently reduce noise generated during image acquisition while preserving (or maintaining) detailed information about the motion area.

[0067] According to one embodiment, in the process of reducing (or removing, suppressing) noise in the current video frame, by applying different noise reduction intensities (or weights) to moving areas and non-moving areas, detailed information can be preserved (or maintained) in moving areas, thereby enabling the acquisition of a clear image.

[0068] According to one embodiment, the temporal noise level can be lowered by blending the current video frame, in which noise has been removed (or reduced, suppressed) using spatial noise reduction results and temporal information for the current video frame, with the previous video frame.

[0069] FIG. 3 is an internal block diagram of an electronic device according to one embodiment.

[0070] Referring to FIG. 3, according to one embodiment, an electronic device (101) (e.g., the electronic device (101) of FIG. 1) may include a camera (380) (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2), a processor (320) (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2), a memory (330) (e.g., the memory (130) of FIG. 1), and a display (360) (e.g., the display module (160) of FIG. 1). According to one embodiment, the electronic device (101) may be implemented identically or similarly to the electronic device (101) of FIG. 1.

[0071] According to one embodiment, the processor (320) can control the overall operation of the electronic device (101). For example, the processor (320) may be implemented as an application processor (AP), a digital signal processor (DSP), or an image signal processor (e.g., the image signal processor (260) of FIG. 2a).

[0072] According to one embodiment, when capturing an image using an electronic device (101) (e.g., in video mode), a processor (320) can acquire an image (e.g., video image) through a camera (380). The camera (380) can receive light using an image sensor of the camera (380) (e.g., image sensor (230) of FIG. 2) and capture an image frame, such as a still image or a video frame. Here, 'image', 'image frame', and 'frame' may be used interchangeably in this disclosure. The electronic device (101) may include processors (e.g., image signal processors) capable of receiving one or more image frames through the camera (380) and processing said one or more image frames.

[0073] According to one embodiment, the processor (320) can acquire images (or image data including multiple image frames) in units of frames (e.g., image frames) in chronological order. For example, multiple image frames may represent images that are continuously captured or simultaneously captured with a time interval between them, targeting the same subject.

[0074] According to one embodiment, the processor (320) may acquire a first image frame and a second image frame. The second image frame is an image frame that is continuous with the first image frame, and may be an image frame acquired (or received) immediately before the first image frame among a plurality of image frames. In this way, any one of the plurality of image frames may be set as a reference image frame, and image frames prior to the reference image frame may be set as reference image frames.

[0075] Hereinafter, for convenience of explanation, the first image frame may be referred to as the current image frame (or the Nth image frame), and the second image frame may be referred to as the previous image frame (or the N-1th image frame). Additionally, the first image frame may be referred to as the reference image frame, and the second image frame may be referred to as the reference image frame. The reference image frame represents an image that serves as a reference in the image processing performed by the electronic device (101), and the reference image frame may represent the remaining images used for image processing. For example, the reference image frame may be the first image frame among the image frames taken with respect to the same subject, and the reference image frame may be an image frame taken after the reference image frame. For example, the second image frame taken at the first time point may be stored in memory (330) (e.g., buffer), and the first image frame taken at the second time point after the first time point may be stored in memory (330) (e.g., buffer). The distinction between the reference image frame and the reference image frame is not limited thereto, and the electronic device (101) may determine the reference image frame among the plurality of image frames arbitrarily or according to a predetermined standard. The reference image frame may be at least one, and for convenience of explanation, the case where there is one reference image frame will be described below.

[0076] According to one embodiment, the electronic device (101) may acquire an image (or image data including a plurality of image frames) through a communication module (e.g., the communication module (190) of FIG. 1). The electronic device (101) may transmit the noise-removed image to an external electronic device through the communication module via a processor (320).

[0077] According to one embodiment, noise may be added to an image sensor (e.g., image sensor (230) of FIG. 2) depending on the characteristics of the sensor during the process of acquiring an image (e.g., a plurality of image frames). Noise may be added during the process of transmitting image data acquired through the image sensor through a channel. The added noise may degrade the image quality and reduce the compression performance of the image data. The performance of the degraded image may be degraded during image processing or post-processing through the processor (320).

[0078] According to one embodiment, the processor (320) can obtain an image-processed output image frame by taking a first image frame (or current image frame, reference image frame) and a second image frame (or previous image frame, reference image frame) as inputs. When photographing a subject, the first image frame and the second image frame may contain image quality degradation caused by the movement of the subject, and the processor (320) can obtain an output image frame with improved image quality by removing (or reducing) the image quality degradation.

[0079] According to one embodiment, a method using spatial information and a method using temporal information may be applied to remove (or reduce) noise. In the embodiments described below, by utilizing the characteristics of continuously acquired video images, noise occurring at corresponding locations or pixels between temporally different video frames can be removed (or reduced) from pixels included in the current video frame by utilizing image information of pixels included in the previous video frame from which noise has been removed.

[0080] According to one embodiment, when noise is removed by utilizing the characteristics of noise occurring at corresponding locations between temporally different video frames, a better noise reduction effect can be obtained than a noise reduction method for still images that uses only spatial information, because the information of previous video frames can be used continuously. According to one embodiment, the processor (320) can receive the noise-removed previous video frame as feedback and remove or reduce the noise of the current video frame. According to one embodiment, the weight of pixel or region-based synthesis (or blending) of the two video frames can be determined based on the difference (e.g., pixel value difference) between the pixels of the previous video frame and the pixels of the current video frame. For example, the difference (e.g., pixel value difference) between the pixels of the previous video frame and the pixels of the current video frame should be understood as being calculated in one of the following units: pixel unit, region unit consisting of one or more pixels, or entire frame unit.

[0081] The processor (320) can obtain a current video frame (e.g., a third video frame different from the first and second video frames) in which noise is removed (or reduced, suppressed) by blending (or synthesizing) the previous video frame and the current video frame.

[0082] In the following description, the operation of removing noise from the current video frame should be understood as referring to the operation of reducing, suppressing, or filtering noise.

[0083] According to one embodiment, the processor (320) may acquire a first image frame through the camera (380). Since the first image frame is prior to image processing, it may be an image frame having a first noise level. Artifacts related to the movement of an object may be left within the first image frame due to the movement of the electronic device (101) or the movement of the subject. An area related to the movement of an object within the first image frame may be referred to as a movement area or a moving area.

[0084] According to one embodiment, the processor (320) may perform image processing to accumulate detailed image information by lowering the noise reduction intensity (or weight) for areas with movement within the first image frame and maintaining the noise reduction intensity (or weight) for areas without movement in order to correct (or prevent) artifacts. Due to this operation of increasing or decreasing the noise reduction intensity, the remaining noise levels in the areas with movement and the areas without movement may differ. The processor (320) may use a second image frame corresponding to a time earlier than the time of acquisition of the first image frame in the image processing process for the first image frame. The second image frame may have a second noise level through image processing, for example, such as noise removal, and may be stored in a buffer after image processing. For example, a plurality of image frames may be acquired (or received) sequentially through the camera (380), and the subsequent next image frame may be received after the initial image frame is captured by the image sensor (230). When the initial image frame is called the reference image frame, the subsequent next image frame is the reference image frame, and the reference image frame corrected immediately before can be used to perform noise removal on the motion area of ​​the reference image frame.

[0085] According to one embodiment, when noise of a different magnitude occurs within a certain temporal deviation at a corresponding pixel of continuously acquired image frames, the processor (320) may use information of a previous pixel from which noise has been removed in a reference image frame to remove noise in a moving area of ​​the reference image frame.

[0086] According to one embodiment, the processor (320) can identify motion information associated with video frames. The processor (320) can estimate motion information related to the difference between the first video frame and the second video frame. For example, it can estimate motion information indicating the degree of movement of the second video frame relative to the first video frame. For example, while the user is holding the electronic device (101) in their hand, movement of the electronic device (101) may occur due to hand shaking, or movement of the subject may occur during filming; the movement of the electronic device (101) itself may be called global movement, and the movement of the subject may be called local movement.

[0087] Motion information may include motion vectors existing between the first image frame and the second image frame, and may include information regarding global motion vectors, local motion vectors, or both. The global motion vector may represent the relative positional difference between the second image frame or the reference region of the second image frame and the first image frame, based on the first image frame or the reference region of the first image frame.

[0088] According to one embodiment, the processor (320) can reduce noise in a moving area of ​​the first image frame using a spatio-temporal filter (or spatio-temporal filter). The processor (320) can use motion information related to the difference between the first image frame and the second image frame to determine how much noise to remove (or reduce) in a moving area of ​​the first image frame. The operation to remove (or reduce) noise in a moving area of ​​the first image frame can be performed on a pixel basis.

[0089] For example, a method for identifying a moving area of ​​a first image frame may be a method for identifying whether the difference between the first image frame and the second image frame is greater than a threshold value. For example, an electronic device (320) may calculate the difference between a first pixel of the first image frame and a second pixel of the second image frame corresponding to the first pixel, and determine that if the difference between the first pixel value and the second pixel value is greater than or equal to a threshold value, the pixel corresponds to a moving area. In this way, a set of pixels composed of pixels where the difference between the first image frame and the second image frame is greater than or equal to a threshold value may be referred to as a moving area.

[0090] For example, reducing noise in a moving area of ​​a first video frame on a pixel-by-pixel basis may mean calculating weights for one or more pixels among a second pixel, third pixels, and fourth pixels based on information (e.g., position, brightness) of third pixels in a first window (or first area) having a first size set around (or based on) a first pixel of the first video frame, and information (e.g., position, brightness) of fourth pixels in a second window (or second area) having a second size set around a second pixel corresponding to the first pixel within the second video frame. The noise level of the first pixel may be lowered based on the calculated weights. The second area of ​​the second video frame may be determined according to a weight map related to motion information, such as the difference between the first video frame and the second video frame.

[0091] According to one embodiment, the processor (320) can generate (or output) a first image frame in which the spatial noise level is reduced for the motion area of ​​the first image frame based on the motion information through a spatiotemporal filter. For example, the strength of the spatiotemporal filter may be determined according to the motion information. The processor (320) can obtain a third image frame through spatial blending by using (or synthesizing, blending) at least a portion of the first image frame including the motion area of ​​the first image frame or the motion area as well as the static area, and the motion area of ​​the first image frame in which the spatial noise level is reduced. Additionally, the processor (320) can obtain a fourth image frame through temporal blending by using (or synthesizing, blending) at least a portion of the third image frame and at least a portion of the second image frame. The fourth image frame may correspond to the first image frame corrected to have a third noise level lower than the first noise level of the first image frame.

[0092] According to one embodiment, the processor (320) may identify (or determine) a weight map representing the degree of local movement of the first image frame based on the difference between the first image frame and the second image frame. For example, the motion information may include a weight map representing the degree of local movement of the first image frame and vector information related to the degree of local movement. The weight map may be used to determine how much noise to remove (or reduce) for the area of ​​movement of the first image frame. Here, before performing the operation to reduce the spatial noise level of the movement area of ​​the first image frame using a spatiotemporal filter, spatial noise reduction may be performed on the first image frame first (e.g., a Gaussian filter). When the first image frame to which spatial noise reduction has been performed is referred to as the first-1 image frame, the processor (320) may generate (or acquire) the first-2 image frame in which the spatial noise level of the movement area of ​​the first-1 image frame has been reduced according to the weight map using a spatiotemporal filter.

[0093] According to one embodiment, the processor (320) can correct the degree of local motion of a second image frame based on a first image frame (e.g., first-1 image frame) that has undergone spatial noise reduction. The corrected second image frame can be used to reduce the spatial noise level of the motion area of ​​the first-1 image frame according to a weight map.

[0094] According to one embodiment, the processor (320) may use a coring function before using a spatiotemporal filter. The coring function may be utilized in the spatial noise reduction (spatial NR) and temporal noise reduction (temporal NR) processes. For example, by using a coring function (e.g., a segment-by-segment linear function), the processor (320) may generate a first-3 image frame by removing signals having values ​​below a threshold among signals representing the difference between the pixels of the first image frame and the pixels of the first-1 image frame. The first-3 image frame may correspond to a first image frame in which only noise below a specific threshold is removed using the coring function, while edge and detail information is preserved.

[0095] According to one embodiment, the processor (320) may use a second image frame (e.g., image frame 2-1) in which global motion information of the second image frame has been corrected, and a first image frame (e.g., image frame 1-3) in which noise below a threshold value has been removed by applying a coring function, as inputs to a spatiotemporal filter. For example, the processor (320) may generate a first image frame (e.g., image frame 1-4) in which the spatial noise level of the static region of the first-3 image frame has been reduced according to a weight map through a spatiotemporal filter based on the second-1 image frame and the first-3 image frame.

[0096] Accordingly, the processor (320) can obtain a third image frame by using (or synthesizing, blending) at least a portion of the first image frame (e.g., the first-4th image frame) that includes a motion area or a motion area as well as a static area of ​​the first image frame through spatial blending, and a first image frame (e.g., the first-2nd image frame) that includes a motion area with reduced spatial noise levels of the first image frame. Additionally, the processor (320) can obtain a fourth image frame by using (or synthesizing, blending) at least a portion of the third image frame and a second image frame (e.g., the second-1st image frame) that corrects the global motion information of the second image frame through temporal blending.

[0097] According to one embodiment, the processor (320) can identify a first window having a first size centered on a first pixel within a first image frame. The processor (320) can use vector information (e.g., a local motion vector) to identify a second pixel in a second image frame corresponding to the first pixel in the first image frame, or a second window having a second size centered on the second pixel. For example, a first window (or first area) having a first size in the first image frame may correspond to a second window (or second area) having a second size centered on a second pixel corresponding to the position of the first pixel within the second image frame.

[0098] According to one embodiment, the processor (320) can determine the similarity based on at least one of a positional difference and a brightness difference between the pixels of the first window having the first size and the pixels of the second window having the second size. For example, the processor (320) can determine the similarity based on at least one of a positional difference and a brightness difference between the pixels of the first window having the first size and the pixels of the second window having the second size. The processor (320) can determine (or determine) a weight based on the determined similarity and reduce the spatial noise level of the motion area of ​​the first image frame (e.g., the first-1 image frame) that has performed spatial noise reduction based on the determined weight.

[0099] According to one embodiment, the processor (320) may generate a third image frame through spatial blending using a second image frame with respect to a first image frame generated through a spatiotemporal filter, and generate a fourth image frame through temporal blending using the third image frame and the second image frame. The fourth image frame may be an image frame in which noise is removed (or reduced) for the motion area of ​​the first image frame.

[0100] According to one embodiment, the generated fourth image frame may be stored in memory (330) as an image frame in which noise is removed (or reduced) from the first image frame, thereby replacing the first image frame. For example, when the processor (320) generates (or acquires) a final resulting image frame (e.g., an image frame with reduced noise levels in the motion area) for the current image frame (e.g., the Nth image frame), it may repeat the above process for the next image frame (e.g., the N+1th image frame).

[0101] According to one embodiment, the processor (320) may store in memory (330) the resulting image frames (e.g., image frames with reduced noise levels in moving areas) by performing image processing such that detailed image information can be accumulated by lowering the noise reduction intensity for moving areas and maintaining the noise reduction intensity for non-moving areas for each image frame. The image frames stored in memory (330) may be displayed (or played) through the display (360) upon a user's request for playback. Detailed embodiments of removing or reducing noise in image frames by increasing or decreasing the noise reduction intensity in the processor (320) will be described later.

[0102] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0103] According to one embodiment, the electronic device (101) may include a camera (380) including an image sensor (230); at least one processor (320) including a processing circuit; and a memory (330) for storing instructions. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire a first image frame having a first noise level from the camera, identify a second image frame having a second noise level preceding the first image frame, reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame, acquire a third image frame using the motion area of ​​the first image frame where the spatial noise level of the first image frame has been reduced and at least a portion of the first image frame, and acquire a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

[0104] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to identify a weight map representing the degree of local motion of the first image frame based on the difference between the first image frame and the second image frame, and to reduce the spatial noise level of the motion area of ​​the first image frame according to the weight map. According to one embodiment, the motion information may include at least one of the weight map or vector information related to the degree of local motion.

[0105] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to generate a first-1 image frame in which spatial noise reduction is performed on the first image frame, and to reduce the spatial noise level of the motion area of ​​the first-1 image frame according to the weight map.

[0106] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to correct the degree of local motion of the second image frame based on the first-1 image frame, and based on the corrected second image frame, generate a first-2 image frame in which the spatial noise level of the motion area of ​​the first-1 image frame is reduced according to the weight map.

[0107] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to generate a first-third image frame by removing signals having a value below a threshold value among signals representing the difference between the pixels of the first image frame and the pixels of the first-first image frame, based on a coring function.

[0108] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to generate a second-1 image frame that corrects the degree of global motion of the second image frame, and to generate a first-4 image frame that reduces the spatial noise level of the static region of the first-3 image frame according to the weight map based on the second-1 image frame and the first-3 image frame.

[0109] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to acquire the third image frame by blending the first-2 image frame and the first-4 image frame, and to acquire the fourth image frame by blending at least a portion of the third image frame and at least a portion of the second-1 image frame.

[0110] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to identify a first window having a first size centered on a first pixel of the first image frame, identify a second pixel of the second-1 image frame corresponding to the first pixel using the vector information, identify a second window having a second size centered on the second pixel within the second-1 image frame, and identify a similarity between the pixels of the first window having the first size and the pixels of the second window having the second size.

[0111] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to check the similarity based on at least one of a positional difference and a brightness difference between the pixels of a first window having a first size and the pixels of a second window having a second size.

[0112] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to determine a weight according to the identified similarity and reduce the spatial noise level of the motion area of ​​the first-1 image frame according to the determined weight.

[0113] FIG. 4 is a flowchart of the operation of an electronic device for noise reduction for video according to one embodiment. Referring to FIG. 4, the operation method may include operations 405 through 425. Each operation of the operation method of FIG. 4 may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 and FIG. 3), and at least one processor of the electronic device (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and the processor (320) of FIG. 3). In one embodiment, at least one of operations 405 through 425 may be omitted, the order of some operations may be changed, or other operations may be added.

[0114] Referring to FIG. 4, in operation 405, the electronic device (101) can acquire a first image frame having a first noise level from a camera.

[0115] In operation 410, the electronic device (101) can identify a second image frame having a second noise level preceding the first image frame. The electronic device (101) can use the second image frame corresponding to a time earlier than the time of acquisition of the first image frame in the image processing process for the first image frame. The second image frame may have a second noise level through image processing, such as noise removal, for example, and may be stored in memory (330) (e.g., buffer) after image processing.

[0116] In operation 415, the electronic device (101) can reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame.

[0117] According to one embodiment, the electronic device (101) can identify a weight map representing the degree of local motion of the first image frame based on the difference between the first image frame and the second image frame, and reduce the spatial noise level of the motion area of ​​the first image frame according to the weight map. The motion information may include at least one of the weight map or vector information related to the degree of local motion. For example, the electronic device (101) may identify motion information associated with the image frames and may estimate motion information representing the degree of motion of the second image frame relative to the first image frame. The motion information may include a motion vector existing between the first image frame and the second image frame, and may include information on a global motion vector, a local motion vector, or both.

[0118] According to one embodiment, the electronic device (101) can perform spatial noise reduction on a first image frame acquired through a camera (380). The electronic device (101) can generate a first-1 image frame in which spatial noise reduction is performed on the first image frame, and can generate (or acquire) a first-2 image frame in which the spatial noise level of the motion area of ​​the first-1 image frame is reduced according to the weight map.

[0119] According to one embodiment, the electronic device (101) can correct the degree of local movement of the second image frame based on the first image frame, and based on the corrected second image frame, generate a first-2 image frame in which the spatial noise level of the movement area of ​​the first-1 image frame is reduced according to the weight map.

[0120] According to one embodiment, the electronic device (101) can generate a first-3 image frame by removing signals having a value below a threshold value among signals representing the difference between the pixels of the first image frame and the pixels of the first-1 image frame, based on a coring function.

[0121] According to one embodiment, the electronic device (101) may obtain motion information (or motion vector) based on the difference between the first image frame and the second image frame when performing a noise removal operation on the first image frame. For example, the electronic device (101) may generate a corrected second image frame (e.g., second-1 image frame) by shifting (or correcting) the second image frame based on the motion vector (e.g., global motion vector).

[0122] According to one embodiment, the electronic device (101) can generate a second-1 image frame that corrects the degree of global motion of the second image frame, and generate a first-4 image frame that reduces the spatial noise level of the static region of the first-3 image frame according to the weight map based on the second-1 image frame and the first-3 image frame.

[0123] According to one embodiment, the electronic device (101) can identify a first window having a first size set around a first pixel of the first image frame. The electronic device (101) can identify a second pixel of the second-1 image frame corresponding to the first pixel using the vector information, and a second window having a second size set around the second pixel within the second-1 image frame. The electronic device (101) can identify the similarity between the pixels of the first window having the first size and the pixels of the second window having the second size.

[0124] According to one embodiment, the electronic device (101) can determine a weight based on the identified similarity and reduce the spatial noise level of the motion area of ​​the first-1 video frame based on the determined weight.

[0125] In operation 420, the electronic device (101) can acquire a third image frame by using the motion area of ​​the first image frame where the spatial noise level is reduced and at least a portion of the first image frame. According to one embodiment, the electronic device (101) can acquire the third image frame by blending the first-2 image frame and the first-4 image frame.

[0126] In operation 425, the electronic device (101) can obtain a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame. According to one embodiment, the electronic device (101) can obtain the fourth image frame by blending at least a portion of the third image frame and at least a portion of the second-1 image frame.

[0127] FIG. 5 is a detailed configuration diagram of an electronic device for noise reduction for video according to one embodiment.

[0128] Referring to FIG. 5, the electronic device (101) may include a global motion estimation / global motion compensation (503), a motion analysis (505), a spatial noise reduction (spatial NR) (507), a local motion estimation / local motion compensation (509), a spatial blending (512), a temporal blending (513), spatiotemporal filters (514-1, 514-2), and a coring function (518). According to one embodiment, the temporal filter (520) may be composed of a spatial blending (512) and a temporal blending (513).

[0129] At least one of the configurations shown in FIG. 5 may be omitted. For example, a configuration in which the spacetime filter (514-2) and the coring function unit (518) of FIG. 5 are omitted is as shown in FIG. 6. Additionally, a configuration in which the spacetime filter (514-1) and the coring function unit (518) of FIG. 5 are omitted is as shown in FIG. 7. Additionally, a configuration in which the spacetime filter (514-1) of FIG. 5 is omitted is as shown in FIG. 8. The operation when at least one of the configurations shown in FIG. 5 is omitted will be described later in FIG. 6 to FIG. 8.

[0130] According to one embodiment, when processing consecutive frames or multiple frames in time (or along the time axis) for a video image, global movement of the image may occur due to camera movement when comparing the image frame by frame along the time axis.

[0131] According to one embodiment, the global motion estimation unit (503) can correct the previous image frame to be mapped to the current image frame by warping each pixel by setting the current image frame (e.g., the first image frame) (e.g., the Nth current image frame) (502) as a reference frame for the previous image frame (e.g., the second image frame, which is an image frame from which noise has been removed or reduced according to the previous noise removal procedure) (e.g., the N-1th previous frame TNR result) (501).

[0132] For example, the global motion estimation unit (503) can calculate a homography matrix representing the relationship between two video frames by using the difference (e.g., pixel value) between pixels of the previous video frame and the current video frame, or by extracting and matching feature points from each of the two video frames. Using the calculated homography matrix, the current video frame can be set as a reference video frame for the previous video frame, and each pixel of the previous video frame can be warped to correspond to the current video frame. The global motion estimation unit (503) can generate (or acquire, output) a second-1 video frame (e.g., N-1th previous frame TNR result) (504) by warping each pixel within the second video frame (e.g., N-1th previous frame TNR result) (501) so as to be mapped to the first video frame (e.g., Nth current video frame) (502).

[0133] For example, the global motion estimation unit (503) can output a second-1 video frame (e.g., N-1st previous video frame TNR result) (504) by estimating motion information indicating the degree of motion that indicates how much of a relative position difference there is between the previous video frame or the reference area of ​​the previous video frame and the current video frame based on the current video frame or the reference area of ​​the current video frame. This is achieved by taking the first video frame (e.g., the Nth current video frame) (502) and the second video frame (e.g., the N-1st previous video frame TNR result) (501) as inputs.

[0134] According to one embodiment, the spatial noise reduction unit (507) can process the current video frame such that the noise level of the current video frame is the same or similar to the noise level of the previous video frame from which noise has been removed. For example, the spatial noise reduction unit (507) performs the role of primarily preprocessing noise for the current video frame and can generate (or acquire, output) a first-1 video frame (e.g., Nth current video frame') (508) from which noise has been removed (or reduced) so that the noise level of the second video frame from which noise has been removed (or reduced) (e.g., the N-1st previous video frame TNR result) (501) and the first video frame containing noise (e.g., Nth current video frame) (502) become the same or similar.

[0135] According to one embodiment, the motion analysis unit (505) may generate (or determine) a weight map (506) representing the degree of local movement to identify areas with local movement and areas without movement caused by an object within a first video frame. The motion analysis unit (505) may verify (or calculate) the degree of difference between a second-1 video frame (e.g., the N-1st previous video frame TNR result) (504) warped by the global motion estimation unit (503) and a first-1 video frame (e.g., the Nth current video frame) (508) from which noise has been preprocessed (or noise removed (or reduced)). By verifying the degree of difference, the motion analysis unit (505) may generate (or obtain, output) a weight map (506) representing the degree of local movement. For example, the weight map (506) may include information on weight values ​​between 0 and 1.

[0136] According to one embodiment, a weight map (506) output from a motion analysis unit (505) may be transmitted to each of a spatial blending unit (512) and a temporal blending unit (513) to determine blending weights. For example, the blending ratio in each of the spatial blending unit (512) and the temporal blending unit (513) may be determined according to the weight map (506). Additionally, the weight map (506) may also be transmitted to each of the spatiotemporal filters (514-1, 514-2).

[0137] According to one embodiment, the local motion estimation unit (509) can identify (or detect) pixels corresponding to a local motion area, such as a motion area, within a first image frame.

[0138] According to one embodiment, the local motion estimation unit (509) can calculate a motion vector (511) in a certain area unit using a block matching algorithm or a dense optical flow algorithm. For example, the local motion estimation unit (509) can generate (or acquire, output) a local motion vector (511) indicating how much movement has occurred in the x and y directions, for instance. The local motion vector (511) can be transmitted to each of the spatiotemporal filters (514-1, 514-2). Additionally, the local motion estimation unit (509) can take the warped 2-1 video frame (e.g., the N-1st previous video frame TNR result) (504) as input and generate (or acquire, output) a 2-2 video frame with local motion corrected (e.g., the N-1st previous video frame TNR result) (510).

[0139] According to one embodiment, a first-1 video frame (e.g., the Nth current video frame) (508), from which noise has been removed (or reduced) by a spatial noise reduction unit (507), may be used to further blend in a spatial blending unit (512) according to a weight map identified by a motion analysis unit (505). However, since noise removal by the spatial noise reduction unit (507) is not complete, the electronic device (201) according to one embodiment may include a spatio-temporal filter (514-1) (STF) to further remove noise in the motion region within the first video frame.

[0140] According to one embodiment, the spatiotemporal filter (514-1) can perform removal (or reduction, suppression) of spatial noise levels for motion regions within a first image frame. For example, the spatiotemporal filter (514-1) can perform removal (or reduction) of spatial noise levels for motion regions of a first-1 image frame (e.g., the Nth current image frame) (508) from which noise has been removed (or reduced) based on a weight map (506) and a local motion vector (511). According to one embodiment, the spatiotemporal filter (514-1) can use a second-2 image frame (e.g., the N-1th previous image frame TNR result) (510) from which local motion has been corrected as input to improve the noise removal performance.

[0141] For example, the spatiotemporal filter (514-1) can adjust the strength of the filter for spatial noise removal based on the weight map. According to one embodiment, the spatiotemporal filter (514-1) can set the strength of the spatiotemporal filter (514-1) for spatial noise removal relatively weak so that detailed image information can be preserved relatively well in areas where there is relatively little local motion, and in areas where there is relatively large motion, the strength of the spatiotemporal filter (514-1) for spatial noise removal can be set relatively strong so that noise can be removed as much as possible rather than preserving detailed image information for the area, taking into account the image blur caused by the motion. According to one embodiment, a weak strength of the spatiotemporal filter (514-1) means that the original data is maintained by applying filtering relatively little, and a strong strength of the filter may mean that the original data is changed by removing noise from the original data. Therefore, by setting the strength of the above spatiotemporal filter (514-1) strongly, the noise level corresponding to the moving area can be lowered.

[0142] According to one embodiment, the spatiotemporal filter (514-1) can generate (or acquire, output) a first-2 image frame (e.g., the Nth current image frame) (515-1) by removing (or reducing) the spatial noise level in the motion area of ​​a first-1 image frame (e.g., the Nth current image frame) (508).

[0143] According to one embodiment, the temporal filter (520) may be composed of a spatial blending unit (512) and a temporal blending unit (513). According to one embodiment, the spatial blending unit (512) may blend a current video frame (e.g., a first video frame) (e.g., the Nth current video frame) (502) and a first-second video frame (e.g., the Nth current video frame) (515-1) output from the spatiotemporal filter (514-1) according to the degree of movement based on a weight map (506).

[0144] According to one embodiment, the electronic device (101) may further include a spatiotemporal filter (514-2) for removing spatial noise for an area (or static area) that has no local movement within a first image frame. The spatiotemporal filter (514-2) may serve to filter noise in the current image frame that contains the noise as is.

[0145] According to one embodiment, the electronic device (101) may further include a coring function (518) that removes noise below a certain level to improve the performance of the spatiotemporal filter (514-2). Since the noise removal processing performance of the resulting image frame output from the spatiotemporal filter (514-2) increases as the noise level of the image frame input to the spatiotemporal filter (514-2) decreases, the quality of the final resulting image frame can be further improved by adding the coring function (518).

[0146] For example, the electronic device (101) can remove low-level noise from a residual image frame representing the difference between a first image frame (e.g., the Nth current image frame) (508) from which noise has been removed (or reduced) and a first image frame (e.g., the Nth current image frame) (502) from which noise has not been removed, by using a coring function part (518). In the residual image frame, the signal and noise are mixed, and there is a high probability that the part with a low residual difference between the two image frames (502, 508) corresponds to the noise. Accordingly, the electronic device (101) can use the coring function unit (518) to input the first-1 video frame (e.g., the Nth current video frame) (508) and the first video frame (e.g., the Nth current video frame) (502), and among the residual signals between the two video frames (502, 508), signals below a threshold value are considered as noise and fixed to 0 or a value adjacent to 0, and only the signal portion above the threshold value is retained to generate the first-3 video frame (e.g., the Nth current video frame) (519). The specific operation of the coring function unit (518) will be described later in FIG. 12.

[0147] According to one embodiment, a first-third image frame (e.g., the Nth current image frame) (519) output from a coring function unit (518) can be input to a spatiotemporal filter (514-2). The spatiotemporal filter (514-2) can generate a first-fourth image frame (e.g., the Nth current image frame) (515-2) by removing noise levels from the motion and static regions of the first-third image frame (e.g., the Nth current image frame) (519).

[0148] According to one embodiment, when the coring function part (518) and the spatiotemporal filter (514-2) are omitted, the spatial blending part (512) can apply a spatial blending operation to a first image frame containing noise (e.g., the Nth current image frame) (502). According to one embodiment, if the electronic device (101) further includes a coring function unit (518) and a spatiotemporal filter (514-2), the spatial blending unit (512) can generate a first-fifth image frame (e.g., the Nth current image frame) (515-1) in which the noise level is lowered for the motion area of ​​the first image frame by referring to a weight map (506), and a first-fourth image frame (e.g., the Nth current image frame) (515-2) refined by the spatiotemporal filter (514-2) by blending (or synthesizing) the first-fifth image frame (e.g., the Nth current image frame) (516). For example, the spatial blending unit (512) can serve to preserve (add-back) detailed image information of the current image frame that may have been removed by the spatial noise reduction unit (507). In this way, the spatial blending unit (512) can preserve detailed image information of the current image frame by blending the current image frame and the spatial noise removal result.

[0149] According to one embodiment, the temporal blending unit (513) can generate the first-sixth video frame (e.g., the Nth current video frame) (516) output from the spatial blending unit (512) by referring to the weight map (506) and blending the second-sixth video frame (e.g., the N-1st previous frame TNR result) (504). For example, the temporal blending unit (513) can blend the first-sixth video frame (e.g., the Nth current video frame) (516), which is the result of spatial blending for the current video frame, with the noise-removed result of the previous video frame at a ratio according to the degree of motion (or presence or absence of motion) (e.g., the weight map (506)). For example, the blending ratio can be adaptively determined based on the presence or absence of movement, and the blending ratio can be determined as the ratio that maximizes noise reduction.

[0150] For areas without movement, detailed information of the image can be accumulated even if noise below a threshold is present, whereas for areas with movement, the intensity (or weight) of noise reduction for the current image frame is maximized to obtain a result image frame with the noise level lowered as much as possible. Accordingly, the electronic device (101) can obtain (or generate, output) a final result image (e.g., the first-sixth image frames (e.g., the Nth current image frame) (517)) in which detailed image information of the first image frame is preserved.

[0151] According to one embodiment, when the electronic device (101) further includes a coring function unit (518) and a spatiotemporal filter (514-2), the electronic device (101) can preserve (or maintain) detailed information regarding the motion area of ​​the first image frame by performing blending according to the degree of motion using a noise-removed first-fourth image frame (e.g., the Nth current image frame) (515-2) rather than blending the noise-removed first image frame (e.g., the Nth current image frame) (502) as is, thereby obtaining an image of clear quality.

[0152] FIG. 6 is a detailed configuration diagram of an electronic device for noise reduction in video when a spatiotemporal filter is applied to a local motion area according to one embodiment.

[0153] Referring to FIG. 6, the electronic device (101) may include a global motion estimation unit (503), a motion analysis unit (505), a spatial noise reduction unit (507), a local motion estimation unit (509), a spatial blending unit (512), a temporal blending unit (513), and a spatiotemporal filter (514-1). According to one embodiment, the temporal filter (520) may be composed of a spatial blending unit (512) and a temporal blending unit (513). FIG. 6 illustrates a case where the coring function unit (518) and the spatiotemporal filter (514-2) of FIG. 5 are omitted.

[0154] Referring to FIG. 6, the spatiotemporal filter (514-1) can reduce the spatial noise level of the first-1 image frame (e.g., the Nth current image frame') (508) from which noise has been removed for the first image frame (e.g., the Nth current image frame) (502). The noise removal result by the spatial noise reduction unit (507) (e.g., the first-1 image frame (e.g., the Nth current image frame') (508)) can be reflected (or applied) more in the area with local movement when blending according to the weight map (506) in the spatial blending unit (512). Even if the first-1 image frame (e.g., the Nth current image frame') (508) is an image frame from which noise has been removed (or reduced) by the spatial noise reduction unit (507), there may be noise that has not been removed. If noise is further removed in the moving area, more image blur may occur in the moving area; however, if a spatiotemporal filter (514-1) is used, noise can be efficiently removed while preserving the details of the first image frame. According to one embodiment, the spatiotemporal filter (514-1) can remove (or reduce) spatial noise in the moving area within the first image frame. Additionally, the spatiotemporal filter (514-1) can take as input a second-second image frame (e.g., the N-1th previous image frame TNR result) (510) in which global motion and local motion have been corrected.

[0155] According to one embodiment, a spatiotemporal filter (514-1) can reduce the spatial noise level of a motion area of ​​a first image frame by using at least a portion of a corrected second-2 image frame (e.g., the N-1st previous image frame TNR result) (510) and a noise-removed (or reduced) first-1 image frame (e.g., the Nth current image frame) (508) by referring to a weight map (506) and a motion vector (511). The spatiotemporal filter (514-1) can generate (or acquire, output) a first-2 image frame (e.g., the Nth current image frame) (515-1) containing a motion area in which the spatial noise level of the first image frame has been reduced.

[0156] According to one embodiment, the spatial blending unit (512) can generate a first-third image frame (e.g., the Nth current image frame) (516) by blending the first-second image frame (e.g., the Nth current image frame) (515-1) output from the spatiotemporal filter (514-1) and the first image frame (e.g., the Nth current image frame) (502) based on the degree of movement according to the weight map (506).

[0157] According to one embodiment, the temporal blending unit (513) can generate a first-fourth video frame (e.g., the Nth current video frame) (516) and a second-first video frame (e.g., the N-1st previous frame TNR result) (504) by blending them based on a weight map (506). For example, the temporal blending unit (513) can generate (or acquire, output) a final result video frame (e.g., the first-to-third video frame (e.g., the Nth current video frame) (516)) by blending the result of spatial blending for the current video frame (e.g., the first-to-third video frame (e.g., the N-1st previous frame TNR result) (504)) according to the degree of motion (e.g., weight map (506)).

[0158] According to one embodiment, by using a spatiotemporal filter (514-1), the electronic device (101) can acquire a final resulting image frame in which the noise level for the current image frame is reduced to the maximum while maintaining a low noise level for areas where there is no movement in the current image frame, while accumulating detailed information for areas where there is movement.

[0159] FIG. 7 is a detailed configuration diagram of an electronic device for noise reduction in video when a spatiotemporal filter is applied to a static area according to one embodiment.

[0160] Referring to FIG. 7, the electronic device (101) may include a global motion estimation unit (503), a motion analysis unit (505), a spatial noise reduction unit (507), a spatial blending unit (512), a temporal blending unit (513), and a spatiotemporal filter (514-2). FIG. 7 illustrates a configuration for noise removal for a static region without motion, in which the local motion estimation unit (509), the coring function unit (518), and the spatiotemporal filter (514-1) of FIG. 5 are omitted. According to one embodiment, the temporal filter (520) may be composed of a spatial blending unit (512) and a temporal blending unit (513).

[0161] According to one embodiment, a spatiotemporal filter (514-2) can generate a first-second image frame (e.g., the Nth current image frame) (515-2) by taking a first image frame containing noise (e.g., the Nth current image frame) (502) and a second image frame with a global motion degree corrected (e.g., the N-1st previous image frame TNR result) (504) as inputs and removing the spatial noise level for a static area without motion of the first image frame (e.g., the Nth current image frame) (502) based on a weight map (506).

[0162] According to one embodiment, the spatial blending unit (512) takes a noise-removed first-1 video frame (e.g., the Nth current video frame) (508) and a first-2 video frame (e.g., the Nth current video frame) (515-2) as inputs, and in a static area where there is no movement, it can primarily perform the role of adding back detailed video information of the current video frame by referring to a weight map (506).

[0163] According to one embodiment, the spatiotemporal filter (514-2) transmits the first-second image frame (e.g., the Nth current image frame) (515-2), which is refined from the first image frame (e.g., the Nth current image frame) (502), to the spatial blending unit (512), thereby preventing temporal noise from being introduced into the resulting image frame (517) output from the temporal blending unit (513).

[0164] FIG. 8 is a detailed configuration diagram of an electronic device for noise reduction for video using a coring function unit according to one embodiment. FIG. 8 illustrates a case in which a coring function (518) is further included in the configuration of FIG. 7. According to one embodiment, the temporal filter (520) may be composed of a spatial blending unit (512) and a temporal blending unit (513).

[0165] According to one embodiment, the coring function unit (518) can remove low-level noise from a residual image frame representing the difference between a noise-removed first-1 image frame (e.g., Nth current image frame) (508) and a first image frame (e.g., Nth current image frame) (502) before noise removal. The coring function unit (518) can generate a first-2 image frame (e.g., Nth current image frame) (519) by adding the first-1 image frame (e.g., Nth current image frame) (508) to the image frame from which low-level noise has been removed.

[0166] According to one embodiment, a first-second video frame (e.g., the Nth current video frame) (519) output from a coring function unit (518) can be input to a spatiotemporal filter (514-2). The spatiotemporal filter (514-2) can generate a first-third video frame (e.g., the Nth current video frame) (515-2) by lowering the noise level in the motion area and static area of ​​the first-second video frame (e.g., the Nth current video frame) (519).

[0167] According to one embodiment, the electronic device (101) can acquire (or generate, output) a final result image (e.g., first-fourth image frames (e.g., the Nth current image frame) (517)) in which detailed image information of the first image frame is preserved through the spatial blending unit (512) and the temporal blending unit (513).

[0168] FIG. 9a is a detailed configuration diagram of a spatiotemporal filter according to one embodiment.

[0169] Referring to FIG. 9a, the spacetime filter (514) may include at least one of the spacetime filter (514-1) or the spacetime filter (514-2) of FIG. 5. For example, if the spacetime filter (514-2) in FIG. 5 is omitted, the spacetime filter (514) of FIG. 9a may correspond to the spacetime filter (514-1).

[0170] Referring to FIG. 9a, the spatiotemporal filter (514) (e.g., the spatiotemporal filter (514-1) of FIG. 5) can use the current image frame (e.g., the Nth image frame) (502) as an input when removing noise from the current image frame (e.g., the Nth image frame) based on the degree of motion.

[0171] According to one embodiment, since the current image frame (e.g., the Nth image frame) (502) is an image frame that is input while containing noise, the spatiotemporal filter (514) (e.g., the spatiotemporal filter (514-1) of FIG. 5) may use the current image frame (e.g., the Nth image frame) (SNR output) (508) that has been noise-removed by the spatial noise reduction unit (507) as input, rather than using the current image frame (e.g., the Nth image frame) (502) containing noise as is, in order to increase noise removal performance.

[0172] According to one embodiment, the spatiotemporal filter (514) (e.g., the spatiotemporal filter (514-2) of FIG. 5) may use as input the current image frame (e.g., the Nth image frame) (coring output) (519) from which low-level noise has been removed by the coring function (518) to increase noise removal performance. Additionally, the spatiotemporal filter (514) (e.g., the spatiotemporal filter (514-1) or the spatiotemporal filter (514-2) of FIG. 5) may use the previous image frame (e.g., the N-1th image frame) to reduce the spatial noise level for the current image frame.

[0173] According to one embodiment, in order to remove noise by distinguishing between a moving area and a static area without movement within the current video frame, the spatiotemporal filter (514) may refer to at least one of a motion vector (511) which is the result of a local motion estimation unit (509) and a weight map (or motion weight) (506) which is the result of a motion analysis unit (505).

[0174] For example, the global motion estimation unit (503) can extract the overall motion between the current video frame and the previous video frame caused by camera motion, and obtain (or generate) a previous video frame (e.g., N-1th previous video frame TNR result) (504) that corrects the degree of global motion corresponding to the overall motion.

[0175] For example, the local motion estimation unit (509) can extract motion caused by an object included in the current video frame and the previous video frame, and obtain a local motion (or degree of local motion) corresponding to the extracted motion as a motion vector (511). Additionally, the local motion estimation unit (509) can obtain (or generate) a previous video frame (e.g., the N-1th video frame) (510) with the local motion corrected. For example, the degree of local motion should be understood as meaning the local motion of an object within the video frame.

[0176] According to one embodiment, the spatiotemporal filter (514) serves to preserve details while removing temporal noise of the current video frame for areas without local motion, and for areas with local motion, since there may be noise that was not removed by the spatial noise reduction unit (507), the noise level can be lowered by using the result of removing temporal noise from the previous video frame. In this way, the spatiotemporal filter (514) can balance the noise levels between areas with local motion and areas without local motion.

[0177] FIG. 9b is a drawing for explaining the detailed operation of a spacetime filter according to one embodiment.

[0178] Referring to FIG. 9b, the spatiotemporal filter (514) can serve to reduce the noise level while preventing artifacts caused by the movement of an object within the current video frame in areas of movement within the current video frame. In order to prevent artifacts caused by the movement of an object in areas of local movement within the current video frame, it may be important to find the corresponding area in the corrected previous video frame by lowering the intensity of the noise reduction.

[0179] Referring to Fig. 9b, it is necessary to set the window size to find the corresponding pixels in the previous image frame that correspond to the pixels in the local motion area of ​​the current image frame.

[0180] According to one embodiment, the spatiotemporal filter (514) can determine the overall degree of motion based on a weight map (506) (e.g., global motion vector). If the overall degree of motion is greater than a threshold value, for example, it may be a situation where shaking occurs due to camera movement. If the overall degree of motion is greater than the threshold value, it is difficult to use the current video frame as a reference video frame, so the motion analysis unit (505) can set the weight map (506) high to prevent the spatiotemporal filter (514) from using the current video frame as a reference video frame.

[0181] The spatiotemporal filter (514) may use a motion vector (511) related to the degree of local movement for window settings for the previous video frame. The larger the degree of local movement according to the motion vector (511), the larger the window size may be set to expand the search area so that more similar pixel information within the previous video frame can be found and reflected.

[0182] The spatiotemporal filter (514) can set the window size and corresponding pixels of the previous video frame based on motion information (e.g., motion vector (511)) (901), and set the window size of the current video frame (902). The window setting (902) of the current video frame can be determined based on the noise level of the result of spatial noise reduction processing, thereby providing the effect of performing additional spatial filtering.

[0183] For example, a first window having a first size set around a first pixel within the current image frame can be identified. The spatiotemporal filter (514) can use vector information (e.g., local motion vector (511)) to identify a second window having a second size set around a second pixel of a previous image frame corresponding to the first pixel of the current image frame.

[0184] Refer to FIGS. 10 and FIGS. 11 to explain how to set the window sizes of the current image frame and the previous image frame. FIG. 10 is an example diagram for explaining how to set the window sizes of the previous image frame and the current image frame for spatiotemporal filtering according to one embodiment, and FIG. 11 is a diagram showing the window areas of the previous image frame and the current image frame according to one embodiment.

[0185] According to one embodiment, not only the local motion vector (511) but also the illuminance value can be used to set the window size. For example, the size of the first window of the current image frame and the size of the second window of the previous image frame can be adaptively set according to the illuminance environment of the image. Since movement may occur more significantly as the illuminance decreases, the size of the second window of the previous image frame can be set larger in inverse proportion to the illuminance. Additionally, the center pixel of the search area can be set based on the local motion estimated using the local motion vector (511).

[0186] Referring to FIG. 10, a first window (or first area) (1020) having a first size of the current image frame (1000b) may correspond to a second window (or second area) (1010) having a second size centered on a center pixel corresponding to the position of the first pixel within the previous image frame (1000a).

[0187] Referring to FIG. 11, the spatiotemporal filter (514) can check (or determine) similarity based on information (903) regarding the size of a second window (or second region) (1010) set around the first pixel and a center pixel corresponding to the position of the first pixel within the previous image frame (1000a), and information (904) regarding the size of a first window (or first region) (1020) of the current image frame (1000b). For example, the spatiotemporal filter (514) can check the similarity of pixel values ​​(intensity) based on at least one of the positional difference and brightness difference between the pixels of the first window and the pixels of the second window.

[0188] The spatiotemporal filter (514) determines (or verifies) weights (905) based on the verified similarity, and can obtain pixel values ​​(906) that reduce the spatial noise level of the local motion area of ​​the current video frame based on the determined weights.

[0189] FIG. 11 illustrates a second window of a second size of a previous image frame (1010) corresponding to a first pixel of a first window of a first size of a current image frame (1020), and the second size is illustrated as being 3 pixels × 3 pixels, but the size of the second window can be determined in various ways, such as 5 pixels × 5 pixels, 7 pixels × 7 pixels, or 9 pixels × 9 pixels, depending on at least one of the lighting environment or the degree of movement (e.g., motion vector (511)).

[0190] According to one embodiment, pixels adjacent to the first pixel within the first window of the current image frame may have few or no pixels aligned within the second window of the previous image frame due to incorrectly estimated motion vectors or occlusion. Therefore, pixels surrounding the first pixel of the current image frame have a relatively higher weight than pixels in the second window of the previous image frame, thereby preventing artifacts that may occur around or inside an object with local movement.

[0191] According to one embodiment, the weight calculation (905) of the spatiotemporal filter (514) may be based on the following mathematical formula 1.

[0192]

[0193] In the above mathematical formula 1, Ω p is the previous video frame, and Ω c is the current image frame, I is the pixel intensity, Im is the current pixel, f is the spatial term, and g is the range term. The functions f and g in Equation 1 above are Gaussian functions.

[0194] For example, the function f (spatial term) can represent that weights are determined based on the positional difference of each pixel, and the function g (range term) can represent that weights are determined based on the difference in brightness intensity.

[0195] k(n) in the above mathematical formula 1 can be determined based on the following mathematical formula 2 as a normalization function.

[0196]

[0197] Referring to the above mathematical formula 2, the strength of the spatiotemporal filter (514) can be determined by setting the sigma parameter of the Gaussian function. In the case of the range term, the noise level according to the brightness intensity is determined, and in the case of the spatial term, the phenomenon where the degree of image blur increases when the degree of movement is large can be taken into account. The spatiotemporal filter (514) can increase the sigma value according to the motion vector (511) or the weight map (506) representing the degree of global movement.

[0198] In order to specifically explain the coring function part (518) used to increase the noise removal performance of the spatiotemporal filter (514) according to one embodiment, we will refer to FIG. 12.

[0199] FIG. 12 is a graph for explaining a linear function by interval according to one embodiment.

[0200] According to one embodiment, the coring function unit (518) may perform the role of refining a current image frame (e.g., the Nth current image frame) (502) containing noise. The coring function unit (518) may use as input the current image frame (e.g., the Nth current image frame) (502) before noise removal and the current image frame (e.g., the Nth current image frame') (508) from which noise has been removed (or reduced) by the spatial noise reduction unit (507). The coring function unit (518) may use a segmented linear function in which, among the residual signals using the difference between two image frames (502, 508), signals below a specific threshold value are considered as noise and fixed to 0 (1201), pixels having residual values ​​above a certain threshold are increased in size by a certain ratio (1202), and pixels corresponding to residual signals containing detailed information are allowed to pass the residual values ​​as they are (1203).

[0201] The above linear function for each interval can be expressed as a formula as in Equation 3 below.

[0202]

[0203] Referring to the above mathematical formula 3, the residual image R(x, y) can be obtained by using the difference between the output result of the spatial noise reduction unit (507) and the current image frame (e.g., the Nth image frame) containing noise (502), and the interval through which the residual signal is differentially passed can be determined by T0 and T1. Here, there may be one or more threshold values, and an anchor interval may be set using one or more threshold values. For example, the threshold value may be set based on image information corresponding to a moving area, a static area without movement, and / or brightness intensity.

[0204] FIG. 13 is a drawing showing an image with a noise removal function applied by a coring function part according to one embodiment.

[0205] Referring to FIG. 13, in the case of images taken in a low-light environment, the image quality may degrade because they are generally dark and noisy, and as illustrated in 1300a, the current image (or captured image) may contain noise. When a spatiotemporal filter (514) is applied to a case where there is no movement in the current image, the performance of the spatiotemporal filter (514) can be improved by lowering the noise level of the current image frame and then using it as an input to the spatiotemporal filter (514). Additionally, if the coring function unit (518) is used, noise that is a signal below a specific threshold can be removed, thereby obtaining a current image frame in which edge and detail information is preserved. The result from the coring function unit (518) can be added to the current image frame from which noise has been removed by the spatial noise reduction unit (507) and used as an input to the spatiotemporal filter (514). In 1300b, the result of adding the result from the coring function unit (518) and the current image frame from which noise has been removed by the spatial noise reduction unit (507) is exemplified. According to one embodiment, if the current image frame from which noise below a specific threshold has been removed using the coring function unit (518) is used as an input to the spatiotemporal filter (514), the noise removal performance of the spatiotemporal filter (514) can be further enhanced.

[0206] FIG. 14 is a diagram illustrating the noise reduction effect resulting from the application of spatiotemporal filtering in a low-light image according to one embodiment.

[0207] In FIG. 14, a spatiotemporal filter (514) can be applied to a captured image (1400) containing multiple regions of interest (ROI) (1401, 1402, 1403, 1404, 1405) as illustrated in 1400a. When comparing the case where the spatiotemporal filter (STF) is applied with the case where it is not applied, an image quality improvement result for the captured image can be obtained when the spatiotemporal filter (STF) is applied, as illustrated in 1400b. For example, FIG. 14 illustrates a case where the improvement value of the peak signal-to-noise ratio (PSNR) as a result of image quality improvement is approximately 2 to 2.5 dB, but the improvement value is merely an example to aid understanding and is not limited thereto.

[0208] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0209] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0210] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0211] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0212] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0213] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0214] According to one embodiment, in a storage medium storing at least one computer-readable instruction, the at least one instruction causes the electronic device (101) to perform at least one operation when executed individually or collectively by at least one processor (120, 320) of the electronic device, and the at least one operation may include the operation of acquiring a first image frame having a first noise level from a camera (180, 380).

[0215] According to one embodiment, the at least one operation may include an operation of identifying a second image frame having a second noise level preceding the first image frame.

[0216] According to one embodiment, the at least one operation may include an operation to reduce the spatial noise level of the motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame.

[0217] According to one embodiment, the at least one operation may include acquiring a third image frame using the motion area of ​​the first image frame where the spatial noise level is reduced and at least a portion of the first image frame.

[0218] According to one embodiment, the at least one operation may include acquiring a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

[0219] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

Claims

Claim 1 In an electronic device (101), a camera (180, 380) including an image sensor (230); at least one processor (120, 320) including a processing circuit; The electronic device comprises a memory (130, 330) for storing instructions, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to acquire a first image frame having a first noise level from the camera, identify a second image frame having a second noise level preceding the first image frame, reduce the spatial noise level of a motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame, acquire a third image frame using the motion area of ​​the first image frame where the spatial noise level of the first image frame has been reduced and at least a portion of the first image frame, and acquire a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame. Claim 2 In claim 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device identifies a weight map representing the degree of local motion of the first image frame based on the difference between the first image frame and the second image frame, reduces the spatial noise level of the motion area of ​​the first image frame according to the weight map, and the motion information includes at least one of the weight map or vector information related to the degree of local motion. Claim 3 An electronic device according to claim 1 or 2, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to generate a first-1 image frame in which spatial noise reduction is performed on the first image frame, and to reduce the spatial noise level of the motion area of ​​the first-1 image frame according to the weight map. Claim 4 An electronic device according to any one of claims 1 to 3, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to correct the degree of local motion of the second image frame based on the first-1 image frame, and, based on the corrected second image frame, generate a first-2 image frame in which the spatial noise level of the motion area of ​​the first-1 image frame is reduced according to the weight map. Claim 5 An electronic device according to any one of claims 1 to 4, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to generate a first-third image frame by removing signals having a value below a threshold value among signals representing the difference between the pixels of the first image frame and the pixels of the first-first image frame based on a coring function. Claim 6 An electronic device according to any one of claims 1 to 5, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to generate a second-1 image frame that corrects the degree of global motion of the second image frame, and generate a first-4 image frame that reduces the spatial noise level of the static region of the first-3 image frame according to the weight map based on the second-1 image frame and the first-3 image frame. Claim 7 An electronic device according to any one of claims 1 to 6, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to acquire the third image frame by blending the first-2 image frame and the first-4 image frame, and to acquire the fourth image frame by blending at least a portion of the third image frame and at least a portion of the second-1 image frame. Claim 8 An electronic device configured such that, in any one of claims 1 to 7, when the instructions are executed individually or collectively by the at least one processor, the electronic device identifies a first window having a first size set around a first pixel of the first image frame, and using the vector information identifies a second pixel of the second-1 image frame corresponding to the first pixel, and a second window having a second size set around the second pixel within the second-1 image frame, and identifies a similarity between the pixels of the first window having the first size and the pixels of the second window having the second size. Claim 9 An electronic device according to any one of claims 1 to 8, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to determine the similarity based on at least one of a positional difference and a brightness difference between pixels of a first window having a first size and pixels of a second window having a second size. Claim 10 An electronic device according to any one of claims 1 to 9, wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to determine a weight according to the identified similarity and reduce the spatial noise level of the motion area of ​​the first-1 image frame according to the determined weight. Claim 11 A method for reducing noise in a video in an electronic device (101), comprising: acquiring a first video frame having a first noise level from a camera; identifying a second video frame having a second noise level preceding the first video frame; reducing the spatial noise level of a motion area of ​​the first video frame based on motion information related to the difference between the first video frame and the second video frame; acquiring a third video frame using the motion area of ​​the first video frame where the spatial noise level is reduced and at least a portion of the first video frame; and acquiring a fourth video frame having a third noise level lower than the first noise level based on at least a portion of the third video frame and at least a portion of the second video frame. Claim 12 A method for noise reduction for a video, wherein, in claim 11, the operation of reducing the spatial noise level of the motion area of ​​the first video frame comprises: the operation of identifying a weight map representing the degree of local motion of the first video frame based on the difference between the first video frame and the second video frame; and the operation of reducing the spatial noise level of the motion area of ​​the first video frame according to the weight map, wherein the motion information comprises at least one of the weight map or vector information related to the degree of local motion. Claim 13 A method for noise reduction for a video, wherein, in claim 11 or 12, the operation of reducing the spatial noise level of the motion area of ​​the first video frame comprises: the operation of generating a first-1 video frame in which spatial noise reduction for the first video frame has been performed; and the operation of reducing the spatial noise level of the motion area of ​​the first-1 video frame according to the weight map. Claim 14 A method for reducing noise in a video, wherein, in any one of claims 11 to 13, the operation of reducing the spatial noise level of the motion area of ​​the first video frame comprises: the operation of correcting the degree of local motion of the second video frame based on the first-1 video frame; and the operation of generating a first-2 video frame in which the spatial noise level of the motion area of ​​the first-1 video frame is reduced according to the weight map based on the corrected second video frame. Claim 15 A method for noise reduction in video, further comprising, in any one of claims 11 to 14, generating a first-3 image frame by removing a signal having a value below a threshold value among a signal representing the difference between the pixels of the first image frame and the pixels of the first-1 image frame based on a coring function. Claim 16 A method for noise reduction in a video, further comprising, in any one of claims 11 to 15, an operation of generating a 2-1 image frame that corrects the degree of global motion of the 2-1 image frame; and an operation of generating a 1-4 image frame that reduces the spatial noise level of a static region of the 1-3 image frame according to the weight map based on the 2-1 image frame and the 1-3 image frame. Claim 17 A method for reducing noise in a video, wherein, in any one of claims 11 to 16, the operation of acquiring the third image frame includes the operation of acquiring the third image frame by blending the first-2 image frame and the first-4 image frame, and the operation of acquiring the fourth image frame includes the operation of acquiring the fourth image frame by blending at least a portion of the third image frame and at least a portion of the second-1 image frame. Claim 18 A method for noise reduction in a video, further comprising, in any one of claims 11 to 17, an operation of identifying a first window having a first size set around a first pixel of the first image frame; an operation of identifying a second pixel of the second-1 image frame corresponding to the first pixel using the vector information, and a second window having a second size set around the second pixel within the second-1 image frame; and an operation of identifying a similarity between the pixels of the first window having the first size and the pixels of the second window having the second size. Claim 19 A method for reducing noise in a video, wherein, in any one of claims 11 to 18, the operation of reducing the spatial noise level of the motion area of ​​the first video frame comprises: the operation of determining a weight according to the identified similarity; and the operation of reducing the spatial noise level of the motion area of ​​the first-1 video frame according to the determined weight. Claim 20 A storage medium for storing at least one instruction readable by a computer, wherein the at least one instruction causes the electronic device (101) to perform at least one operation when executed individually or collectively by at least one processor (120, 320) of the electronic device, and the at least one operation comprises: acquiring a first image frame having a first noise level from a camera; identifying a second image frame having a second noise level preceding the first image frame; reducing the spatial noise level of a motion area of ​​the first image frame based on motion information related to the difference between the first image frame and the second image frame; acquiring a third image frame using the motion area of ​​the first image frame where the spatial noise level has been reduced and at least a portion of the first image frame; and acquiring a fourth image frame having a third noise level lower than the first noise level based on at least a portion of the third image frame and at least a portion of the second image frame.

Images