Electronic device and multi-frame fusing method

Abstract

An electronic device (400) according to various embodiments of the present disclosure may comprise: a camera (430); a memory (420); and at least one processor (410) operatively connected to the camera and the memory. The memory may include instructions that may be executed by the at least one processor and may cause, when executed, the electronic device to: set a first capture window (620) including a plurality of frames used to generate a first fused frame (626) in response to a first input for image capturing; set a second capture window (610) including a plurality of frames used to generate a second fused frame (636) in response to a second input detected after the first input; and generate a second fused frame on the basis of the frames of the second capture window and a third fused frame (848 or 866) generated by fusing at least some of the frames of the first capture window. Various other embodiments are possible.

Description

Electronic devices and multi-frame synthesis methods

[0001] This article relates to electronic devices, and for example, to a method for compositing multiple frames in an electronic device.

[0002] Portable electronic devices (hereinafter, "electronic devices"), such as smartphones and tablet PCs, can provide diverse user experiences using various applications. Electronic devices may include at least one camera, and images captured by the camera can be utilized in various applications. Electronic devices may have size constraints for portability, and thus, compared to standard cameras, the lenses and image sensors of electronic devices may be relatively smaller.

[0003] When configuring a camera's lens and sensor to be smaller, noise can increase and the dynamic range can narrow. To improve this, electronic devices can provide a multi-frame fusing function. Multi-frame fusing can be a function that synthesizes multiple sequentially acquired frames to create a single composite frame (fused frame).

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] The multi-frame compositing feature requires physical time to acquire and synthesize multiple image frames. Therefore, compared to standard shooting, it can take longer to complete the process from the start of the recording to the end. Consequently, when a user attempts multi-frame compositing, the delay between pressing the shutter button and the actual start of the recording, as well as the compositing process, can significantly delay the completion of the recording, potentially reducing the camera's usability.

[0006] An electronic device (400) according to this disclosure (or specification) may include a camera (430), a memory (420), and at least one processor (410) operatively connected to the camera and the memory.

[0007] According to one embodiment, the memory may store instructions that are executable by the at least one processor and, when executed, cause the electronic device to, in response to a first input for capturing an image, set a first capture window (620) including a plurality of frames used for generating a first composite frame (626), and in response to a second input detected after the first input, set a second capture window (610) including a plurality of frames used for generating a second composite frame (636).

[0008] According to one embodiment, the memory may store instructions that are executable by the at least one processor and, when executed, cause the electronic device to generate the second composite frame based on a third composite frame (848 or 866) generated by synthesizing the frames of the second capture window and at least some of the frames of the first capture window.

[0009] A method performed by an electronic device according to various embodiments of the present document may include, in response to a first input for capturing an image, setting a first capture window including a plurality of frames used for generating a first composite frame, and, in response to a second input detected after the first input, setting a second capture window including a plurality of frames used for generating a second composite frame.

[0010] According to one embodiment, the method may include generating the second composite frame based on a third composite frame generated by synthesizing at least some of the frames of the second capture window and the frames of the first capture window.

[0011] A computer-readable, non-transitory recording medium according to various embodiments of the present document may store instructions that, when executed by an electronic device, cause the electronic device to perform, in response to a first input for capturing an image, an operation of setting a first capture window including a plurality of frames used for generating a first composite frame, and, in response to a second input detected after the first input, an operation of setting a second capture window including a plurality of frames used for generating a second composite frame.

[0012] According to one embodiment, the recording medium may store instructions for generating the second composite frame based on a third composite frame generated by synthesizing at least some of the frames of the second capture window and the frames of the first capture window.

[0013] Various embodiments of this document can provide an electronic device and a multi-frame synthesis method that can reduce shutter delay and improve image quality when capturing an image using a multi-frame synthesis function of a camera.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] FIG. 2 is a block diagram of a camera module according to various embodiments.

[0016] FIGS. 3A and 3B illustrate capture windows used for multi-frame composite shooting according to one embodiment.

[0017] FIG. 4 is a block diagram of an electronic device according to various embodiments.

[0018] FIG. 5 illustrates a software and hardware configuration for performing multi-frame synthesis in an electronic device according to various embodiments.

[0019] FIG. 6 illustrates a method by which an electronic device according to one embodiment generates a current composite frame using frames of a current capture window and a previous composite frame.

[0020] FIG. 7 is a flowchart of a multi-frame synthesis method of an electronic device according to one embodiment.

[0021] FIGS. 8A, 8B and 8C illustrate a method for an electronic device to generate a composite frame when a current capture window and a previous capture window overlap, according to one embodiment.

[0022] FIG. 9 illustrates a method for an electronic device to generate a composite frame when a current capture window and a previous capture window do not overlap, according to one embodiment.

[0023] FIGS. 10A, 10B and 10C illustrate a method by which an electronic device generates a composite frame during long exposure photography, according to one embodiment.

[0024] FIG. 11 illustrates a method for an electronic device to generate composite frames during continuous shooting by a long press input, according to one embodiment.

[0025] FIG. 12 is a flowchart of a multi-frame synthesis method of an electronic device according to one embodiment.

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0027] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0028] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0029] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0030] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0034] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0035] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0036] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0040] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0042] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0045] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0047] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0050] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0051] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display device (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0052] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display device (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured to perform a control operation of the processor (120). It may be configured as a separate processor that is at least partially composed of, or operates independently of, the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display device (160) as is or after undergoing additional image processing by the processor (120).

[0053] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0054] FIGS. 3A and 3B illustrate capture windows used for multi-frame composite shooting according to one embodiment.

[0055] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may include at least one camera (e.g., the camera module (180) of FIGS. 1 and 2) disposed on the front and / or rear of a housing. In the case of portable electronic devices such as smart phones and tablet PCs, the size of the lens and image sensor of the camera may be relatively small compared to that of a general camera device due to size constraints. Due to such size constraints, noise may relatively increase and the dynamic range may be narrow when capturing an image.

[0056] According to one embodiment, the electronic device may provide a multi-frame fusion technology to compensate for the above-described noise increase and dynamic range limitations. Here, the multi-frame fusion technology may refer to an image processing method of synthesizing a plurality of sequentially acquired image frames into a single image frame. According to one embodiment, the electronic device may set a plurality of image frames used to generate a single fused frame as a capture window through the multi-frame fusion technology.

[0057] Referring to FIG. 3A, the electronic device can set five image frames, including an image frame N acquired at the time when a user input (or shutter input) for capturing an image is detected, and four image frames N-1, N-2, N-3, and N-4 acquired sequentially before the image frame N, as one capture window (310). Although FIG. 3A and some drawings to be described later describe image frames acquired before the time when the user input is detected and stored in a memory (e.g., the memory (250) of FIG. 2) as being set as one capture window, the present invention is not limited thereto, and the capture window may be configured with the image frame N and the image frames acquired thereafter (e.g., N+1, N+2, N+3, N+4), or the capture window may be configured including the image frames before and after the image frame N (e.g., N-2, N-1, N+1, N+2). In addition, although FIG. 3A and some drawings to be described later illustrate that one capture window includes five image frames, the number of image frames constituting the capture window is not limited thereto, and a capture window may be configured using a greater number (e.g., 30) of image frames. In addition, the number of image frames constituting the capture window may change depending on the setting mode and the shooting environment (e.g., ambient lighting, camera movement) during image shooting.

[0058] Referring to FIG. 3A, the electronic device can generate a composite frame (316) corresponding to a user input for capturing an image based on image frames N, N-1, N-2, N-3, and N-4 included in a capture window (310). When using a multi-frame synthesis technique in this way, noise removal and / or dynamic range enhancement can be provided. For example, a noise removal method of an image can remove noise by selecting an average or median value of surrounding pixels, but according to this, noise may not be properly removed from an actual image, the resolution may be lowered, and / or the modulation transfer function (MTF), which represents resolution and contrast performance, may be lowered. In contrast, when using a multi-frame synthesis technique, noise can be removed while minimizing image quality degradation by utilizing information from multiple image frames. In addition, when using a multi-frame synthesis technique, image frames with various exposures can be synthesized to express both bright and dark areas within a single image frame, thereby implementing HDR (high dynamic range).

[0059] FIG. 3B illustrates a case where a user input for capturing an image is input within a short period of time. Referring to FIG. 3B, the electronic device may designate image frames N-7, N-6, N-5, N-4, and N-3 as capture windows (340) based on a user input detected at the time of acquiring image frame N-3 to generate a previous composite frame (346), and may designate image frames N-4, N-3, N-2, N-1, and N as current capture windows (330) based on a current user input detected at the time of acquiring image frame N to generate a current composite frame (336). In this way, when a user input is input within a short period of time, at least one image frame (e.g., N-4, N-3) may overlap in the previous capture window (340) and the current capture window (330).

[0060] According to one embodiment, when an electronic device synthesizes image frames constituting capture windows (310, 330, 340) using a multi-frame synthesis technique, the synthesis process may take time. Referring to FIG. 3B, the previous synthesized frame (346) may be generated at the time of acquisition of image frame N-1, and the current synthesized frame (336) may be generated at the time of acquisition of image frame N+4. In this way, when user inputs are continuously input to the extent that two capture windows (330, 340) overlap, the synthesis of image frames may take longer, which may be perceived by the user as a shutter delay.

[0061] Below, various embodiments for reducing shutter delay and improving image quality when capturing an image using the multi-frame synthesis function of a camera will be described with reference to FIGS. 4 to 12.

[0062] In this document, the current capture window (330) set according to the current user input and the current fused frame (336) generated by synthesizing the same may be referred to as a second capture window and a second fused frame, respectively, and the previous capture window (340) set immediately before the current capture window (330) and the previous fused frame (346) generated by synthesizing the same may be referred to as a first capture window and a first fused frame. In addition, the user input that triggers the generation of the previous fused frame (or the first fused frame) may be referred to as the first input, and the user input that triggers the generation of the current fused frame (or the second fused frame) may be referred to as the second input.

[0063] FIG. 4 is a block diagram of an electronic device according to various embodiments.

[0064] Referring to FIG. 4, the electronic device (400) may include a display (450), a camera (430), a processor (410), a memory (420), and sensors (440). Even if at least some of the illustrated components are omitted or replaced with other components, various embodiments of the present document may be implemented. In addition to the illustrated components, the electronic device (400) may further include at least some of the components and / or functions of the electronic device (101) of FIG. 1. At least some of the components of the illustrated (or not illustrated) electronic device (400) (e.g., the processor (410), the memory (420), the sensors (440)) may be disposed within the housing of the electronic device (400), and at least some of the other components (e.g., the display (450), the camera (430)) may be at least partially visually exposed to the outside of the housing. At least some of the components of the electronic device (400) may be operatively, functionally and / or electrically connected to one another.

[0065] According to one embodiment, the display (450) can display image information provided from the processor (410). The display (450) can be implemented as any one of a liquid crystal display (LCD), a light-emitting diode (LED) display, and an organic light-emitting diode (OLED) display, but is not limited thereto. The display (450) can be configured as a touch screen that detects a touch and / or proximity touch (or hovering) input using a part of a user's body (e.g., a finger) or an input device (e.g., a stylus pen). The display (450) can include at least some of the configurations and / or functions of the display module (160) of FIG. 1. At least a part of the display (450) can be flexible, and can also be implemented as a foldable display or a rollable display.

[0066] According to one embodiment, the electronic device (400) may include at least one camera (430) (e.g., the camera module (180) of FIG. 2) on the front and / or rear. According to one embodiment, the camera (430) may include an image sensor (432), an image signal processor (436), and a camera memory (434). The camera (430) may further include at least some of the components and / or functions of the camera module (180) of FIG. 2, such as a lens assembly (210), a flash (220), and an image stabilizer (240).

[0067] According to one embodiment, the image sensor (432) can convert light emitted or reflected from a subject into an electrical signal to obtain an image corresponding to the subject. The image sensor (432) may be configured as a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor, but is not limited thereto. The image sensor (432) may further include at least some of the configurations and / or functions of the image sensor (230) of FIG. 2.

[0068] According to one embodiment, the image signal processor (436) may perform various image processing on image data acquired by the image sensor (432). For example, the image signal processor (436) may perform image processing such as depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening), and may perform control functions of the image sensor (432) such as exposure time control or read-out timing control. The image signal processor (436) may further include at least some of the configurations and / or functions of the image signal processor (260) of FIG. 2.

[0069] In one embodiment, the camera memory (434) may include a storage space independent of the memory (420), or may include a portion of a physical or virtual area of ​​the memory (420). When the camera (430) is activated to sequentially acquire image frames from the image sensor (432), they may be sequentially / parallelly stored in the camera memory (434). The camera memory (434) may further include at least some of the configuration and / or functions of the memory (250) of FIG. 2.

[0070] According to one embodiment, the electronic device (400) may include various types of sensors (440). For example, the electronic device (400) may include at least one of an illumination sensor, a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, or a humidity sensor. According to one embodiment, sensor data acquired from at least some of the sensors (440) may be transmitted to the camera (430) and utilized for the operation of the camera (430). For example, the camera (430) may receive ambient illumination information from the illumination sensor to determine an exposure time, and may receive movement information of the electronic device (400) from the gyro sensor and / or the acceleration sensor to utilize for image processing. The electronic device (400) may further include at least some of the configurations and / or functions of the sensor module (176) of FIG. 1.

[0071] According to one embodiment, the memory (420) may temporarily or permanently store various data, including volatile memory and non-volatile memory. The memory (420) may include at least some of the configuration and / or functions of the memory (130) of FIG. 1, and may store the program (140) of FIG. 1.

[0072] According to one embodiment, the memory (420) may store various instructions that may be performed by the processor (410). Such instructions may include control commands such as arithmetic and logical operations, data movement, and input / output that may be recognized by the processor (410).

[0073] According to one embodiment, the processor (410) may be configured as one or more processors capable of performing calculations or data processing related to control and / or communication of each component of the electronic device (400). The processor (410) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. The processor (410) may be operatively, functionally, and / or electrically connected to at least some of the components of the electronic device (400), such as the display (450), the camera (430), the sensors (440), and the memory (420).

[0074] According to one embodiment, there is no limitation to the computational and data processing functions that the processor (410) can implement on the electronic device (400), but in this document, the operation of the processor (410) related to multi-frame composite photography will be described. The operations of the processor (410) described below can be performed by loading instructions stored in the memory (420). At least some of the operations of the processor (410) described below may also be performed by the image signal processor (436) of the camera (430).

[0075] In this document, the description that a processor (410) can perform a certain operation may be interpreted to mean that instructions (or computer programs) that cause an electronic device (400) (or processor (410), image signal processor (436)) to perform the corresponding operation are stored in a memory (420) (e.g., non-volatile memory, storage) (or camera memory (434)). In addition, the description that a processor (410) can perform a certain operation may be interpreted to mean that at least one processor (410) can perform the corresponding operation.

[0076] According to one embodiment, the processor (410) may execute a camera application based on a user input. The processor (410) may display a screen of the camera application on the display (450), and the screen of the camera application may include a preview image acquired in real time from the camera (430), a shutter button for a shooting input, and at least one menu button for setting shooting options such as brightness, exposure, resolution, quality, and / or auto-focusing of an image to be captured.

[0077] According to one embodiment, when a camera application is executed, the processor (410) may sequentially acquire image frames according to a set frame rate using the image sensor (432) and store the acquired image frames in the memory (420) (or camera memory (434)). For example, a certain area of ​​the memory (420) may be designated as a ring buffer for storing image frames, and the sequentially acquired image frames may be stored in the ring buffer, and when a new image frame is stored and exceeds the capacity of the ring buffer, the image frames may be sequentially deleted from the ring buffer starting from the image frame that was stored most recently.

[0078] According to one embodiment, when the multi-frame synthesis shooting mode is selected in the options menu, the processor (410) can synthesize multiple frames stored in the memory (420) into a single composite frame. The multi-frame synthesis technique has been described with reference to FIGS. 3A and 3B.

[0079] According to one embodiment, the processor (410) can detect a user input (or shutter input) for capturing an image. Here, the user input for capturing an image may be a touch input for a shutter button displayed on the display (450), or may be various forms of input, such as a voice input or a motion input of the electronic device (400).

[0080] According to one embodiment, the processor (410) may set a capture window including a plurality of frames used for generating a composite frame based on the user input. According to one embodiment, the processor (410) may set a predetermined number of image frames as capture windows of the capture window. For example, as described above with reference to FIG. 3A, the processor (410) may set five image frames as the capture window, but the number of image frames constituting the capture window is not limited thereto, and a greater number (e.g., 30) of image frames may be used to configure the capture window. In addition, the number of image frames constituting the capture window may change depending on the user's settings and the shooting environment (e.g., ambient illumination, movement of the camera (430)) during the image shooting. For example, if the image quality priority mode is selected in the setting menu related to multi-frame composite shooting according to user settings, the processor (410) can set the capture window to a relatively larger number of image frames, and if the low delay priority mode is selected, the processor (410) can set the capture window to a relatively smaller number of image frames.

[0081] According to one embodiment, the processor (410) may determine image frames to be used for generating a composite frame based on whether frames of a current capture window and a previous capture window overlap each other, whether frames of the current capture window and a previous capture window are close to each other, whether the previous composite frame has been completed at the time of the current user input, and / or whether to wait until the previous composite frame is completed.

[0082] According to one embodiment, the processor (410) may, in response to a second input for capturing an image, set a second capture window including a plurality of frames used for generating a second composite frame. According to one embodiment, when the number of image frames to be included in the capture window is determined, the processor (410) may determine the number of image frames to be included in the second capture window based on whether frames of the second capture window and a previous first capture window overlap or are adjacent to each other.

[0083] According to one embodiment, the processor (410) can determine whether the image frames of the current second capture window and the immediately preceding first capture window overlap. The processor (410) can determine whether the image frames overlap based on the time between a first input that triggers the generation of the first capture window and the first composite frame and a second input that triggers the generation of the second capture window and the second composite frame. For example, when the number of image frames of the capture window is set to n, and the time between the first input and the second input is less than the time for acquiring n image frames, at least one of the image frames of the second capture window and the first capture window may overlap.

[0084] According to one embodiment, the processor (410) can determine whether the generation of the first composite frame generated by synthesizing frames of the first capture window at the time the second input is received is completed. When the first composite frame is generated in response to the first input, it may take time to synthesize multiple image frames included in the first capture window, and when shutter input is frequent, i.e., when the second input is input within a short time from the first input, the generation of the first composite frame may not be completed before the time when the second input is received. In addition, even when synthesizing the same number of image frames, the time required to generate the composite frame may vary depending on information in the image frames or parameters used during synthesis.

[0085] According to one embodiment, when the generation of the first composite frame is completed at the time the second input is received, the processor (410) may generate the second composite frame based on the frames of the second capture window and the first composite frame. In this case, the number of frames of the second capture window may be less than the number of frames of the first capture window. For example, when the number of image frames of the capture window is set to 5 for generation of the composite frame, and the second input is input after the time when 3 image frames are acquired from the first input, the first capture window and the second capture window may overlap each other by 2 image frames based on 5 image frames. In this case, the processor (410) may include only the remaining 3 image frames excluding the 2 overlapping image frames in the second capture window, and generate the second composite frame by synthesizing 4 image frames including the 3 image frames of the second capture window and the first composite frame. This embodiment will be described in detail with reference to Fig. 8a.

[0086] According to one embodiment, if the generation of the first composite frame is not completed at the time when the second input is received, the processor (410) may use a first partial composite frame generated by synthesizing some of the image frames of the first capture window, or wait until the generation of the first composite frame is completed and then use the first composite frame that has been completely generated to generate the second composite frame. For example, when synthesizing a plurality of image frames, the processor (410) may sequentially synthesize some of the image frames one by one. Accordingly, a first partial composite frame may be generated by synthesizing only some of the image frames in the process of synthesizing the image frames of the first capture window to generate the first composite frame. According to one embodiment, the processor (410) may generate the second composite frame by using the image frames of the second capture window and the first partial composite frame generated at the time when the second input is received. Alternatively, the processor (410) may wait without generating the first partial composite frame and without performing the process of generating the second composite frame until the generation of the first composite frame is completed, and then generate the second composite frame using the image frames of the second capture window and the first composite frame.

[0087] According to one embodiment, if the generation of the first composite frame is not completed at the time the second input is received, the processor (410) may determine whether to use the first partial composite frame or wait until the generation of the first composite frame is completed, based on user settings or the shooting environment. For example, if the image quality priority mode is selected in the setting menu related to multi-frame composite shooting according to user settings, the processor (410) may wait until the generation of the first composite frame is completed and then generate the second composite frame using the image frames of the second capture window and the first composite frame, thereby generating the second composite frame with high image quality. Alternatively, if the low-latency priority mode is selected according to user settings, the processor (410) may generate the second composite frame using the image frames of the second capture window and the first partial composite frame without waiting until the generation of the first composite frame is completed for fast processing. According to one embodiment, if the first partial composite frame has not been generated at the time the second input is received, the processor (410) waits until the generation of the first partial composite frame or the first composite frame is completed, and then synthesizes the generated first partial composite frame or the first composite frame with the image frames of the second capture window to generate the second composite frame. This embodiment will be described in detail with reference to FIGS. 8B and 8C.

[0088] According to one embodiment, the processor (410) may determine whether the image frames of the current second capture window and the immediately preceding first capture window are adjacent to each other if they do not overlap. For example, the processor (410) may determine that the first capture window and the second capture window are adjacent to each other if the number of frames acquired between the first input and the second input is less than or equal to a reference number.

[0089] According to one embodiment, the processor (410) may determine a reference number for determining whether the first capture window and the second capture window are close to each other based on the degree of movement between each image frame. For example, if the movement is large, the subject changes significantly within a short period of time, so when previously acquired frames are synthesized, the change may be large compared to the information on the subject at the current time. Therefore, the processor (410) may set the reference number relatively small when the degree of movement between each image frame is large, and may set the reference number relatively large when the degree of movement is small. According to one embodiment, the processor (410) may recognize the degree of movement between the image frames through changes in pixel values ​​between sequentially acquired image frames, or based on movement information of the electronic device (400) acquired through a sensor (e.g., an acceleration sensor).

[0090] According to one embodiment, when the first capture window and the second capture window are not adjacent, the processor (410) may generate a second composite frame using only the image frames of the second capture window without using the first composite frame in response to the second input.

[0091] According to one embodiment, the processor (410) may generate a second composite frame based on the image frames of the second capture window and the first composite frame when the first capture window and the second capture window are close to each other. In this case, the processor (410) may set the number of image frames of the second capture window to be smaller than that of the first capture window. This embodiment will be described in more detail with reference to FIG. 9.

[0092] According to one embodiment, in a situation where shutter inputs are frequently received from a camera application, both long exposure frames and short exposure frames may be required when synthesizing image frames based on the judgment of the exposure control system (or auto exposure system) of the camera (430). For example, the electronic device (400) may synthesize long exposure frames and short exposure frames to implement HDR (high dynamic range), and in a situation where the current shooting environment has a large dynamic range, strong backlighting, or rapidly changing brightness intensity, the electronic device (400) may synthesize long exposure frames and short exposure frames to generate a composite frame.

[0093] According to one embodiment, when performing long exposure photography in response to a second input, the processor (410) may generate a second composite frame based on a first composite frame generated by synthesizing short exposure frames of a second capture window and frames of a first capture window including at least one long exposure frame. An embodiment of generating a composite frame during long exposure photography will be described in more detail with reference to FIGS. 10A to 10C.

[0094] According to one embodiment, the electronic device (400) may provide a function (e.g., burst shot) to capture multiple images in succession during a press of the shutter button when a long press input is input for the shutter button of the camera application.

[0095] According to one embodiment, when performing multi-frame composite shooting, the electronic device (400) may configure a capture window using frames continuously acquired in response to a long press input, and generate a composite frame of the current capture window using the current capture window image frames and the composite frame of the previous capture window. An embodiment of generating a composite frame during continuous shooting by a long press input will be described in more detail with reference to FIG. 11.

[0096] Instructions for performing the operations of the above-described processor (410) may be stored on a computer-readable recording medium. The recording medium may be tangible and non-transitory. The recording medium may store one or more computer programs including the instructions.

[0097] FIG. 5 illustrates a software and hardware architecture for performing multi-frame synthesis in an electronic device according to various embodiments.

[0098] The operation of the processor (or image signal processor) described above through FIG. 4 can be realized through the architecture illustrated in FIG. 5.

[0099] According to one embodiment, the camera application (CAM application) layer (510) may execute a camera application and perform an operation of outputting the application screen on a display (e.g., display (450) of FIG. 4). The camera application layer (510) may detect a user input for a shutter button displayed on the application screen and provide the input to the framework layer (520).

[0100] According to one embodiment, the framework layer (520) may perform operations for setting a capture window and compositing image frames related to multi-frame composite shooting. According to one embodiment, the operations of the processor (410) (or image signal processor (436)) related to multi-frame composite shooting described through FIG. 4 may be performed by instructions defined on the framework layer (520).

[0101] According to one embodiment, the framework layer (520) may set a capture window in response to user input transmitted from the camera application layer (510) and request the system layer (530) to transmit image frames included in the capture window.

[0102] According to one embodiment, the framework layer (520) may generate a current composite frame (or a second composite frame) based on image frames of the current capture window (or the second capture window) in response to a current shutter input, and the immediately preceding composite frame (or the first composite frame) and / or the immediately preceding partial composite frame (or the first partial composite frame). To this end, the framework layer (520) may temporarily store the immediately preceding composite frame and / or the immediately preceding partial composite frame on a frame buffer for use in generating the current composite frame.

[0103] According to one embodiment, the system layer (530) may perform an operation of transmitting image frames input from the camera module (540) to the framework layer (520). The system layer (530) may be a hardware abstraction layer (HAL) that acts as an interface between the software layer and the hardware layer. The system layer (530) may provide the framework with image frames that constitute a capture window set in the framework among the image frames transmitted from the camera module (540).

[0104] According to one embodiment, the camera module (540) can generate images acquired from the camera's image sensor in frame units and transmit them to the system layer (530).

[0105] FIG. 6 illustrates a method for an electronic device according to one embodiment to generate a current composite frame using frames of a current capture window and a previous composite frame.

[0106] According to one embodiment, an electronic device (e.g., an electronic device (400) of FIG. 4) may set a first capture window (620) including a plurality of image frames used to generate a first composite frame (626) in response to a user's first input (or first shutter input) for capturing an image. The electronic device may generate the first composite frame (626) by synthesizing the image frames of the first capture window (620).

[0107] Referring to FIG. 6, the electronic device can designate five image frames N-7 to N-3 as the first capture window (620) based on a first input input at the time of acquiring image frame N-3. Here, it is assumed that there is no user input for a predetermined period of time prior to the first input or while a predetermined number of image frames are acquired.

[0108] According to one embodiment, the electronic device can generate a first composite frame (626) using image frames of the first capture window (620). It takes time to generate the first composite frame (626) from five image frames, and as shown in FIG. 6, the first composite frame (626) can be synthesized at point N-1.

[0109] According to one embodiment, the electronic device may set a second capture window (610) including a plurality of image frames used for generating a second composite frame (636) in response to a second input (or a second shutter input) of a user for capturing an image after a first input. According to one embodiment, the electronic device may select at least one frame for the second capture window (610) so as not to overlap with the frames of the first capture window (620). For example, in a state where five frames are set to be configured as one capture window, the second capture window (610) input at the time of acquiring image frame N may be determined to be a total of five, from N-4 to N. In this case, since image frames N-4 and N-3 are also included in the first capture window (620), the electronic device may configure the second capture window (610) with three image frames N-2, N-1, and N, excluding image frames N-4 and N-3.

[0110] According to one embodiment, the electronic device may generate a second composite frame (636) based on a third composite frame generated by synthesizing frames of the second capture window (610) (e.g., image frames N-2, N-1, and N) and at least some of the frames of the first capture window (620). Here, when the first composite frame (626) is completed by synthesizing the image frames of the first capture window (620), the third composite frame may be the same as the first composite frame (626). Referring to FIG. 6, the first composite frame (626) may be completed at a time point N-1 before a time point N at which the second input is input. In this case, the electronic device may generate the second composite frame (636) by synthesizing a total of four image frames including three image frames N-2, N-1, and N of the second capture window (610) and the first composite frame (626).

[0111] Even in situations where shutter inputs are frequently and continuously received from a camera application, the electronic device can reduce the number of image frames in the capture window used for the composite frame. For example, if the framework determines that the composite result of the previous multiple frames can be utilized at the current capture window point in time, only a total of four image frames, including three image frames, such as the second capture window (610) of FIG. 6, and the first composite frame (626) generated by the previous shutter input, can be utilized. Referring to FIG. 6, a first composite frame (626) synthesized from image frames N-7 to N-3 of a first capture window (620) is generated at point N-1 before a second input, and a second composite frame (636) is generated using the first composite frame (626) synthesized from five image frames corresponding to the second input and three image frames (N-2, N-1, N) of the second capture window. Therefore, when generating the second composite frame (636), a smaller number (e.g., four) of image frames (the first composite frame (626), N-2, N-1, N) can be used, while still utilizing image information of a larger number of image frames (e.g., five image frames N-7 to N-3 of the first capture window (620) and three image frames N-2 to N of the second capture window (610). Accordingly, the synthesis time can be shortened and the synthesis quality can be maintained by using a smaller number of image frames when generating a composite frame rather than setting the capture window to five image frames.

[0112] FIG. 7 is a flowchart of a multi-frame synthesis method of an electronic device according to one embodiment.

[0113] The illustrated method can be performed by an electronic device (e.g., an electronic device (400) of FIG. 4), and the technical features described above may be omitted below.

[0114] According to one embodiment, in operation 710, the electronic device may initiate multi-frame synthesis photography. Here, the multi-frame synthesis technique is an image processing method that synthesizes multiple sequentially acquired image frames into a single image frame, as described in detail with reference to FIGS. 3A and 3B.

[0115] According to one embodiment, in operation 720, the electronic device can determine whether the current capture window (or the second capture window) and the previous capture window (or the first capture window) overlap each other. For example, when the number of image frames of the capture window is set to n, and the time between the previous shutter input (or the first input) that triggers image capture and the current shutter input (or the second input) is less than the time for acquiring n image frames, at least one of the image frames of the current capture window and the previous capture window may overlap.

[0116] According to one embodiment, when the current capture window and the previous capture window overlap (operation 720 - YES), in operation 725, the electronic device can determine whether the previous composite frame (or the first composite frame) is completed. When the previous composite frame is generated in response to the previous shutter input, it may take time to synthesize multiple image frames included in the previous capture window, and when the shutter input is frequent, i.e., when the current shutter input is input within a short time from the previous shutter input, the generation of the previous composite frame may not be completed before the time point at which the current shutter input is received.

[0117] According to one embodiment, if the previous composite frame is completed (operation 725 - YES), in operation 730, the electronic device can generate a current composite frame using the frames of the current capture window and the previous composite frame. In this case, the electronic device can configure the current capture window by excluding the image frames that overlap in the current capture window and the previous composite frame, and thus the number of image frames in the current capture window can be less than the number of image frames in the previous capture window.

[0118] According to one embodiment, if the previous composite frame is not completed (operation 725 - NO), in operation 735, the electronic device may determine whether to wait until the previous composite frame is completed. For example, if a quality priority mode is selected in a setting menu related to multi-frame composite shooting according to a user setting, the electronic device may determine to wait until the previous composite frame is completed, and if a low-latency priority mode is selected, the electronic device may determine to generate the current composite frame using the previous partial composite frame without waiting.

[0119] According to one embodiment, if it is determined to wait until the previous composite frame is completed (operation 735 - YES), then in operation 740, the electronic device waits from the current shutter input until the completion of the previous composite frame, and then, when the previous composite frame is completed, generates a current composite frame using the frames of the current capture window and the previous composite frame.

[0120] According to one embodiment, if it is determined not to wait until the previous composite frame is completed (operation 735 - NO), then in operation 745, the electronic device can generate the current composite frame using the frames of the current capture window and the previous partial composite frame.

[0121] According to one embodiment, if the result of the check in operation 720 is that the current capture window and the previous capture window do not overlap (operation 720 - NO), in operation 750, the electronic device can check whether the current capture window and the previous capture window are close to each other. For example, the electronic device can determine that the current capture window and the previous capture window are close to each other if the number of image frames acquired between the previous shutter input and the current shutter input is less than or equal to a reference number. The reference number can be increased or decreased based on the degree of movement between the image frames.

[0122] According to one embodiment, if the current capture window and the previous capture window are not adjacent (operation 750 - NO), in operation 755, the electronic device can generate the current composite frame using only the frames of the current capture window, without using the frames of the previous capture window.

[0123] According to one embodiment, if the current capture window and the previous capture window are close to each other (operation 750 - YES), in operation 760, the electronic device may generate a current composite frame using the frames of the current capture window and the previous composite frame. In this case, the electronic device may set the number of image frames of the current capture window to be less than that of the previous composite frame.

[0124] The multi-frame synthesis method of the electronic device described above can be implemented as a computer program including instructions to be performed on the electronic device. The instructions for performing the method described above can be stored on a computer-readable recording medium. The recording medium can be tangible and non-transitory. The recording medium can store one or more computer programs including the instructions.

[0125] FIGS. 8A, 8B and 8C illustrate a method for an electronic device to generate a composite frame when a current capture window and a previous capture window overlap, according to one embodiment.

[0126] According to one embodiment, an electronic device (e.g., the electronic device (400) of FIG. 4) can determine whether a current capture window (or a second capture window) to be set based on a user's current shutter input (or a second input) for image capture overlaps with a previous capture window (or a first capture window) when performing multi-frame composite shooting. The electronic device can determine whether image frames overlap based on a time between a first input that triggers generation of the first capture window and the first composite frame and a second input that triggers generation of the second capture window and the second composite frame.

[0127] Referring to FIGS. 8A to 8C, when five image frames are set as one capture window, five image frames N-7 to N-3 can be set as the first capture window according to the first input at time point N-3. When a second input is input at time point N, when the second capture window is set with five image frames, N-4 to N can be selected, and in this case, image frames N-4 and N-3 can overlap each other in the first capture window and the second capture window.

[0128] According to one embodiment, if the generation of the first composite frame is completed at the time the second input is received, the electronic device can generate the second composite frame based on the frames of the second capture window and the first composite frame. For example, if the generation of the first composite frame has already been completed in a framework (e.g., framework (520) of FIG. 5), the result of the previous capture can be immediately used in the current capture corresponding to the second input.

[0129] Referring to FIG. 8A, a first composite frame (826) synthesized from image frames N-7 to N-3 of the first capture window (820) may be generated before time point N, which is the input time point of the second input. That is, a second input of a user for capturing an image may be input after the generation of the first composite frame (826). In this case, the electronic device may set the second capture window (810) including three image frames N-2, N-1, and N, excluding image frames N-4 and N-3 that overlap with the first capture window (820). The electronic device may generate the second composite frame (816) by synthesizing the three image frames of the second capture window (810) and the first composite frame (826) that has been generated, for a total of four image frames.

[0130] According to one embodiment, if the generation of the first composite frame is not completed at the time when the second input is received, the electronic device may generate the second composite frame by using a first partial composite frame generated by synthesizing some of the image frames of the first capture window, or by waiting until the generation of the first composite frame is completed and then using the first composite frame that has been completed. For example, when synthesizing a plurality of image frames, the electronic device may sequentially synthesize some of the image frames. Accordingly, a first partial composite frame may be generated by synthesizing only some of the image frames in the process of synthesizing the image frames of the first capture window to generate the first composite frame.

[0131] Referring to FIG. 8B, at time N when the second input is received, the generation of the first composite frame (not shown) is not completed, but the first partial composite frame (848) synthesized by synthesizing only some of the five image frames of the first capture window (840) may be completed. For example, the electronic device may select some of the image frames of the first capture window (840) and sequentially synthesize them, and at time N when the second input is received, the first partial composite frame (848) synthesized by synthesizing three image frames (842) (e.g., N-3, N-4, and N-5) may be completed. According to one embodiment, the electronic device may set a second capture window (830) including three image frames N-2, N-1, and N in response to the second input, and may generate a second composite frame (836) by synthesizing the three image frames of the second capture window (830) and the first partial composite frame (848) that has been generated, for a total of four image frames. According to one embodiment, when the low-latency priority mode is selected according to a user setting, the electronic device may generate the second composite frame (836) using the first partial composite frame (848) without waiting until the generation of the first composite frame is completed in order to improve the speed of image synthesis, as shown in FIG. 8B.

[0132] Referring to FIG. 8C, similarly to FIG. 8B, at time N when the second input is received, the generation of the first composite frame (866) is not completed, and the first partial composite frame (868) synthesized by synthesizing three image frames (862) (e.g., N-3, N-4, and N-5) among the image frames of the first capture window (860) may be completed. According to one embodiment, the electronic device may wait until the generation of the first composite frame (866) is completed without performing an operation of generating the second composite frame (856) in response to the second input. When the generation of the first composite frame (866) is completed, the electronic device can set a second capture window (850) including three image frames N-2, N-1, and N, and generate a second composite frame (856) by synthesizing the three image frames of the second capture window (850) and the first composite frame (866) that has been completed, for a total of four image frames.

[0133] According to one embodiment, if the electronic device determines that waiting for the final completed synthesis result by utilizing all frames of the first capture window (860) is helpful for improving image quality compared to improving speed, the electronic device may wait until the generation of the first synthesis frame (866) is completed, as shown in FIG. 8C, and then generate the second synthesis frame (856) using the first synthesis frame (866). For example, if the image quality priority mode is selected according to user settings, the electronic device may wait until the generation of the first synthesis frame (866) is completed, and then generate the second synthesis frame (856) using the first synthesis frame (866) in order to obtain a high-quality image.

[0134] FIG. 9 illustrates a method for an electronic device to generate a composite frame when a current capture window and a previous capture window do not overlap, according to one embodiment.

[0135] According to one embodiment, an electronic device (e.g., the electronic device (400) of FIG. 4) may determine whether the image frames of the current second capture window (910) and the immediately preceding first capture window (920) are close to each other if they do not overlap. According to one embodiment, the electronic device may determine that the first capture window (920) and the second capture window (910) are close to each other if the number of frames acquired between the first input and the second input is less than or equal to a reference number. For example, referring to FIG. 9, when the reference number is set to 10 image frames, the electronic device may determine that the first capture window (920) and the second capture window (910) are close to each other because the number of frames acquired between the first input at time N-6 and the second input at time N is less than the reference number, which is 5.

[0136] According to one embodiment, the electronic device may determine a reference number for determining whether the first capture window (920) and the second capture window (910) are close to each other based on the degree of movement between each image frame. For example, if the movement is large, the subject changes significantly within a short period of time, so when previously acquired frames are synthesized, the change may be large compared to the information on the subject at the current time. Accordingly, if the degree of movement between each image frame is large, the electronic device may set the reference number to a relatively small number, and if the degree of movement is small, the reference number may be set to a relatively large number. According to one embodiment, the electronic device may recognize the degree of movement between the image frames through a change in pixel values ​​between sequentially acquired image frames, or based on movement information of the electronic device acquired through a sensor (e.g., an acceleration sensor).

[0137] According to one embodiment, the electronic device may determine the reference number, which serves as a criterion for determining whether the capture windows (910, 920) are close, based on the number of image frames each of the capture windows (910, 920) includes. For example, when the capture windows (910, 920) are configured with a relatively large number of image frames, the reference number may be set to a relatively large number.

[0138] According to one embodiment, the electronic device may determine the reference number, which serves as a criterion for determining whether the capture windows (910, 920) are close, based on the exposure times of the image frames included in the capture windows (910, 920). For example, if at least one of the capture windows (910, 920) includes a long exposure frame captured with a long exposure time, the reference number may be set to be smaller than if it includes only short exposure frames.

[0139] According to one embodiment, when the first capture window (920) and the second capture window (910) are close to each other, the electronic device may generate a second composite frame (916) based on the image frames of the second capture window (910) and the first composite frame (926). In this case, the processor may set the number of image frames of the second capture window (910) to be less than that of the first capture window (920).

[0140] Referring to FIG. 9, the electronic device can set a second capture window (910) with three image frames N-2, N-1, and N, and generate a second composite frame (916) by synthesizing a total of four image frames including the first composite frame (926).

[0141] FIGS. 10A, 10B and 10C illustrate a method by which an electronic device generates a composite frame during long exposure photography, according to one embodiment.

[0142] In one embodiment, in a situation where shutter inputs are frequently received from a camera application, both long exposure frames and short exposure frames may be required when synthesizing image frames based on the judgment of the camera's exposure control system (or auto exposure system). For example, the electronic device may synthesize long exposure frames and short exposure frames to implement HDR (high dynamic range), and in a situation where the current shooting environment has a large dynamic range, strong backlighting, or rapidly changing brightness intensity, the electronic device may synthesize long exposure frames and short exposure frames to generate a composite frame.

[0143] Referring to FIG. 10A, a first input for capturing an image is received at time N-4, and a long exposure composite frame including a long exposure frame can be generated according to the judgment of the exposure control system. The electronic device can acquire image frame N-4 as a long exposure frame by increasing the exposure time, and acquire four image frames N-3 to N as short exposure frames. The electronic device can configure one long exposure frame N-4 and four short exposure frames N-3 to N as a first capture window (1020), and synthesize them to generate a first composite frame (1026). When a second input for capturing an image is received at time N, in order to avoid overlapping with the image frames of the first capture window (1020), an image frame N+1 constituting the second capture window (1010) can be acquired as a long exposure frame by increasing the exposure time, and four image frames N+2 to N+5 can be acquired as short exposure frames. The electronic device can generate a second composite frame (1016) by synthesizing one long exposure frame N+1 and four short exposure frames N+2 to N+5.

[0144] When generating a long-exposure composite frame in response to a second input as in FIG. 10A, the electronic device may set a second capture window (1010) including five image frames from N+1 after the end of the first capture window (1020) to obtain a long-exposure frame. When generating a second composite frame (1016) by setting the second capture window (1010) in this way, since the current shooting process starts after composing the image frames required for the previous shooting in a situation where shutter input is frequently input from the camera application, a shutter delay may occur due to the difference between the shutter input time and the actual start of the synthesis, and it may also take a long time for the second composite frame (1016) to be completed.

[0145] According to one embodiment, when the electronic device performs long exposure photography in response to a second input, the electronic device can generate a second composite frame based on a first composite frame generated by synthesizing short exposure frames of a second capture window and frames of a first capture window including at least one long exposure frame.

[0146] Referring to FIG. 10B, the electronic device can generate a first composite frame (1046) by synthesizing one long exposure frame N-4 and four short exposure frames N-3 to N at a time point N. When a second input for capturing an image is received at a time point N, the electronic device can configure a second capture window (1030) including short exposure frames N, N+1, and N+2 without acquiring an additional long exposure frame in response to the second input. The electronic device can generate a second composite frame (1036) by synthesizing the first composite frame (1046) obtained by synthesizing three image frames N, N+1, and N+2 of the second capture window (1030) and at least one image frame including the long exposure frame N-4.

[0147] According to one embodiment, the electronic device may generate a second composite frame (1036) based on a first composite frame (1046) generated using frames (N to N+2) of a second capture window (1030) in response to a second input, a long exposure frame N-4 included in a first capture window (1040), and four image frames (e.g., four short exposure frames N-3 to N).

[0148] According to one embodiment, when the generation of the first composite frame (1046) is not completed at the time of the second input, and a first partial composite frame (not shown) is generated by synthesizing only some frames including the long exposure frame N-4 among the image frames of the first capture window (1040), the electronic device may generate the second composite frame (1036) by synthesizing the image frames (N to N+2) of the second capture window (1030) and the first partial composite frame (some frames including the long exposure frame N-4) that has been completed, or wait until the generation of the first composite frame (1046) is completed and then synthesize the first composite frame (1046) and the image frames (N to N+2) of the second capture window (1030) to generate the second composite frame (1036).

[0149] In FIG. 10a, a total of five image frames, including a long exposure frame N+1 and short exposure frames N+2 to N+5, are synthesized, whereas in FIG. 10b, a total of four image frames, including three image frames N, N+1, and N+2 of the second capture window (1030) and the first synthesized frame (1046), are synthesized, so that the number of image frames used for synthesis is reduced, and since a long exposure frame is not acquired in the second capture window (1030), the speed for generating the second synthesized frame (1036) can be relatively fast. In addition, since the long exposure information of the first synthesized frame (1046) can be utilized, image quality improvement can also be provided.

[0150] Referring to FIG. 10c, the electronic device can generate a first composite frame (1066) by synthesizing one long exposure frame N-4 and four short exposure frames N-3 to N of the first capture window (1060). When a second input for capturing an image is received at a time point N, the electronic device can configure a second capture window (1050) including a long exposure frame N+1 and a short exposure frame N+2 in response to the second input. The electronic device can generate a second composite frame (1056) by synthesizing two image frames of the second capture window (1050) and the first composite frame (1066). In the case of configuring as in Fig. 10c, the number of image frames of the second capture window (1050) can be reduced to two, so the speed for generating the second composite frame (1056) can be relatively fast, and since the long exposure information of the first composite frame (1066) and the long exposure frame N+1 can be utilized, image quality improvement can also be provided.

[0151] FIG. 11 illustrates a method for an electronic device to generate composite frames during continuous shooting by a long press input, according to one embodiment.

[0152] According to one embodiment, an electronic device (e.g., electronic device (400) of FIG. 4) may provide a function to capture multiple images in succession during a press of a shutter button when a long press input is input for a shutter button of a camera application (e.g., burst shot).

[0153] According to one embodiment, when performing multi-frame composite shooting, the electronic device may configure a capture window using at least one frame continuously acquired in response to a long press input, and generate a composite frame of the current capture window using the image frames of the current capture window and the composite frame of the previous capture window. In this case, the electronic device may set the number of image frames constituting the capture window to be relatively small.

[0154] According to one embodiment, if the previous composite image of the previous capture window is completed at the time of acquisition of the last image frame of the current capture window, the image frames of the current capture window and the previous composite image can be synthesized to generate the current composite image.

[0155] Referring to Fig. 11, a long press input for a shutter button can be input from time points N to N+9. In response to the long press input, the electronic device can configure image frames N and N+1 as a first capture window (1110), image frames N+1 and N+2 as a second capture window (1120), image frames N+2 and N+3 as a third capture window (1130), image frames N+3 and N+4 as a fourth capture window (1140), image frames N+4 and N+5 as a fifth capture window (1150), image frame N+7 as a sixth capture window (1160), and image frames N+8 and N+9 as a seventh capture window (1170).

[0156] According to one embodiment, the electronic device can generate a first composite frame using image frames N and N+1 of the first capture window (1110). When generation of the first composite frame (1126) is completed at time N+2, the electronic device can generate a second composite frame (1136) using image frames N+1 and N+2 of the second capture window (1120) and the first composite frame (1126). When generation of the second composite frame (1136) is completed at time N+3, the electronic device can generate a third composite frame (1146) using image frames N+2 and N+3 of the third capture window (1130) and the second composite frame (1136). If the generation of the third composite frame (1146) is completed at time N+4, the electronic device can generate the fourth composite frame (1148) using image frames N+3 and N+4 of the fourth capture window (1140) and the third composite frame (1146). If the generation of the fourth composite frame (1148) is not completed at time N+5, the electronic device can generate the fifth composite frame (1156) using only image frames N+4 and N+5 of the fifth capture window (1150) without using the fourth composite frame (1148).

[0157] According to one embodiment, the electronic device can generate a sixth composite frame (1166) using image frames N+8 and N+9 of the seventh capture window (1170). The electronic device can generate a seventh composite frame (1176) by synthesizing image frame N+7 of the sixth capture window (1160), which is composed of one image frame, and the sixth composite frame (1166).

[0158] In this way, since information from previous composite frames can be used even in short shooting intervals, it can help improve image quality when generating composite frames.

[0159] FIG. 12 is a flowchart of a multi-frame synthesis method of an electronic device according to one embodiment.

[0160] The illustrated method can be performed by an electronic device (e.g., an electronic device (400) of FIG. 4), and the technical features described above may be omitted below.

[0161] According to one embodiment, in operation 1210, the electronic device may detect a first input for capturing an image. Here, the user input for capturing an image may be a touch input for a shutter button displayed on a display, or may be various forms of input, such as a voice input or a motion input of the electronic device.

[0162] According to one embodiment, in operation 1220, the electronic device may set a first capture window including a plurality of frames used to generate a first composite frame. According to one embodiment, the electronic device may set a predetermined number of image frames as the capture window of the capture window, and the predetermined number may change depending on the shooting environment when capturing the image.

[0163] In one embodiment, at operation 1230, the electronic device may detect a second input for capturing an image. The second input may be detected after the first input and may be an input of the same type as the first input (e.g., a touch input on a shutter button).

[0164] According to one embodiment, in operation 1240, the electronic device may set a second capture window including a plurality of frames used to generate a second composite frame. According to one embodiment, the electronic device may configure the second capture window with a smaller number of image frames than the number of image frames of the first capture window. For example, if the image frames of the first capture window and the second capture window partially overlap, the second capture window may be set excluding some of the overlapping image frames.

[0165] According to one embodiment, in operation 1250, the electronic device may generate a second composite frame based on a third composite frame generated by synthesizing at least some of the frames of the second capture window and the frames of the first capture window. Here, the third composite frame may be a first composite frame generated by synthesizing the frames of the first capture window, or a first partial composite frame generated by synthesizing some of the frames of the first capture window.

[0166] According to one embodiment, the electronic device can determine whether the generation of the first composite frame is completed at the time the second input is received, if some of the image frames of the current second capture window and the immediately preceding first capture window overlap. If the generation of the first composite frame is completed, the electronic device can generate the second composite frame based on the frames of the second capture window and the first composite frame. If the generation of the first composite frame is not completed, the electronic device can generate the second composite frame by using a first partial composite frame generated by synthesizing some of the image frames of the first capture window, or by waiting until the generation of the first composite frame is completed and then using the first composite frame that has been completely generated.

[0167] The multi-frame synthesis method of the electronic device described above can be implemented as a computer program including instructions to be performed on the electronic device. The instructions for performing the method described above can be stored on a computer-readable recording medium. The recording medium can be tangible and non-transitory. The recording medium can store one or more computer programs including the instructions.

[0168] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0169] An electronic device (400) according to various embodiments of the present document may include a camera (430), a memory (420), and at least one processor (410) operatively connected to the camera and the memory.

[0170] According to one embodiment, the memory may include instructions that are executable by the at least one processor and, when executed, cause the electronic device to, in response to a first input for capturing an image, set a first capture window (620) including a plurality of frames used for generating a first composite frame (626), in response to a second input detected after the first input, set a second capture window (610) including a plurality of frames used for generating a second composite frame (636), and generate the second composite frame based on the frames of the second capture window and a third composite frame (848 or 866) generated by synthesizing at least some of the frames of the first capture window.

[0171] According to one embodiment, the third composite frame may be the first composite frame generated by synthesizing frames of the first capture window, or may be a first partial composite frame generated by synthesizing some of the frames of the first capture window.

[0172] According to one embodiment, the instructions stored in the memory may cause the electronic device to generate the second composite frame based on the frames of the second capture window and the first composite frame when the generation of the first composite frame is completed at the time the second input is received.

[0173] According to one embodiment, the instructions stored in the memory may cause the electronic device to delay generation of the second composite frame until generation of the first composite frame is completed, if generation of the first composite frame is not completed at the time the second input is received, and to generate the second composite frame based on the frames of the second capture window and the first composite frame in response to completion of generation of the first composite frame.

[0174] According to one embodiment, the instructions stored in the memory may cause the electronic device to generate the second composite frame based on the frames of the second capture window and the first partial composite frame, if the generation of the first composite frame is not completed at the time the second input is received.

[0175] According to one embodiment, the instructions stored in the memory may cause the electronic device to delay generation of the second composite frame until generation of the first composite frame is completed to improve image quality of the second composite frame, based on user setting information, or to determine whether to generate the second composite frame based on frames of the second capture window and the first partial composite frame to increase the speed of generation of the second composite frame.

[0176] According to one embodiment, the instructions stored in the memory may cause the electronic device to generate the second composite frame based on the frames of the second capture window and the third composite frame when the number of frames acquired between the first input and the second input is less than or equal to a reference number, and to generate the second composite frame based on the frames of the second capture window when the number of frames acquired between the first input and the second input is greater than or equal to a reference number.

[0177] According to one embodiment, the instructions stored in the memory may cause the electronic device, when performing long exposure photography in response to the second input, to generate the second composite frame based on the first composite frame generated by synthesizing short exposure frames of the second capture window and frames of the first capture window including at least one long exposure frame.

[0178] According to one embodiment, the number of frames of the second capture window may be less than the number of frames of the first capture window.

[0179] According to one embodiment, the number of frames used to generate the second composite frame may be less than the number of frames used to generate the first composite frame.

[0180] According to one embodiment, the camera stores sequentially acquired frames in the memory, and the first capture window may include a frame acquired at the time when the first input occurs and at least one frame acquired sequentially before the frame, or may include a frame acquired at the time when the first input occurs and at least one frame acquired sequentially after the frame.

[0181] A method performed by an electronic device according to various embodiments of the present document may include, in response to a first input for capturing an image, setting a first capture window including a plurality of frames used for generating a first composite frame; in response to a second input detected after the first input, setting a second capture window including a plurality of frames used for generating a second composite frame; and generating the second composite frame based on the frames of the second capture window and a third composite frame generated by synthesizing at least some of the frames of the first capture window.

[0182] According to one embodiment, the third composite frame may be the first composite frame generated by synthesizing frames of the first capture window, or may be a first partial composite frame generated by synthesizing some of the frames of the first capture window.

[0183] According to one embodiment, the operation of generating the second composite frame may include an operation of generating the second composite frame based on the frames of the second capture window and the first composite frame when the generation of the first composite frame is completed at the time the second input is received.

[0184] According to one embodiment, the operation of generating the second composite frame may include, if the generation of the first composite frame is not completed at the time the second input is received, delaying the generation of the second composite frame until the generation of the first composite frame is completed, and, in response to the completion of the generation of the first composite frame, generating the second composite frame based on the frames of the second capture window and the first composite frame.

[0185] According to one embodiment, the operation of generating the second composite frame may include an operation of generating the second composite frame based on the frames of the second capture window and the first partial composite frame, if the generation of the first composite frame is not completed at the time the second input is received.

[0186] According to one embodiment, the operation of generating the second composite frame may include an operation of generating the second composite frame based on the frames of the second capture window and the third composite frame when the number of frames acquired between the first input and the second input is less than or equal to a reference number, or an operation of generating the second composite frame based on the frames of the second capture window when the number of frames acquired between the first input and the second input is greater than or equal to a reference number.

[0187] According to one embodiment, the operation of generating the second composite frame may include an operation of generating the second composite frame based on the first composite frame generated by synthesizing short exposure frames of the second capture window and frames of the first capture window including at least one long exposure frame when performing long exposure photography in response to the second input.

[0188] According to one embodiment, the number of frames of the second capture window may be less than the number of frames of the first capture window, and the number of frames used to generate the second composite frame may be less than the number of frames used to generate the first composite frame.

[0189] A computer-readable, non-transitory recording medium according to various embodiments of the present document may store instructions that, when executed by an electronic device, cause the electronic device to perform an operation of setting a first capture window including a plurality of frames used for generating a first composite frame in response to a first input for capturing an image, an operation of setting a second capture window including a plurality of frames used for generating a second composite frame in response to a second input detected after the first input, and an operation of generating the second composite frame based on the frames of the second capture window and a third composite frame generated by synthesizing at least some of the frames of the first capture window.

[0190] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0191] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0192] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0193] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0194] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0195] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0196] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0197] As used herein, the term "if" will be understood to mean "when, upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, "if it is determined to," or "if [the stated condition or event] is detected," will optionally be understood to mean "upon determining," or "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]."

[0198] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. A processing device (or processing circuit) may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0199] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0200] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0201] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0202] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In an electronic device (400), Camera (430); memory (420); and At least one processor (410) operatively connected to the camera and the memory, The above memory is executable by the at least one processor, and when executed, the electronic device: In response to a first input for image capture, a first capture window (620) is set up including a plurality of frames used for generating a first composite frame (626), In response to a second input detected after the first input, a second capture window (610) is set up including a plurality of frames used for generating a second composite frame (636), and An electronic device including instructions for generating a second composite frame based on a third composite frame (848 or 866) generated by synthesizing at least some of the frames of the second capture window and the frames of the first capture window.

2. In paragraph 1, The above third synthetic frame is, An electronic device, wherein the first composite frame is generated by synthesizing frames of the first capture window, or the first partial composite frame is generated by synthesizing some of the frames of the first capture window.

3. In paragraph 2, The instructions stored in the above memory are, An electronic device that generates the second composite frame based on the frames of the second capture window and the first composite frame when the generation of the first composite frame is completed at the time the second input is received.

4. In paragraph 2, The instructions stored in the above memory are, If the generation of the first composite frame is not completed at the time the second input is received, An electronic device that delays generation of the second composite frame until generation of the first composite frame is completed, and in response to completion of generation of the first composite frame, generates the second composite frame based on the frames of the second capture window and the first composite frame.

5. In paragraph 2, The instructions stored in the above memory are, If the generation of the first composite frame is not completed at the time the second input is received, An electronic device that generates the second composite frame based on the frames of the second capture window and the first partial composite frame.

6. In paragraph 1, The instructions stored in the above memory are, If the number of frames acquired between the first input and the second input is less than or equal to a reference number, the second composite frame is generated based on the frames of the second capture window and the third composite frame, and An electronic device that generates the second composite frame based on the frames of the second capture window when the number of frames acquired between the first input and the second input exceeds a reference number.

7. In paragraph 1, The instructions stored in the above memory are, An electronic device that, when performing long exposure photography in response to the second input, generates the second composite frame based on the first composite frame generated by synthesizing short exposure frames of the second capture window and frames of the first capture window including at least one long exposure frame.

8. In any one of paragraphs 1 to 7, An electronic device wherein the number of frames of the second capture window is less than the number of frames of the first capture window.

9. In any one of paragraphs 1 to 8, The above camera stores sequentially acquired frames in the above memory, An electronic device wherein the first capture window includes a frame acquired at the time when the first input occurs and at least one frame acquired sequentially before the frame, or includes a frame acquired at the time when the first input occurs and at least one frame acquired sequentially after the frame.

10. In a method performed by an electronic device, In response to a first input for capturing an image, an operation of setting a first capture window including a plurality of frames used for generating a first composite frame; An operation of setting a second capture window including a plurality of frames used for generating a second composite frame in response to a second input detected after the first input; and A method comprising an operation of generating the second composite frame based on a third composite frame generated by synthesizing frames of the second capture window and at least some of the frames of the first capture window.

11. In paragraph 10, The above third synthetic frame is, A method wherein the first composite frame is generated by synthesizing frames of the first capture window, or the first partial composite frame is generated by synthesizing some of the frames of the first capture window.

12. In paragraph 11, The operation of generating the above second synthetic frame is: When the generation of the first composite frame is completed at the time the second input is received, an operation of generating the second composite frame based on the frames of the second capture window and the first composite frame; and A method comprising: delaying the generation of the second composite frame until the generation of the first composite frame is completed if the generation of the first composite frame is not completed at the time the second input is received; and generating the second composite frame based on the frames of the second capture window and the first composite frame in response to the completion of the generation of the first composite frame.

13. In paragraph 11, The operation of generating the above second synthetic frame is: If the generation of the first composite frame is not completed at the time the second input is received, A method comprising an operation of generating the second composite frame based on the frames of the second capture window and the first partial composite frame.

14. In paragraph 10, The operation of generating the above second synthetic frame is: If the number of frames acquired between the first input and the second input is less than or equal to a reference number, an operation of generating the second composite frame based on the frames of the second capture window and the third composite frame, or A method including an operation of generating the second composite frame based on the frames of the second capture window when the number of frames acquired between the first input and the second input exceeds a reference number.

15. When executed by an electronic device, said electronic device: In response to a first input for capturing an image, an operation of setting a first capture window including a plurality of frames used for generating a first composite frame; An operation of setting a second capture window including a plurality of frames used for generating a second composite frame in response to a second input detected after the first input; and A computer-readable non-transitory recording medium storing instructions that cause an operation of generating a second composite frame based on a third composite frame generated by synthesizing frames of the second capture window and at least some of the frames of the first capture window.

Images