Electronic device for obtaining multi-exposure image and method for operating same

Abstract

An electronic device according to one embodiment comprises: a camera module including an image sensor; a memory in which images are stored; and at least one processor connected to the camera module and the memory. The at least one processor may be configured to perform an operation of acquiring a first image on the basis of image data outputted from the image sensor, an operation of acquiring a second image stored in the memory, an operation of generating a third image on the basis of the second image on the basis of an aspect ratio of the second image, and an operation of generating a fourth image by synthesizing the first image and the third image.

Description

Electronic device and operating method for acquiring multiple exposure images

[0001] Various embodiments of the present disclosure relate to an electronic device for acquiring a multiple exposure image and a method of operating the same.

[0002] When taking a photograph by exposing a film multiple times, a multiple exposure photograph can be obtained in which multiple images are superimposed. Electronic devices including digital cameras can acquire multiple images through an image sensor and synthesize them to obtain a multiple exposure image. In this case, the multiple exposure image can be generated based on a user-set synthesis mode and the number of images used for synthesis. The electronic device can store an image file containing the acquired multiple exposure image on a storage medium.

[0003] During the process of creating a multiple exposure image, the user can decide whether to use previously stored images for compositing. If not, the shooting settings (ISO, white balance, exposure time, etc.) are set and the shooting begins. The images acquired from the image sensor during the shooting process are then passed on to the compositing process and can be used to create the multiple exposure image.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] Various embodiments of the present disclosure may have as a first object to provide an electronic device and an operating method thereof for obtaining a multiple exposure image.

[0006] An electronic device according to one embodiment may include a camera module including an image sensor, a memory storing images, and at least one processor connected to the camera module and the memory. The at least one processor may be configured to perform an operation of acquiring a first image based on image data output from the image sensor. The at least one processor may be configured to perform an operation of acquiring a second image stored in the memory. The at least one processor may be configured to perform an operation of generating a third image capable of multiple exposure synthesis with the first image based on the second image, based on an aspect ratio of the second image. The at least one processor may be configured to perform an operation of synthesizing the first image and the third image to generate a fourth image having a second aspect ratio.

[0007] A method of operating an electronic device including a camera module including an image sensor according to one embodiment may include an operation of acquiring a first image based on image data output from the image sensor. The method may include an operation of acquiring a second image stored in a memory included in the electronic device. The method may include an operation of generating a third image capable of multiple exposure synthesis with the first image based on the second image, based on an aspect ratio of the second image. The method may include an operation of generating a fourth image having a second aspect ratio by synthesizing the first image and the third image.

[0008] A computer-readable recording medium having recorded thereon a program according to one embodiment of the present invention can, when executed, cause an electronic device to perform an operation of acquiring a first image based on image data output from an image sensor. In addition, the recording medium can, when executed, cause an electronic device to perform an operation of acquiring a second image stored in a memory. The recording medium can, when executed, cause an electronic device to perform an operation of generating a third image capable of multiple exposure synthesis with a first image based on the second image, based on an aspect ratio of the second image. The recording medium can, when executed, cause an electronic device to perform an operation of synthesizing the first image and the third image to generate a fourth image having a second aspect ratio.

[0009] According to one embodiment of the present disclosure, a multiple exposure image can be obtained by utilizing a captured image or a stored image.

[0010] Additionally, according to one embodiment of the present disclosure, the operation for obtaining a multiple exposure image can be made intuitive and simplified, thereby improving user convenience.

[0011] In addition, according to one embodiment of the present disclosure, synthesis can be applied to previously stored images having different formats from the synthesized image, thereby increasing the versatility of multiple exposure images.

[0012] The effects according to the embodiments of the present disclosure are not limited to the above-described contents, and various effects can be derived from the contents described below.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0014] FIG. 2 is a block diagram illustrating a camera module according to one embodiment.

[0015] Figure 3 is a block diagram of an electronic device according to one embodiment.

[0016] FIG. 4 is a block diagram illustrating an example of obtaining a basic image and a post-processed image from image sensor data in an electronic device according to one embodiment.

[0017] FIG. 5 is a block diagram illustrating elements included in an image file stored by an electronic device according to one embodiment.

[0018] Figure 6 illustrates a method for obtaining a multiple exposure image according to one embodiment.

[0019] FIG. 7 is a block diagram of an electronic device performing a method for acquiring multiple exposure images according to one embodiment.

[0020] FIG. 8 is a block diagram of an electronic device performing a method for acquiring multiple exposure images according to one embodiment.

[0021] FIG. 9 is a block diagram of an electronic device performing a method for acquiring multiple exposure images according to one embodiment.

[0022] FIG. 10 is a block diagram of an electronic device performing a method for acquiring multiple exposure images according to one embodiment.

[0023] FIG. 11 illustrates an image transformation method for obtaining a multiple exposure image according to one embodiment.

[0024] Figure 12 illustrates an image cropping operation according to one embodiment.

[0025] FIG. 13 illustrates an image resizing operation and an image positioning operation according to one embodiment.

[0026] Figure 14 shows an example of region segmentation of a multiple exposure image according to one embodiment.

[0027] FIG. 15 illustrates a flowchart for determining values ​​of pixels corresponding to x and y coordinates of a synthetic image according to one embodiment.

[0028] Figure 16 illustrates an example of region-based image synthesis according to one embodiment.

[0029] Figure 17 illustrates an example of region-based image synthesis according to one embodiment.

[0030] FIG. 18a illustrates a UX / UI of a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0031] FIG. 18b illustrates a UX / UI of a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0032] FIG. 18c illustrates a UX / UI of a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0033] FIG. 18d illustrates a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0034] FIG. 19 is a flowchart illustrating a method for obtaining a multiple exposure image according to one embodiment.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0036] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0037] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0038] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0042] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0043] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0044] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0048] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0050] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0053] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0054] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0055] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0056] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0058] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0059] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0060] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0061] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0062] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0063] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to one embodiment. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0064] The flash (220) can emit light used to enhance light emitted or reflected from a subject. In one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. In one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as, for example, an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0065] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0066] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). In one embodiment, the image signal processor (260) may be at least a part of the processor (120). It may be configured as a separate processor that is configured or operates independently from the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0067] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0068] FIG. 3 is a block diagram of an electronic device (101) according to one embodiment.

[0069] An electronic device (101) according to one embodiment may include a display (310) (e.g., the display module (160) of FIG. 1), a camera module (320) (e.g., the camera module (180) of FIG. 2), a memory (330) (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2), and at least one processor (340) (e.g., the processor (120) of FIG. 1). The camera module (320) according to one embodiment may include an image sensor (230) and an image signal processor (ISP) (260). FIG. 3 is provided to explain one embodiment, and some components illustrated in FIG. 3 may be replaced with other components or omitted.

[0070] When using a previously saved image file for multiple exposure synthesis, a list of saved image files is displayed, and any one of the image files can be selected by the user. The selected image file can be used to create a multiple exposure image, and the generated multiple exposure image can be saved depending on whether the multiple exposure termination condition is satisfied. Furthermore, the aforementioned processes can be repeatedly performed depending on whether the multiple exposure mode is terminated.

[0071] When an electronic device provides a list of stored image files, if the format does not match the format of the image currently used for multiple exposure synthesis, the images may be filtered out and processed so that the user cannot select them. Here, the image format for multiple exposure synthesis can be defined in various ways. For example, sensor type, resolution, color space, bits used per pixel, and lens information may be applied. The electronic device can use only some of the previously stored images for multiple exposure synthesis by filtering out images with formats that are difficult to synthesize for multiple exposure synthesis. Therefore, for images acquired in various environments (cloud, other cameras and sensors, etc.), problems may arise in utilizing them for creating multiple exposure images. For example, if the resolution of the shooting environment is 4000x3000, and only images with a resolution of 4000x3000 among the stored images can be synthesized, the number of files that can be used for multiple exposure synthesis among the stored image files may be significantly reduced.

[0072] Embodiments of the present disclosure propose a method and device capable of effectively generating multiple exposure images for the above problems.

[0073] In one embodiment, at least one processor (340) may execute a camera application for capturing images. At least one processor (340) may execute a multi-capture mode that synthesizes images captured by the camera application with other images and provides the images. For example, at least one processor (340) may execute the multi-capture mode in response to a user input selecting a multi-capture mode menu included in a user interface provided by the camera application.

[0074] In one embodiment, at least one processor (340) may obtain a base image and a post-processed image. In this document, a 'base image (e.g., raw image)' may refer to image data before image processing is performed to display brightness and color recognizable by the human eye through image processing. For example, the base image may include image data acquired by the image signal processor (260) during or as a result of a preprocessing process and a demosaicing process performed on image sensor data output from the image sensor (230). The base image may be configured in a raw image file format. For example, the base image may be configured according to a raw image in the DNG (digital negative) format defined by Adobe™. A 'post-processed image (e.g., processed image)' may refer to image data generated by performing image processing on the base image to display brightness and color recognizable by the human eye, and may be configured in a post-processed image format (e.g., JPG, PNG, etc.).

[0075] It is difficult to determine whether the brightness and color of a base image are identical or similar to what the human eye perceives based solely on the information contained in the base image. Therefore, the electronic device (101) can perform additional operations using the base image and metadata (e.g., raw image metadata) related to the base image to render the base image. For example, the electronic device (101) can perform demosaicing or white balancing on the base image based on the metadata.

[0076] In one embodiment, the image signal processor (260) may generate metadata including image sensor data transmitted from the image sensor (230) or information acquired in the process of processing the image sensor data. For example, the metadata may include white balancing parameter values ​​corresponding to a color temperature for performing white balancing on a base image acquired based on the image sensor data. For example, if the first image sensor data relates to an image captured around a red light source, the image signal processor (260) may acquire first metadata including a value for performing white balancing based on a color temperature associated with the red light source on a first base image acquired based on the first image sensor data.

[0077] In one embodiment, the method of acquiring a base image may be implemented in various ways. For example, at least one processor (340) may receive a user input for selecting a capture button of a user interface displayed through the display (310) while the multiple exposure shooting mode is running. Based on the received user input, the image signal processor (260) may generate a first base image and first metadata from the first image sensor data output from the image sensor (230). As another example, at least one processor (340) may display an image list including the first base image stored in the memory (330) through the display (310). The stored first base image may be stored in the memory (330) in a raw image file format. An image file stored in the raw image file format may include the base image and metadata. At least one processor (340) may acquire the first base image and first metadata from the memory (330) based on a user input for selecting the first base image displayed through the display (310). However, the present invention is not limited thereto. The first basic image may be generated based on first image sensor data output from an image sensor (230) included in the electronic device (101), or may be generated based on image sensor data output from an image sensor (not shown) included in an external electronic device (not shown). As another example, at least one processor (340) may display an image list including post-processing images stored in the memory (330) through the display (310). At least one processor (340) may obtain the first basic image based on a user input selecting a stored post-processing image displayed through the display (310).

[0078] In one embodiment, at least one processor (340) may display an image including a selected first base image and a user interface for modifying the same through a display (310), and may obtain information input by a user to modify the image displayed on the display (310). In addition, the processor (340) may synthesize the first base image with another image based on the image modification information.

[0079] In one embodiment, at least one processor (340) may obtain a second base image and second metadata corresponding to the second base image. The second base image and the second metadata may be obtained based on second image sensor data output from the image sensor (230) at a different time from the time at which the first image sensor data is output from the image sensor (230).

[0080] In one embodiment, at least one processor (340) may obtain reference metadata based on at least a portion of the first metadata or the second metadata. For example, at least one processor (340) may determine the earliest obtained base image among the base images to be synthesized (e.g., the first base image and the second base image) as the reference image. At least one processor (340) may determine metadata corresponding to the reference image as the reference metadata. For another example, at least one processor (340) may determine metadata corresponding to any base image among the base images to be synthesized (e.g., the first base image and the second base image) as the reference metadata. For another example, at least one processor (340) may determine reference metadata including a value (e.g., an average value) obtained by calculating a parameter included in the first metadata and a parameter included in the second metadata.

[0081] In one embodiment, at least one processor (340) can synthesize a first base image and a second base image based on parameters corresponding to reference metadata. At least one processor (340) can synthesize the first base image and the second base image to obtain a third base image. If no additional images are to be synthesized with the third base image, the at least one processor (340) can store an image file generated based on the third base image and the reference metadata. For example, the image file generated based on the third base image and the reference metadata can be stored in the memory (330). In one embodiment, the at least one processor (340) can display a screen including a preview image through the display (310). The preview image can be displayed by overlaying an image streamed from the image sensor (230) on a recently acquired image (a single image or a synthesized image). For example, when the multi-exposure shooting mode is running while the first basic image is acquired through the display (310), at least one processor (340) can overlay and output an image corresponding to the first basic image with an image in which the image frames acquired through the image sensor (230) are continuously updated. When the multi-exposure shooting mode is running after the first basic image and the second basic image are acquired and a third basic image is synthesized, at least one processor (340) can overlay and output an image corresponding to the third basic image with an image in which the image frames acquired through the image sensor (230) are continuously updated.

[0082] In one embodiment, at least one processor (340) may further obtain a post-processing image used to provide a preview image. The post-processing image may refer to an image obtained by performing image processing on the base image by the image signal processor (260) or at least one processor (340). For example, the post-processing image may include image data obtained by performing tone mapping, noise removal, color correction, or gamma correction on the base image. The post-processing image may be configured in a format (e.g., JPG (Joint Photographic Experts Group) format) that includes information having luminance and color that are the same or similar to those perceived by the human eye. The post-processing image may have a lower bit depth than the base image. The at least one processor (340) may synthesize a plurality of post-processing images. For example, the at least one processor (340) may synthesize a first post-processing image corresponding to a first base image and a second post-processing image corresponding to a second base image, thereby generating a third post-processing image corresponding to a third base image obtained by synthesizing the first base image and the second base image. At least one processor (340) may configure the image file to include a final acquired post-processed image when generating the image file. The post-processed image included in the image file may be used as a thumbnail image or cover image for the image file.

[0083] FIG. 4 is a block diagram illustrating an example of an electronic device (e.g., the electronic device (101) of FIG. 1 and FIG. 3) obtaining a basic image and a post-processed image from image sensor data according to one embodiment.

[0084] In one embodiment, an image signal processor (e.g., an image signal processor (260) of FIG. 2) of an electronic device (e.g., an electronic device (101) of FIGS. 1 and 3) may process image sensor data (400) via a Bayer / raw processing engine (410) and an image processing engine (420).

[0085] The Bayer / raw processing engine (410) may include a Bayer preprocessing module (411) and a Bayer demosaic module (412). Image sensor data (400) may be primarily processed in the Bayer preprocessing module (411). The Bayer demosaic module (412) may convert data (Bayer image sensor data) primarily processed in the Bayer preprocessing module (411) into RGB format. A base image (430) may be acquired from the processing process or result of the Bayer / raw processing engine (410). Metadata may be acquired based on information about the base image (430) acquired from the image sensor data (400) or the process of processing the image sensor data (400).

[0086] In one embodiment, the result of the Bayer / raw processing engine (410) processing the image sensor data (400) may be transmitted to the image processing engine (420). The image processing engine (420) may include, for example, at least one of a tone mapping module (421), a noise reduction module (422), a color correction module (423), a gamma correction module (424), a spatial filter module (425), or a grain add module (426). As a result of the image processing performed by the image processing engine (420), a post-processed image (440) having brightness and color that are the same as or similar to those perceived by the eye may be obtained.

[0087] The configuration of the Bayer / raw processing engine (410) and the image processing engine (420) illustrated in FIG. 4 is for illustrative purposes only and is not limited thereto.

[0088] FIG. 5 is a block diagram illustrating elements included in an image file (500) stored by an electronic device (e.g., the electronic device (101) of FIG. 1 and FIG. 3) according to one embodiment.

[0089] In one embodiment, an image file (500) (e.g., a raw image file) may include a base image (510) and metadata (520). The image file (500) may further include a post-processed image (530) corresponding to the base image (510). The post-processed image (530) may be omitted.

[0090] Metadata (520) may include information that can be used to render an image contained in an image file. The metadata may include information acquired from image sensor data output from an image sensor or acquired during the process of processing the image sensor data. It is difficult to determine brightness and color discernible by the human eye using only the image data of an image file stored in a raw format. Therefore, an electronic device may perform additional operations (e.g., demosaicing or white balance) based on the metadata.

[0091] The image file may be, for example, in the DNG (digital negative) format defined by Adobe™. In this case, the base image (510) may be a raw image (NewSubFileType value is 0) defined in DNG. The metadata (520) may be data that provides information about the raw image defined in DNG. The post-processing image (530) may be a rendered preview (NewSubFileType value is 1) defined in DNG. The post-processing image (530) may be provided in the form of a thumbnail image for easily browsing the image stored in the image file (500). Alternatively, if there is no software for checking the base image (510), it may be provided in the form of a cover image of the base image (510). An electronic device (e.g., the electronic device (101) of FIGS. 1 and 3) may render the base image (510) by executing dedicated software using the base image (510) and metadata (520) in the image file (500).

[0092] A multiple exposure image can be obtained by synthesizing images captured using an image sensor at different times. Since the images used to obtain a multiple exposure image can be generated at different times, the metadata corresponding to each image can be different. For example, if the color temperature of the ambient light source of an image captured at a first time point is different from the color temperature of the ambient light source of an image captured at a second time point, different white balancing-related parameters may be required. However, since the parameters stored in the metadata correspond to only one condition, the result of applying white balancing to the composite image may differ from the color of the required image.

[0093] To acquire a multiple exposure image, images can be acquired from images previously stored in memory. For example, a base image or a post-processed image contained in an image file stored in memory can be used to acquire a multiple exposure image. However, if the format required to acquire a multiple exposure image does not match the format of the image, the image may be filtered before being displayed to the user, making it unselectable. This filtering may reduce the number of images available for acquiring a multiple exposure image.

[0094] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0095] FIG. 6 illustrates a method for acquiring a multiple exposure image in an electronic device (e.g., the electronic device (101) of FIG. 1 or 3) according to one embodiment. In various embodiments of the present disclosure, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1 or 3) may be understood as being performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, or at least one processor (340) of FIG. 3) of the electronic device (e.g., the electronic device (101) of FIG. 1 or 3)) performing a calculation or controlling another component of the electronic device (e.g., the electronic device (101) of FIG. 1 or 3).

[0096] In one embodiment, an electronic device (e.g., the electronic device (101) of FIGS. 1 and 3) may enter a multiple exposure mode and determine multiple exposure mode setting values ​​for performing multiple exposure mode shooting (operation 605). For example, the electronic device (e.g., the electronic device (101) of FIGS. 1 and 3) may receive at least one user input for selecting a quantity of images to be used to generate a composite image or selecting a composite mode that defines a method of synthesizing the images. The electronic device (e.g., the electronic device (101) of FIGS. 1 and 3) may determine shooting setting values ​​for shooting an image based on the set multiple exposure mode setting values. For example, the shooting setting values ​​may include at least one of sensitivity, white balance, or exposure time. The shooting setting values ​​may be changed based on user input for each shooting while the multiple exposure mode is in operation. Alternatively, the shooting setting values ​​may be automatically set by the electronic device (e.g., the electronic device (101) of FIGS. 1 and 3).

[0097] In one embodiment, an electronic device (e.g., the electronic device (101) of FIGS. 1 and 3) may acquire a first basic image and first metadata corresponding to the first basic image. In addition, the electronic device may acquire a first post-processed image based on the first basic image. In operation 610, the electronic device according to one embodiment may determine whether to use an image stored in a memory to acquire a multiple exposure image. For example, the electronic device may receive an input from a user as to whether to use a captured image or a pre-stored image to acquire a multiple exposure image.

[0098] In operation 610, if the electronic device receives a selection input from a user who wants to use a captured image, in operation 630, the electronic device can set setting values ​​for capturing (e.g., ISO, White balance, etc.). In addition, in operation 632, the electronic device can perform capturing and obtain a base image and a post-processed image from image sensor data output from an image sensor. In operation 640, the electronic device can perform synthesis based on the obtained images (base image and post-processed image) to obtain a multiple exposure image (or, composite image). For example, the electronic device can synthesize a base image and a post-processed image obtained by capturing with another base image and another post-processed image obtained previously, respectively, to obtain a composite base image and a composite post-processed image.

[0099] Meanwhile, in operation 610, if the electronic device receives a selection input from the user who wants to use an image previously stored in the memory, the electronic device displays a list of stored images on the display (operation 620), receives an image selection input from the user to acquire an image (operation 622), displays the acquired image and an interface for modifying the image on the display (operation 623), and acquires information input by the user to modify the image (operation 623-1). In addition, the electronic device can analyze the acquired image based on the acquired image and / or the acquired user's image modification information (operation 624). In addition, the electronic device can determine the necessity of generating a basic image or a post-processed image based on the analysis result (operation 624-1), and, if necessary, generate a basic image or a post-processed image from the acquired image (operation 626). In addition, the electronic device can determine whether multiple exposure synthesis is possible for the acquired image or the image generated in operation 626 (operation 625). If it is determined that multiple exposure synthesis is possible in operation 625, the electronic device can perform synthesis based on the acquired image or the image generated in operation 626 (operation 640). If it is determined that multiple exposure synthesis is not possible in operation 625, the electronic device can generate a modified image from the acquired image or the image generated in operation 626 (operation 628). Then, in operation 640, the electronic device can perform synthesis based on the modified image (the modified basic image or the modified post-processed image) to obtain a multiple exposure image (or, a synthesized image). The image to be synthesized may include a photographed image, another previously stored image, or a previously synthesized image.

[0100] Specifically, in operation 620, the electronic device may display a list of image files (e.g., image files (500) of FIG. 5) or post-processed images stored in memory via a display. The image files may refer to a format (e.g., DNG) containing a basic image, and the post-processed images may represent images composed of a format such as JPG or PNG. However, the present invention is not limited thereto.

[0101] In operation 622, the electronic device may acquire an image by receiving a user's selection input for an image included in a list. The acquired image may include a base image or a post-processed image. For example, the acquired image may be a base image or a post-processed image extracted from an image file selected by the user, and the acquired image may be a post-processed image selected by the user by being included in a post-processed image list through a display. If the images stored in the memory of the electronic device are images acquired by various environments (e.g., cloud, other cameras and sensors), these images exist in the form of post-processed images rather than raw files, and thus a corresponding base image may not exist.

[0102] In operation 623, the electronic device may display the image acquired in operation 622 and an interface for modifying the image through a display. The interface that may be provided for modifying the image may include image enlargement, image reduction, image rotation, image inversion, etc. The electronic device may clearly display the area to be synthesized for the acquired image. Accordingly, when the user modifies the image using the provided interface, the user may clearly recognize the area in which the synthesis is performed in the image. For example, when the user modifies a selected image through an enlargement or reduction interface, the electronic device may display the synthesized area and the non-synthesized area of ​​the selected image by distinguishing them by brightness difference or blurriness, and may display the edge of the synthesized area.

[0103] In operation 623-1, the electronic device can obtain image modification information input by the user through operation 623. The image modification information can include information indicating image modification, such as image enlargement or reduction, image rotation, image inversion, and image position adjustment. When the user inputs an image modification through an interface displayed on the display of the electronic device, the electronic device can receive the user's input and obtain image modification information.

[0104] In operation 624, the electronic device may analyze the image acquired in operation 622. For example, the electronic device may determine the validity of the acquired image. The operation of determining the validity of the acquired image may determine whether the base image or the post-processed image is valid for the acquired image.

[0105] In operation 624-1, the electronic device may determine whether it is necessary to generate a base image or a post-processed image based on the analysis result of operation 624. In operation 624, if it is determined that the acquired image is invalid, in operation 624-1, the electronic device may determine that generation of the base image or the post-processed image is necessary. Then, in operation 626, the electronic device may generate a post-processed image or a base image based on the acquired image. For example, if the base image exists in the image file acquired in operation 622 but the post-processed image does not exist, the validity of the base image may be acknowledged and the validity of the post-processed image may be denied. Accordingly, in operation 626, the electronic device may generate the post-processed image based on the valid base image. Conversely, if the post-processed image exists in the acquired image file but the base image does not exist, the validity of the base image may be denied and the validity of the post-processed image may be acknowledged. At this time, in operation 626, the electronic device may generate the base image based on the valid post-processed image. For example, if a post-processed image, such as a JPG or PNG format, is acquired in operation 622, the base image does not exist. Therefore, the validity of the base image is denied and the validity of the post-processed image is recognized, and in operation 626, the electronic device can generate the base image based on the valid post-processed image.

[0106] In operation 624, if the acquired image is determined to be valid, the electronic device may determine in operation 624-1 that generation of a base image or a post-processed image is not necessary. In this case, operation 626 may be omitted.

[0107] In operation 625, the electronic device can determine whether multiple exposure synthesis is possible for the acquired image or the image generated in operation 626. The operation of determining whether the acquired image can be used for multiple exposure synthesis can be determined based on the sensor type, resolution, color space, bits used per pixel, or lens information of the acquired image and the user's image modification information acquired in operation 623-1, but is not limited thereto.

[0108] If, as a result of the analysis of operation 625, the electronic device determines that the acquired image or the image generated in operation 626 is not suitable for use in multiple exposure synthesis, in operation 628, the electronic device may modify the acquired basic image or the post-processed image to generate a modified image suitable for multiple exposure synthesis. Alternatively, the electronic device may modify the basic image or the post-processed image generated in operation 626 to generate a modified image suitable for multiple exposure synthesis. In operation 628, the electronic device may generate the modified image based on the user's image modification information acquired in operation 623-1. For example, the electronic device may rotate the basic image or the post-processed image. In addition, the electronic device may convert the color space of the image and remove a portion of the image. The electronic device may adjust the size of the image or change the position of the image. The image modification methods described above may be omitted or the order may be changed as needed. Specific image modification methods related to operation 628 are described below with reference to FIGS. 11 to 13.

[0109] In operation 640, the electronic device may perform synthesis based on the transformed images (base image and post-processed image) generated in operation 628 to obtain a multi-exposure image (or, synthesized image). For example, the electronic device may perform synthesis of the transformed base image and the transformed post-processed image with another base image and another post-processed image, respectively, to obtain a synthesized base image and a synthesized post-processed image. In another example, the electronic device may perform synthesis of the transformed base image and the transformed post-processed image with another base image and another post-processed image acquired by photographing (in other words, generated by processing image sensor data output from an image sensor), respectively, to obtain a synthesized base image and a synthesized post-processed image.

[0110] In addition, in operation 640, the electronic device may synthesize the transformed image and the image to be synthesized based on an area and generate a synthesized image. Here, the 'transformed image' may refer to the basic image transformed in operation 628 or the transformed post-processed image, and the 'image to be synthesized' may refer to a previously acquired image or an image acquired by photographing. The electronic device may define an area where the transformed image and the image to be synthesized overlap as a first area, and define an area where they do not overlap as a second area. In addition, the electronic device may apply a method of synthesizing images in the first area and a method of synthesizing images in the second area differently. With respect to operation 640, a specific synthesis method is described later with reference to FIGS. 14 to 17.

[0111] After the composite image is generated, the electronic device may determine whether the condition for terminating the multiple exposure mode is satisfied (operation 650). For example, if the number of image synthesis included in the multiple exposure mode setting value is set to N, the electronic device may determine that the condition for terminating the multiple exposure mode is satisfied when the number of times the image is synthesized by repeating the operation reaches N. In another example, the electronic device may determine that the multiple exposure mode is terminated based on receiving a user input corresponding to a command to terminate the multiple exposure mode.

[0112] If the multiple exposure mode termination condition is not satisfied, the electronic device may perform one or more of operations 610 to 640. For example, if the composite number is set to 3, the electronic device may acquire a first image by shooting, and then perform one or more of operations 610 to 640 to acquire a composite image. At this point in time, the composite number is 2, which is less than the set composite number of 3, and therefore one or more of operations 610 to 640 may be performed again. Thereafter, the composite number becomes 3, the multiple exposure termination condition is satisfied, and the electronic device may store the final generated composite image result in the memory (operation 660). In addition, the electronic device may check whether the multiple exposure mode is terminated (operation 670) and terminate the multiple exposure mode accordingly.

[0113] FIG. 7 is a block diagram of an electronic device that performs a method for acquiring multiple exposure images according to one embodiment. Referring to FIG. 7, the electronic device may include an image parsing module (710), an image validation module (7121, 7122), an image generation module (7141, 7142), an image transformation module (7161, 7162), an image data processing module (720), a reference image determination module (7201), an image blending module (7221, 7222), or an image file encoding module (730).

[0114] The image parsing module (710) can extract the selected image from the user's image selection input. For example, the image parsing module (710) can extract the basic image and the post-processing image included in the image file selected by the user (e.g., the image file (500) of FIG. 5). Alternatively, if the image selected by the user corresponds to a post-processing image, the image parsing module (710) can transmit the post-processing image as is. The image parsing module (710) can perform an operation of acquiring an image in operation 622 of FIG. 6. The image file input to the image parsing module (710) can include an image file configured in a raw image file format. The image file configured in the raw image file format can be stored in the memory of the electronic device (101). In one embodiment, the electronic device (101) can also generate an image file configured in a raw image file format from the post-processing image.

[0115] The image validation module (7121, 7122) can check the validity of the base image and post-processing image extracted from the image parsing module (710). The image validation module (7121, 7122) can include a base image validation module (7121) and a post-processing image validation module (7122). The base image validation module (7121) can perform a validation on the base image extracted from the image parsing module (710). If the extracted base image is invalid or does not exist, the base image validation module (7121) can determine that the base image is invalid. The post-processing image validation module (7122) can perform a validation on the post-processing image extracted from the image parsing module (710). If the extracted post-processing image is invalid or does not exist, the post-processing image validation module (7122) can determine that the post-processing image is invalid. The image validation module (7121, 7122) can perform an operation of analyzing the image acquired in operation 624 of FIG. 6.

[0116] The image generation module (7141, 7142) can generate a base image or a post-processing image based on the validation result performed by the image validation module (7121, 7122). For example, if the base image is valid and the post-processing image is invalid in the validation result of the image validation module (7121, 7122), the image generation module (7141, 7142) can generate a post-processing image based on the valid base image. For another example, if the base image is invalid and the post-processing image is valid in the validation result of the image validation module (7121, 7122), the image generation module (7141, 7142) can generate a base image based on the valid post-processing image. The image generation module (7141, 7142) can include a base image generation module (7141) and a post-processing image generation module (7142). The image generation module (7141, 7142) can perform operation 626 of FIG. 6.

[0117] The image transformation module (7161, 7162) can transform a valid base image or a valid post-processing image to be suitable for synthesis to obtain a multiple exposure image. For example, the image transformation module (7161, 7162) can transform an image to be suitable for multiple exposure synthesis by performing at least one of the following operations: rotating an image, converting a color space of an image, removing a portion of an image (cropping an image), resizing an image, and changing a position of an image. The image transformed by the image transformation module (7161, 7162) can be synthesized with an image based on data output from an image sensor by shooting or with a previously synthesized image. The image transformation module (7161, 7162) includes a base image transformation module (7161) that transforms a base image and a post-processing image transformation module (7162) that transforms a post-processing image. The image transformation module (7161, 7162) can perform operation 628 of FIG. 6.

[0118] The image data processing module (720) can process image sensor data output from an image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) to generate a basic image, metadata, and a post-processing image. The image data processing module (720) can correspond to the image signal processor (260) of FIG. 2, the Bayer / raw processing engine (410) and the image processing engine (420) of FIG. 4. The basic image and metadata generated by the image data processing module (720) are transferred to the reference image determination module, and the post-processing image generated by the image data processing module (720) is transferred to the post-processing image blending module (7222). The image data processing module (710) can perform operation 632 in FIG. 6.

[0119] The reference image determination module (7201) receives a base image transformed by the base image transformation module (7161) and a base image and related metadata generated by the image data processing module (720). The reference image determination module can determine a reference base image and reference metadata based on the received base images and metadata.

[0120] The image blending module (7221, 7222) includes a base image blending module (7221) and a post-processing image blending module (7222). The base image blending module (7221) can synthesize a base image transformed by the base image transformation module (7161) and a base image generated by the image data processing module (720) based on an area. The post-processing image blending module (7222) can synthesize a post-processing image transformed by the post-processing image transformation module (7162) and a post-processing image generated by the image data processing module (720) based on an area. The image blending modules (7221, 7222) can perform operation 640 of FIG. 6, and are described in FIGS. 14 to 17 with respect to the operation of generating a composite image based on an area.

[0121] The image file encoding module (730) can receive the aforementioned reference metadata and the synthesized base image and synthesized post-processed image generated from the image blending modules (7221, 7222) to generate an image file. The generated image file can include the synthesized base image, the synthesized post-processed image, and reference metadata.

[0122] FIG. 8 is a block diagram of an electronic device that performs a method for acquiring multiple exposure images according to one embodiment. Referring to FIG. 8, the electronic device may include an image parsing module (810), an image validation module (8121, 8122), an image generation module (8141, 8142), an image transformation module (8161, 8162), an image data processing module (820), a reference image determination module (8201), an image blending module (8221, 8222), or an image file encoding module (830). The operations of the above components are omitted as they have been described in FIG. 7.

[0123] Referring to FIG. 8, the electronic device may further include a preview overlay module (840) and a preview rendering module (850).

[0124] In one embodiment, the preview overlay module (840) can receive a post-processing image synthesized by the post-processing image blending module (8222). The preview overlay module (840) can overlay a post-processing image output from the image data processing module (820) on the synthesized post-processing image. The image data processing module (820) can stream the post-processing image to the preview overlay module (840) based on the rendering cycle of the preview image. The preview rendering module (850) can render an image in which the post-processing image, which is updated at each rendering cycle, is overlaid on the synthesized post-processing image, thereby outputting the preview image. Accordingly, the electronic device can output a preview image for the synthesized image through a display.

[0125] FIG. 9 is a block diagram of an electronic device that performs a method for acquiring multiple exposure images according to one embodiment. The electronic device may include an image parsing module (910), an image validation module (9121, 9122), an image generation module (9141, 9142), an image transformation module (9161, 9162), an image data processing module (920), a reference image determination module (9201), an image blending module (9221, 9222), or an image file encoding module (930). The operations of the above components are omitted as they have been described in FIG. 7.

[0126] Referring to FIG. 9, the electronic device may further include a white balance application module (921) and a reverse white balance application module (923).

[0127] In one embodiment, image sensor data (800) output from the image sensor module (230) may be input to the image data processing module (920). When a shooting event occurs in multiple exposure mode, the image data processing module (920) may process the image sensor data (800) to output a basic image and metadata.

[0128] The reference image determination module (9201) can determine a reference image from among a plurality of base images output from the image data processing module (920) and the transformed base images output from the base image transformation module (9161). Reference metadata corresponding to the reference image can be included in an image file generated by the image file encoding module (930).

[0129] The white balance application module (921) can correct the color information of the base image by applying white balancing to the base image based on metadata corresponding to the base image. The white balance application module (921) can extract white balancing parameters from the metadata and correct the base image so that it has a value that takes white balancing into account. This process can be applied to all base images captured based on the multiple exposure mode and base images modified by the base image modification module (9161).

[0130] The base image with white balancing applied can be passed to the base image blending module (9221). The base image blending module (9221) can composite the base images with white balancing applied based on the area according to the set composite mode.

[0131] In one embodiment, an inverse white balance applying module (923) can extract white balancing parameters from reference metadata. When the termination condition for shooting based on the multiple exposure mode is met, the final synthesized base image can be inversely transformed using the white balancing parameters extracted from the reference metadata. The inverse transformation process can be performed based on the transformation used in the white balance applying module (921). The inverse transformation process can also be configured by approximation. The synthesized base image output by the inverse white balance applying module (923) can be transmitted to the image file encoding module (930).

[0132] In one embodiment, the image file encoding module (930) can encode an image file including a synthesized base image and reference metadata.

[0133] In one embodiment, the image data processing module (920) may perform image processing on image sensor data to further output a post-processed image obtained. The post-processed image output from the image data processing module (920) and the post-processed image transformed by the post-processed image transformation module (9162) may be synthesized on a region basis by the post-processed image blending module (9222). The synthesized post-processed image may be included in an image file by the image file encoding module (930).

[0134] FIG. 10 is a block diagram of an electronic device that performs a method for acquiring a multiple exposure image according to one embodiment. Referring to FIG. 10, the electronic device may include an image parsing module (1010), an image validation module (10121, 10122), an image generation module (10141, 10142), an image transformation module (10161, 10162), an image data processing module (1020), a reference image determination module (10201), an image blending module (10221, 10222), or an image file encoding module (1030). In addition, the electronic device may further include a white balance application module (1021) and an inverse white balance application module (1023), and may further include a preview overlay module (1040) and a preview rendering module (1050). The operation of the component blocks represented in FIG. 10 has been described above with reference to FIGS. 7 to 9.

[0135] In one embodiment, the image parsing module (1010) can obtain a base image or a post-processed image from an image previously stored in memory, and the image validation module (10121, 10122) can check the validity of the obtained image. The image generation module (10141, 10142) can generate a post-processed image from the base image or generate a base image from the post-processed image based on the image validation result. In addition, the image transformation module (10161, 10162) can transform a valid base image or a post-processed image to be suitable for multiple exposure synthesis.

[0136] In one embodiment, the image data processing module (1020) may process image sensor data output from an image sensor by capturing to generate a base image, metadata, and a post-processed image. The reference image determination module (10201) may receive the base image generated by the image data processing module (1020) and the base image transformed by the base image transformation module (10161) to determine a reference base image and reference metadata. The white balance application module (1021) may apply white balancing to the received base image, and the base image blending module (10221) may generate a synthesized base image by synthesizing base images to which white balancing has been applied based on an area. The inverse white balance application module (1023) may inversely transform the synthesized base image using white balancing parameters extracted from the reference metadata. The inverse transformation process may be performed based on the transformation used in the white balance application module (1021).

[0137] In one embodiment, the post-processing image blending module (10222) may receive a post-processing image generated by the image data processing module (1020) and a post-processing image transformed by the post-processing image transformation module (10162), and generate a synthesized post-processing image by synthesizing the post-processing images based on an area.

[0138] In one embodiment, the image file encoding module (1030) may receive a synthesized base image, a synthesized post-processed image, and reference metadata to generate an image file.

[0139] In one embodiment, the preview overlay module (1040) may receive a post-processed image synthesized from the post-processing image blending module (10222). The preview overlay module (1040) may overlay the post-processed image output from the image data processing module (1020) on the synthesized post-processed image. The preview rendering module (1050) may render an image with the post-processed image overlaid to output a preview image. Accordingly, the electronic device may output a preview image for the synthesized image through a display.

[0140] FIG. 11 illustrates an image deformation method for obtaining a multiple exposure image according to one embodiment. Referring to FIG. 11, the image deformation method according to one embodiment may be performed by an image deformation module (1100).

[0141] In one embodiment, the image transformation module (1100) may determine whether to perform an operation module for image transformation and parameters for performing the operation module by considering an image to be transformed (or an input image) and its corresponding metadata, a shooting environment and result, image file information, a user's image selection information, and image modification information (operation 1105).

[0142] In one embodiment, the image transformation module (1100) can transform an image selected by the user based on the user's image selection information and image modification information applied to the selected image. The user's image modification information may represent information input by the user through a screen provided in the form of an interface to utilize the selected image for synthesis. For example, the user may perform an input to rotate, enlarge, reduce, or invert the selected image to determine an area to be synthesized in the selected image. The electronic device can receive input information regarding the rotation angle, enlarged or reduced area, and inversion operation set by the user through a graphical interface. The image transformation module (1100) can determine a transformation operation and the parameters required for image transformation based on the user's image selection information and image modification information, and can transform the image according to the content modified by the user. The electronic device can obtain the user's image selection information and image modification information through operations 622 and 623-1 of FIG. 6, respectively.

[0143] In one embodiment, the image transformation module (1100) determines whether the image needs to be rotated (operation 1115), and if necessary, can rotate the image (operation 1110). For example, in the case of image rotation, the image transformation module (1100) can determine whether the image needs to be rotated based on the orientation information of the device in which the input image was captured and the orientation information of the current device. If the image needs to be rotated, the image transformation module (1100) can determine a rotation angle (e.g., 90, 180°, 270°, etc.) and rotate the image. In addition, the image transformation module can determine whether the image needs to be rotated based on the user's image modification information, and if necessary, can rotate the image.

[0144] In one embodiment, the image transformation module (1100) determines whether a color space conversion of an image is required (operation 1125), and if necessary, can convert the color space of the image (operation 1120). For example, if the color space in which the input image is expressed is different from the color space to be included in the resulting image file (e.g., Raw File), the image transformation module (1100) can convert the color space of the input image. For example, if the color space of the input image is in Bayer format and the image file (e.g., DNG) should be stored in a color space of Linear RGB format, the image transformation module (1100) can convert the color space of the input image from Bayer format to Linear RGB format. In addition, the image transformation module can determine whether a color space conversion of an image is required based on the user's image modification information, and if necessary, can convert the color space of the image.

[0145] In one embodiment, the image transformation module (1100) determines whether cropping of the image is required (operation 1135), and if necessary, may remove a portion of the image (operation 1130). If the aspect ratio (or aspect ratio) of the input image is different from the aspect ratio of the resulting image (or composite image), the image transformation module (1100) may remove a portion of the input image to make the aspect ratio of the input image the same as the aspect ratio of the resulting image. The portion to be removed from the input image may be determined by the user or automatically determined by the system. The size of the composite image may be determined by the multiple exposure mode setting. In addition, the image transformation module may determine whether cropping of the image is required based on the user's image modification information, and may crop the image if necessary.

[0146] In this regard, FIG. 12 shows an example of performing an image cropping operation in an image deformation module (1100) according to one embodiment.

[0147] Referring to FIG. 12, the size of the image selected by the user input (hereinafter, referred to as the input image) is larger than that of the composite image (or the multiple exposure image). For example, the input image may be an image with a width of 4000 pixels and a height of 3000 pixels, and the composite image may be an image with a width of 3900 pixels and a height of 3000 pixels. In this case, the heights of the images are the same, but the width of the input image is longer than that of the composite image. Therefore, the image transformation module (1100) may remove a portion of the input image to transform the shape of the input image to be the same as that of the composite image.

[0148] In one embodiment, the image transformation module (1100) may remove the left area of ​​the input image (a), remove areas of the same size from the left and right sides of the input image (b), or remove the right area of ​​the input image. However, the location of the area removed by the image transformation module (1100) is not limited thereto.

[0149] In one embodiment, the image transformation module (1100) determines whether the image needs to be resized (operation 1145), and if necessary, can resize the image (operation 1140). For example, if the aspect ratio of the input image is different from the aspect ratio of the composite image, the image transformation module (1100) can resize the input image so that the entire area of ​​the input image is represented in the composite image. That is, the image transformation module (1100) can resize the input image so that the width and height of the input image do not exceed the width and height of the composite image, respectively. Accordingly, the resized input image has a width or height that is smaller than or equal to that of the composite image. In addition, the image transformation module can determine whether the image needs to be resized based on the user's image modification information, and can resize the image if necessary.

[0150] In one embodiment, the image transformation module (1100) determines whether the image needs to be repositioned (operation 1155) and, if necessary, can adjust the image's position (operation 1150). If the input image is resized while maintaining its aspect ratio, the resized input image may have a smaller size than the composite image. Accordingly, the image transformation module (1100) can adjust where the resized input image is positioned within the area of ​​the composite image. In addition, the image transformation module can determine whether the image needs to be repositioned based on the user's image modification information and, if necessary, can adjust the image's position.

[0151] In this regard, FIG. 13 illustrates an example of performing an image size adjustment operation and an image position adjustment operation in an image deformation module (1100) according to one embodiment.

[0152] Referring to FIG. 13, the image transformation module (1100) can adjust the size of the input image based on the synthetic image. For example, if the size of the input image is 4000 x 3000 pixels and the size of the synthetic image is 3900 x 3000 pixels, the image transformation module (1100) can adjust the size of the input image to obtain an image with a size of 3900 x 2925 pixels. That is, the image transformation module (1100) can adjust the size of the input image based on the horizontal length of the synthetic image. In the above example, since the aspect ratio of the resized input image is maintained, the vertical length of the resized input image is smaller than that of the synthetic image. The image transformation module (1100) can adjust the position of the resized input image. For example, the image transformation module (1100) can position the resized input image to align with the top of the composite image (a), position the resized input image to the middle of the composite image (b), or position the resized input image to align with the bottom of the composite image. The image transformation module (1100) can position one corner of the input image to align with one corner of the composite image, or position the input image within an area of ​​the composite image. The position of the input image can be determined by the user or automatically determined by the system.

[0153] Figure 14 shows an example of region segmentation of a multiple exposure image according to one embodiment.

[0154] In one embodiment, an image transformed by an image transformation module (e.g., the image transformation module of FIG. 7) and an image generated by an image data processing module (e.g., the image data processing module of FIG. 7) may be used for region-based multi-exposure synthesis. In this case, regions according to the embodiment may be distinguished as follows.

[0155] Referring to FIG. 14, cases where an input image used for synthesis fills an area of ​​a synthesized image and cases where it does not are illustrated. In this case, an area where the area of ​​the synthesized image overlaps with an area of ​​the input image may be identified as a first area, and an area where the area of ​​the synthesized image does not overlap with an area of ​​the input image may be identified as a second area. According to one embodiment of the present disclosure, when a pre-stored image is used for multiple exposure synthesis, if the aspect ratio of the pre-stored image is different from that of the synthesized image, the size of the input image may not fill the area of ​​the synthesized image. In this case, a part where a pre-stored image selected by a user's input overlaps an area of ​​the synthesized image may be identified as a first area, and a part where the overlap does not occur may be identified as a second area.

[0156] An electronic device and method according to one embodiment of the present disclosure can perform multiple exposure synthesis by dividing an area as described above and generate a composite image.

[0157] Fig. 15 illustrates a flowchart for determining x, y coordinate pixel values ​​of a composite image according to one embodiment. Referring to Fig. 15, an electronic device can perform synthesis of images by distinguishing area types.

[0158] The first region type may represent a region comprised of pixels having values ​​based on an image selected by the user (either a base image or a post-processed image). For example, in the case of an input image that has undergone a transformation process (such as resizing or cropping) based on an image selected by the user, a region comprised of pixels having values ​​based on the pixel values ​​of the image before the transformation may correspond to the first region type.

[0159] The second region type may represent a region composed of pixels based on values ​​added during the process of generating the input image. For example, a region composed of pixels other than the first region type in the input image may correspond to the second region type. Pixels corresponding to the second region type may have no value or may be set to a specific value representing an arbitrary color (such as white or black).

[0160] The null region type can represent an area where pixel values ​​do not exist. The base image can represent the image that will be used to synthesize the input image. The base image can initially consist of a null region with no pixel values. Furthermore, the base image can be an image captured by a camera module, or it can include a composite image produced by synthesizing two or more images.

[0161] Referring to FIG. 15, the electronic device can determine pixel values ​​of a synthesized image based on the types of regions of the input image and the base image (or, the synthesized target image).

[0162] In operation 1510, the electronic device can determine whether the pixel corresponding to the x, y coordinates of the input image is of the first region type. That is, the electronic device can determine whether the pixel corresponding to the x, y coordinates of the input image is located within the first region. If the pixel corresponding to the x, y coordinates of the input image is of the first region type, the electronic device can determine whether the pixel corresponding to the x, y coordinates of the base image is of the null region type (operation 1520). If the pixel corresponding to the x, y coordinates of the base image is of the null region type, the electronic device can determine the value of the pixel corresponding to the x, y coordinates of the composite image as the pixel value corresponding to the x, y coordinates of the input image (operation 1532). If the pixel corresponding to the x, y coordinates of the base image does not correspond to the null region type, the electronic device can determine whether the pixel corresponding to the x, y coordinates of the base image is of the first region type (operation 1522). If the pixel corresponding to the x, y coordinates of the base image is not of the first region type, the electronic device may perform operation 1532. If the pixel corresponding to the x, y coordinates of the base image is of the first region type, the electronic device may determine the value of the pixel corresponding to the x, y coordinates of the composite image based on the pixel value corresponding to the x, y coordinates of the input image and the pixel value corresponding to the x, y coordinates of the base image (operation 1534). In addition, the pixel corresponding to the x, y coordinates of the composite image may be determined to be of the first region type (operation 1540).

[0163] If the pixel corresponding to the (x, y) coordinate of the input image is not of the first region type, the electronic device can determine whether the pixel corresponding to the (x, y) coordinate of the base image is of the null region type (operation 1524). If the pixel corresponding to the (x, y) coordinate of the base image is of the null region type, the electronic device can determine the value of the pixel corresponding to the (x, y) coordinate of the composite image as the pixel value corresponding to the (x, y) coordinate of the input image (operation 1536). In addition, the region type of the pixel corresponding to the (x, y) coordinate of the composite image can be determined as the second region type.

[0164] In operation 1524, if the pixel corresponding to the (x, y) coordinate of the base image does not correspond to the null region type, the electronic device may determine whether the pixel corresponding to the (x, y) coordinate of the base image is of the first region type (operation 1526). If the pixel corresponding to the (x, y) coordinate of the base image is of the first region type, the electronic device may determine the value of the pixel corresponding to the (x, y) coordinate of the composite image as the pixel value corresponding to the (x, y) coordinate of the base image (operation 1538). Alternatively, if the pixel corresponding to the (x, y) coordinate of the base image is of the first region type, the electronic device may determine the value of the pixel corresponding to the (x, y) coordinate of the composite image based on the pixel value corresponding to the (x, y) coordinate of the base image and the value of the pixel corresponding to the (x, y) coordinate of the input image. At this time, the value of the pixel corresponding to the (x, y) coordinate of the input image may be set to a specific value indicating an arbitrary color. A pixel corresponding to the (x, y) coordinate of the composite image can be determined as a first region type (operation 1544).

[0165] Figures 16 and 17 illustrate examples of region-based image synthesis according to one embodiment.

[0166] Fig. 16 illustrates an example in which an electronic device, after entering a multiple exposure mode, acquires a captured image (1604) through 'capture' and synthesizes an input image (1602) selected by a user to generate a composite image (1606). The captured image (1604) may represent a basic image or a post-processed image generated by processing image sensor data output from an image sensor. The input image (1602) may represent a basic image or a post-processed image acquired based on a user's selection from images previously stored in a memory.

[0167] In one embodiment, since the shooting operation is performed based on the setting of the multiple exposure mode, the entire area of ​​the shot image (1604) may overlap with the entire area of ​​the composite image. Accordingly, the entire area of ​​the shot image (1604) may be identified as the first area. In addition, when the entire area of ​​the input image (1602) overlaps with the area of ​​the composite image as shown in the drawing, the entire area of ​​the input image (1602) may be identified as the first area. The area of ​​the initially set composite image is a blank area without data, and when the shot image (1604) is synthesized into the area of ​​the composite image, the composite image may be produced in the same manner as the shot image (1604). In addition, when the input image (1602) is additionally synthesized into the area of ​​the composite image into which the shot image (1604) is synthesized, a composite image in which the shot image (1604) and the input image (1602) are synthesized may be produced. At this time, the pixel corresponding to the first area of ​​the composite image may have a pixel value based on the data of the input image (1602) and the captured image (1604) and the synthesis mode. For example, if the synthesis mode is determined by the user as an average or a corresponding value, the pixel corresponding to the first area of ​​the composite image may have an average value of the data of the input image (1602) and the captured image (1604). However, the synthesis method is not limited thereto, and the pixel value corresponding to the first area of ​​the composite image may be calculated based on the maximum value, minimum value, median value, etc. of the data of the input image (1602) and the captured image (1604) depending on the synthesis mode.

[0168] In one embodiment, FIG. 17 illustrates an example in which an electronic device, after entering a multiple exposure mode, divides an input image (1704) and a captured image (1702) into regions and synthesizes them. When the electronic device performs multiple exposure synthesis based on an image previously stored in a memory, the format, such as the aspect ratio (or height-to-height ratio) or size, of the acquired image may be different from that of the synthesized image. Accordingly, the electronic device may generate an input image (1704) for multiple exposure synthesis by modifying the acquired image. The generated input image may include a first region composed of pixels based on pixel values ​​of the original image, and a second region, which is the remaining region excluding the first region. In the input image (1704), pixels belonging to the second region may have no pixel value or may have a specific value representing an arbitrary color.

[0169] In one embodiment, the area of ​​the initially set synthetic image may be a blank area without data. The electronic device may synthesize the input image (1704) into the area of ​​the initially set synthetic image (first synthesis). Then, the electronic device may additionally synthesize the captured image (1702) into the composite image synthesized with the input image (1704) (second synthesis). At this time, the pixel corresponding to the first area of ​​the synthetic image (1706) may have a pixel value based on the pixel values ​​of the input image (1704) and the captured image (1702). For example, the pixel corresponding to the first area of ​​the input image (1704) in the synthetic image (1706) may be calculated based on the input image (1704) and the captured image (1702). Then, the pixel corresponding to the second area of ​​the input image (1704) in the synthetic image (1706) may be calculated based on the captured image (1702) or a value representing an arbitrary color of the captured image (1702). And the entire composite image (1706) image can be defined as the first region for the next synthesis.

[0170] As described above, the electronic devices and methods according to embodiments of the present disclosure can distinguish regions during multiple exposure synthesis and apply different synthesis methods to each region. Accordingly, when performing synthesis using pre-stored images, the electronic devices and methods according to embodiments can apply separate synthesis methods to regions in the input image where pixel values ​​are absent or arbitrarily set, thereby generating a composite image.

[0171] FIG. 18a illustrates a UX / UI sequence of multiple exposure mode screens displayed on a display of an electronic device according to one embodiment.

[0172] Referring to FIG. 18A, the electronic device may display a multiple exposure mode screen through a display (e.g., display (310) of FIG. 3). According to one embodiment, the multiple exposure mode screen may include an interface for adjusting setting values ​​of the multiple exposure mode (number of composite images, composite method, etc.) and a button for displaying images stored in a memory (e.g., a “Select image” button).

[0173] When a user selects the "Select image" button and the electronic device receives the input, the electronic device may display an image selection screen that lists images stored in the memory. According to one embodiment, the image selection screen displays multiple images in appropriately sized images, and the user may view the images of the multiple images. The user may select any one of the displayed images. The images displayed on the image selection screen may represent image files (e.g., image files (500) of FIG. 5) or post-processed images (e.g., post-processed images (530) of FIG. 5).

[0174] When a user selects one of the images listed through an image selection screen, the electronic device receives an input regarding the user's selection. The electronic device can then use the selected image as an input image for multiple exposure synthesis to generate a composite image. Alternatively, the electronic device can generate or modify an image based on the selected image to generate an input image for multiple exposure synthesis. The input image can be synthesized with a captured image or can be further synthesized with a previously synthesized composite image. The electronic device can then display a preview screen of the generated composite image through a display. The user can then confirm the synthesis result through the preview screen. This operation of displaying the preview screen can be performed by the preview overlay module or the preview rendering module of FIG. 8 or FIG. 10.

[0175] FIG. 18b illustrates a UX / UI of a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0176] Referring to FIG. 18B, when a user selects a multi-exposure synthesis function button on a camera shooting mode screen, the electronic device may display a setting area on the display that allows settings for the multi-exposure synthesis mode. This setting area may include an image selection button for multi-exposure synthesis (“Select picture for multi expo”). When the user selects the image selection button and the electronic device receives the input, the electronic device may display an image selection screen that displays a list of images stored in the memory. According to one embodiment, the image selection screen includes an image list display area in which a plurality of images are displayed as images of an appropriate size, and the user may check the images of the plurality of images. In addition, the user may select any one of the displayed images. The images displayed on the image selection screen may represent image files (e.g., an image file (500) of FIG. 5) or post-processed images (e.g., a post-processed image (530) of FIG. 5).

[0177] When a user selects an image, the electronic device may display the selected image along with an interface for editing the selected image. At this time, the electronic device may display edge lines to indicate the area where the composite will be performed in the selected image. If the selected image has the same aspect ratio (e.g., 3:4) as the area where the composite will be performed, the entire area of ​​the selected image may correspond to the composite area.

[0178] In the image editing screen, when the user performs a zoom-in operation or selects an image rotation button or an image inversion button, the electronic device can display the selected image by enlarging, rotating, or inverting it. The electronic device can display the selected image within the area where the synthesis is performed (the area within the edge of the synthesis area). Then, when the user selects the "Done" button to terminate the image editing, the electronic device can display a preview screen on the display in which the modified image from the selected image and the image currently received through the image sensor are synthesized. When the user selects the "Capture" button, a synthesized image, similar to the preview screen, is generated and stored in the memory of the electronic device.

[0179] FIG. 18c illustrates a UX / UI of a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0180] Referring to Fig. 18c, a case is illustrated where the horizontal and vertical ratios of an image selected by a user are different from the ratios of the composite area. Since the aspect ratios of the selected image and the composite area are different, some areas of the selected image may not be located within the composite area. That is, even if the user zooms out to reduce the selected image, the electronic device may display the selected image so that it fills the composite area. Accordingly, some areas of the selected image may be located outside the composite area. When the user adjusts the size or position of the selected image, the size or position of the selected image may be adjusted so that no empty areas are created within the composite area.

[0181] When a user zooms out, the electronic device can display the entire area of ​​the selected image and the composite area separately. For example, the electronic device can display areas outside the composite area of ​​the selected image as darker than areas within the composite area. In other words, the electronic device can display the entire image area, which displays the entire area of ​​the selected image, and the image composite area, which displays the area of ​​the selected image where the composite is performed. Therefore, the user can clearly recognize which area of ​​the selected image will be composited.

[0182] In one embodiment, when a user changes the size or position of a selected image, the user may change the size or position of the selected image without causing blank areas within the composite area.

[0183] FIG. 18d illustrates a multiple exposure composite mode screen displayed on a display of an electronic device according to one embodiment.

[0184] Referring to Fig. 18d, an image displayed when an electronic device is rotated on an interface screen for modifying a selected image is shown. When an electronic device (e.g., a terminal) is rotated from portrait mode (left side of Fig. 18d) to landscape mode (right side of Fig. 18d), the interface displayed on the display of the electronic device may be transformed into a form appropriate for a landscape mode screen to reflect the rotation of the electronic device. However, for an image selected for multiple exposure synthesis, the electronic device can display the selected image as is regardless of the rotation. Therefore, the electronic device can display the selected image on the display in its original state regardless of whether the mode is changed from landscape mode to portrait mode.

[0185] The UX / UI screens described in FIGS. 18a to 18d may be UX / UI screens corresponding to the contents described in operations 620 to 623-1 of FIG. 6.

[0186] FIG. 19 is a flowchart illustrating a method for acquiring a multiple exposure image according to one embodiment. The operations illustrated in FIG. 19 may be performed by the electronic device described in FIGS. 1 and 3 . Furthermore, a processor included in the electronic device may be configured to perform the operations described in FIG. 19 .

[0187] Referring to FIG. 19, in one embodiment, a method of acquiring a multiple exposure image may include an operation of acquiring a first image (1910), an operation of acquiring a second image stored in a memory (1920), an operation of generating a third image (1930), and an operation of generating a fourth image (1940).

[0188] The operation of acquiring a first image (1910) may acquire a first image based on image data output from an image sensor. Operation 1910 may mean that an electronic device acquires an image by photographing. In this case, the acquired image may be a raw image included in a raw image file (e.g., the image file of FIG. 5) or a processed image.

[0189] The 1920 operation may indicate that the electronic device acquires an image previously stored in memory. The image previously stored in memory may be a raw image contained in a raw image file (e.g., the image file of FIG. 5) or a processed image. Furthermore, the image previously stored in memory may be a processed image acquired from various environments (e.g., an external server or another electronic device) and stored in memory.

[0190] Operation 1930 can generate a third image capable of multiple exposure synthesis with the first image based on the aspect ratio of the second image. For example, operation 1930 can generate the third image by modifying the second image, such as resizing the second image or removing a portion of the second image by comparing the aspect ratio of the second image with the aspect ratio of the synthesized image. The generated third image can be a second image modified to enable multiple exposure synthesis. The operation of modifying the second image is described in FIGS. 11 to 13.

[0191] The 1940 operation may generate a fourth image having a second aspect ratio by synthesizing the first image acquired in the 1910 operation and the third image generated in the 1930 operation. The fourth image may represent a composite image obtained by synthesizing the first image and the third image. In the 1940 operation, the first image and the third image may be synthesized based on the type of region. For example, different synthesis methods may be applied depending on the type of region. A synthesis method based on the type of region is described in FIGS. 14 to 17.

[0192] Embodiments according to the present disclosure describe an electronic device and operations performed by the electronic device that enable a user to create a multi-exposure composite image by selecting images captured or previously stored. Embodiments according to the present disclosure utilize information about the current shooting environment and image files (e.g., raw files) generated as a result of the multi-exposure, thereby utilizing previously stored images (e.g., base images and post-processed images of raw files) for multi-exposure shooting.

[0193] The embodiments included in this disclosure are not mutually exclusive, and some components between the embodiments may be implemented in combination with each other.

[0194] In one embodiment, an electronic device (e.g., the electronic device of FIGS. 1 and 3) includes a camera module including an image sensor, a memory storing images, and at least one processor connected to the camera module and the memory. The at least one processor may perform an operation of acquiring a first image based on image data output from the image sensor. The at least one processor may perform an operation of acquiring a second image stored in the memory. The at least one processor may perform an operation of analyzing the second image. The at least one processor may perform an operation of generating a third image capable of multiple exposure synthesis with the first image based on an aspect ratio of the second image, based on an analysis result of the second image. The at least one processor may perform an operation of generating a fourth image by synthesizing the first image and the third image.

[0195] In one embodiment, the electronic device may further include a display. The operation of obtaining a second image stored in the memory by at least one processor may include controlling the display to display a list of image files including image files stored in the memory. The operation of obtaining the second image may include receiving a user input for selecting an image file from the list of image files and obtaining a second image from the image files based on the user input, wherein the second image may include a raw image or a processed image.

[0196] In one embodiment, the operation of generating a third image based on the second image may include the operation of generating a base image or a post-processed image based on the second image. The operation of generating the third image may include at least one of the following operations: rotating the second image, the generated base image, or the generated post-processed image, resizing the image, removing a portion of the image, or changing the position of the second image, the generated base image, or the generated post-processed image.

[0197] In one embodiment, the processor may further perform an operation of obtaining correction information for the second image. The operation of obtaining correction information for the second image may include controlling a display to display the second image and an interface for correcting the second image, and receiving a user input for correcting the second image to obtain image correction information. Furthermore, the operation of generating the third image may generate the third image based on the image correction information.

[0198] In one embodiment, the electronic device may further include a display. The operation of obtaining a second image stored in the memory may include: controlling the display to display an image list including images stored in the memory; receiving a user input for selecting an image from the image list; and obtaining a second image based on the user input, wherein the second image may include a processed image.

[0199] In one embodiment, the third image (e.g., the input image) may include a first region comprised of pixels having values ​​based on pixel values ​​of the second image. Pixels of the fourth image (e.g., the composite image) corresponding to pixels belonging to the first region of the third image may have pixel values ​​based on the pixel values ​​of the first image and the pixel values ​​of the third image.

[0200] In one embodiment, the third image (e.g., the input image) includes a second region, which is the remaining region excluding the first region. A pixel of a fourth image (e.g., a composite image) corresponding to a pixel belonging to the second region of the third image may have a pixel value based on a pixel value of the first image. Alternatively, a pixel of the fourth image (e.g., a composite image) corresponding to a pixel belonging to the second region of the third image may be determined based on a pixel value of the first image and a pixel value representing an arbitrary color (e.g., white or black).

[0201] In one embodiment, the operation of generating a fourth image by synthesizing the first image and the third image may include generating a first corrected image by applying white balancing to the first image and a second corrected image by applying white balancing to the third image. The operation of generating the fourth image may include generating a third corrected image by synthesizing the first corrected image and the second corrected image. The operation of generating the fourth image may include generating the fourth image by applying an inverse transformation of white balancing to the third corrected image.

[0202] In one embodiment, the electronic device further includes a display, and the at least one processor may be further configured to perform an operation of controlling the display to display the fourth image as a preview image.

[0203] In one embodiment, the operation of generating the fourth image may determine pixel values ​​of the fourth image by referring to at least one of the first image or the third image based on the type of region to which a pixel of the third image corresponds.

[0204] The method of obtaining a multiple exposure image according to one embodiment of the present disclosure may correspond to operations performed by at least one processor of the electronic device described above.

[0205] According to various embodiments, an electronic device and an operating method thereof capable of improving the quality of a multiple exposure image may be provided.

[0206] According to various embodiments, an electronic device and a method of operating the same may be provided that render and display a multiple exposure image stored in a raw format so that the result has a color that is the same as or similar to a color of a thumbnail image, a cover image, or a preview image for the multiple exposure image.

[0207] According to various embodiments, an electronic device and a method of operating the same may be provided that provides a preview image predicted to be generated when a shooting command is input.

[0208] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0209] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0210] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims or specification of the present disclosure.

[0211] These programs (software modules, software) may be stored in a random access memory, a non-volatile memory including a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), a digital versatile disc (DVD) or other forms of optical storage, a magnetic cassette, or a memory formed by a combination of some or all of these. In addition, each configuration memory may be included in multiple numbers.

[0212] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0213] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0214] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0215] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0216] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0217] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, including both a and b, including both b and c, including both a and c, or including all of a, b, and c.”

[0218] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In electronic devices, A camera module including an image sensor; Memory where images are stored; and comprising at least one processor connected to the camera module and the memory; The above memory, when executed by the at least one processor, causes the electronic device to: Obtaining a first image based on image data output from the image sensor, Acquire a second image stored in the above memory, Based on the aspect ratio of the second image, a third image capable of multiple exposure synthesis with the first image is generated based on the second image, and Including instructions for synthesizing the first image and the third image to generate a fourth image having the second aspect ratio. Electronic devices.

2. In claim 1, The electronic device further comprises a display, The above memory, when executed by the at least one processor, causes the electronic device to: Controlling the display to display a list of image files including image files stored in the memory; Receive user input for selecting an image file from the list of image files above, Further comprising instructions for obtaining the second image from the image file based on the user input; The second image includes a raw image or a processed image. Electronic devices.

3. In claim 2, The above memory, when executed by the at least one processor, causes the electronic device to: To generate the third image, a base image or a post-processing image is generated based on the second image, Performing at least one operation among the operation of rotating the second image, the generated basic image, or the generated post-processed image, the operation of adjusting the size, the operation of removing a portion, and the operation of changing the position, Controlling the display to display the second image and an interface for modifying the second image to obtain modification information for the second image; Further comprising instructions for receiving a user's input for modifying the second image and obtaining the modification information; The third image is generated based on the above modification information. Electronic devices.

4. In claim 1, The electronic device further comprises a display, The above memory, when executed by the at least one processor, causes the electronic device to: Controlling the display to display an image list including images stored in the memory to obtain the second image; Receive user input for selecting an image from the above image list, Obtaining the second image based on the user input, To generate the third image, a base image is generated based on the second image, Further comprising instructions for performing at least one of the following operations: rotating the second image or the generated base image, resizing the image, removing a portion of the image, or changing the position of the image; The second image above includes a processed image, Electronic devices.

5. In claim 1, The third image includes a first region composed of pixels having values ​​based on pixel values ​​of the second image, and a second region that is a remaining region excluding the first region, A pixel of the fourth image corresponding to a pixel belonging to the first area of ​​the third image has a pixel value based on a pixel value of the first image and a pixel value of the third image, The pixels of the fourth image corresponding to the pixels belonging to the second area of ​​the third image have pixel values ​​based on the pixel values ​​of the first image. Electronic devices.

6. In claim 1, The above memory, when executed by the at least one processor, causes the electronic device to: To generate the fourth image by synthesizing the first image and the third image, a first corrected image is generated by applying white balancing to the first image, and a second corrected image is generated by applying white balancing to the third image. A third correction image is generated by synthesizing the first correction image and the second correction image, Further comprising instructions for applying an inverse transformation of white balancing to the third correction image to generate the fourth image. Electronic devices.

7. In claim 1, The above memory, when executed by the at least one processor, causes the electronic device to: Further comprising instructions for determining pixel values ​​of the fourth image by referring to at least one of the first image or the third image based on the area type to which a pixel of the third image corresponds to generate the fourth image. Electronic devices.

8. A method for operating an electronic device including a camera module including an image sensor, An operation of acquiring a first image based on image data output from the image sensor; An operation of acquiring a second image stored in a memory included in said electronic device; An operation of generating a third image capable of multiple exposure synthesis with the first image based on the aspect ratio of the second image; and Including an operation of generating a fourth image by synthesizing the first image and the third image. method.

9. In claim 8, The operation of obtaining the second image is as follows: An operation for controlling a display to display a list of image files including image files stored in the above memory, An action for receiving user input for selecting an image file from the above image file list, An operation of obtaining the second image from the image file based on the user input, The second image includes a raw image or a processed image. method.

10. In claim 9, The action of generating the above third image is: An operation for generating a base image or a post-processing image based on the second image, It includes at least one operation among the operation of rotating the second image, the generated basic image or the generated post-processed image, the operation of adjusting the size, the operation of removing a part of the image or the operation of changing the position, The action of generating the above third image is: Further comprising an operation of obtaining modification information for the second image, The operation of obtaining modification information for the above second image is as follows: An operation of controlling the display to display the second image and an interface for modifying the second image; Including an operation of receiving a user's input for modifying the second image and obtaining the modification information; The third image is generated based on the above modification information. method.

11. In claim 8, The operation of obtaining the second image is as follows: An operation for controlling a display to display an image list including images stored in the above memory, An action for receiving user input for selecting an image from the above image list, comprising an action of obtaining the second image based on the user input; The second image above includes a processed image, The action of generating the above third image is: An operation for generating a base image based on the second image, Including at least one of the following actions: rotating the second image or the generated basic image, adjusting the image, removing a portion of the image, or changing the position of the image; method.

12. In claim 8, The third image comprises a first region composed of pixels having values ​​based on pixel values ​​of the second image, A pixel of the fourth image corresponding to a pixel belonging to the first area of ​​the third image has a pixel value based on a pixel value of the first image and a pixel value of the third image. method.

13. In claim 12, The third image includes a second area, which is the remaining area excluding the first area, The pixels of the fourth image corresponding to the pixels belonging to the second area of ​​the third image have pixel values ​​based on the pixel values ​​of the first image. method.

14. In claim 8, The operation of generating the fourth image by synthesizing the first image and the third image is, An operation for generating a first correction image with white balancing applied to the first image and a second correction image with white balancing applied to the third image; An operation of generating a third correction image by synthesizing the first correction image and the second correction image, Including an operation of generating the fourth image by applying an inverse transformation of white balancing to the third correction image. method.

15. In claim 8, The operation of generating the above fourth image is: Determining the pixel value of the fourth image by referring to at least one of the first image or the third image based on the area type to which the pixel of the third image corresponds, method.

Images