Electronic device, method, and storage medium for generating information for enhancing brightness of display for displaying image

Abstract

This electronic device may comprise at least one camera, a memory for storing instructions, and at least one processor. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: acquire a first image including a visual object in a first area and a second image including the visual object in a second area partially overlapping the first area; generate, as map information available for adjusting a brightness range for displaying the first image, a third image including the visual object in a third area corresponding to the first area on the basis of the acquiring of the first image and the second image; and store a file including the first image and the third image.

Description

Electronic device, method, and storage medium for generating information for enhancing the brightness of a display displaying an image

[0001] The following descriptions relate to electronic devices, methods, and storage media for generating information for enhancing the brightness of a display displaying an image.

[0002] Digital information generated for visualizing images and / or videos (e.g., image files in the format of the Joint Photographic Experts Group (JPEG) and / or Moving Picture Experts Group (MPEG)) can be generated to represent colors using a limited number of bits. For example, within the digital information, the brightness of a particular primary color (e.g., one of red, green, or blue) can be stored using eight bits. In the above example, the digital information can be generated to represent the brightness of a particular primary color using a total of 256 brightness levels.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] An electronic device may include at least one camera. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, using the at least one camera, a first image including a visual object in a first area and a second image including the visual object in a second area partially overlapping the first area and being darker than the first image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate, based on obtaining the first image and the second image, a third image including the visual object in a third area corresponding to the first area, using map information usable for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to store a file comprising the first image and the third image.

[0005] An electronic device may include at least one camera. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, using the at least one camera, a first image including a visual object in a first area and a second image including the visual object in a second area partially overlapping the first area and being darker than the first image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate, based on obtaining the first image and the second image, a third image including the visual object in a third area corresponding to the first area, using map information usable for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range. The global color tone of the third image may be different from the global color tone of the first image. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to store a file including the first image and the third image.

[0006] A method performed by an electronic device may include acquiring a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region and being darker than the first image. The method may include generating a third image including the visual object within a third region corresponding to the first region as map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, based on acquiring the first image and the second image. The method may include storing a file including the first image and the third image.

[0007] A non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by at least one processor of an electronic device including at least one camera, cause the electronic device to obtain, using the at least one camera, a first image including a visual object in a first area and a second image including the visual object in a second area partially overlapping the first area and being darker than the first image. One or more programs comprising instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to generate, based on obtaining the first image and the second image, a third image including the visual object in a third area corresponding to the first area using map information usable for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range. The electronic device may store one or more programs comprising instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to store a file comprising the first image and the third image.

[0008] Figure 1a illustrates an example of an electronic device that displays an image using a file.

[0009] Figure 1b illustrates an example of ghosting distortion resulting from a high dynamic range (HDR) image generated using multiple images acquired through multiple exposure photography.

[0010] Figure 1c illustrates an example of a method for generating an HDR image using multiple images acquired using at least one camera without ghosting distortion.

[0011] Figure 2 illustrates an exemplary block diagram of an electronic device.

[0012] Figure 3 illustrates an example of a flow of operations for displaying an HDR image while shaking multiple images within it.

[0013] Figure 4a shows map information for an HDR image using multiple images.

[0014] An example of how to create it is shown.

[0015] FIG. 4b illustrates an example of a method for generating a virtual image for a main image using an auxiliary image among a plurality of images, and generating map information for an HDR image using the main image and the virtual image.

[0016] Figure 5 illustrates an example of a method for generating a virtual image using global tone information of a main image and global tone information of an auxiliary image.

[0017] Figure 6 illustrates an exemplary structure of a file generated by an electronic device.

[0018] Figure 7a shows an example of map information in which distortion has been formed.

[0019] Figure 7b illustrates an example of an operational flow for a method of generating map information based on a flattening process.

[0020] Figure 7c shows an example of map information changed according to the flattening process.

[0021] Figure 8a illustrates an example of an operational flow for a method of generating a virtual image using filtered transformation information.

[0022] Figure 8b shows an example of a graph representing the grayscale range of a virtual image generated according to filtered transformation information.

[0023] Figure 9 illustrates an example of an operational flow for a method of storing a file containing an additional image used as map information for adjusting a brightness range for displaying an image.

[0024] FIG. 10 is a block diagram of an electronic device within a network environment according to various embodiments.

[0025] FIG. 11 is a block diagram of a display module according to various embodiments.

[0026] FIG. 12 is a block diagram of a camera module according to various embodiments.

[0027] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0028] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0029] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0030] Figure 1a illustrates an example of an electronic device that displays an image using a file.

[0031] FIG. 1A illustrates an example of an electronic device (101) that displays an image on a display (103) (e.g., a display module (1060) of FIG. 10). For example, the electronic device (101) may have various form factors, such as a smartphone, a laptop personal computer (PC), a tablet PC, a head-mounted display (HMD) device, a watch, and other computing devices. The electronic device (101) may also be referred to as a mobile device, a user terminal, a user equipment (UE), a multi-function device, a portable communication device, and / or a portable device. The form factor of the electronic device (101) is not limited to the exemplary form factor illustrated in FIG. 1A (e.g., an electronic device including a bar-type display (103). In one example, the electronic device (101) may be a device including a display (103) that is a flexible display. For example, the electronic device (101) may be referred to as a foldable electronic device, a rollable electronic device, or a multi-foldable electronic device.

[0032] In one embodiment, the electronic device (101) may generate a file (110), and / or visualize or display media content (e.g., media content referred to as a photograph and / or image) of the generated file (110). For example, the media content may be displayed on a display (103).

[0033] For example, the file (110) may include a joint photographic experts group (JPEG) file, a high efficiency image file format (HEIF) file, a high efficiency image container (HEIC) file, a file for storing raw data (e.g., a digital negative image (DNG) file), a portable network graphic (PNG) file, and / or a graphics interchange format (GIF) file. An exemplary hardware configuration of an electronic device (101) for executing functions related to generating and / or processing the file (110) is illustrated and described with reference to FIG. 2.

[0034] In the present disclosure, the dynamic range of an image may refer to the ratio between the minimum brightness and the maximum brightness, which is represented by the color distribution of the image. The dynamic range may include the contrast ratio. Referring to FIG. 1A, the file (110)

[0035] An image (112) associated with a standard dynamic range (SDR) may be included. In the example of FIG. 1A, an example of a file (110) including an image (112) associated with an SDR is illustrated, but the present disclosure is not limited thereto. For example, the file (110) may also include an image associated with a low dynamic range (LDR).

[0036] For example, the brightness (or luma value) of each pixel of an image (112) having SDR can be represented as a binary value with a bit depth of 8 bits. Using a bit depth of 8 bits, the image (112) having SDR can be represented in 256 steps ( ) can represent the brightness. For example, an image (112) displayed using SDR can have a contrast ratio of about 250:1.

[0037] Referring to FIG. 1A, a brightness range for displaying an image (112) may be determined based on a minimum brightness l1 and a maximum brightness l2. For example, the brightness range may represent the brightness of the display (103) for displaying the image (112). For example, the brightness range for displaying the image (112) may correspond to a grayscale range of the image (112). For example, the grayscale range may be determined based on at least one of a tone, a brightness, and a contrast of pixels of the image (112). For example, the electronic device (101) may display the image (112) on the display (103) using the brightness range according to the minimum brightness l1 and the maximum brightness l2 corresponding to the grayscale range of the image (112). For example, the light generated to display an image (112) having SDR may have a deviation of up to 100 nits (e.g., l2-l1) in different parts of the image (112). The image (112) associated with SDR may have color information based on, for example, the sRGB (standard RGB) color space.

[0038] In one embodiment, the electronic device (101) can perform operations related to a wider dynamic range than SDR (e.g., high dynamic range (HDR)). For example, the electronic device (101) can output light with a contrast ratio different from 250:1 (e.g., about 10,000:1) by controlling a display (103) that supports HDR mode. For example, the electronic device (101) can generate a file (110) that can support both SDR and HDR by synthesizing (e.g., bracketing) images acquired using a camera. For example, the file (110) can include an image (112) that can be displayed by another electronic device (or display device) that supports only SDR among SDR or HDR. For example, an electronic device (101) may include information that can scale visual information of an image (112) to HDR while including an image (112) of SDR that is compatible with a legacy display device that only supports SDR. An exemplary operation of the electronic device (101) to generate a file (110) is illustrated and described with reference to FIGS. 1B, 1C, 4A, and 4B. An exemplary structure of a file (110) generated by the electronic device (101) is illustrated and described with reference to FIG. 6.

[0039] Referring to FIG. 1A, according to one embodiment, a file (110) generated by an electronic device (101) may include metadata used by an electronic device (101) (or a display device) supporting an HDR mode to synthesize an image (130) having an HDR effect from an image (112) having an SDR. The metadata may include map information (115) related to the image (112) having an SDR. However, the present disclosure is not limited thereto. For example, the metadata may further include additional information (e.g., photo shooting information based on EXIF ​​(EXchangeable Image File)) and / or display control information.

[0040] For example, the map information (115) may be a two-dimensional array having a width and a height less than or equal to the width and height of the image (112), respectively. For example, the size of the map information (115) may be smaller than or equal to the size of the image (112). For example, the map information (115) may be referred to as an additional image or a gain image. For example, the brightness (or global brightness) of the map information (115) may be darker than the brightness (or global brightness) of the image (112). For example, the global brightness may be identified by an average value of the brightness for all pixels of the map information (115), a grayscale range (or contrast) for the pixels of the map information (115), or global tone information. Elements of the two-dimensional array (e.g., pixels of the map information (115)) may include numerical values ​​(e.g., coefficients and / or increments) used to change the brightness of different portions of the image (112). By changing the values ​​(e.g., brightness values) of the pixels of the image (112) using the map information (115), the electronic device (101) may obtain an image (130) having an HDR effect. For example, the image (130) may be referred to as an HDR image. For example, an image (130) synthesized from the image (112) using the map information (115) may have a grayscale range represented using a bit depth greater than the bit depth of the image (112).

[0041] According to one embodiment, the electronic device (101) may control the display (103) based on the metadata when displaying an image (130) on the display (103). The electronic device (101) may display an image (130) having an HDR effect, synthesized from an image (112) having SDR, included in a file (110), on the display (103). For example, when displaying the image (130) using the display (103), the brightness range for displaying the image (130) may be determined based on a minimum brightness l3 and a maximum brightness l4. For example, the electronic device (101) may display the image (130) on the display (103) using a brightness range according to the minimum brightness l3 and the maximum brightness l4. The contrast ratio associated with HDR may exceed (e.g., about 10,000:1) the contrast ratio associated with SDR (e.g., about 250:1). When displaying an image (130) through a display (103), the electronic device (101) may display the image (130) based on a wider color space than SDR (e.g., digital cinema initiatives (DCI)-P3 and / or REC.2020).

[0042] For example, when the brightness range for displaying an image (112) is a first brightness range, the brightness range for displaying an image (130) may have a second brightness range that is wider than the first brightness range. For example, the electronic device (101) may adjust the brightness range of the display (103) for displaying an image from the first brightness range to the second brightness range using map information (115).

[0043] Although not illustrated in FIG. 1A, the display control information included in the file (110) may be used to control the display (103). For example, the display control information may include a value related to the maximum brightness l4 required to display an image (130) having an HDR effect based on the file (110). For example, the display control information may include a ratio between the maximum brightness l2 of pixels of the image (112) of SDR and the maximum brightness l4 of pixels of the image (130) having an HDR effect.

[0044] As described above, according to one embodiment, the electronic device (101) can clearly display (or visualize) an image (130) having an HDR effect in order to at least partially increase (or enhance) the brightness of the display (103). The electronic device (101) can generate a file (110) including all of the metadata (e.g., map information (115) and / or display control information) for synthesizing the image (130) of the HDR effect and the image (112) of the SDR. Using the image (112) of the SDR, the electronic device (101) can generate a file (110) that is compatible with a legacy image rendering pipeline that uses a bit depth of 8 bits (e.g., a hardware accelerator that performs rendering on a file having a format of JPEG, and / or a software application configured to emulate the hardware accelerator). The electronic device (101) can cause a display device supporting HDR (e.g., an electronic device (101) including a display (103)) to display the image (130) using a wider dynamic range than SDR by providing information used to synthesize an image (130) supporting HDR effect to a file (110).

[0045] For example, the HDR mode (or a mode for generating (or storing) and / or displaying an image (130) that supports the HDR effect) may be set within the electronic device (101). For example, the electronic device (101) may display a screen (150) on the display (103) while executing a software application (e.g., a camera application) for controlling at least one camera.

[0046] For example, the electronic device (101) may display a visual object (140) mapped to a function for changing an option associated with at least one camera (e.g., the camera module (1080) of FIG. 10). A visual object (140) including an image and / or an icon in the shape of a gear is illustrated as an example, but the embodiment is not limited thereto. In response to an input associated with the visual object (140) (e.g., a touch input to a portion of the display (103) on which the visual object (140) is displayed), the electronic device (101) may display a screen (150) on the display (103). The screen (150) may include a settings screen provided by a software application for controlling the at least one camera.

[0047] Referring to an exemplary screen (150) of FIG. 1A, the electronic device (101) may provide an option (152) for determining whether to generate a file including information for displaying an image (130) with an HDR effect when saving a file (110) related to an image. Using a visual object (154) (e.g., a radio button and / or a toggle switch) corresponding to the option (152), the electronic device (101) may receive an input for activating or deactivating the option (152). When the option (152) is activated, the HDR mode may be set (or activated). For example, when the option (152) is activated by the visual object (154), the electronic device (101) may determine whether to store an image with an HDR effect based on a deviation and / or distribution of the amount of light in the external environment detected by at least one camera.

[0048] For example, the electronic device (101) may receive a shooting input when the HDR mode is set. The shooting input may include an input for acquiring images continuously acquired through at least one camera and storing a file related to the images. For example, the shooting input may be detected by a gesture (e.g., a tap gesture) for a shooting input icon displayed on the display (103). For example, the shooting input icon may be referred to as a shooting button. However, the present disclosure is not limited thereto.

[0049] For example, the above-described photographing input may be detected by a gesture of pressing a physical button (e.g., a button for adjusting the volume of the electronic device (101)) exposed externally through one side of the electronic device (101) (e.g., a front side and a side connecting the rear side opposite the front side).

[0050] Alternatively, for example, the photographing input may be detected in response to a body part (e.g., a palm) detected by at least one camera and / or a user gesture associated with the body part while displaying a preview image acquired through at least one camera on the display (103). For example, if a palm in an open posture is detected using an image acquired from at least one camera, the electronic device (101) may determine that a gesture indicating a photographing input has been detected. The electronic device (101) may display an indicator (e.g., a visual object in the form of a rectangular line) indicating the location where the palm is detected within the preview image. To detect the palm from the image, the electronic device (101) may perform an algorithm for object recognition.

[0051] Alternatively, for example, the photographing input may be detected based on an audio signal obtained from a microphone of the electronic device (101). For example, the electronic device (101) may determine that a voice command indicating the photographing input has been detected if it obtains a natural language sentence (e.g., “photograph” and / or “smile”) indicating the photographing input from the audio signal. In order to recognize the natural language sentence, the electronic device (101) may process the audio signal obtained from the microphone by performing an algorithm such as STT (speech to text) while displaying the preview image.

[0052] For example, an electronic device (101) that receives a photographing input may generate or store a file (110). In the following FIGS. 1B and 1C, examples of ghosting distortion caused in an HDR image based on a file generated by the electronic device (101) and a method for suppressing (or removing, preventing) the ghosting distortion are described.

[0053] Figure 1b illustrates an example of ghosting distortion resulting from a high dynamic range (HDR) image generated using multiple images acquired through multiple exposure photography.

[0054] FIG. 1b illustrates an example of ghosting distortion caused in an image (130a) having an HDR effect generated using images (112, 113) acquired according to multiple exposure photography (or multiple exposure photography technique).

[0055] For example, the multiple exposure shooting may be a shooting technique in which the electronic device (101) acquires images (112, 113) according to different exposure values ​​using at least one camera in a state in which the HDR mode is set (or activated). For example, the exposure value may include a property of the at least one camera. For example, the property may include an International Standardization Organization (ISO) sensitivity, an aperture opening degree (e.g., f-stop), and / or a shutter speed. As different exposure values ​​are used, the plurality of images (112, 113) may have different grayscale ranges. For example, an image (112) acquired according to a first exposure value may have a first grayscale range. For example, an image (113) acquired according to a second exposure value lower than the first exposure value may have a second grayscale range different from the first grayscale range. For example, image (112) may be brighter than image (113). For example, image (113) may have relatively higher contrast because it was acquired with a lower second exposure value than image (112). For example, the contrast may represent a difference in brightness between a relatively brighter area (or highlight area) within the image (e.g., an area where a visual object corresponding to a fluorescent light is located) and a relatively darker area (or non-highlight area) within the image (e.g., an area where a visual object corresponding to a plant is located). Image (112) used as an SDR image may be referred to as a main image, a base image, or a first image. Image (113) may be referred to as a sub image, or a second image.

[0056] In one embodiment, the multiple exposure shooting can be used to acquire (or display) an HDR image. To acquire (or display) an HDR image, a relatively wider dynamic range (DR) can be secured by using multiple exposure shooting through bracketing. In this case, the HDR image can be synthesized from a plurality of images (112, 113) acquired according to different exposure values.

[0057] For example, the electronic device (101) may acquire images (112, 113) using at least one camera while the HDR mode is set. However, while the images (112, 113) are acquired, a time difference may occur between the timing at which the image (112) is acquired and the timing at which the image (113) is acquired. Within the time difference, the electronic device (101) may be moved (or the electronic device (101) may be moved by the user), or the position of a visual object (or subject) within the image may change. For example, a visual object (112a) within the image (112) and a visual object (113a) within the image (113) may represent the same external object (e.g., a plant).

[0058] For example, the electronic device (101) can identify whether shaking has occurred in the images (112, 113). For example, when shaking has occurred, a first region of the image (112) where a visual object (112a) in the image (112) is located may partially overlap with a second region of the image (113) where a visual object (113a) in the image (113) is located. In other words, position information indicating the first region of the image (112) may not correspond to (or be identical to) position information indicating the second region of the image (113). Alternatively, at least some deviation may exist between the position of the visual object (112a) in the image (112) and the position of the visual object (113a) in the image (113). In the above example, it is described as determining whether the position of the visual object (112a) in the image (112) corresponds (or is the same) with the position of the visual object (113a) in the image (113), but the present disclosure is not limited thereto. For example, if the difference (or deviation) between the first region where the visual object (112a) in the image (112) is located and the second region where the visual object (113a) in the image (113) is located is less than the reference difference, it can be determined that the position of the visual object (112a) in the image (112) is substantially corresponding (or the same) with the position between the positions of the visual object (113a) in the image (113) (or there is no shaking). Alternatively, for example, if the difference (or deviation) between the representative position of the visual object (112a) in the image (112) and the representative position of the visual object (113a) in the image (113) is less than the reference difference, the position of the visual object (112a) in the image (112) may be determined to substantially correspond to (or be identical to) the position between the positions of the visual object (113a) in the image (113).

[0059] For example, the position information indicating the first area where the visual object (112a) in the image (112) is located and the position information indicating the second area where the visual object (113a) in the image (113) is located may each include relative position information. For example, the relative position information may include information for indicating a relative position with respect to the visual object in the image. For example, the relative position information may include a ratio of the distance from a reference position (or reference coordinates) of the image to a position (or representative position) of the visual object. Or, for example, the relative position information may include a normalized value for the position (or representative position) of the visual object in the image. By using the relative position information, even when the sizes of the images are different, the electronic device (101) can compare the positions between the visual objects in the images. For example, an algorithm for obtaining the relative position information may include optical flow and oriented fast and rotated brief (ORB). In one example, the electronic device (101) can identify a motion value by measuring the distance between key points for a visual object (or any location or area). For example, the electronic device (101) can identify that shaking (or ghosting distortion) is likely to occur if the motion value exceeds a reference value.

[0060] For example, the electronic device (101) may generate a file (110a) to support the HDR effect. For example, the electronic device (101) may generate the file (110a) using images (112, 113). For example, the file (110a) may include an image (112) supporting SDR and map information (117). In the example of FIG. 1B, the file (110a) includes map information (117), but the present disclosure is not limited thereto. For example, the file (110a) may include metadata including map information (117).

[0061] As described above, since the map information (117) is generated by synthesizing the images (112, 113), if shaking occurs in the images (112, 113), ghosting distortion may occur in the map information (117). For example, ghosting distortion may occur in the map information (117) due to a positional deviation between the visual object (112a) of the image (112) and the visual object (113a) of the image (113). For example, the ghosting distortion may be a distortion in which an effect such as an afterimage of a visual object is displayed due to a mismatch in the positions (or areas) of corresponding visual objects when synthesizing multiple images. For example, the ghosting distortion may be caused by a user's hand shaking or a movement of a visual object when taking a picture using the electronic device (101), as described above.

[0062] For example, the electronic device (101) may generate an image (130a), which is an HDR image, using a file (110a) to support an HDR effect. At this time, since ghosting distortion occurs in the map information (117), ghosting distortion may also occur in the image (130a).

[0063] As described above, the electronic device (101) may acquire images (112, 113) using at least one camera in a state where the HDR mode is set, and generate an HDR image (or file (110a)). However, as described above, if it is determined that ghosting distortion will occur, the electronic device (101) may perform image processing using only the image (112) (or main image) supporting SDR, without using the images (112, 113) acquired through multiple exposure shooting, and may then generate an HDR image. In this case, the advantage of multiple exposures in which a highlight area (e.g., fluorescent light) and a non-highlight area (e.g., plant) within the image (112) are clearly distinguished may not be utilized. Accordingly, when generating an HDR image using only the image (112), a gain (or weight) for generating an HDR image from the image (112) may be set relatively low. In other words, when only the image (112), which is an SDR image, is used, the highlight area may be relatively less bright compared to when a multiple exposure shooting technique is used. When ghosting distortion occurs, even though there are images (112, 113) actually acquired using multiple exposure values, the image (113), which is an auxiliary image among the images (112, 113), may not be used. In addition, when the visual object is a word, the standard for determining whether ghosting distortion occurs for generating (or displaying) an HDR image (or the standard for determining shaking, the standard for determining movement) may be relatively strictly determined depending on the visual saliency.

[0064] Hereinafter, the present disclosure can generate an HDR image (or a file for generating an HDR image) using acquired images even when shake is caused while acquiring images according to a multiple exposure shooting technique in HDR mode. An electronic device, a method, and a storage medium according to the present disclosure can generate and store an HDR image (or a file for generating an HDR image) by using information of acquired images even when shake is identified while acquiring images. Accordingly, the electronic device, the method, and the storage medium according to the present disclosure can provide a wider dynamic range. In addition, the electronic device, the method, and the storage medium according to the present disclosure can further perform a process for removing distortion that may be caused by using information of acquired images. Accordingly, the electronic device, the method, and the storage medium according to the present disclosure can display an image that is clearer and provides a wider dynamic range. The electronic device, method, and storage medium according to the present disclosure can provide a more dynamic HDR effect to a user by improving the brightness range (or luminance) of a display for displaying an image when displaying the image.

[0065] When shaking occurs during image acquisition, specific details on how to create an HDR image (or file) using information from the acquired images are exemplified and described with reference to FIG. 1c below.

[0066] Figure 1c illustrates an example of a method for generating an HDR image using multiple images acquired using at least one camera without ghosting distortion.

[0067] FIG. 1c illustrates an example of a method for generating an image (130) that is an HDR image and a file (110) for generating an HDR image without ghosting distortion, using images (112, 113) acquired according to multiple exposure shooting (or multiple exposure shooting technique).

[0068] For example, the electronic device (101) can acquire images (112, 113) using the at least one camera in a state where the HDR mode is set. In FIG. 1C, for convenience of explanation, it is assumed that shaking occurs in the images (112, 113) (or, the time between the timing at which the image (112) is acquired and the timing at which the image (113) is acquired). For example, when the shaking occurs, the first area of ​​the image (112) where the visual object (112a) in the image (112) is located may partially overlap with the second area of ​​the image (113) where the visual object (113a) in the image (113) is located. For example, when the shaking occurs, the position information of the visual object (112a) in the image (112) (or the position information indicating the first area) may not correspond to (or be identical to) the position information of the visual object (113a) in the image (113) (or the position information indicating the second area).

[0069] For example, when the electronic device (101) identifies that the shaking has occurred, it can generate an image (114) using the image (112) and the image (113). For example, the image (114) can be used to generate map information (115). The image (114) can be referred to as a virtual image, a fake image, or a virtual auxiliary image. For example, the image (114) can be generated by applying information of the image (113) to the image (112). Accordingly, the first region in which the visual object (112a) in the image (114) is located can correspond to (or be identical to) the first region in which the visual object (112a) in the image (112) is located. The visual object (112a) in the image (114) can be identical to the visual object (112a) in the image (112). For example, since the image (114) is generated by applying the information of the image (113), the brightness of the image (114) may be darker than the brightness of the image (112). At this time, the information of the image (113) may be referred to as global tone information or global color tone of the image (113). For example, the global tone information of the image may be related to (or used to define) a grayscale range for all pixels of the image. For example, the grayscale range may be determined according to at least one of hue, brightness, or contrast for all pixels of the image. For example, the image (114) may be generated by applying transformation information for adjusting the global tone information of the image (112) to the global tone information of the image (113) to the image (112). For example, for specific details on how to generate an image (114) using the above conversion information, reference may be made to FIG. 4b and FIG. 5.

[0070] For example, the electronic device (101) may generate a file (110) to support the HDR effect. For example, the electronic device (101) may generate the file (110) using images (112, 114). In the example of FIG. 1C, the file (110) includes map information (115), but the present disclosure is not limited thereto. For example, the file (110) may include metadata including map information (115).

[0071] As described above, since the map information (115) is generated by synthesizing the images (112, 114), even if shaking occurs in the images (112, 113) acquired (or photographed) using at least one camera, ghosting distortion may not occur in the map information (115). More specifically, since the map information (115) is generated using the image (112) and the image (114) in which the gradation range is changed and there is no positional deviation with respect to the image (112), the ghosting distortion may not occur. For example, the third region where the visual object (115a) of the map information (115) is located may correspond to the first region where the visual object (112a) of the image (112) is located. For example, the third area may indicate substantially the same location as the first area, depending on the ratio between the size of the map information (115) and the size of the image (112). For example, the size of the map information (115) may be smaller than the size of the image (112). The location information indicating the third area where the visual object (115a) is located may correspond to (or be identical to) the location information indicating the first area where the visual object (112a) is located. This may be because the location information indicating the third area and the location information indicating the first area each include relative location information.

[0072] For example, the electronic device (101) may generate an image (130), which is an HDR image, using the file (110) to support the HDR effect. At this time, since ghosting distortion is not caused in the map information (115), ghosting distortion may not be caused in the image (130) either. At this time, even if shaking is identified while acquiring images (112, 113) using at least one camera, the electronic device (101) may generate an image (114) using global tone information of the image (113) acquired according to different exposure values, thereby generating and storing an image (130) (or a file (110) for the image (130)) that provides a wider dynamic range. Although not illustrated in FIG. 1B, the electronic device (101) may display the image (130) on the display (103), as in the example of FIG. 1A. At this time, the brightness range of the display (103) for displaying the image (130) can be adjusted from a first brightness range corresponding to the gradation range of the image (130) to a second brightness range wider than the first brightness range according to the map information (115).

[0073] Figure 2 illustrates an exemplary block diagram of an electronic device.

[0074] The electronic device (101) of FIG. 2 may be an example of the electronic device (101) of FIG. 1A. For example, the electronic device (101) of FIG. 2 may correspond to or include at least a portion of the electronic device (1001) of FIG. 10.

[0075] Referring to FIG. 2, the electronic device (101) may include a processor (210), a camera (220), a memory (230), and a display (240). The hardware configuration of the electronic device (101) is not limited to the example of FIG. 2. For example, the electronic device (101) may further include at least some of the electronic components described with reference to FIG. 10. For example, some of the electronic components of FIG. 2 (e.g., the display (240)) may be excluded from the electronic device (101).

[0076] For example, the processor (210) may be operably coupled with the display (240) or display driver circuitry within the display (240) (e.g., display driver IC (integrated circuitry) (1130) of FIG. 11). For example, the processor (210) being operably coupled with the display (240) (or display driver circuitry) may indicate that the processor (210) is directly connected to the display (240) (or display driver circuitry). For example, the processor (210) being operably coupled with the display (240) (or display driver circuitry) may indicate that the processor (210) is connected to the display (240) (or display driver circuitry) via another component of the electronic device (101). For example, the fact that the processor (210) is operatively coupled with the display (240) (or the display driving circuit) may indicate that the state of the processor (210) is a state in which the display (240) (or the display driving circuit) can be controlled. For example, the fact that the processor (210) is operatively coupled with the display (240) (or the display driving circuit) may indicate that the operation of the display (240) (or the display driving circuit) is caused based on information, data, signals, or commands obtained from the processor (210). However, the present invention is not limited thereto. The processor (210) of FIG. 2 may be an example of the processor (1020) of FIG. 10.

[0077] For example, the processor (210) of the electronic device (101) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). For example, the number of processors may be one or more. The processing circuit of the processor that loads (or fetches) instructions and performs calculations corresponding to the loaded instructions may be referred to as or referred to as a core circuit (or core). For example, the processor may have a multi-core processor structure including a plurality of core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure may be performed individually or collectively by one or more processing circuits included in the processor (210).

[0078] For example, a display (240) of an electronic device (101) (e.g., display (103) of FIG. 1A) can output visualized information (e.g., image (112) or image (130) of FIG. 1A) to a user. For example, the display (240) can be controlled by a controller, such as a graphic processing unit (GPU), to output visualized information to the user. The display (240) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (240) can include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display (240) can have an at least partially curved shape or a deformable shape. A display (240) having a deformable shape may be referred to as a flexible display. The display (240) of FIG. 2 may be an example of the display module (1060) of FIG. 10.

[0079] For example, the display (240) of the electronic device (101) may include a sensor (e.g., a touch sensor panel (TSP)) for detecting an external object (e.g., a user's finger) on the display (240). For example, based on the TSP, the processor (210) may detect an external object that is in contact with the display (240) or floating on the display (240). In response to detecting the external object, the processor (210) may execute a function related to a specific visual object corresponding to a location of the external object on the display (240) among visual objects displayed on the display (240).

[0080] For example, the display (240) may include a display driving circuit and a display panel (e.g., the display panel (1110) of FIG. 11). For example, the display driving circuit may be operatively coupled to the display panel. For example, when the display panel includes a plurality of LEDs arranged in a two-dimensional matrix form, the display driving circuit may be configured to control at least one LED included in a corresponding row or column among the plurality of LEDs. The display driving circuit controlling the at least one LED may include an operation of adjusting the luminance (or light amount, brightness) of the LEDs. In the present disclosure, luminance may mean the intensity of light emitted from pixels of the display (240) (e.g., nit (or cd / cm2)). ) of light intensity measured in units of bit depth). In the present disclosure, brightness (or brightness range) may include a relative value (or relative values) expressed in bits of bit depth, referred to as brightness level, as brightness for an image to be displayed through the display (240).

[0081] For example, the memory (230) of the electronic device (101) may include a circuit and / or a storage medium for storing data and / or instructions input and / or output to the processor (210). The memory may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The non-volatile memory may be referred to as storage. The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC). The processor (210) of the electronic device (101) may execute instructions of the memory (230) within the electronic device (101) to perform functions and / or operations indicated by the instructions. For example, if the electronic device (101) includes at least one processor, the at least one processor may be configured to collectively or individually execute the instructions. The memory (230) of FIG. 2 may be an example of the memory (1030) of FIG. 10.

[0082] For example, the camera (220) of the electronic device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal representing the color and / or brightness of light. The plurality of optical sensors included in the camera (220) may be arranged in the form of a two-dimensional array. The camera (220) may acquire the electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (220) may mean one (a) two-dimensional frame data acquired from the camera (220). For example, video data captured using the camera (220) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (220). For example, the camera (220) may include at least one camera. The camera (220) of FIG. 2 may be an example of the camera module (1080) of FIG. 10.

[0083] Referring to FIG. 2, information (e.g., file (231)) and / or programs (e.g., image renderer (233) and / or display brightness determiner (235)) stored in a memory (230) of an electronic device (101) are illustrated. A processor (210) that executes instructions included in a program may perform functions and / or operations indicated by the instructions. A file (231) (e.g., file (110) of FIG. 1A or file (110) of FIG. 1C) may include color information representing colors of pixels of an image (e.g., image (112) of FIG. 1A or image (112) of FIG. 1C) according to a color space such as YUV, RGB, and / or HSV. For example, a file (231) based on a color space of RGB may represent the color of a specific pixel using the intensities of three primary colors of red, green, and blue. For example, a file (231) based on the color space of YUV can represent the color of a specific pixel using three components including a brightness component (e.g., Y component) and chrominance components (e.g., Cb component and / or Cr component).

[0084] For example, at least three channels can be used to represent the colors of pixels of an image. From a file (231) in JPEG format, the processor (210) can identify the colors of a plurality of pixels expressed by three channels having a bit depth of 8 bits. The file (231) can additionally include map information (e.g., map information (115) of FIG. 1A or map information (115) of FIG. 1C) corresponding to a specific component (e.g., brightness component). The map information corresponding to a specific component can be used to increase the bit depth of the specific component (e.g., to a number of bits greater than 8 bits).

[0085] By executing the image renderer (233), the processor (210) can generate or synthesize an image having an HDR effect (e.g., an image (130) of FIG. 1A or an image (130) of FIG. 1C) from an image having an SDR included in a file (231) (e.g., an image (112) of FIG. 1A or an image (112) of FIG. 1C). For example, the processor (210) can restore an image having an HDR effect from an image of an SDR using map information included in metadata of the file (231) (e.g., a map information (115) of FIG. 1A or a map information (115) of FIG. 1C). The restoration can include scaling (e.g., amplifying and / or attenuating) the brightness of pixels of the image of the SDR using information included in the map information. The processor (210) executing the image renderer (233) can generate or obtain information for displaying an image having a bit depth of HDR.

[0086] By executing the display luminance determiner (235), the processor (210) can increase, at least partially, the luminance of the pixels of the display (240). The processor (210) executing the display luminance determiner (235) can control the display driving circuit (122) to increase, at least partially, the luminance of the display (240). For example, in order to visually emphasize a portion of an image to be displayed on the display (240) over another portion, the processor (210) can cause (or control) the luminance of at least one pixel of the display (240) corresponding to the portion to exceed the luminance of at least one pixel of the display (240) corresponding to the other portion. With the display luminance determiner (235) executing, the processor (210) can determine the luminance increase amount of the entire or at least a portion of the display (240) by using map information (e.g., the map information (115) of FIG. 1A or the map information (115) of FIG. 1C). Using the increased luminance increase, the processor (210) can perform operations such as gamma adjustment.

[0087] The processor (210) that executes the display brightness determiner (235) can adjust the gamma of the image rendered by the image renderer (233). The gamma may refer to the relationship between the brightness of the pixels of the image and the brightness of the pixels of the display (240) that displays the image. The gamma may be expressed as a function referred to as a gamma curve. When displaying an HDR image synthesized from a file (231), the processor (210) may control the display driving circuit to display the image using the gamma related to the metadata (e.g., display control information) of the file (231).

[0088] As described above, according to one embodiment, the electronic device (101) can, when displaying a file (231) based on SDR, restore or synthesize an HDR image using metadata of the file (231). When displaying the synthesized image (or HDR image), the electronic device (101) can control the display (240) using metadata of the file (231) so that the brightness of the display (240) is at least partially increased.

[0089] Figure 3 illustrates an example of a flow of operations for displaying an HDR image while shaking multiple images within it.

[0090] At least some of the methods of FIG. 3 may be performed by the electronic device (101) of FIG. 2. For example, at least some of the methods may be controlled by the processor (210) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0091] In operation (300), the electronic device (101) may detect an event for displaying an image. For example, the event may include acquisition of a photographing input within a state in which the HDR mode is set. For example, the state in which the HDR mode is set may indicate a state in which an option (152) is activated within the screen (150) of FIG. 1A. For example, the photographing input may include an input for acquiring (or capturing) an image of an external environment while a camera application is running. For example, the electronic device (101) may detect acquisition of a photographing input within a state in which the HDR mode is set.

[0092] In operation (305), the electronic device (101) may acquire a plurality of images. For example, the electronic device (101) may acquire the images using the camera (220) based on detecting the event. For example, the electronic device (101) may acquire the images by changing the exposure value (or the property of the camera (220)) each time each of the images is acquired. Hereinafter, for convenience of explanation, it is assumed that the images include a first image (e.g., image (112) of FIGS. 1A, 1B, and 1C) and a second image (e.g., image (113) of FIGS. 1B and 1C). For example, the first image may be acquired using the camera (220) according to a first exposure value, and the second image may be acquired using the camera (220) according to a second exposure value lower than the first exposure value. For example, the first image may be referred to as a main image. For example, the second image may be referred to as a secondary image.

[0093] In operation (310), the electronic device (101) may determine whether shaking is identified within a plurality of images. For example, the electronic device (101) may determine whether shaking is identified within the first image and the second image. It is assumed that the first image includes a visual object corresponding to an external object (e.g., a visual object (112a) of FIGS. 1B and 1C), and the second image includes a visual object corresponding to the external object (e.g., a visual object (113a) of FIGS. 1B and 1C).

[0094] For example, the electronic device (101) can determine whether the position information of the visual object in the first image corresponds to (or is the same as) the position information of the visual object in the second image.

[0095] For example, the electronic device (101) may determine that the shaking is identified when the first area where the visual object is located in the first image partially overlaps the second area where the visual object is located in the second image. In other words, the electronic device (101) may determine that the shaking is identified when the first area and the second area are different. Or, for example, the electronic device (101) may determine that the shaking is identified when the location information indicating the first area where the visual object is located in the first image (or the location information of the visual object in the first image) and the location information indicating the second area where the visual object is located in the second image (or the location information of the visual object in the second image) are different (or do not correspond).

[0096] In contrast, the electronic device (101) may determine that the shaking is identified if the first region in which the visual object is located in the first image completely overlaps the second region in which the visual object is located in the second image. In other words, the electronic device (101) may determine that the shaking is not identified if the first region and the second region are the same (or substantially the same). Or, for example, the electronic device (101) may determine that the shaking is not identified if the location information indicating the first region in which the visual object is located in the first image (or the location information of the visual object in the first image) is the same as (or corresponds to) the location information indicating the second region in which the visual object is located in the second image (or the location information of the visual object in the second image).

[0097] The fact that the position information of the visual object in the first image does not correspond to the position information of the visual object in the second image may indicate that there is a deviation between the position of the visual object in the first image and the position of the visual object in the second image.

[0098] In the above example, it is described as determining whether the location of the visual object in the first image corresponds (or is the same) to the location of the visual object in the second image, but the present disclosure is not limited thereto. For example, if the difference (or deviation) between the first region where the visual object in the first image is located and the second region where the visual object in the second image is located is less than a reference difference, the location of the visual object in the first image may be determined to be substantially corresponding (or the same) to the location between the locations of the visual object in the second image (or there is no shaking). Alternatively, for example, if the difference (or deviation) between the location of the visual object in the first image and the location of the visual object in the second image is less than a reference difference, the location of the visual object in the first image may be determined to be substantially corresponding (or the same) to the location of the visual object in the second image.

[0099] For example, the position information indicating the first area where the visual object is located in the first image and the position information indicating the second area where the visual object is located in the second image may each include relative position information. For example, the relative position information may include information for indicating a relative position with respect to a visual object in the image. For example, the relative position information may include a ratio of the distance from a reference position (or reference coordinates) of the image to a position (or representative position) of the visual object. Or, for example, the relative position information may include a normalized value for the position (or representative position) of the visual object in the image. By using the relative position information, even when the sizes of the images are different, the electronic device (101) can compare the positions between the visual objects in the images. For example, an algorithm for obtaining the relative position information may include optical flow and oriented fast and rotated brief (ORB). In one example, the electronic device (101) can identify a motion value by measuring the distance between key points for a visual object (or any location or area). For example, the electronic device (101) can identify that shaking (or ghosting distortion) is likely to occur if the motion value exceeds a reference value.

[0100] In operation (310), the electronic device (101) may perform operation (315) upon determining that the shaking is not identified. Alternatively, in operation (310), the electronic device (101) may perform operation (320) upon determining that the shaking is identified.

[0101] In operation (315), the electronic device (101) may generate a third image using the first image and the second image. For example, if the shaking is not identified, the electronic device (101) may generate the third image using the first image and the second image, which are images acquired from the camera (220). For example, the third image may be generated using the grayscale range (or first global tone information) of the first image and the grayscale range (or second global tone information) of the second image.

[0102] For example, the third image may be referenced as map information (e.g., map information (115) of FIGS. 1A and 1C), an additional image, or a gain image. For example, the third image may be used as the map information to adjust the brightness range for displaying the first image from the first brightness range corresponding to the gradation range of the first image to a second brightness range wider than the first brightness range. For specific details related to a method for generating the third image when the shaking is not identified, reference may be made to FIG. 4A.

[0103] Figure 4a illustrates an example of a method for generating map information for an HDR image using multiple images.

[0104] FIG. 4A illustrates an example (400) of a method for obtaining map information (430) using a main image (410) and an auxiliary image (420). For example, the main image (410) may be the first image of FIG. 3 . For example, the auxiliary image (420) may be the second image of FIG. 3 . For example, the map information (430) may be the third image of FIG. 3 .

[0105] For example, the electronic device (101) can obtain a main image (410) and an auxiliary image (420) using the camera (220). For example, the main image (410) can be obtained according to the first exposure value. For example, the auxiliary image (420) can be obtained according to the second exposure value. For example, the auxiliary image (420) can be darker than the main image (410).

[0106] For example, if shaking is not identified within the main image (410) and the auxiliary image (420), the electronic device (101) may generate map information (430) by performing synthesis of the main image (410) and the auxiliary image (420). For example, the map information (430) may be darker than the main image (410). Although not illustrated in FIG. 4A, the electronic device (101) may generate a file including the main image (410) and the map information (430).

[0107] Referring back to FIG. 3, in operation (320), the electronic device (101) may generate a virtual image (e.g., image (114) of FIG. 1C) using the first image and the second image. If the shaking is identified, the electronic device (101) may generate a virtual image using the first image and the second image. For example, the virtual image may be generated from the first image using information of the second image.

[0108] In operation (325), the electronic device (101) may generate the third image using the first image and the virtual image. For example, if the shaking is identified, the electronic device (101) may generate the third image using the first image and the virtual image instead of generating the third image using the first image and the second image. For specific details regarding the method for generating the third image when the shaking is identified, reference may be made to FIG. 4B.

[0109] FIG. 4b illustrates an example of a method for generating a virtual image for a main image using an auxiliary image among a plurality of images, and generating map information for an HDR image using the main image and the virtual image.

[0110] FIG. 4B illustrates an example (450) of a method for generating a virtual image (470) using a main image (410) and an auxiliary image (460), and obtaining map information (480) using the main image (410) and the virtual image (470). For example, the main image (410) may be the first image of FIG. 3 . For example, the auxiliary image (460) may be the second image of FIG. 3 . Unlike the main image (410) of FIG. 4A , the auxiliary image (460) of FIG. 4B may be an image having a positional deviation with respect to the main image (410). For example, a second area in which a visual object is located in the auxiliary image (460) may partially overlap with a first area in which a visual object is located in the main image (410). The visual object in the auxiliary image (460) and the visual object in the main image (410) may be visual information corresponding to the same external object. For example, the map information (480) may be the third image of FIG. 3.

[0111] For example, the electronic device (101) can obtain a main image (410) and an auxiliary image (460) using the camera (220). For example, the main image (410) can be obtained according to the first exposure value. For example, the auxiliary image (460) can be obtained according to the second exposure value. For example, the auxiliary image (460) can be darker than the main image (410). For example, the electronic device (101) can generate a virtual image (470) when shaking is identified within the main image (410) and the auxiliary image (460).

[0112] For example, to generate a virtual image (470), the electronic device (101) may identify first global tone information based on the grayscale range of the main image (410) and second global tone information based on the grayscale range of the auxiliary image (460). For example, the first global tone information may represent information determined based on the hue, brightness, or contrast of each pixel of the main image (410). For example, the second global tone information may represent information determined based on the hue, brightness, or contrast of each pixel of the auxiliary image (460). For example, each of the first global tone information and the second global tone information may be related to luminance (and / or contrast) for expressing the image when the image is displayed.

[0113] For example, the electronic device (101) can identify (or learn, determine, or generate) transformation information (455) for adjusting the first global tone information of the main image (410) to the second global tone information of the auxiliary image (460). For example, the transformation information (455) can be referenced as a transformation function, a transformation value, or a transformation parameter for adjusting the first global tone information to the second global tone information. For example, the electronic device (101) can identify the transformation information (455) according to a histogram specification technique using a cumulative distribution function (CDF) for the first global tone information and a CDF for the second global tone information. Specific details related thereto are illustrated and described below with reference to FIG. 5.

[0114] Figure 5 illustrates an example of a method for generating a virtual image using global tone information of a main image and global tone information of an auxiliary image.

[0115] FIG. 5 illustrates an example (500) of a method for generating a virtual image (470) using a main image (410) and an auxiliary image (460) according to the histogram specification technique.

[0116] Referring to example (500), the electronic device (101) can identify a first CDF (510) based on the grayscale range of the main image (410) and identify a second CDF (520) based on the grayscale range of the auxiliary image (460). In example (500) of FIG. 5, for convenience of explanation, only an example of the second CDF (520) of the auxiliary image (460) is illustrated, but specific details regarding the second CDF (520) can be substantially equally applied to the first CDF (510).

[0117] For example, the second CDF (520) may be a histogram of the grayscale (or grayscale level) for all pixels of the auxiliary image (460). For example, the second CDF (520) may be identified by classifying all pixels of the auxiliary image (460) by grayscale level. For example, the electronic device (101) may identify a look-up table (LUT) (525) representing the grayscale according to the quantized cumulative frequency by performing an inverse transformation on the second CDF (520). For example, the LUT (525) may be referenced as the transformation information (455).

[0118] For example, the electronic device (101) can adjust the first CDF (510) to the second CDF (520) by using the LUT (525) representing the relationship between the gradation and the cumulative frequency on the histogram. For example, the electronic device (101) can identify the third CDF (530) by calculating the LUT (525) on the first CDF (510). For example, the third CDF (530) can be the CDF of the virtual image (470). For example, the third CDF (530) can have values ​​that are substantially the same as (or similar to) those of the second CDF (520).

[0119] Referring to the above, the electronic device (101) can generate the third CDF (530), which is global tone information of the virtual image (470), by adjusting the first CDF (510), which is global tone information of the main image (410), using the second CDF (520), which is global tone information of the auxiliary image (460).

[0120] Referring back to FIG. 4b, FIG. 5 illustrates a case where transformation information (455) is identified according to the histogram specification technique, but the present disclosure is not limited thereto. For example, the electronic device (101) may identify transformation information (455) based on a neural network trained to input the first global tone information and the second global tone information and output the transformation information.

[0121] For example, even if the contrast (or grayscale range) of the main image (410) and the contrast (or grayscale range) of the auxiliary image (460) are different, the higher the correlation between the grayscale of the pixel of the auxiliary image (460) corresponding to the pixel of the main image (410) and the grayscale of the pixel of the main image (410), the more likely it is that the conversion information (455) that causes the first global tone information to approximate the second global tone information can be generated. For example, the correlation between the grayscale of the pixel of the auxiliary image (460) corresponding to the pixel of the main image (410) and the grayscale of the pixel of the main image (410) can include linearity or nonlinearity between the grayscales. For example, if the relationship between the grayscale of a pixel of an auxiliary image (460) corresponding to a pixel of a main image (410) and the grayscale of a pixel of a main image (410) is linear or nonlinear, the first global tone information can be adjusted to be substantially identical to the second global tone information by using the conversion information (455).

[0122] For example, the electronic device (101) can generate a virtual image (470) by applying the identified conversion information (455) to the main image (410). For example, since the virtual image (470) is generated by applying the conversion information (455) to the main image (410), it can have the same location information as the location information of the main image (410). For example, the area where a visual object is located in the virtual image (470) can be the first area where a visual object is located in the main image (410). In addition, the virtual image (470) can have a different gradation range (or the second global tone information) than the gradation range (or the first global tone information) of the main image (410). For example, the virtual image (470) can be darker than the main image (410). In one example, the virtual image (470) may have global tone information that is substantially the same as (or similar to) the second global tone information of the auxiliary image (460).

[0123] In one example, the electronic device (101) may generate a virtual image (470) from the main image (410) by partially using the second global tone information of the auxiliary image (460) when the second global tone information of the auxiliary image (460) indicates a brightness below a certain level (or when the auxiliary image (460) is darker than a certain level). Partially using the second global tone information may include correcting and using the second global tone information of the auxiliary image (460), using global tone information identified after processing the auxiliary image (460) to make it brighter, or modifying conversion information (455) generated based on the second global tone information of the auxiliary image (460).

[0124] For example, the electronic device (101) can generate map information (480) by synthesizing the main image (410) and the virtual image (470). Unlike FIG. 4A, the electronic device (101) can generate map information (480) by synthesizing the virtual image (470) with the main image (410) instead of the auxiliary image (460) when the shaking is identified. Although not illustrated in FIG. 4B, for example, the electronic device (101) can generate a file including the main image (410) and the map information (480).

[0125] Referring back to FIG. 3, in operation (315) or operation (325), the electronic device (101) may generate the third image. For example, a weight value (or pixel value) of the third image may be adjusted according to an exposure value (e.g., the second exposure value) used to obtain the second image. For example, each of the weight values ​​of the third image may be a weight value to be applied to one or more pixels of the first image. For example, each of the weight values ​​of the third image may be set to a lower value as the second exposure value used to obtain the second image decreases. This may be to prevent (or remove, limit) distortion (or spot distortion) that may occur because the lower the second exposure value, the greater the contrast between a relatively bright area (or highlight area) and a relatively dark area (or non-highlight area) of the second image. In other words, the electronic device (101) can generate (or modify) the third image by adjusting at least some of the weight values ​​of the third image according to the second exposure value when generating the third image.

[0126] In operation (330), the electronic device (101) may store a file including the first image and the third image. For example, the electronic device (101) may store the file including the first image (e.g., the main image (410) of FIG. 4A) and the third image (e.g., the map information (430)) within the electronic device (101) (or the memory (230)) according to operation (315). Alternatively, for example, the electronic device (101) may store the file including the first image (e.g., the main image (410) of FIG. 4B) and the third image (e.g., the map information (480)) within the electronic device (101) (or the memory (230)) according to operations (320) and (325).

[0127] In operation (335), the electronic device (101) may display the first image based on the stored file. For example, the electronic device (101) may display the first image, which is an SDR image, on the display (240) based on the first image of the stored file. Or, for example, the electronic device (101) may display the first image having an HDR effect on the display (240) based on the first image and the third image of the stored file. For example, the brightness range of the display (240) for displaying the first image, which is an SDR image, may be narrower than the brightness range of the display (240) for displaying the first image having the HDR effect.

[0128] Figure 6 illustrates an exemplary structure of a file generated by an electronic device.

[0129] The electronic device (101) of FIGS. 1A and 2, and / or the processor (210) of FIG. 2, may perform the operation of the electronic device (101) described with reference to FIG. 6. The file (600) of FIG. 6 may be generated by the operation of the electronic device (101) described with reference to FIGS. 3, 4A, 4B, and 5. For example, the file (600) may be an example of the file (110) of FIG. 1A or 1C.

[0130] Referring to FIG. 6, the structure of a file (600) based on the format of the International Standardization Organization (ISO) referred to as EXIF ​​is illustrated. The file (600) stored in a memory (e.g., memory (230) of FIG. 2) may start from an area (M1) in which a designated value (e.g., a value of Table 1 below) indicating the start of the file (600) is stored. After the area (M1), application areas (APPlication segments) (e.g., M2, M3, M4, ..., M10) may be formed within the file (600).

[0131] Tag information may be stored in the first application area (M2) of the file (600). The tag information may include one or more character strings used for indexing the file (600). Content list data (M3) and / or stream data (M4 to M10) may be stored in the second application area (APP2) of the file (600). Within the file (600), after the application areas (M2, M3, ..., M10), a JPEG table area (M11, M12, ..., M15) may be formed. The JPEG table area may include a Define-Quantization-Tables (DQT) area (M11) (e.g., a variable area starting with a value in Table 1 below), a Define-Huffman-Tables (DHT) area (M12) (e.g., a variable area starting with a value in Table 1 below), a Define-Restart-Interval (DRI) area (M13) (e.g., a fixed area starting with a value in Table 1 below), a Start-of-Frame (SOF) area (M14), and / or a Start-Of-Scan (SOS) area (M15) (e.g., a variable area starting with a value in Table 1 below). Within the file (600), a JPEG compressed data area (M16) may be formed after the JPEG table area. The file (600) may include, after the JPEG compressed data area (M16), an area (M17) in which a designated value is stored to indicate the end of the file (600) (e.g., a fixed area starting with the value in Table 1 below).

[0132] In one embodiment, information stored in a file (600) based on the format of EXIF ​​is not limited to the example of FIG. 6. For example, the file (600) may be a high efficiency image file format (HEIF) file, a high efficiency image container (HEIC) file, a file for storing raw data (e.g., a digital negative image (DNG) file), a portable network graphic (PNG) file, and / or a graphics interchange format (GIF) file. As a non-limiting example, a file (600) that is a HEIF file or an HEIC file may include information of 8 bits or more. For example, the file (600) may include information having a name in Table 1.

[0133] Abbreviation file (600) Value (hexadecimal based) Length of information (or payload) Name SOI 0xFF, 0xD8 None Start of image SOF 00xFF, 0xC0 Variable Start of frame (baseline DCT (discrete cosine transform)) SOF 2 0xFF, 0xC2 Variable Start of frame (progressive DCT) DHT 0xFF, 0xC4 Variable Definition of Huffman table DQT 0xFF, 0xDB Variable Definition of quantization table DRI 0xFF, 0xDD4 Definition of byte restart period SOS 0xFF, 0xDA Variable Start of scan RSTn 0xFF, 0xDn (n = 0, ..., 7) None Restart point APPn 0xFF, 0xEn Variable Application area (EXIF, APP1, etc.) COM 0xFF, 0xFE variable comment EOI 0xFF, 0xD9 none End point of image

[0134] In one embodiment, a main image (610) (e.g., image (112) of FIGS. 1A and 1C, or main image (410) of FIGS. 4A and 4B) among images acquired from at least one camera (e.g., camera (220) of FIG. 2) may be stored in a JPEG compressed data area (M16) within a file (600). Metadata may be stored in another area (e.g., M2 to M10) of the file (600) that is different from the JPEG compressed data area (M16). For example, shooting information of a camera used to restore and / or display an HDR image, a third image (620) (e.g., map information (115) of FIGS. 1A and 1C, map information (430) of FIG. 4A, or map information (480) of FIG. 4B), and / or display control information may be stored in a second application area (APP2) of the file (600).

[0135] Figure 7a shows an example of map information in which distortion has been formed.

[0136] Figure 7a illustrates an example of a distortion (705) formed within map information (700) generated using a main image. For example, the map information (700) may be referenced as an additional image, or a third image.

[0137] For example, map information (700) can be generated by synthesizing a main image and an auxiliary image (or a virtual image). For example, map information (700) can be generated by remapping global tone information of an auxiliary image (or a virtual image) with respect to the main image. The brightness of each pixel of each image can be expressed using a specific pipeline. For example, when using the specific pipeline, the brightness of each pixel of each image can be expressed as a binary value having a bit depth of 8 bits per channel. However, in generating the map information (700), the brightness of each pixel of the map information (700) can be expressed using at least 9 bits according to the synthesis of the main image defined using an 8-bit pipeline and the auxiliary image defined using an 8-bit pipeline. However, since the number (or size) of bits used for displaying an image on the display (240) is 8 bits, the 9 bits for expressing the map information (700) need to be compressed again into 8 bits. At this time, when the compression is performed, the step between the gradations within the highlight area (701) (or bright area) of the map information (700) may increase, and the resulting contrast may be perceived by the user as distortion (705). Similarly, when the compression is performed, the step between the gradations within the non-highlight area (702) (or dark area) of the map information (700) also increases, but the resulting contrast may not be relatively perceived by the user.

[0138] For specific details on the flattening process for the highlight area (701) performed to correct distortion (705) (or, stain, stain distortion), reference may be made to FIGS. 7b and 7c.

[0139] Figure 7b illustrates an example of an operational flow for a method of generating map information based on a flattening process.

[0140] At least some of the methods of FIG. 7B may be performed by the electronic device (101) of FIG. 2. For example, at least some of the methods may be controlled by the processor (210) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0141] In operation (710), the electronic device (101) may reduce the brightness levels of the composite image. For example, the composite image may be an image obtained by synthesizing a first image (e.g., the main image (410) of FIGS. 4A and 4B) and a second image (e.g., the auxiliary image (420) of FIG. 4A or the virtual image (470) of FIG. 4B). In one example, the composite image may be referenced as map information (or an additional image, a third image) before being changed according to the flattening process. For example, the electronic device (101) may obtain a first processed image based on reducing the brightness levels of a brightness component (e.g., a Y component) of the composite image. Although not illustrated in FIG. 7B, the electronic device (101) may also adjust the size of the brightness value (or Y value) of the first processed image. For example, the electronic device (101) can resize (or reduce) the luminance value of the first processed image to 1 / 4.

[0142] In operation (721), the electronic device (101) may perform binarization on the first processed image. For example, the electronic device (101) may set the brightness level to a first value (e.g., '1') based on whether the brightness level exceeds a reference brightness level. For example, the electronic device (101) may set the brightness level to a second value (e.g., '0') based on whether the brightness level is less than or equal to the reference brightness level. The electronic device (101) may perform binarization by setting each of the brightness values ​​of the pixels of the first processed image to one of the first value and the second value.

[0143] In operation (722), the electronic device (101) may perform a morphology operation. For example, the electronic device (101) may reduce noise (e.g., a misidentified light source) based on reducing and then increasing the size of the first processed image on which operation (721) is performed. For example, the electronic device (101) may reduce noise (e.g., a misidentified light source) based on performing dilation and / or erosion of the first processed image on which operation (721) is performed according to the morphology operation.

[0144] In operation (723), the electronic device (101) may perform a connected component operation. For example, the electronic device (101) may set labels of objects in the first processed image for which operation (722) was performed based on performing the connected component operation. The electronic device (101) may obtain a label map of objects included in the first processed image for which operation (722) was performed based on performing the connected component operation. In one example, the electronic device (101) may perform correction on the label map. The electronic device (101) may perform correction on the label map based on an aspect ratio or size. In one example, the electronic device (101) can identify (or obtain, generate) statistical values ​​(e.g., average values) of labels included in the label map by further utilizing resized luminance values ​​(e.g., 1 / 4Y) and / or RGB values ​​(e.g., 1 / 4 RGB) along with the corrected (or redefined) label map.

[0145] In operation (724), the electronic device (101) can identify a non-light source. For example, the electronic device (101) can identify a non-light source based on statistical values ​​(e.g., average values) of labels included in the label map.

[0146] In operation (725), the electronic device (101) may correct the brightness levels of each label. For example, the electronic device (101) may identify a ceiling line (or maximum value) of the brightness levels of the labels included in the label map. For example, the electronic device (101) may change the brightness levels of the labels to the ceiling line (or maximum value).

[0147] In operation (726), the electronic device (101) may apply a weight to the brightness levels of each of the labels. For example, the electronic device (101) may apply a weight to the brightness levels of each of the labels based on at least one of the flatness of the brightness levels of each of the labels, the average value of the brightness levels, and / or the amount of gradation change. By performing operation (726), the electronic device (101) may obtain a second processed image.

[0148] In operation (730), the electronic device (101) may increase the brightness levels of the second processed image. For example, the electronic device (101) may remove noise and / or distortion included in the composite image by increasing the brightness levels of the second processed image again (e.g., x4). The electronic device (101) may obtain map information (or additional image) based on removing noise and / or distortion included in the composite image.

[0149] The map information (or additional image) acquired in operation (730) according to the flattening process of FIG. 7b may be referred to as map information (or additional image) with distortion removed. A specific example of map information changed according to the flattening process may be referred to FIG. 7c.

[0150] Figure 7c shows an example of map information changed according to the flattening process.

[0151] Referring to FIG. 7C, the electronic device (101) can convert the image (751) into the image (752) based on the flattening process. For example, the image (751) can be an example of a composite image (or map information (or additional image, third image) before being changed according to the flattening process). For example, the image (752) can be an example of map information (or changed map information (or additional image, third image)).

[0152] For example, the image (751) may include at least one object including a visual object (761). The visual object (761) may represent an external object (e.g., a fluorescent light). Even when the visual object (761) represents the same external object, the grayscale range of the visual object (761) may be configured to be wider than a specified grayscale range. If the grayscale range is wide, the visual object (761) may be displayed in a blotchy form. For example, the grayscale levels of an area (763) of the visual object (761) and the grayscale levels of an area (764) may be distinguished. If the grayscale levels of an area (763) and the grayscale levels of an area (764) are distinguished, a blotchy distortion may be formed within the visual object (761). Therefore, the electronic device (101) may perform a smoothing process to relatively narrow the grayscale range of the visual object (761).

[0153] The graph (770) represents the detection frequency according to the grayscale level of the visual object (761). The visual object (761) may have grayscale levels in the grayscale range (771). The electronic device (101) may perform a flattening process on the image (751). The electronic device (101) may obtain the image (752) based on the flattening process. For example, the electronic device (101) may obtain the image (752) by changing the visual object (761) into the visual object (762). The graph (780) represents the detection frequency according to the grayscale level of the visual object (762). The visual object (762) may have grayscale levels in the grayscale range (781). The grayscale range (781) may be narrower than the grayscale range (771).

[0154] The electronic device (101) can obtain a visual object (762) having identical or similar grayscale levels by narrowing the grayscale range of the visual object (761) through a flattening process. When the grayscale levels of the visual object (762) are configured identically or similarly, the visual object (762) may not be distorted in a blotchy form.

[0155] As described above, as the contrast within a highlight area (e.g., a visual object (761)) increases, the user may perceive that distortion occurs in the highlight area. The electronic device (101) may perform smoothing to prevent the user from perceiving the distortion (or spot distortion) by reducing the contrast within the highlight area. While FIGS. 7A to 7C illustrate a smoothing process performed to correct (or remove, prevent, or reduce) distortion, the present disclosure is not limited thereto. To correct distortion, filtering may be performed on transformation information used when generating a virtual auxiliary image using the main image. For example, specific details on a method for identifying filtered transformation information and removing distortion accordingly may be referred to FIGS. 8A and 8B below.

[0156] Figure 8a illustrates an example of an operational flow for a method of generating a virtual image using filtered transformation information.

[0157] At least some of the methods of FIG. 8A may be performed by the electronic device (101) of FIG. 2. For example, at least some of the methods may be controlled by the processor (210) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0158] In operation (810), the electronic device (101) can identify a partial range of the grayscale range of the image. For example, the electronic device (101) can identify the grayscale range of the image, which is a main image among images acquired using the camera (220). For example, the image, which is a main image, can be referred to as a first image. For example, a partial range of the grayscale range can be identified. For example, when the grayscale range representing the range of grayscale levels constituting the image is 0 to 255, the partial range can represent a part included within the grayscale range. In one example, the partial range can be between 200 and 255.

[0159] In operation (820), the electronic device (101) can identify filtered transformation information for distributing the grayscale levels within the partial range more evenly within the partial range. For example, the electronic device (101) can identify the transformation information for adjusting the global tone information of the image, which is the main image, to the global tone information of another image, which is an auxiliary image among the plurality of images. For example, the other image, which is the auxiliary image, can be referred to as a second image. For example, the electronic device (101) can distribute the grayscale levels within the partial range more evenly within the partial range by performing filtering on the transformation information.

[0160] In operation (830), the electronic device (101) may generate a virtual image by applying filtered transformation information to the image. For example, the virtual image may be generated to generate map information when shaking is identified in images acquired using the camera (220).

[0161] For example, by performing filtering on the above transformation information, the grayscale levels within a subrange of the grayscale range of the virtual image can be more evenly distributed within the subrange. For specific details related thereto, reference may be made to FIG. 8B below.

[0162] Figure 8b shows an example of a graph representing the grayscale range of a virtual image generated according to filtered transformation information.

[0163] The graph (850) of FIG. 8B illustrates a first set (860) representing a grayscale range of a virtual image generated by applying transformation information to a main image, and a second set (870) representing a grayscale range of a virtual image generated by applying filtered transformation information to the main image. For example, each of the horizontal and vertical axes of the graph (850) may be a grayscale level (e.g., 0 to 255) (or a grayscale value). It is assumed that the grayscale range of each of the first set (860) and the second set (870) of the graph (850) is 0 to 255. However, the present disclosure is not limited thereto. For example, depending on the grayscale levels constituting each virtual image, the grayscale range may be determined from a grayscale level exceeding 0 to a grayscale level less than 255.

[0164] Referring to graph (850), a partial range (880) of the first set (860) and the second set (870) may include grayscale levels between 240 and 255. For example, the partial range (880) may include grayscale levels of a highlight area within a lyric image. For example, within a range outside of the partial range (880), the grayscale levels of the first set (860) may be substantially similar to the grayscale levels of the second set (870).

[0165] For example, within a sub-range (880), a first set (860) may include grayscale levels (861, 862). For example, within a sub-range (880), a second set (870) may include grayscale levels (871).

[0166] For example, the interval between the tone levels within the tone levels (861, 862) within the first set (860) may be relatively larger than the interval between the tone levels within the tone levels (871) within the second set (870). For example, within the sub-range (880), the tone levels (861) within the first set (860) may be distributed spaced apart from the tone levels (862). In contrast, within the sub-range (880), the tone levels (871) within the second set (870) may be relatively evenly distributed. In other words, the interval (or difference, deviation) between the tone levels within the tone levels (861, 862) may be relatively larger than the interval between the tone levels within the tone levels (871).

[0167] Referring to FIG. 8b, the grayscale levels of a virtual image generated by applying filtered transformation information to a main image may be relatively more evenly distributed compared to the grayscale levels of a virtual image generated by applying unfiltered transformation information to the main image. Accordingly, distortion (or blotch, blotch distortion) in map information generated by synthesizing the virtual image and the main image, and in an HDR image generated using the map information and the main image may be relatively less noticeable to a user.

[0168] Figure 9 illustrates an example of an operational flow for a method of storing a file containing an additional image used as map information for adjusting a brightness range for displaying an image.

[0169] At least some of the methods of FIG. 9 may be performed by the electronic device (101) of FIG. 2. For example, at least some of the methods may be controlled by the processor (210) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0170] In operation (910), the electronic device (101) may acquire a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region. For example, the electronic device (101) may acquire the first image and the second image using a camera (220).

[0171] For example, the electronic device (101) may detect an event for displaying an image. For example, the event may include acquisition of a photographing input within a state in which the HDR mode is set. For example, the state in which the HDR mode is set may indicate a state in which an option (152) is activated within the screen (150) of FIG. 1A. For example, the photographing input may include an input for acquiring (or capturing) an image of the external environment while a camera application is running. For example, the electronic device (101) may detect acquisition of a photographing input within a state in which the HDR mode is set.

[0172] For example, the electronic device (101) may acquire the first image and the second image using the camera (220) based on detecting the event. For example, the electronic device (101) may acquire the first image and the second image by changing an exposure value (or an attribute of the camera (220)). For example, the first image may be acquired using the camera (220) according to a first exposure value, and the second image may be acquired using the camera (220) according to a second exposure value lower than the first exposure value. For example, the second image may be darker than the first image. For example, the first image may be referred to as a main image. For example, the second image may be referred to as a secondary image.

[0173] In operation (920), the electronic device (101) may generate a third image including the visual object within a third area corresponding to the first area, using map information available for adjusting the brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range. For example, the electronic device (101) may generate the third image based on acquiring the first image and the second image.

[0174] For example, the electronic device (101) can determine whether the location information of the visual object in the first image corresponds to (or is the same as) the location information of the visual object in the second image. For example, the electronic device (101) can determine that the location information of the visual object in the first image is different from the location information of the visual object in the second image by identifying that the first area where the visual object in the first image is located partially overlaps the second area where the visual object in the second image is located.

[0175] In the above example, the shaking is identified by identifying whether the first region in the first image where the visual object is located and the second region in the second image where the visual object is located partially overlap, but the present disclosure is not limited thereto. For example, if the difference (or deviation) between the first region in the first image where the visual object is located and the second region in the second image where the visual object is located is less than a reference difference, the location of the visual object in the first image may be determined to substantially correspond to (or be identical to) (or be free of shaking) the location between the locations of the visual object in the second image.

[0176] For example, each of the position information indicating the first area where the visual object is located in the first image and the position information indicating the second area where the visual object is located in the second image may include relative position information. For example, the relative position information may include information for indicating a relative position with respect to a visual object in the image. For example, the relative position information may include a ratio of the distance from a reference position (or reference coordinates) of the image to a position (or representative position) of the visual object. Or, for example, the relative position information may include a normalized value for the position (or representative position) of the visual object in the image. By using the relative position information, even when the sizes of the images are different, the electronic device (101) can compare the positions between the visual objects in the images. For example, an algorithm for obtaining the relative position information may include optical flow and oriented fast and rotated brief (ORB). In one example, the electronic device (101) can identify a motion value by measuring the distance between key points for a visual object (or any location or area). For example, the electronic device (101) can identify that shaking (or ghosting distortion) is likely to occur if the motion value exceeds a reference value.

[0177] For example, the electronic device (101) may generate a virtual image (e.g., image (114) of FIG. 1C) using the first image and the second image based on determining that shaking is identified by identifying that the first region and the second region partially overlap. If the shaking is identified, the electronic device (101) may generate a virtual image using the first image and the second image. For example, the virtual image may be generated from the first image using information of the second image.

[0178] For example, in order to generate the virtual image, the electronic device (101) may identify first global tone information based on the grayscale range of the first image and second global tone information based on the grayscale range of the second image. For example, the first global tone information may represent information determined based on the hue, brightness, or contrast of each pixel of the first image. For example, the second global tone information may represent information determined based on the hue, brightness, or contrast of each pixel of the second image. For example, each of the first global tone information and the second global tone information may be related to luminance (and / or contrast) for expressing the image when the image is displayed.

[0179] For example, the electronic device (101) can identify transformation information for adjusting the first global tone information of the first image to the second global tone information of the second image. For example, the transformation information can be referenced as a transformation function, a transformation value, and a transformation parameter for adjusting the first global tone information to the second global tone information. For example, the electronic device (101) can identify the transformation information according to a histogram specification technique using a cumulative distribution function (CDF) for the first global tone information and a CDF for the second global tone information.

[0180] Alternatively, for example, the electronic device (101) may identify the transformation information based on a neural network trained to use the first global tone information and the second global tone information as inputs and output the transformation information.

[0181] For example, the electronic device (101) can generate the virtual image by applying the identified transformation information to the first image. For example, since the virtual image is generated by applying the transformation information to the first image, it can have the same location information as the location information of the first image. For example, the area where the visual object is located in the virtual image can be the first area where the visual object is located in the first image. For example, the virtual image can be darker than the first image.

[0182] For example, the electronic device (101) may generate the third image using the first image and the virtual image. For example, when the shaking is identified, the electronic device (101) may generate the third image using the first image and the virtual image instead of generating the third image using the first image and the second image. As a non-limiting example, the third image may be darker than the first image. Also, as a non-limiting example, the third image may be generated to have a brighter value than the first image.

[0183] In operation (930), the electronic device (101) may store a file including the first image and the third image. For example, the electronic device (101) may generate the file including the first image and the third image and store it in the electronic device (101) (or memory (230)).

[0184] Although not illustrated in FIG. 9, for example, the electronic device (101) may display the first image based on the stored file. For example, the electronic device (101) may display the first image, which is an SDR image, on the display (240) based on the first image of the stored file. Or, for example, the electronic device (101) may display the first image having an HDR effect on the display (240) based on the first image and the third image of the stored file. For example, the brightness range of the display (240) for displaying the first image, which is an SDR image, may be narrower than the brightness range of the display (240) for displaying the first image having the HDR effect.

[0185] Referring to FIGS. 1A to 9, the present disclosure can improve the brightness (or luminance) of a display (240) by generating map information at a level similar to that in the absence of shaking, even when shaking occurs between images acquired according to a multiple exposure shooting technique for HDR effect.

[0186] For example, an image having an HDR effect can be generated based on one image (or a main image) among a plurality of images when an external object (or subject) moves rapidly while continuously capturing a plurality of images according to a multiple exposure photography technique. Alternatively, an image having an HDR effect can be generated based on a plurality of images when an external object (or subject) is stationary while continuously capturing a plurality of images according to a multiple exposure photography technique. In contrast, the present disclosure can generate an image having an HDR effect based on a plurality of images regardless of the movement of the external object. For example, the present disclosure can generate a virtual image by applying information of an auxiliary image to a main image among the plurality of images, and then generate the image having an HDR effect. Therefore, in the present disclosure, since the same information (e.g., information of an auxiliary image) is used regardless of the movement of the external object, the global tone information of an image having an HDR effect when there is movement can be similar to the global tone information of an image having an HDR effect when there is no movement.

[0187] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0188] As described above, the present disclosure can be applied to electronic devices disclosed in the drawings below.

[0189] FIG. 10 is a block diagram of an electronic device within a network environment according to various embodiments.

[0190] Referring to FIG. 10, in a network environment (1000), an electronic device (1001) may communicate with an electronic device (1002) via a first network (1098) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1004) or a server (1008) via a second network (1099) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1001) may communicate with the electronic device (1004) via the server (1008). According to one embodiment, the electronic device (1001) may include a processor (1020), a memory (1030), an input module (1050), an audio output module (1055), a display module (1060), an audio module (1070), a sensor module (1076), an interface (1077), a connection terminal (1078), a haptic module (1079), a camera module (1080), a power management module (1088), a battery (1089), a communication module (1090), a subscriber identification module (1096), or an antenna module (1097). In some embodiments, the electronic device (1001) may omit at least one of these components (e.g., the connection terminal (1078)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1076), camera module (1080), or antenna module (1097)) may be integrated into a single component (e.g., display module (1060)).

[0191] The processor (1020) may, for example, execute software (e.g., a program (1040)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1001) connected to the processor (1020) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1020) may store commands or data received from other components (e.g., a sensor module (1076) or a communication module (1090)) in the volatile memory (1032), process the commands or data stored in the volatile memory (1032), and store result data in the non-volatile memory (1034). According to one embodiment, the processor (1020) may include a main processor (1021) (e.g., a central processing unit or an application processor) or a secondary processor (1023) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1021). For example, when the electronic device (1001) includes the main processor (1021) and the secondary processor (1023), the secondary processor (1023) may be configured to use less power than the main processor (1021) or to be specialized for a given function. The secondary processor (1023) may be implemented separately from the main processor (1021) or as a part thereof.

[0192] The auxiliary processor (1023) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1060), the sensor module (1076), or the communication module (1090)) of the electronic device (1001), for example, on behalf of the main processor (1021) while the main processor (1021) is in an inactive (e.g., sleep) state, or together with the main processor (1021) while the main processor (1021) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1023) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1080) or a communication module (1090)). In one embodiment, the auxiliary processor (1023) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1001) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1008)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0193] The memory (1030) can store various data used by at least one component (e.g., the processor (1020) or the sensor module (1076)) of the electronic device (1001). The data can include, for example, software (e.g., the program (1040)) and input data or output data for commands related thereto. The memory (1030) can include volatile memory (1032) or non-volatile memory (1034).

[0194] The program (1040) may be stored as software in memory (1030) and may include, for example, an operating system (1042), middleware (1044), or an application (1046).

[0195] The input module (1050) can receive commands or data to be used in a component of the electronic device (1001) (e.g., a processor (1020)) from an external source (e.g., a user) of the electronic device (1001). The input module (1050) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0196] The audio output module (1055) can output audio signals to the outside of the electronic device (1001). The audio output module (1055) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0197] The display module (1060) can visually provide information to an external party (e.g., a user) of the electronic device (1001). The display module (1060) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0198] The audio module (1070) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1070) can acquire sound through the input module (1050), output sound through the sound output module (1055), or an external electronic device (e.g., electronic device (1002)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1001).

[0199] The sensor module (1076) can detect the operating status (e.g., power or temperature) of the electronic device (1001) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1076) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0200] The interface (1077) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1001) with an external electronic device (e.g., the electronic device (1002)). In one embodiment, the interface (1077) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0201] The connection terminal (1078) may include a connector through which the electronic device (1001) may be physically connected to an external electronic device (e.g., the electronic device (1002)). According to one embodiment, the connection terminal (1078) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0202] The haptic module (1079) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1079) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0203] The camera module (1080) can capture still images and videos. According to one embodiment, the camera module (1080) may include one or more lenses, image sensors, image signal processors, or flashes.

[0204] The power management module (1088) can manage power supplied to the electronic device (1001). According to one embodiment, the power management module (1088) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0205] A battery (1089) may power at least one component of the electronic device (1001). In one embodiment, the battery (1089) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0206] The communication module (1090) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1001) and an external electronic device (e.g., electronic device (1002), electronic device (1004), or server (1008)), and the performance of communication through the established communication channel. The communication module (1090) may operate independently from the processor (1020) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1090) may include a wireless communication module (1092) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1094) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module can communicate with an external electronic device (1004) via a first network (1098) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1099) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1092) can verify or authenticate the electronic device (1001) within a communication network such as the first network (1098) or the second network (1099) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1096).

[0207] The wireless communication module (1092) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1092) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1092) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1092) may support various requirements specified in the electronic device (1001), an external electronic device (e.g., the electronic device (1004)), or a network system (e.g., the second network (1099)). According to one embodiment, the wireless communication module (1092) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

[0208] The antenna module (1097) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1097) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1097) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1098) or the second network (1099), may be selected from the plurality of antennas by, for example, the communication module (1090). A signal or power may be transmitted or received between the communication module (1090) and an external electronic device through the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1097).

[0209] According to various embodiments, the antenna module (1097) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0210] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0211] According to one embodiment, commands or data may be transmitted or received between the electronic device (1001) and an external electronic device (1004) via a server (1008) connected to a second network (1099). Each of the external electronic devices (1002 or 1004) may be the same or a different type of device as the electronic device (1001). According to one embodiment, all or part of the operations executed in the electronic device (1001) may be executed in one or more of the external electronic devices (1002, 1004, or 1008). For example, when the electronic device (1001) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1001) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1001). The electronic device (1001) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1001) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (1004) may include an Internet of Things (IoT) device. The server (1008) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1004) or server (1008) may be included in the second network (1099). The electronic device (1001) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0212] FIG. 11 is a block diagram of a display module according to various embodiments.

[0213] Referring to FIG. 11, a display module (1060) may include a display panel (1110) and a display driver IC (DDI) (1130) for controlling the same. The DDI (1130) may include an interface module (1131), a memory (1133) (e.g., a buffer memory), an image processing module (1135), or a mapping module (1137). The DDI (1130) may receive, for example, image data or image control signals corresponding to commands for controlling the image data, from other components of the electronic device (1001) through the interface module (1131). For example, according to one embodiment, image information may be received from a processor (1020) (e.g., a main processor (1021) (e.g., an application processor) or an auxiliary processor (1023) (e.g., a graphics processing unit) that operates independently of the function of the main processor (1021). The DDI (1130) may communicate with a touch circuit (1150) or a sensor module (1076) through the interface module (1131). In addition, the DDI (1130) may store at least a part of the received image information in the memory (1133), for example, in units of frames. The image processing module (1135) may, for example, perform preprocessing or postprocessing (e.g., resolution, brightness, or size adjustment) on at least a part of the image data based at least on the characteristics of the image data or the characteristics of the display panel (1110). The mapping module (1137) may output a voltage value corresponding to the image data preprocessed or postprocessed through the image processing module (1135). Alternatively, a current value may be generated. In one embodiment, the generation of the voltage value or current value may be performed at least in part based on, for example, the properties of the pixels of the display panel (1110), such as the arrangement of the pixels (RGB stripe or pentile structure), or the size of each sub-pixel.At least some pixels of the display panel (1110) may be driven based at least in part on, for example, the voltage value or current value, so that visual information (e.g., text, an image, or an icon) corresponding to the image data may be displayed through the display panel (1110).

[0214] According to one embodiment, the display module (1060) may further include a touch circuit (1150). The touch circuit (1150) may include a touch sensor (1151) and a touch sensor IC (1153) for controlling the same. The touch sensor IC (1153) may control the touch sensor (1151) to detect, for example, a touch input or a hovering input for a specific location of the display panel (1110). For example, the touch sensor IC (1153) may detect a touch input or a hovering input by measuring a change in a signal (e.g., voltage, light quantity, resistance, or charge quantity) for a specific location of the display panel (1110). The touch sensor IC (1153) may provide information (e.g., location, area, pressure, or time) regarding the detected touch input or hovering input to the processor (1020). According to one embodiment, at least a portion of the touch circuit (1150) (e.g., touch sensor IC (1153)) may be included as part of the display driver IC (1130), or as part of the display panel (1110), or as part of another component (e.g., auxiliary processor (1023)) disposed external to the display module (1060).

[0215] According to one embodiment, the display module (1060) may further include at least one sensor (e.g., a fingerprint sensor, an iris sensor, a pressure sensor, or an illuminance sensor) of the sensor module (1076), or a control circuit therefor. In this case, the at least one sensor or the control circuit therefor may be embedded in a part of the display module (1060) (e.g., the display panel (1110) or the DDI (1130)) or a part of the touch circuit (1150). For example, when the sensor module (1076) embedded in the display module (1060) includes a biometric sensor (e.g., a fingerprint sensor), the biometric sensor may obtain biometric information (e.g., a fingerprint image) associated with a touch input through a part of the display panel (1110). For another example, when the sensor module (1076) embedded in the display module (1060) includes a pressure sensor, the pressure sensor can obtain pressure information associated with a touch input through a part or the entire area of ​​the display panel (1110). According to one embodiment, the touch sensor (1151) or the sensor module (1076) can be placed between pixels of a pixel layer of the display panel (1110), or above or below the pixel layer.

[0216] FIG. 12 is a block diagram of a camera module according to various embodiments.

[0217] Referring to FIG. 12, the camera module (1080) may include a lens assembly (1210), a flash (1220), an image sensor (1230), an image stabilizer (1240), a memory (1250) (e.g., a buffer memory), or an image signal processor (1260). The lens assembly (1210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (1210) may include one or more lenses. According to one embodiment, the camera module (1080) may include a plurality of lens assemblies (1210). In this case, the camera module (1080) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (1210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (1210) may include, for example, a wide-angle lens or a telephoto lens.

[0218] The flash (1220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (1220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (1230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (1210) into an electrical signal. According to one embodiment, the image sensor (1230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (1230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0219] The image stabilizer (1240) can move at least one lens or image sensor (1230) included in the lens assembly (1210) in a specific direction or control the operating characteristics of the image sensor (1230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (1080) or the electronic device (1001) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (1240) can detect such movement of the camera module (1080) or the electronic device (1001) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (1080). According to one embodiment, the image stabilizer (1240) may be implemented as, for example, an optical image stabilizer. The memory (1250) may temporarily store at least a portion of the image acquired through the image sensor (1230) for the next image processing task. For example, when image acquisition is delayed due to a shutter or a plurality of images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) may be stored in the memory (1250), and a corresponding copy image (e.g., a low-resolution image) may be previewed through the display module (1060). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (1250) may be acquired and processed by, for example, the image signal processor (1260). According to one embodiment, the memory (1250) may be configured as at least a portion of the memory (1030), or as a separate memory that operates independently therefrom.

[0220] The image signal processor (1260) can perform one or more image processing operations on an image acquired through an image sensor (1230) or an image stored in a memory (1250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (1260) may perform control (e.g., exposure time control, read-out timing control, etc.) on at least one of the components included in the camera module (1080) (e.g., image sensor (1230)). An image processed by the image signal processor (1260) may be stored back in the memory (1250) for further processing or provided to an external component of the camera module (1080) (e.g., memory (1030), display module (1060), electronic device (1002), electronic device (1004), or server (1008)). According to one embodiment, the image signal The processor (1260) may be configured as at least a part of the processor (1020), or may be configured as a separate processor that operates independently of the processor (1020). If the image signal processor (1260) is configured as a separate processor from the processor (1020), at least one image processed by the image signal processor (1260) may be displayed through the display module (1060) by the processor (1020) as is or after undergoing additional image processing.

[0221] According to one embodiment, the electronic device (1001) may include a plurality of camera modules (1080), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (1080) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (1080) may be a front camera, and at least another may be a rear camera.

[0222] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0223] As described above, the electronic device (101) may include at least one camera (220). The electronic device (101) may include a memory (230) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (210) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to obtain, using the at least one camera (220), a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region and being darker than the first image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate a third image including the visual object within a third area corresponding to the first area, using map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, based on acquiring the first image and the second image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to store a file including the first image and the third image.

[0224] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to acquire, using the at least one camera (220), a first image according to a first exposure value and, using the at least one camera (220), a second image according to a second exposure value less than the first exposure value, based on detecting an event for displaying an image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to determine whether the first area in which the visual object is located in the first image and the second area in which the visual object is located in the second image are the same. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate the third image based on determining that the first region and the second region are different by identifying that the second region partially overlaps the first region.

[0225] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to identify first global tone information indicating the grayscale range of the first image and second global tone information indicating the grayscale range of the second image based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to identify transformation information for adjusting the first global tone information to the second global tone information based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate a pseudo image including the visual object within the first area by applying the transformation information to the first image based on determining that the first area in which the visual object within the first image is located and the second area in which the visual object within the second image is located are different.The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate the third image using the first image and the virtual image as map information usable for adjusting the brightness range from the first brightness range to the second brightness range based on determining that the first area in which the visual object is located in the first image and the second area in which the visual object is located in the second image are different.

[0226] According to one embodiment, each of the first image, the second image, and the virtual image may have a first size. The third image may have a second size smaller than the first size.

[0227] According to one embodiment, the transformation information may be identified using a first cumulative distribution function for the first global tone information and a second cumulative distribution function for the second global tone information.

[0228] In one embodiment, the transformation information may be identified based on a neural network trained to adjust the first global tone information to the second global tone information.

[0229] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to identify a subrange of the grayscale range of the first image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to identify filtered transformation information from the transformation information to more uniformly distribute grayscale levels within the subrange. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate the virtual image by applying the filtered transformation information to the first image.

[0230] According to one embodiment, the event may include acquisition of a photographing input within a state in which a high dynamic range (HDR) mode of the electronic device (101) is set.

[0231] According to one embodiment, the weight value of the map information may be adjusted according to the second exposure value for obtaining the second image.

[0232] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to further acquire a fourth image including the visual object within the first area according to the second exposure value using the at least one camera (220). The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate the third image as map information usable for adjusting the brightness range from the first brightness range to the second brightness range using the first image and the fourth image, based on acquiring the first image and the fourth image.

[0233] According to one embodiment, the position information indicating the third area where the visual object in the third image is located may include relative position information for each of the coordinates included in the third area from the reference coordinates of the third image. The position information indicating the first area where the visual object in the first image is located may include relative position information for each of the coordinates included in the first area from the reference coordinates of the first image.

[0234] According to one embodiment, the electronic device (101) may further include a display. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to display the first image on the display based on the stored file. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to adjust the brightness range of the display to the second brightness range adjusted using the third image while displaying the first image.

[0235] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to remove distortion formed within the third region representing a relatively bright region of the third image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate the third image from which the distortion has been removed.

[0236] In one embodiment, the file may include a joint photographic experts group (JPEG).

[0237] As described above, the electronic device (101) may include at least one camera (220). The electronic device (101) may include a memory (230) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (210) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to obtain, using the at least one camera (220), a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region and being darker than the first image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to generate a third image including the visual object within a third area corresponding to the first area, using map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, based on acquiring the first image and the second image. A global color tone of the third image may be different from a global color tone of the first image. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (101) to store a file including the first image and the third image.

[0238] As described above, the method performed by the electronic device (101) may include an operation of acquiring a first image including a visual object in a first area and a second image including the visual object in a second area partially overlapping the first area and being darker than the first image. The method may include an operation of generating a third image including the visual object in a third area corresponding to the first area as map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, based on acquiring the first image and the second image. The method may include an operation of storing a file including the first image and the third image.

[0239] In one embodiment, the method may include an operation of acquiring a first image according to a first exposure value and a second image according to a second exposure value less than the first exposure value based on detecting an event for displaying an image. The method may include an operation of determining whether a first area in which the visual object is located in the first image and a second area in which the visual object is located in the second image are the same. The method may include an operation of generating the third image based on determining that the first area and the second area are different by identifying that the second area partially overlaps the first area.

[0240] According to one embodiment, the method may include an operation of identifying first global tone information indicating the grayscale range of the first image and second global tone information indicating the grayscale range of the second image based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different. The method may include an operation of identifying transformation information for adjusting the first global tone information to the second global tone information based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different. The method may include an operation of generating a pseudo image including the visual object in the first region by applying the transformation information to the first image based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different. The method may include generating the third image using the first image and the virtual image as map information available for adjusting the brightness range from the first brightness range to the second brightness range, based on determining that the first area in the first image where the visual object is located and the second area in the second image where the visual object is located are different.

[0241] According to one embodiment, each of the first image, the second image, and the virtual image may have a first size. The third image may have a second size smaller than the first size.

[0242] The non-transitory computer-readable storage medium as described above can store one or more programs including instructions that, when individually or collectively executed by at least one processor (210) of an electronic device (101) including at least one camera (220), cause the electronic device (101) to obtain, using the at least one camera (220), a first image including a visual object within a first area and a second image including the visual object within a second area partially overlapping the first area and being darker than the first image. One or more programs including instructions that cause the electronic device (101) to generate a third image including the visual object within a third area corresponding to the first area as map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, based on acquiring the first image and the second image, when executed individually or collectively by the at least one processor (210) may store one or more programs including instructions that cause the electronic device (101) to store a file including the first image and the third image.

[0243] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0244] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0245] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0246] Various embodiments of the present document may be implemented as software (e.g., a program (1040)) including one or more instructions stored in a storage medium (e.g., an internal memory (1036) or an external memory (1038)) readable by a machine (e.g., an electronic device (1001)). For example, a processor (e.g., a processor (1020)) of the machine (e.g., an electronic device (1001)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0247] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0248] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, At least one camera; A memory storing instructions and including one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor: Using at least one camera, obtaining a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region and being darker than the first image; Based on acquiring the first image and the second image, generating a third image including the visual object within a third area corresponding to the first area as map information available for adjusting the brightness range for displaying the first image from a first brightness range corresponding to the gradation range of the first image to a second brightness range wider than the first brightness range; and To save a file including the first image and the third image, causing the above electronic device, Electronic devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor: Based on detecting an event for displaying an image, acquiring a first image according to a first exposure value using the at least one camera and a second image according to a second exposure value smaller than the first exposure value using the at least one camera; Determining whether the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are the same; and By identifying that the second region partially overlaps the first region, and thereby determining that the first region and the second region are different, the third image is generated. causing the above electronic device, Electronic devices.

3. In claim 2, The above instructions, when individually or collectively executed by the at least one processor: Based on determining that the first region in which the visual object is located in the first image and the second region in which the visual object is located in the second image are different: Identifying first global tone information indicating the tone range of the first image and second global tone information indicating the tone range of the second image; Identifying conversion information for adjusting the first global tone information to the second global tone information; By applying the transformation information to the first image, a pseudo image including the visual object within the first region is generated; and Using the first image and the virtual image, generate the third image as map information available for adjusting the brightness range from the first brightness range to the second brightness range. causing the above electronic device, Electronic devices.

4. In claim 3, Each of the first image, the second image, and the virtual image has a first size, and The third image above has a second size smaller than the first size, Electronic devices.

5. In claim 3, The above transformation information is identified using a first cumulative distribution function for the first global tone information and a second cumulative distribution function for the second global tone information. Electronic devices.

6. In claim 3, The above transformation information is identified based on a neural network learned to adjust the first global tone information to the second global tone information. Electronic devices.

7. In claim 3, The above instructions, when individually or collectively executed by the at least one processor: Identifying a partial range of the grayscale range of the first image; To more uniformly distribute the grayscale levels within the above partial range, filtered transformation information is identified from the above transformation information; and By applying the filtered transformation information to the first image, the virtual image is generated. causing the above electronic device, Electronic devices.

8. In claim 2, The above event includes acquisition of a shooting input within a state in which the high dynamic range (HDR) mode of the electronic device is set. Electronic devices.

9. In claim 2, The weight value of the above map information is adjusted according to the second exposure value for obtaining the second image. Electronic devices.

10. In claim 1, The above instructions, when individually or collectively executed by the at least one processor: further acquiring a fourth image including the visual object within the first area according to the second exposure value using at least one camera; and Based on obtaining the first image and the fourth image, using the first image and the fourth image, generate the third image as map information available for adjusting the brightness range from the first brightness range to the second brightness range. causing the above electronic device, Electronic devices.

11. In claim 1, The position information indicating the third area where the visual object in the third image is located includes relative position information for each of the coordinates included in the third area from the reference coordinate of the third image, and The position information indicating the first area where the visual object in the first image is located includes relative position information for each of the coordinates included in the first area from the reference coordinate of the first image. Electronic devices.

12. In claim 1, The electronic device further comprises a display, The above instructions, when individually or collectively executed by the at least one processor: Based on the stored file, displaying the first image on the display; and While displaying the first image, adjust the brightness range of the display to the second brightness range adjusted using the third image. causing the above electronic device, Electronic devices.

13. In claim 1, The above instructions, when individually or collectively executed by the at least one processor: Removing distortion formed within the third region representing a relatively bright region of the third image; and To generate the third image with the above distortion removed, causing the above electronic device, Electronic devices.

14. In claim 1, The above file contains JPEG (joint photographic experts group), Electronic devices.

15. In electronic devices, At least one camera; A memory storing instructions and including one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor: Using at least one camera, obtaining a first image including a visual object within a first region and a second image including the visual object within a second region partially overlapping the first region and being darker than the first image; Based on acquiring the first image and the second image, generating a third image including the visual object within a third area corresponding to the first area as map information available for adjusting a brightness range for displaying the first image from a first brightness range corresponding to a gradation range of the first image to a second brightness range wider than the first brightness range, wherein a global color tone of the third image is different from a global color tone of the first image; and To save a file including the first image and the third image, causing the above electronic device, Electronic devices.

Images