Electronic device and method for providing flare image by electronic device

Abstract

A method for providing a flare image by an electronic device may be disclosed, the method comprising the steps of: obtaining an input image including at least one light source object; obtaining a luminance image corresponding to the input image by using luminance data of the input image; obtaining a light source candidate map including at least one light source candidate region having a luminance value equal to or greater than a reference luminance value in the luminance image; obtaining a flare layer including at least one horizontally elongated flare corresponding to the at least one light source candidate region by performing a convolution operation on the light source candidate map by using a plurality of kernels; and outputting a flare image, in which at least one flare is superimposed on the at least one light source object, by applying the flare layer to the input image.

Description

Electronic device and method of the electronic device providing a flare image

[0001] The present disclosure relates to a method for providing a flare image with a flare effect applied and an electronic device for the same.

[0002] While standard camera sensors have a 16:9 aspect ratio, using an anamorphic lens allows you to obtain theater-style widescreen ratios such as 2.39:1 or 2.35:1, making anamorphic lenses suitable for creating a cinematic feel like Hollywood movies.

[0003] Anamorphic lenses produce elliptical bokeh rather than round bokeh due to the vertical compression effect. Additionally, anamorphic lenses emphasize depth by creating a stronger sense of compression between the subject and the background even at the same focal length. On the other hand, anamorphic lenses produce a distinctive horizontal lens flare when there is a strong light source. Therefore, anamorphic lenses are frequently used in movies to create a sci-fi feel or an emotional atmosphere.

[0004] A method for providing a flare image using an electronic device according to one embodiment of the present disclosure may include: acquiring an input image including at least one light source object; acquiring a luminance image corresponding to the input image using luminance data of the input image; acquiring a light source candidate map including at least one light source candidate region that exceeds a reference luminance value in the luminance image; acquiring a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region by performing a convolution operation on the light source candidate map using a plurality of kernels; and applying the flare layer to the input image to output a flare image in which at least one flare is superimposed on at least one light source object.

[0005] An electronic device according to one embodiment of the present disclosure may include a camera; a display unit; a memory for storing at least one instruction; and at least one processor. The electronic device may acquire an input image including at least one light source object through the camera by executing at least one instruction individually or collectively by at least one processor. The electronic device may acquire a luminance image corresponding to the input image by using luminance data of the input image. The electronic device may acquire a light source candidate map including at least one light source candidate region that exceeds a reference luminance value in the luminance image. By performing a convolution operation on the light source candidate map using a plurality of kernels, a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region may be acquired. The electronic device may apply the flare layer to the input image and output a flare image through the display unit in which at least one flare is superimposed on at least one light source object.

[0006] FIG. 1 is a drawing for illustrating an electronic device that provides a flare image according to one embodiment of the present disclosure.

[0007] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to provide a flare image.

[0008] FIG. 3 is a drawing for explaining a luminance image according to one embodiment of the present disclosure.

[0009] FIG. 4 is a diagram illustrating a morphology operation according to one embodiment of the present disclosure.

[0010] FIG. 5 is a diagram illustrating the operation of converting a light source candidate region into the shape of a circular object according to one embodiment of the present disclosure.

[0011] FIG. 6 is a drawing for explaining a light source candidate map according to one embodiment of the present disclosure.

[0012] FIG. 7 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to apply a first weight associated with a ghost object to a light source candidate region.

[0013] FIG. 8 is a drawing for explaining a ghost object according to one embodiment of the present disclosure.

[0014] FIG. 9 is a diagram illustrating a second weight corresponding to the distance between a light source candidate region and the center of a lens according to one embodiment of the present disclosure.

[0015] FIG. 10 is a drawing for explaining a flare image according to whether a second weight is applied according to one embodiment of the present disclosure.

[0016] FIG. 11 is a drawing for explaining a plurality of kernels according to one embodiment of the present disclosure.

[0017] FIG. 12 is a drawing for explaining the operation of generating a flare according to one embodiment of the present disclosure.

[0018] FIG. 13 is a drawing for explaining a flare layer according to one embodiment of the present disclosure.

[0019] FIG. 14 is a drawing for illustrating a flare image according to one embodiment of the present disclosure.

[0020] FIG. 15 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure providing a flare image corresponding to a preview image.

[0021] FIG. 16 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure providing a flare image corresponding to a stored image.

[0022] FIG. 17 is a block diagram illustrating the function of an electronic device according to one embodiment of the present disclosure.

[0023] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0024] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0025] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0026] Throughout the entire disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as described in the disclosure refer to a unit that processes at least one function or operation, and "...part" or "module" may be implemented in hardware or software, or a combination of hardware and software.

[0027] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored entirely in a single memory or may be partitioned and stored in multiple different memories.

[0028] Unless the context clearly indicates otherwise, the singular forms (e.g., "a," "an," and "the") may be understood to include plural objects. Thus, for example, the description "a component surface" may include cases where it refers to one or more of such surfaces.

[0029] All functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing and may include circuitry such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.

[0030] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0031] FIG. 1 is a drawing for explaining an electronic device (1000) that provides a flare image according to one embodiment of the present disclosure.

[0032] An electronic device (1000) according to one embodiment of the present disclosure may be a portable handheld device. For example, the electronic device (1000) may be a portable telephone, a smartphone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), etc., but is not limited thereto. The electronic device (1000) may be a wearable device that can be worn by a user. The wearable device may include at least one of an accessory-type device (e.g., a watch, a ring, a wristband, an ankle band, a necklace, glasses, a contact lens), a head-mounted device (HMD), a fabric or clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit), but is not limited thereto.

[0033] An electronic device (1000) according to one embodiment of the present disclosure may include a camera (image sensor). However, due to the size limitations of the electronic device (1000), the camera of the electronic device (1000) may include a small optical system. There are limitations in implementing flare (10), which is one of the effects of an anamorphic lens (100), in an image captured by a small optical system.

[0034] Flare (10) appears when a high-intensity object (e.g., a light source) is included in the subject, and may mean a mass of light that was not present when viewing the image with the naked eye. In particular, the flare (10) of the anamorphic lens (100) may be elongated in the horizontal direction and may appear more strongly when the light source is strong and the background is dark. Meanwhile, the flare (10) of the anamorphic lens (100) may have a unique color. For example, the flare (10) of the anamorphic lens (100) may have a blue color or a gold color. When using the flare (10) of the anamorphic lens (100), a cool cinematic feel can be provided. However, it is difficult to mount the anamorphic lens (100) on an electronic device (1000). Accordingly, according to one embodiment of the present disclosure, the electronic device (1000) may use computer vision technology or a deep learning-based AI image processing algorithm to emulate the flare (10) effect of the anamorphic lens (100).

[0035] Below, we will specifically examine, with reference to FIG. 2, a method for an electronic device (1000) to software-implement an optical flare (10) that occurs according to the characteristics of an anamorphic lens (100).

[0036] FIG. 2 is a flowchart illustrating a method for an electronic device (1000) according to one embodiment of the present disclosure to provide a flare image. In the present disclosure, the flare image may be an image including at least one flare in a horizontally elongated shape.

[0037] Referring to FIG. 2, the method of the electronic device (1000) providing a flare image may include steps S210 to S250. In one embodiment of the present disclosure, steps S210 to S250 may be executed by at least one processor included in the electronic device (1000). The method of the electronic device (1000) providing a flare image is not limited to that illustrated in FIG. 2, and in one or more embodiments, additional steps not illustrated in FIG. 2 may be included, or some steps may be omitted.

[0038] In step S210, an electronic device (1000) according to one embodiment of the present disclosure can acquire an input image including at least one light source object.

[0039] According to one embodiment of the present disclosure, an electronic device (1000) may acquire an input image for generating a flare image based on original image data acquired through a camera. The input image may be a raster image composed of pixels. Each pixel of the input image has a color depth value that can be displayed on the electronic device (1000). At this time, the bit depth of each pixel of the input image may be 8 bits (approximately 16.8 million colors), but is not limited thereto. For example, the bit depth of the input image may be 10 bits (approximately 1 billion colors) or 12 bits (approximately 68 billion colors). Bit depth refers to the number of bits used to indicate the color of a single pixel in a bitmap image or video frame buffer, or the number of bits used for each color element of a single pixel. Bit depth may also be expressed as bit length.

[0040] According to one embodiment of the present disclosure, an input image may include at least one light source object. The light source object may include, for example, a car headlight, a lighting device (e.g., a light bulb), a mobile phone flashlight, a street light, the sun, the moon, etc., but is not limited thereto, and may include various types of objects that emit light.

[0041] According to one embodiment of the present disclosure, the input image may be a still image (e.g., portrait, landscape, nightscape, etc.) or a video (e.g., cinematic video, short form, etc.). The video may include a plurality of frame images.

[0042] According to one embodiment of the present disclosure, the input image may be a preview image (e.g., live image, viewfinder image) or an image stored in memory. For example, if a user wants to generate a flare image for a real-time captured image, a preview image may be used as the input image. Additionally, if a user wants to edit a portion of the stored images into a flare image, an image selected by the user from a list of stored images may be used as the input image.

[0043] In step S220, an electronic device (1000) according to one embodiment of the present disclosure can acquire a luminance image corresponding to an input image.

[0044] According to one embodiment of the present disclosure, the luminance image may be a one-dimensional map in which only the luminance component is extracted by preprocessing raw image data. The luminance image may have the same scale as the input image, but in this case, memory loading issues may easily occur. Accordingly, the electronic device (1000) can generate a luminance image with a scale smaller than that of the input image.

[0045] According to one embodiment of the present disclosure, an electronic device (1000) can obtain a downscaled luminance image by applying a max pooling method to luminance data extracted from preprocessed original image data. Pooling may mean lowering the resolution of a feature map by reducing the height and width of the feature map, while maintaining the map features necessary for classification. Pooling may also be expressed as downsampling. Max pooling refers to a method of extracting only the maximum value within an area overlapping with a filter (kernel). For example, the electronic device (1000) can generate a downscaled luminance image by using a max pooling method to extract the maximum luminance value of each area overlapping with the filter (kernel). Meanwhile, according to one embodiment of the present disclosure, average pooling may be applied instead of max pooling.

[0046] According to one embodiment of the present disclosure, an electronic device (1000) can generate a luminance image using original image data before gamma is applied or original image data that has undergone de-gamma processing. For example, the electronic device (1000) can perform de-gamma processing on the pre-processed original image data when gamma is applied to the pre-processed original image data corresponding to the input image. The electronic device (1000) can obtain a downscaled luminance image by applying a max pooling method to the luminance data extracted from the de-gamma processed original image data.

[0047] Gamma can refer to the nonlinear transformation of a light intensity signal using a nonlinear transfer function. Since human vision responds nonlinearly to brightness according to Weber's law, if light brightness is recorded linearly within a limited bit depth—such as 8 bits per channel—posterization occurs in dark areas, to which the human eye is sensitive; changes in brightness do not appear smooth but rather discontinuous. Therefore, to display optimal image quality within the limitations of a given bit depth, it is necessary to encode nonlinearly to record dark areas in greater detail.

[0048] De-gamma can mean transforming a light intensity signal that has been non-linearly transformed back into a linear one. Since the light intensity (strength) must be accurately reflected for the flare to appear in the form of an actual optical effect, the electronic device (1000) can generate a luminance image using the original image data before gamma processing or the original image data after de-gamma processing.

[0049] According to one embodiment of the present disclosure, the luminance image may have the same number of bits (bit depth) as the input image, or may have a higher number of bits than the input image. In order for an electronic device (1000) having a small optical system to operate similarly to an actual optical lens, luminance information with a high number of bits (bit depth or bit length) is required. However, during the process of storing original image data as an input image in JPEG or TIFF format, there may be cases where the luminance information of the original image data cannot all be contained in the input image due to the limit of the number of bits. Therefore, the electronic device (1000) may generate a luminance image having a higher number of bits than the input image for reference when generating a flare image. For example, if the number of bits of the input image is 8 bits, the electronic device (1000) may generate a luminance image with 10 to 16 bits.

[0050] In step S230, an electronic device (1000) according to one embodiment of the present disclosure can obtain a light source candidate map including at least one light source candidate region that is greater than or equal to a reference luminance value in a luminance image.

[0051] According to one embodiment of the present disclosure, an electronic device (1000) can extract at least one light source candidate region that is greater than or equal to a reference luminance value in a luminance image. The reference luminance value may be a luminance value that serves as a reference for when flare may occur in an anamorphic lens. For example, when the number of bits in the luminance image is 8 bits, the luminance image has luminance values ​​between 0 and 255, so the reference luminance value may be set to 254 to extract the brightest region in the luminance image, but is not limited thereto. When the reference luminance value is 254, the electronic device (1000) can extract pixel regions with luminance values ​​of 254 and 255 in the luminance image. At this time, the extracted pixel regions may be light source candidate regions where flare may occur. In the luminance image, there may be one light source candidate region or multiple light source candidate regions.

[0052] Meanwhile, light source candidate regions with a reference luminance value or higher may contain noise and vary in shape and size. Therefore, when the input image is a video, the shape or size of the light source candidate regions may change continuously in each frame, which can lead to very unstable flare flickering.

[0053] According to one embodiment of the present disclosure, in order to prevent flare flickering, the electronic device (1000) can perform stabilization and simplification on at least one candidate light source region. For example, the electronic device (1000) can remove noise by performing morphology operations on at least one candidate light source region.

[0054] Morphological operation is a technique in image processing that performs shape-based operations on binary images or grayscale images. Morphological operation operates by transforming the pixels of an image using a kernel. The shape of the kernel for performing morphological operation can vary. For example, the kernel for performing morphological operation can be circular, square, cross-shaped, etc., but is not limited to these. Meanwhile, the size of the kernel for performing morphological operation may vary depending on the size of the luminance image. For example, a 5x5 kernel may be used for a luminance image (2000x1500) that is downscaled to 1 / 4 of the input image (4000x3000), and a 10x10 kernel may be used for a luminance image (4000x3000) that is the same scale as the input image. In other words, if the resize ratio is reduced, the luminance image itself is scaled down, so the kernel size can also be set to a smaller size.

[0055] Morphological operations can include erosion, dilation, opening, and closing operations. Erosion refers to an operation that cuts away from an image. In erosion, if the kernel does not completely overlap with an area filled with 1s in the input image, it changes the value to 0. Dilation is an operation that expands the surroundings of an object, conversely to erosion. In dilation, if there is even one 1 in the area containing the kernel, the corresponding pixel is changed to 1. Applying dilation after erosion is called opening, and applying erosion after dilation is called closing. For the sake of convenience, the following explanation will use the case where the morphological operation is erosion as an example.

[0056] According to one embodiment of the present disclosure, an electronic device (1000) can perform a simplification operation on at least one light source candidate region on which a morphological operation (e.g., an erosion operation) has been performed. When a morphological operation is performed on a light source candidate region, noise can be removed and the surface can be made smooth. However, due to the characteristics of the image, the shape of the light source candidate region may be distorted instantaneously. When a flare is generated based on this, the quality of the image may be degraded, such as by the flare continuously flickering or moving rapidly, so a process of simplifying the light source candidate region is performed.

[0057] According to one embodiment of the present disclosure, the electronic device (1000) can convert a light source candidate region into the form of a circular object to simplify the light source candidate region. For example, the electronic device (1000) can determine the center of gravity of the light source candidate region as the center of the circular object and determine the radius of the circular object so that the circular object has the same width as the light source candidate region. If there are multiple light source candidate regions, the electronic device (1000) can convert each of the multiple light source candidate regions into the form of a circular object.

[0058] When the light source candidate region is transformed into the shape of a circular object, the flare can be maintained at the same location because there is no significant difference in radius even if the shape of the light source candidate region changes slightly in each frame.

[0059] Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) may apply weights to a light source candidate region. For example, the electronic device (1000) may apply a weight based on a ghost object (hereinafter referred to as the first weight) or a weight based on the distance from the center of the luminance image (the center of the lens) (hereinafter referred to as the second weight) to the light source candidate region in order to create a flare of a darker color the stronger the light.

[0060] According to one embodiment of the present disclosure, an electronic device (1000) may apply a first weight (hereinafter also referred to as a ghost weight) to a light source candidate region that is point-symmetric with respect to a ghost object among light source candidate regions. In the case of a light source that is very strongly directed toward a lens, some of the light reflected from the lens causes a ghost to appear at a point-symmetric position with respect to the center of the lens. Although the ghost cannot be seen with the naked eye, it may be a round, faint light of a specific color captured in an input image. Therefore, when a ghost object is detected in an input image, the electronic device (1000) may assign a high weight to a light source candidate object that exists at a point-symmetric position with respect to the ghost object, as such object may be a strong light source. For example, the electronic device (1000) may increase the flare strength of a light source candidate object that is point-symmetric with respect to the ghost object from 1 to 3. The operation of the electronic device (1000) applying a first weight to a light source candidate region that is point-symmetric with the ghost object among the light source candidate regions will be examined in more detail later with reference to FIG. 7.

[0061] When strong light is incident on a lens, ghosting may occur in a point-symmetric manner, but no matter how strong the light is, ghosting is almost non-existent if it is far from the center of the lens. In other words, the intensity of flare caused by an anamorphic lens also weakens as it moves further away from the center of the lens. Accordingly, according to one embodiment of the present disclosure, an electronic device (1000) may apply a second weight to at least one light source candidate region based on the distance between at least one light source candidate region and the center of the luminance image (the center of the lens). The second weight may become smaller as the distance between the light source candidate region and the center of the luminance image (the center of the lens) increases, and may become larger as the distance between the light source candidate region and the center of the luminance image (the center of the lens) decreases. In the following, the second weight may be expressed as a lens center weight. For example, the electronic device (1000) may subdivide the region of the luminance image according to the distance relative to the center of the luminance image and define a different lens center weight (second weight) for each region. For example, a weight value on a logarithmic scale from 0 to 1 can be defined for each region. The lens center weight (second weight) can be subdivided into 12 steps, but is not limited thereto.

[0062] If lens center weighting is not applied to light source candidate regions, strong flares are created in all regions, resulting in a very complex image composition of the flare image. However, if lens center weighting is applied to light source candidate regions, only the flares at the center of the lens are kept strong, allowing for a simple image composition.

[0063] Accordingly, according to one embodiment of the present disclosure, the electronic device (1000) can perform a morphological operation on a light source candidate region extracted from a luminance image at a reference luminance value or higher, convert it into the form of a circular object, and generate a light source candidate map including a stabilized and simplified light source candidate region by reflecting a ghost weight or a lens center weight.

[0064] In step S240, an electronic device (1000) according to one embodiment of the present disclosure can obtain a flare layer including at least one flare by performing a convolution operation on a light source candidate map using a plurality of kernels. The at least one flare may correspond to each of at least one light source candidate region and may be horizontally elongated.

[0065] According to one embodiment of the present disclosure, a plurality of kernels may include a first kernel (hereinafter referred to as kernel X) having a one-dimensional array long in the X-axis direction and a second kernel (hereinafter referred to as kernel Y) having a one-dimensional array long in the Y-axis direction. In this case, the first kernel and the second kernel may be of an exponential form. For example, the element values ​​of the first kernel and the second kernel may decrease exponentially as they move away from the kernel center.

[0066] According to one embodiment of the present disclosure, an electronic device (1000) can perform convolution by creating a second kernel (Kernel Y) and a first kernel (Kernel X) in the order of the Y-axis direction and the X-axis direction on a light source candidate map. Since the sizes of the first kernel (Kernel X) and the second kernel (Kernel Y) are involved in the shape and size of the flare, the electronic device (1000) can set the sizes of the first kernel (Kernel X) and the second kernel (Kernel Y) based on the size of the flare to be created. For example, if the electronic device (1000) wants to create a flare with a horizontal length of 1000 and a central vertical length of 100, it can set the size of the first kernel (kernel X) to 1000 and the size of the second kernel (Kernel Y) to 100.

[0067] According to one embodiment of the present disclosure, the electronic device (1000) can set the value of each element of the first kernel (Kernel X) (value of each array, value of each weight) using Equation (1), and can set the value of each element of the second kernel (Kernel Y) (value of each array, value of each weight) using Equation (2).

[0068]

[0069] Equation (1):

[0070]

[0071] Equation (2):

[0072]

[0073] In equation (1), x can be a horizontally elongated one-dimensional matrix in the range from -x0 to x0, and motion xis a matrix in which the weight values ​​to be finally set in the first kernel (Kernel X) are stored, and a may be a parameter that controls the degree of spread of the weight values. As the value of a increases, the weight values ​​spread more widely, and as the value of a decreases, the weight values ​​spread more narrowly. That is, when the value of a is small, the weight values ​​can converge to 0 even if they deviate only slightly from the center of the kernel. In Equation (1), Max and min are, the denominator is e 1 ~e 10 It serves the role of restricting.

[0074] For example, if the size of the first kernel (kernel X) is set to 3000, the x matrix can be [-1500, -1499, -1498, … … … , 1498, 1499, 1500]. In this case, assuming a is 1500, if we substitute the values ​​(indexes) of the x matrix into Equation (1), motion x It can be as follows.

[0075]

[0076] That is, the weight value of the center of the first kernel (kernel X) is 1 (e / e 1 It can be set to ), and as it deviates from the center, the weight value decreases exponentially, becoming almost 0( It converges to ).

[0077] In equation (2), y can be a vertically elongated 1-dimensional matrix existing in the range from -y0 to y0, and motion y is a matrix in which the weight values ​​to be finally set in the second kernel (Kernel Y) are stored, and b may be a parameter that controls the degree of spread of the weight values. As the value of b increases, the weight values ​​spread more widely, and as the value of b decreases, the weight values ​​spread more narrowly. That is, when the value of b is small, the weight values ​​can converge to 0 even if they deviate only slightly from the center of the kernel. In Equation (2), Max and min are, the denominator is e 1 ~e10 It serves the role of restricting.

[0078] For example, if the size of the second kernel (kernel Y) is set to 100, the y matrix can be [-50, -49, -48, … … 0 … … , 48, 49, 50]. In this case, assuming b is 50, if we substitute the values ​​(indexes) of the y matrix into Equation (1), motion y It can be as follows.

[0079]

[0080] That is, the weight value of the center of the second kernel (kernel Y) is 1 (e / e 1 It can be set to ), and as it deviates from the center, the weight value decreases exponentially, becoming almost 0( It converges to ).

[0081] According to one embodiment of the present disclosure, an electronic device (1000) can generate a flare with a tapered tail by performing two convolution operations sequentially using a second kernel of an exponential form and a first kernel. A flare with a tapered tail cannot be created with only one convolution operation. For example, the electronic device (1000) can perform a first convolution operation in the Y-axis direction on a light source candidate map using a second kernel (Kernel Y). Then, the electronic device (1000) can perform a second convolution operation in the X-axis direction on the result of performing the first convolution operation using a first kernel (Kernel X). At this time, the electronic device (1000) can obtain at least one flare based on the result of performing the second convolution operation. That is, the electronic device (1000) can generate a flare with a thinning tail corresponding to a light source candidate region by sequentially performing a first convolution operation in the Y-axis direction and a second convolution operation in the X-axis direction.

[0082] According to one embodiment of the present disclosure, the electronic device (1000) can select the color of at least one flare. For example, the electronic device (1000) may select a default color or select a color based on user input. According to one embodiment of the present disclosure, the color may include cyan or gold, which are the flare colors of an anamorphic lens, but is not limited thereto. The electronic device (1000) can adjust the RGB values ​​of at least one flare based on the selected color. Since the flare layer is currently in a state of one channel (luminance), the electronic device (1000) can change the flare layer to three RGB channels and then adjust the RGB values ​​to match the selected color. For example, if cyan is selected, the electronic device (1000) can adjust the RGB values ​​of the flare to R : G : B = 62 : 173 : 211. Additionally, the electronic device (1000) can adjust the RGB values ​​of the flare to R : G : B = 255 : 220 : 115 when gold is selected.

[0083] According to one embodiment of the present disclosure, a flare layer including a flare of a specific color (e.g., cyan) corresponding to each light source candidate region may be generated. According to one embodiment of the present disclosure, since a first weight and a second weight are applied to the light source candidate map, a flare corresponding to a light source candidate region that is point-symmetric with respect to a ghost object may be expressed intensely in the flare layer. Additionally, a flare corresponding to a light source candidate region located at the center of the light source candidate map may appear intensely (strongly) in the flare layer, while a flare corresponding to a light source candidate region located at the edge of the light source candidate map may appear faintly (weakly).

[0084] In step S250, an electronic device (1000) according to one embodiment of the present disclosure can apply a flare layer to an input image to output a flare image in which at least one flare is superimposed on at least one light source object.

[0085] According to one embodiment of the present disclosure, an electronic device (1000) can generate a flare image by merging an input image and a flare layer. When the electronic device (1000) merges the input image and the flare layer, the flare is superimposed on a light source object included in the input image, thereby emulating the flare effect of an anamorphic lens. When a plurality of light source objects are included in the input image, each of the plurality of light source objects can be superimposed with a corresponding flare.

[0086] According to one embodiment of the present disclosure, when a flare layer is generated using a downscaled luminance image, the electronic device (1000) can upscale the flare layer to correspond to the size of the input image and merge the upscaled flare layer with the input image to generate a flare image.

[0087] According to one embodiment of the present disclosure, an electronic device (1000) can output a flare image in real time to a preview screen (or live screen, viewfinder). At this time, the user can check the flare image displayed on the preview screen and select a shooting button. When the electronic device (1000) receives input from the user to select a shooting button, it can capture the flare image currently displayed on the preview screen and store the flare image in memory.

[0088] According to one embodiment of the present disclosure, the electronic device (1000) may display a flare image corresponding to an input image stored in memory on an editing screen. The user may check the flare image through the editing screen and complete the editing if the flare image is satisfactory. At this time, the electronic device (1000) may store the edited flare image corresponding to the input image in memory.

[0089] According to one embodiment of the present disclosure, when the input image is a video, the electronic device (1000) generates flare images corresponding to each of the image frames included in the video and can play the video with the flare effect applied through a shooting screen or an editing screen.

[0090] Accordingly, according to one embodiment of the present disclosure, by emulating the flare effect of an anamorphic lens even in a smartphone having a small optical system, the user can lightly shoot movie footage or short-form content through the smartphone without a large and heavy camera system.

[0091] Below, with reference to FIGS. 3 to FIGS. 14, we will examine in more detail the operation of the electronic device (1000) providing a flare image.

[0092] FIG. 3 is a drawing for explaining a luminance image according to one embodiment of the present disclosure.

[0093] Referring to FIG. 3, an electronic device (1000) can acquire an input image (200) and generate a luminance image (300) corresponding to the input image (200). According to one embodiment of the present disclosure, the luminance image (300) may have the same number of bits (bit depth) as the input image (200) displayed on the screen, or it may have a higher number of bits (bit depth) than the input image (200) displayed on the screen. For example, if the original image data is 10 to 16 bits, the electronic device (1000) can acquire a luminance image (300) of 10 to 16 bits even if the input image (200) displayed on the screen is 8 bits. On the other hand, if the original image data is 8 bits, both the input image (200) displayed on the screen and the luminance image (300) may be 8 bits.

[0094] The electronic device (1000) can preprocess raw image data and extract a luminance component from the preprocessed raw image data to generate a luminance image (300). Since the luminance image (300) is intended to generate a flare object, it does not need to maintain a large size, and it is sufficient if it is large enough to extract an area greater than or equal to a reference luminance value. Accordingly, the electronic device (1000) can downscale the luminance image (300) to have a smaller size than the input image (200). For example, the electronic device (1000) can obtain a downscaled luminance image (300) by applying a max pooling method to the luminance data extracted from the preprocessed raw image data.

[0095] Referring to 301 in FIG. 3, when an electronic device (1000) generates a luminance image (300) downscaled using a max pooling method with a 2x2 filter, only the maximum luminance value can be extracted within the area overlapping with the filter (see Equation 3). Since a 2x2 filter is used, the scale of the luminance image (300) can be reduced to 1 / 4 compared to the input image (200). For example, if the input image (200) has 4000x3000 pixels, the luminance image (300) can have 2000x1500 pixels.

[0096]

[0097] Equation (3):

[0098]

[0099] FIG. 4 is a diagram illustrating a morphology operation according to one embodiment of the present disclosure.

[0100] According to one embodiment of the present disclosure, an electronic device (1000) can extract at least one light source candidate region that is greater than or equal to a reference luminance value (e.g., 254) in the luminance image (300) of FIG. 3. The electronic device (1000) can perform a morphological operation (e.g., erosion operation) on at least one light source candidate region to remove noise from the extracted at least one light source candidate region.

[0101] 400 in FIG. 4 is an original image representing light source candidate regions extracted from the luminance image (300) of FIG. 3. Since the light source candidate regions extracted from the luminance image (300) have only exceeded a reference threshold value, they may be in an unrefined state (unenhanced state). If the light source candidate regions are not refined, the shape or size of the light source candidate regions may change continuously in each frame, which can lead to very unstable flare flickering. Therefore, to prevent flare flickering, the electronic device (1000) can remove noise by performing an erosion operation on at least one light source candidate region.

[0102] The shape of the kernel for performing the erosion operation can vary. Figure 410 of FIG. 4 shows the result image of performing the erosion operation using a circular kernel (erosion disk). Figure 420 shows the result image of performing the erosion operation using a square kernel (erosion square). It can be seen that the light source candidate region is reduced slightly less in the result image (420) of performing the erosion operation using a square kernel compared to the result image (410) of performing the erosion operation using a circular kernel.

[0103] Meanwhile, it can be seen that the resulting images (410, 420) after performing the erosion operation have much less noise removed compared to the original image (400). That is, when the erosion operation is performed on the light source candidate region, the shape of the light source candidate region is neatly refined, so that messy flare effects can be prevented.

[0104] Although FIG. 4 describes an example of performing an erosion operation, it is not limited thereto. For example, the electronic device (1000) may sequentially perform an opening operation and a closing operation on a light source candidate region.

[0105] FIG. 5 is a diagram illustrating the operation of converting a light source candidate region into the shape of a circular object according to one embodiment of the present disclosure.

[0106] According to one embodiment of the present disclosure, an electronic device (1000) can convert light source candidate regions (e.g., 410) on which the morphological operation of FIG. 4 has been performed into the form of a circular object.

[0107] In the case of a light source candidate region with a somewhat complex shape, as shown in 510 of FIG. 5, the range of flare generation may change continuously from frame to frame. However, as shown in 520 of FIG. 5, if the light source candidate region is made into a circular object of the same width, there is no significant difference in the radius of the circular object even if there is a change in the shape of the light source candidate region, so the flare may appear in almost the same location from frame to frame.

[0108] According to one embodiment of the present disclosure, the position and size of a circular object are determined by an existing light source candidate area. For example, the center of the circular object is set to the center of gravity of the light source candidate area, and the width of the circular object can be set to be equal to the width of the light source candidate area. That is, the radius of the circular object can be set so that the circular object has a width equal to the width of the light source candidate area.

[0109] According to one embodiment of the present disclosure, the electronic device (1000) can simplify each light source candidate region on which morphological operations are performed into a circular object.

[0110] FIG. 6 is a drawing for explaining a light source candidate map according to one embodiment of the present disclosure.

[0111] Referring to FIG. 6, the electronic device (1000) can extract a first light source candidate region (601), a second light source candidate region (602), a third light source candidate region (603), and a fourth light source candidate region (604) that are greater than or equal to a reference luminance value in a luminance image (300). The first light source candidate region (601), the second light source candidate region (602), the third light source candidate region (603), and the fourth light source candidate region (604) may have complex shapes as they are not refined.

[0112] According to one embodiment of the present disclosure, an electronic device (1000) can perform a morphological operation (e.g., an erosion operation) on each of the first light source candidate region (601), the second light source candidate region (602), the third light source candidate region (603), and the fourth light source candidate region (604). Additionally, the electronic device (1000) can convert the first light source candidate region (601), the second light source candidate region (602), the third light source candidate region (603), and the fourth light source candidate region (604) into the form of a circular object to obtain an advanced first light source candidate region (610), an advanced second light source candidate region (620), an advanced third light source candidate region (630), and an advanced fourth light source candidate region (640).

[0113] And the electronic device (1000) can generate a light source candidate map (600) including an advanced first light source candidate region (610), an advanced second light source candidate region (620), an advanced third light source candidate region (630), and an advanced fourth light source candidate region (640).

[0114] Meanwhile, the electronic device (1000) may apply a ghost weight (first weight) and a lens center weight (second weight) to the advanced first light source candidate region (610), the advanced second light source candidate region (620), the advanced third light source candidate region (630), and the advanced fourth light source candidate region (640). Below, we will examine in detail the operation of the electronic device (1000) applying the ghost weight (first weight) and the lens center weight (second weight) to the light source candidate map (600) with reference to FIGS. 7 to 10.

[0115] FIG. 7 is a flowchart illustrating a method for an electronic device (1000) according to one embodiment of the present disclosure to apply a first weight associated with a ghost object to a light source candidate region.

[0116] Referring to FIG. 7, a method for an electronic device (1000) to apply a first weight associated with a ghost object to a light source candidate region may include steps S710 through S740. In one embodiment of the present disclosure, steps S710 through S740 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to apply a first weight associated with a ghost object to a light source candidate region is not limited to that illustrated in FIG. 7, and in one or more embodiments, additional steps not illustrated in FIG. 7 may be included, or some steps may be omitted.

[0117] In step S710, an electronic device (1000) according to one embodiment of the present disclosure can determine whether the brightness value around the electronic device (1000) is greater than or equal to a threshold brightness value when capturing an input image, based on metadata of the input image.

[0118] According to one embodiment of the present disclosure, an electronic device (1000) can store an ambient brightness value as metadata when capturing an image. For example, the electronic device (1000) can acquire an ambient brightness value using an illuminance sensor when capturing an image and store the ambient brightness value as metadata corresponding to the image being captured.

[0119] Accordingly, the electronic device (1000) can determine whether the brightness value around the electronic device (1000) is greater than or equal to the threshold brightness value by comparing the brightness value included in the metadata of the input image with the threshold brightness value. When a light source is directed toward the electronic device (1000) during the input image capture, the brightness value around the electronic device (1000) may be high. The threshold brightness value may be the brightness value of a light source that is likely to cause flare in an anamorphic lens.

[0120] In steps S720 and S730, an electronic device (1000) according to one embodiment of the present disclosure can perform ghost object detection having a predetermined color in an input image when the brightness value is greater than or equal to a threshold brightness value (Yes in step S720).

[0121] According to one embodiment of the present disclosure, when the brightness value around the electronic device (1000) is greater than or equal to a threshold brightness value, there is a high probability that a ghost object will be detected in the input image, so the electronic device (1000) can perform ghost object detection on the input image. Since the ghost object has a unique color (e.g., cyan), the electronic device (1000) can search for whether a ghost object having a specific color (e.g., cyan) exists at a point-symmetric position of the light source candidate regions.

[0122] Meanwhile, an electronic device (1000) according to one embodiment of the present disclosure may not perform ghost object detection when the brightness value around the electronic device (1000) is less than a threshold brightness value (No of S720), because there is a low probability that a ghost object will be detected in the input image.

[0123] In steps S740 and S750, an electronic device (1000) according to one embodiment of the present disclosure may apply a first weight (hereinafter referred to as a ghost weight) to a light source candidate region that is point-symmetric with respect to the ghost object among at least one light source candidate region when a ghost object is detected in an input image (Yes in S740). Since a light source object located in a light source candidate region that is point-symmetric with respect to the ghost object is highly likely to generate a flare of much stronger intensity, the electronic device (1000) may assign a high weight to the light source candidate region that is point-symmetric with respect to the ghost object.

[0124] For example, the electronic device (1000) can increase the flare strength of a ghosted light source candidate area to 3 when the flare strength of a normal light source candidate area where no ghost object is generated is set to a default value of 1.

[0125] Meanwhile, according to one embodiment of the present disclosure, when no ghost object is detected in the input image (No of S740), since there is no light source strong enough to cause a ghost object in the input image, the electronic device (1000) may not apply a first weight (ghost weight) to at least one light source candidate region extracted from the luminance image.

[0126] FIG. 8 is a drawing for explaining a ghost object according to one embodiment of the present disclosure.

[0127] Referring to FIG. 8, the electronic device (1000) can take a picture of a mobile phone flash. At this time, since the light of the mobile phone flash, which is the light source (810), is strong, a blue ghost object (820) may appear at a point-symmetric position with respect to the mobile phone flash.

[0128] The electronic device (1000) can detect a blue ghost object (820) in an input image captured by a mobile phone flash based on the determination that the brightness value is greater than or equal to a threshold brightness value when capturing an input image. When a ghost object (820) is detected, the electronic device (1000) can apply a ghost weight to a light source candidate region (circular object) that is point-symmetric with respect to the ghost object (820).

[0129] Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) may additionally apply a second weight (lens center weight) corresponding to the distance between the light source candidate region and the lens center (center of the luminance image) to the light source candidate region. The operation of the electronic device (1000) applying the lens center weight to the light source candidate region will be examined with reference to FIG. 9.

[0130] FIG. 9 is a diagram illustrating a second weight corresponding to the distance between a light source candidate region and the center of a lens according to one embodiment of the present disclosure.

[0131] Referring to 900 in FIG. 9, the luminance image can be divided into multiple weighting regions based on the center of the luminance image. For example, it can be divided into a first weighting region (910), a second weighting region (920), a third weighting region (930), and a fourth weighting region (940) starting from the center. Additionally, the second weight (hereinafter referred to as the lens center weight) can be set lower as it moves further away from the center. For example, the lens center weight of the first weighting region (910) can be set to 1, the lens center weight of the second weighting region (920) can be set to 0.5, the lens center weight of the third weighting region (930) can be set to 0.25, and the lens center weight of the fourth weighting region (940) can be set to 0.05.

[0132] Meanwhile, when a first light source candidate region (901), a second light source candidate region (902), a third light source candidate region (903), and a fourth light source candidate region (904) are extracted from a luminance image, the electronic device (1000) can perform morphological operations on the first light source candidate region (901), the second light source candidate region (902), the third light source candidate region (903), and the fourth light source candidate region (904) and convert them into the form of a circular object. The electronic device (1000) can then apply ghost weights and / or lens center weights to the first light source candidate region (901), the second light source candidate region (902), the third light source candidate region (903), and the fourth light source candidate region (904) that have been converted into the form of a circular object. For example, if the second light source candidate region (902) exists at a point-symmetric position of the ghost object and is included in the first weighting region (910), the electronic device (1000) may assign a ghost weight of '3' and a lens center weight of '1' to the second light source candidate region (902). Additionally, if the third light source candidate region (903) is included in the second weighting region (920), the electronic device (1000) may assign a lens center weight of '0.5' to the third light source candidate region (903). If the fourth light source candidate region (904) is included in the third weighting region (930), the electronic device (1000) may assign a lens center weight of '0.25' to the fourth light source candidate region (904). If the first light source candidate region (901) is included in the fourth weighting region (940), the electronic device (1000) can assign a lens center weight of '0.05' to the first light source candidate region (901).

[0133] Accordingly, the electronic device (1000) can generate a light source candidate map (600) including a first light source candidate region (901), a second light source candidate region (902), a third light source candidate region (903), and a fourth light source candidate region (904) to which different weights are applied. The intensity (darkness) of the flare corresponding to each light source candidate region can be determined according to the weight of each light source candidate region. If the weight is high, the flare may be expressed darkly in the flare image, and if the weight is low, the flare may be expressed weakly in the flare image. Accordingly, according to one embodiment of the present disclosure, the electronic device (1000) can maintain only the flares at the center of the lens strongly and express the flares at the edge of the lens weakly, and express the flare of a strong light source darkly by applying a lens center weight and / or ghost weight when generating the flare.

[0134] In FIG. 9, the case where the lens center weighting region is classified into four is described as an example, but it is not limited thereto. For example, the lens center weighting region can be subdivided into 12 or more, and the lens center weight (second weight) of each region can have a value made on a logarithmic scale between 0 and 1.

[0135] FIG. 10 is a drawing for explaining a flare image according to whether a second weight is applied according to one embodiment of the present disclosure.

[0136] Referring to 1010 in FIG. 10, if the electronic device (1000) does not apply lens center weights to each of the multiple light source candidate regions, strong flares (101) are created in all regions of the input image, so that the flare image (1001) may have a very complex screen composition.

[0137] On the other hand, referring to 1020 in FIG. 10, when the electronic device (1000) assigns a lens center weight to each of the plurality of light source candidate regions, only the flares (102) at the lens center are strongly maintained, allowing the screen of the flare image (1002) to be constructed concisely.

[0138] FIG. 11 is a drawing for explaining a plurality of kernels according to one embodiment of the present disclosure.

[0139] According to one embodiment of the present disclosure, an electronic device (1000) can obtain a flare layer including a horizontally elongated flare by performing a convolution operation on a light source candidate map (600, see FIG. 9) using a plurality of kernels.

[0140] According to one embodiment of the present disclosure, a plurality of kernels may include a first kernel (Kernel X, 1101) in a one-dimensional array long in the X-axis direction and a second kernel (Kernel Y, 1102) in a one-dimensional array long in the Y-axis direction. In this case, the first kernel (1101) and the second kernel (1102) may be of an exponential form. For example, the element values ​​of the first kernel (1101) and the second kernel (1102) may be largest at the center of the kernel and decrease exponentially as they move away from the center of the kernel. For example, the element values ​​of the first kernel (1101) and the second kernel (1102) may be 1 at the center of the kernel and close to 0 at both ends of the kernel.

[0141] According to one embodiment of the present disclosure, an electronic device (1000) can perform convolution operations by creating a second kernel (Kernel Y, 1102) and a first kernel (Kernel X, 1101) in the order of the Y-axis direction and the X-axis direction on a light source candidate map (600). Since the sizes of the first kernel (Kernel X, 1101) and the second kernel (Kernel Y, 1102) are involved in the shape and size of the flare, the electronic device (1000) can set the sizes of the first kernel (Kernel X, 1101) and the second kernel (Kernel Y, 1102) based on the size of the flare to be generated. Since the flare is long in the horizontal direction, the size of the first kernel (Kernel X, 1101) may be much larger than the size of the second kernel (Kernel Y, 1102).

[0142] Referring to FIG. 11, if the electronic device (1000) wants to create a flare with a central horizontal length of 1500 and a central vertical length of 100, the size of the first kernel (kernel X, 1101) can be set to 1500 pixels and the size of the second kernel (kernel Y, 1102) can be set to 100 pixels.

[0143] The electronic device (1000) can create a flare with a thin tail by performing two convolution operations on the light source candidate map (600) using the second kernel (1102) and the first kernel (1101). We will examine the shape of the flare further with reference to FIG. 12.

[0144] FIG. 12 is a drawing for explaining the operation of generating a flare according to one embodiment of the present disclosure.

[0145] Referring to 1201 in FIG. 12, the electronic device (1000) can obtain a light source candidate map including a light source candidate region in the shape of a circular object. The electronic device (1000) can then perform a first convolution operation in the Y-axis direction on the light source candidate map using a second kernel (1102).

[0146] Referring to 1202 in FIG. 12, as a result of performing a first convolution operation in the Y-axis direction using a second kernel (1102), the light source candidate region can be slightly spread in the vertical direction. At this time, since the size of the second kernel (1102) is 100 pixels, the light source candidate region can be spread up to 100 pixels in the vertical direction.

[0147] The electronic device (1000) can perform a second convolution operation in the X-axis direction using a first kernel (1101) on the result of performing a first convolution operation.

[0148] Referring to 1203 in FIG. 12, as a result of performing a second convolution operation in the X-axis direction using the first kernel (1101), a flare with a tail that thins in the horizontal direction can be generated. At this time, since the size of the first kernel (1101) is 1500 pixels, the horizontal length of the flare can be 1500 pixels.

[0149] FIG. 13 is a drawing for explaining a flare layer (700) according to one embodiment of the present disclosure.

[0150] According to one embodiment of the present disclosure, an electronic device (1000) can perform morphological operations on a first light source candidate region (1301), a second light source candidate region (1302), a third light source candidate region (1303), and a fourth light source candidate region (1304) extracted from a luminance image (300), and then convert them into the form of a circular object. The electronic device (1000) can generate a light source candidate map (600) by applying ghost weights and / or lens center weights to the first light source candidate region (1301), the second light source candidate region (1302), the third light source candidate region (1303), and the fourth light source candidate region (1304) converted into the form of a circular object. When the electronic device (1000) performs two convolution operations using multiple kernels on a light source candidate map (600), it can generate a first flare (1310), a second flare (1320), a third flare (1330), and a fourth flare (1340) corresponding to each of the first light source candidate region (1301), the second light source candidate region (1302), the third light source candidate region (1303), and the fourth light source candidate region (1304). The electronic device (1000) can generate a flare layer (700) by applying a specific color (e.g., cyan) or a base color selected by the user to the first flare (1310), the second flare (1320), the third flare (1330), and the fourth flare (1340).

[0151] In the flare layer (700), the first flare (1310) is located far from the center of the lens so it can be expressed faintly, and the second flare (1320) is close to the center of the lens so it can be expressed intensely.

[0152] FIG. 14 is a drawing for illustrating a flare image according to one embodiment of the present disclosure.

[0153] Referring to FIG. 14, according to one embodiment of the present disclosure, an electronic device (1000) can generate a flare image (1401) by merging an input image (200) and a flare layer (700). When the electronic device (1000) merges the input image (200) and the flare layer (700), a flare included in the flare layer (700) may be superimposed on a light source object included in the input image (200). If a plurality of light source objects are included in the input image (200), each of the plurality of light source objects may be superimposed on a corresponding flare.

[0154] According to one embodiment of the present disclosure, when a flare layer (700) is generated using a downscaled luminance image, the electronic device (1000) can upscale the flare layer (700) to correspond to the size of the input image (200), and merge the upscaled flare layer (700) with the input image (200) to generate a flare image (1401).

[0155] Meanwhile, just as an anamorphic lens generates a strong flare for light incident on the center of the lens, the electronic device (1000) can generate a flare image (1401) in which a strong flare appears closer to the center of the image by reflecting the weighting of the center of the lens.

[0156] Accordingly, according to one embodiment of the present disclosure, optical flare occurring according to the characteristics of an anamorphic lens in an electronic device (1000) can be emulated in software.

[0157] FIG. 15 is a diagram illustrating the operation of an electronic device (1000) according to one embodiment of the present disclosure providing a flare image corresponding to a preview image.

[0158] Referring to screen 1510 of FIG. 15, the electronic device (1000) can execute a camera application according to user input. At this time, the user can select a person video mode (1501) in the execution window of the camera application. When the user selects the person video mode (1501), the electronic device (1000) can adjust the camera setting value to a default value corresponding to the person video mode (1501).

[0159] Meanwhile, referring to screen 1520 of FIG. 15, the electronic device (1000) may display an icon (1502) for setting effects related to the person video mode (1501) on the preview screen. When the user selects the icon (1502), the electronic device (1000) may provide a list of various effects related to the person video mode (1501). For example, the electronic device (1000) may provide a list including bokeh (blur) (applying a blur effect to the background), studio (applying a lighting effect to the person and a blur effect to the background), high-key mono (emphasizing the person through a lighting effect in a black and white photo with a white background), low-key mono (emphasizing the person through a lighting effect in a black and white photo with a black background), color background (changing the background color and applying a lighting effect to the person, and the background color is automatically set to a color that matches the photo by detecting the color of the person's clothes), color point (maintaining the color of the person and converting the background to black and white), flare, etc.

[0160] Referring to screen 1530 of FIG. 15, the user can select an icon (1503) representing a flare effect from the effect list. The user can also select the intensity of the flare effect. In response to the input of selecting the icon (1503), the electronic device (1000) can generate a flare image with the flare effect applied to the preview image and output the flare image to the display unit. The user can view the flare image and select a shooting button. When the user selects the shooting button, the electronic device (1000) can capture the flare image with the flare effect applied and store it in memory.

[0161] In FIG. 15, an example is described in which a flared image with a flared effect applied is displayed on a preview screen after the user selects the person video mode (1501), but this is not limited thereto. When the user selects the person video mode (1501), the electronic device (1000) may automatically perform a convolution operation on the preview image to display a flared image corresponding to the preview image.

[0162] FIG. 16 is a diagram illustrating the operation of an electronic device (1000) according to one embodiment of the present disclosure providing a flare image corresponding to a stored image.

[0163] Referring to screen 1610 of FIG. 16, the user can select one from a list of previously stored images. At this time, the electronic device (1000) can display the stored image selected by the user on the screen. The user can check the stored image displayed on the screen and select an icon (1601) for changing background effects. The electronic device (1000) can provide an effect list in response to the user input selecting the icon (1601) for changing background effects. If the user selects an icon (1602) representing a flare effect from the effect list, the electronic device (1000) can generate a flare image by applying a flare layer to the stored image selected by the user. For example, the electronic device (1000) can generate a luminance image based on preprocessed original image data stored together with the stored image selected by the user. The electronic device (1000) can generate a light source candidate map by extracting at least one light source candidate region from a luminance image, performing a morphological operation on at least one light source candidate region, converting it into the shape of a circular object, and applying a ghost weight and a lens center weight. The electronic device (1000) can generate a flare layer containing at least one flare corresponding to at least one light source candidate region by performing a first convolution operation in the Y-axis direction on the light source candidate map and then performing a second convolution operation in the X-axis direction. The electronic device (1000) can generate a flare image by combining a saved image selected by a user with the flare layer.

[0164] Referring to screen 1620 of FIG. 16, the electronic device (1000) can output a flare image to a display unit in which a flare layer is applied to a saved image selected by the user. Accordingly, the electronic device (1000) can easily provide a flare image that implements an effect similar to the optical flare of an anamorphic lens in response to a user's request to edit a saved image.

[0165] FIG. 17 is a block diagram for explaining the function of an electronic device (1000) according to one embodiment of the present disclosure.

[0166] As illustrated in FIG. 17, an electronic device (1000) according to one embodiment of the present disclosure may include an output unit (1100), a sensor unit (1200), a processor (1300), a communication interface (1400), an A / V input unit (1500), a user input unit (1600), and a memory (1700). However, not all components illustrated in FIG. 17 are essential components. The electronic device (1000) may be implemented with more components than those illustrated, or with fewer components. For example, the electronic device (1000) may be implemented with a memory (1700) and a processor (1300).

[0167] The above components will be examined in turn below.

[0168] The output unit (1100) is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit (1111), an audio output unit (1112), a vibration motor (1113), etc.

[0169] The sound output unit (1112) outputs audio data received from the communication interface (1400) or stored in the memory (1700). Additionally, the sound output unit (1112) outputs an acoustic signal related to a function performed by the electronic device (1000) (e.g., a call signal reception sound, a message reception sound, a notification sound). The sound output unit (1112) may include a speaker, a buzzer, etc.

[0170] The vibration motor (1113) can output a vibration signal. For example, the vibration motor (1113) can output a vibration signal corresponding to the output of audio data or video data (e.g., a call signal reception sound, a message reception sound, etc.). Additionally, the vibration motor (1113) can output a vibration signal when a touch is input to the touchscreen.

[0171] The output unit (1100) can output an image of a 3D space. At this time, the image of the 3D space may include an image indicator displayed on the screen in a predetermined size. The image indicator may be a UI element for controlling an IoT device (3000). For example, the image indicator may be an icon image, but is not limited thereto.

[0172] The sensor unit (1200) may include at least one of a magnetic sensor (1211), an acceleration sensor (1212), a tilt sensor (1213), an infrared sensor (1214), a gyroscope sensor (1215), a position sensor (e.g., GPS) (1216), a temperature and humidity sensor (1217), a proximity sensor (1218), and a light sensor (1219), but is not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description is omitted.

[0173] The processor (1300) typically controls the overall operation of the electronic device (1000). For example, the processor (1300) can control the output unit (1100), sensor unit (1200), communication interface (1400), A / V input unit (1500), user input unit (1600), memory (1700), etc., by executing programs stored in memory (1700).

[0174] The processor (1300) may be composed of one or more processors. The one or more processors included in the processor (1300) may be circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). The one or more processors included in the processor (1300) may be general-purpose processors such as a CPU (Central Processing Unit), MPU (Micro Processor Unit), AP (Application Processor), or DSP (Digital Signal Processor); graphics-dedicated processors such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit); artificial intelligence-dedicated processors such as an NPU (Neural Processing Unit); or communication-dedicated processors such as a CP (Communication Processor). If the one or more processors included in the processor (1300) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The processor (1300) may be implemented as a single core processor or as a multicore processor.

[0175] The processor (1300) can write data to memory (1700) or read data stored in memory (1700), and in particular, can process data according to a predefined operation rule or artificial intelligence model by executing a program or at least one instruction stored in memory (1700).

[0176] The communication interface (1400) may include one or more components that enable communication between an electronic device (1000) and an IoT device (3000) or between an electronic device (1000) and a server (2000). For example, the communication interface (1400) may include a short-range communication unit (1411), a mobile communication unit (1412), and a broadcast receiving unit (1413).

[0177] The short-range wireless communication unit (221) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra-wideband (UWB) communication unit, an Ant+ communication unit, etc.

[0178] The mobile communication unit (1412) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various forms of data such as voice call signals, video call call signals, or text / multimedia message transmission and reception.

[0179] The broadcast receiver (1413) receives broadcast signals and / or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Depending on the implementation example, the electronic device (1000) may not include the broadcast receiver (1413).

[0180] The A / V (Audio / Video) input unit (1500) is for inputting audio signals or video signals and may include a camera (1511), a microphone (1512), etc. The camera (1511) can obtain image frames, such as still images or video, through an image sensor in a video call mode or a shooting mode. Images captured through the image sensor may be processed through a processor (1300) or a separate image processing unit (not shown). Image frames processed by the camera (1511) may be stored in a memory (1700) or transmitted externally through a communication interface (1400). According to one embodiment of the present disclosure, the camera (1511) may include at least one of a telephoto camera, a wide-angle camera, and a standard camera, but is not limited thereto.

[0181] The microphone (1512) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1512) can receive an acoustic signal from an external device or a speaker. The microphone (1512) can use various noise removal algorithms to remove noise generated during the process of receiving the external acoustic signal.

[0182] The user input unit (1600) refers to a means for a user to input data for controlling an electronic device (1000). For example, the user input unit (1600) may include a key pad, a dome switch, a touch pad (contact capacitive method, pressure resistive method, infrared detection method, surface ultrasonic conduction method, integral tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, etc., but is not limited thereto.

[0183] The memory (1700) may store a program for processing and controlling the processor (1300), and may also store input / output data (e.g., voice data, photo images, memo data, user biometric information, etc.).

[0184] The memory (1700) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0185] The memory (1700) may not exist separately but may be configured to be included in the processor (1300). The memory (1700) may be composed of volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. The memory (1700) may store a program or at least one instruction for performing operations according to one embodiment of the present disclosure. The memory (1700) may provide stored data to the processor (1300) upon the request of the processor (1300).

[0186] According to one embodiment of the present disclosure, the electronic device (1000) can apply a cinematic effect to an image obtained through a standard lens of the electronic device (1000) by software-implementing the flare function of an anamorphic lens used in actual movies.

[0187] A method for providing a flare image by an electronic device (1000) according to one embodiment of the present disclosure may include: acquiring an input image including at least one light source object; acquiring a luminance image corresponding to the input image using luminance data of the input image; acquiring a light source candidate map including at least one light source candidate region that is greater than or equal to a reference luminance value in the luminance image; acquiring a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region by performing a convolution operation on the light source candidate map using a plurality of kernels; and applying the flare layer to the input image to output a flare image in which at least one flare is superimposed on at least one light source object.

[0188] The step of acquiring a luminance image according to one embodiment of the present disclosure may include the step of acquiring a luminance image downscaled from an input image by applying a max pooling method to the luminance data.

[0189] The step of obtaining a light source candidate map according to one embodiment of the present disclosure may include the step of performing a morphological operation on at least one light source candidate region to remove noise from at least one light source candidate region.

[0190] A step of obtaining a light source candidate map according to one embodiment of the present disclosure may include: a step of converting at least one light source candidate region into the form of a circular object; and a step of obtaining a light source candidate map including at least one light source candidate region converted into the form of a circular object.

[0191] A step of converting at least one light source candidate region into the shape of a circular object according to one embodiment of the present disclosure may include: determining the center of gravity of at least one light source candidate region as the center of the circular object; and determining the radius of the circular object such that the circular object has the same area as at least one light source candidate region.

[0192] A step of acquiring a light source candidate map according to one embodiment of the present disclosure may include: a step of detecting a ghost object of a predetermined color in an input image; and a step of applying a first weight to a light source candidate region that is point-symmetric with respect to the ghost object among at least one light source candidate region.

[0193] A step of detecting a ghost object according to one embodiment of the present disclosure may include: a step of determining whether a brightness value around an electronic device (1000) is greater than or equal to a threshold brightness value when capturing an input image based on metadata of an input image; and a step of detecting a ghost object having a predetermined color in the input image based on the fact that the brightness value is greater than or equal to the threshold brightness value.

[0194] A step of acquiring a light source candidate map according to one embodiment of the present disclosure may include acquiring a light source candidate map by applying a second weight to at least one light source candidate region based on the distance between at least one light source candidate region and the center of a luminance image.

[0195] According to one embodiment of the present disclosure, the second weight may become smaller as the distance between at least one light source candidate region and the center of the luminance image increases, and may become larger as the distance between at least one light source candidate region and the center of the luminance image decreases.

[0196] A plurality of kernels according to one embodiment of the present disclosure may include a first kernel having a one-dimensional array long in the X-axis direction and a second kernel having a one-dimensional array long in the Y-axis direction.

[0197] A step of acquiring a flare layer according to one embodiment of the present disclosure may include: a step of performing a first convolution operation in the Y-axis direction on a light source candidate map using a second kernel; a step of performing a second convolution operation in the X-axis direction on the result of performing the first convolution operation using the first kernel; and a step of acquiring at least one flare based on the result of performing the second convolution operation.

[0198] According to one embodiment of the present disclosure, the element value of the first kernel and the element value of the second kernel may decrease exponentially as they move away from the kernel center.

[0199] The shape and size of at least one flare according to one embodiment of the present disclosure may be determined according to the size of the first kernel and the size of the second kernel.

[0200] The step of obtaining a flare layer according to one embodiment of the present disclosure may include: selecting a color of at least one flare; and adjusting the RGB values ​​of at least one flare based on the selected color.

[0201] A color according to one embodiment of the present disclosure may include cyan or gold.

[0202] An electronic device (1000) according to one embodiment of the present disclosure may include a camera (1511); a display unit (1111); a memory (1700) for storing at least one instruction; and at least one processor (1300).

[0203] An electronic device (1000) according to one embodiment of the present disclosure can acquire an input image including at least one light source object through a camera (1511) by executing at least one instruction individually or collectively by at least one processor (1300). The electronic device (1000) can acquire a luminance image corresponding to the input image by using luminance data of the input image. The electronic device (1000) can acquire a light source candidate map including at least one light source candidate region that is greater than or equal to a reference luminance value in the luminance image. The electronic device (1000) can acquire a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region by performing a convolution operation on the light source candidate map using a plurality of kernels. The electronic device (1000) can apply the flare layer to the input image and output a flare image in which at least one flare is superimposed on at least one light source object through a display unit (1111).

[0204] An electronic device (1000) according to one embodiment of the present disclosure can remove noise from at least one light source candidate region by performing a morphological operation on at least one light source candidate region.

[0205] An electronic device (1000) according to one embodiment of the present disclosure can convert at least one light source candidate region into the form of a circular object. The electronic device (1000) can obtain a light source candidate map including at least one light source candidate region converted into the form of a circular object.

[0206] An electronic device (1000) according to one embodiment of the present disclosure can detect a ghost object of a predetermined color in an input image. The electronic device (1000) can apply a first weight to a light source candidate region that is point-symmetric with respect to the ghost object among at least one light source candidate region.

[0207] An electronic device (1000) according to one embodiment of the present disclosure can obtain a light source candidate map by applying a second weight to at least one light source candidate region based on the distance between at least one light source candidate region and the center of a luminance image.

[0208] A plurality of kernels according to one embodiment of the present disclosure may include a first kernel having a one-dimensional array long in the X-axis direction and a second kernel having a one-dimensional array long in the Y-axis direction. An electronic device (1000) according to one embodiment of the present disclosure may perform a first convolution operation in the Y-axis direction on a light source candidate map using the second kernel. The electronic device (1000) may perform a second convolution operation in the X-axis direction on the result of performing the first convolution operation using the first kernel. The electronic device (1000) may obtain at least one flare based on the result of performing the second convolution operation.

[0209] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0210] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

1. A method in which an electronic device (1000) provides a flare image, A step of acquiring an input image including at least one light source object (S210); A step (S220) of obtaining a luminance image corresponding to the input image using the luminance data of the input image; A step (S230) of obtaining a light source candidate map including at least one light source candidate region having a reference luminance value greater than or equal to the luminance value in the above luminance image; A step (S240) of obtaining a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region by performing a convolution operation on the light source candidate map using a plurality of kernels; and A method comprising the step (S250) of applying the flare layer to the input image and outputting a flare image in which the at least one flare is superimposed on the at least one light source object.

2. In claim 1, the step of acquiring the luminance image is, A method comprising the step of obtaining a luminance image downscaled from the input image by applying a max pooling method to the luminance data.

3. In claim 1 or 2, the step of acquiring the light source candidate map is, A method comprising the step of performing a morphological operation on the at least one light source candidate region to remove noise from the at least one light source candidate region.

4. In any one of claims 1 to 3, the step of acquiring the light source candidate map is, The step of converting the above at least one light source candidate region into the shape of a circular object; and A method comprising the step of obtaining a light source candidate map including at least one light source candidate region converted into the shape of the above-mentioned circular object.

5. In paragraph 4, the step of converting the at least one light source candidate region into the shape of the circular object is, The step of determining the center of gravity of the at least one light source candidate region as the center of the circular object; and A method comprising the step of determining the radius of the circular object such that the circular object has the same area as the at least one light source candidate region.

6. In any one of claims 1 to 5, the step of acquiring the light source candidate map is, A step of detecting a ghost object of a predetermined color in the above input image; and A method comprising the step of applying a first weight to a light source candidate region that is point-symmetric with respect to the ghost object among the above at least one light source candidate region.

7. In paragraph 6, the step of detecting the ghost object is, A step of determining whether the brightness value around the electronic device (1000) is greater than or equal to a threshold brightness value when capturing the input image, based on the metadata of the input image; and A method comprising the step of detecting a ghost object having a predetermined color in an input image based on the brightness value being greater than or equal to a threshold brightness value.

8. In any one of claims 1 to 7, the step of acquiring the light source candidate map is, A method comprising the step of obtaining a light source candidate map by applying a second weight to the at least one light source candidate region based on the distance between the at least one light source candidate region and the center of the luminance image.

9. In paragraph 8, the second weighting factor is, A method in which the distance between the at least one light source candidate region and the center of the luminance image becomes smaller as the distance between them increases, and the distance between the at least one light source candidate region and the center of the luminance image becomes smaller as the distance between them increases.

10. In any one of paragraphs 1 through 9, The plurality of kernels above include a first kernel of a one-dimensional array long in the X-axis direction and a second kernel of a one-dimensional array long in the Y-axis direction, and The step of acquiring the above flare layer is, A step of performing a first convolution operation in the Y-axis direction on the light source candidate map using the second kernel above; A step of performing a second convolution operation in the X-axis direction on the result of performing the first convolution operation using the first kernel; and A method comprising the step of obtaining at least one flare based on the result of performing the second convolution operation.

11. A method according to paragraph 10, wherein the element value of the first kernel and the element value of the second kernel decrease exponentially as they move away from the kernel center.

12. A method according to claim 10, wherein the shape and size of at least one flare are determined according to the size of the first kernel and the size of the second kernel.

13. In any one of claims 1 to 12, the step of obtaining the flare layer is, Step of selecting the color of at least one flare; and A method comprising the step of adjusting the RGB values ​​of at least one flare based on the selected color.

14. In paragraph 13, the above color is cyan or gold.

15. Camera (1511); Display unit (1111); Memory (1700) for storing at least one instruction; and It includes at least one processor (1300), By having the above at least one instruction executed individually or collectively by the above at least one processor (1300), the electronic device (1000) is, An input image including at least one light source object is obtained through the camera (1511), and Using the luminance data of the input image above, a luminance image corresponding to the input image is obtained, and A light source candidate map is obtained that includes at least one light source candidate region having a reference luminance value greater than or equal to the luminance value in the above luminance image, and By performing a convolution operation on the light source candidate map using multiple kernels, a flare layer including at least one horizontally elongated flare corresponding to at least one light source candidate region is obtained, and An electronic device that applies the flare layer to the input image and outputs a flare image in which the at least one flare is superimposed on the at least one light source object through the display unit (1111).

Images