Electronic device and method for providing bokeh image by means of electronic device

Abstract

Provided may be a method for: acquiring a luminance map having a higher bit depth than an input image; determining a distance difference between a reference pixel in focus and the current pixel in the input image by using a depth map corresponding to the input image; determining the size of a geometric kernel corresponding to the current pixel by using the distance difference between the reference pixel and the current pixel; and generating and outputting a bokeh image corresponding to the input image by using the luminance differences between the current pixel and neighboring pixels in the geometric kernel having the determined size and luminance weights based on the luminance map.

Description

Electronic devices and how they provide bokeh images

[0001] The present disclosure relates to a method for providing a bokeh image with a bokeh effect (optical background blur) applied and an electronic device therefor.

[0002] Optical background blur (bokeh) is a phenomenon that occurs when a lens with a large aperture has a small circle of confusion (CoC) in the focused area and a gradually larger circle of confusion in the unfocused area with increasing distance. It is often used as a technique to aesthetically separate the subject and background in a photograph.

[0003] Since the size of these circles of confusion is related to distance and also to the intensity of light, the distance and intensity of light must be well reflected to reproduce a beautiful bokeh image.

[0004] A method for providing a bokeh image by an electronic device according to an embodiment of the present disclosure may include: acquiring an input image; acquiring a luminance map corresponding to the input image having a higher bit depth than the input image; determining a distance difference between a reference pixel in focus and a current pixel in the input image using the depth map corresponding to the input image; determining a size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel; generating a bokeh image corresponding to the input image using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map; and outputting the bokeh image corresponding to the input image.

[0005] An electronic device according to one embodiment of the present disclosure may include a camera; a display unit; a memory storing a program or one or more instructions; and at least one processor. The electronic device may acquire an input image through the camera by the at least one processor executing the program or one or more instructions stored in the memory, acquire a luminance map corresponding to the input image having a higher bit depth than the input image, determine a distance difference between a reference pixel in focus and a current pixel in the input image using a depth map corresponding to the input image, determine a size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel, generate a bokeh image corresponding to the input image using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map, and output the bokeh image corresponding to the input image through the display unit.

[0006] FIG. 1 is a drawing for explaining an electronic device that provides bokeh images according to one embodiment of the present disclosure.

[0007] FIG. 2 is a flowchart illustrating a method for an electronic device to provide a bokeh image according to an embodiment of the present disclosure.

[0008] FIG. 3 is a drawing for explaining a luminance map according to one embodiment of the present disclosure.

[0009] FIG. 4 is a drawing for explaining a depth map according to one embodiment of the present disclosure.

[0010] FIG. 5 is a drawing for explaining the size of a geometric kernel according to one embodiment of the present disclosure.

[0011] FIG. 6 is a drawing for explaining the shape of a geometric kernel according to one embodiment of the present disclosure.

[0012] FIG. 7a is an operation illustrating an operation of determining weights of surrounding pixels according to one embodiment of the present disclosure.

[0013] FIG. 7b is a diagram illustrating an operation of determining weights for pixels within a geometric kernel according to one embodiment of the present disclosure.

[0014] FIG. 8 is a diagram for explaining a luminance weight based on a luminance map according to one embodiment of the present disclosure.

[0015] FIG. 9 is a diagram for explaining an operation of performing a bokeh operation on an input image according to one embodiment of the present disclosure.

[0016] FIG. 10 is a diagram for explaining an operation of outputting a bokeh image according to one embodiment of the present disclosure.

[0017] FIG. 11 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to provide a bokeh image corresponding to a preview image.

[0018] FIG. 12 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to provide a bokeh image corresponding to a stored image.

[0019] FIG. 13 is a flowchart illustrating a method for an electronic device according to an embodiment of the present disclosure to provide a bokeh image by adjusting the scale of an input image.

[0020] FIG. 14 is a block diagram illustrating the function of an electronic device according to one embodiment of the present disclosure.

[0021] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0022] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0023] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0024] Throughout this disclosure, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components may be included, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used in this disclosure refer to a unit that processes at least one function or operation, and "part" and "module" may be implemented as hardware or software, or as a combination of hardware and software.

[0025] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be stored in separate portions across multiple different memories.

[0026] Unless the context clearly dictates otherwise, singular forms (e.g., "a," "an," and "the") are to be understood to include plural referents. Thus, for example, the description "a component surface" may also include reference to one or more of such surfaces.

[0027] All functions or operations described in this document may be performed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing, and may include circuitry such as an Application Processor (AP), a Communication Processor (CP), a Graphical Processing Unit (GPU), a Neural Processing Unit (NPU), a Microprocessor Unit (MPU), a System on Chip (SoC), or an Integrated Chip (IC).

[0028] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0029] FIG. 1 is a drawing for explaining an electronic device (1000) that provides bokeh images according to one embodiment of the present disclosure.

[0030] An electronic device (1000) according to an embodiment of the present disclosure may be a portable handheld device. For example, the electronic device (1000) may include, but is not limited to, a portable telephone, a smart phone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), etc. The electronic device (1000) may be a device that can be worn by a user (wearable device). A wearable device may include, but is not limited to, at least one of an accessory type device (e.g., a watch, a ring, an wristband, an ankleband, a necklace, glasses, a contact lens), a head-mounted device (HMD), a fabric or clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit).

[0031] An electronic device (1000) according to one embodiment of the present disclosure may include a camera (image sensor). However, due to the size limitation of the electronic device (1000), the camera of the electronic device (1000) may include a small optical system. There is a limitation in implementing bokeh (optical background blur), which is one of the depth effects of a lens, in an image captured with a small optical system. Bokeh (optical background blur) may refer to a pleasing visual defect that occurs around a bright area of ​​an image when it is out of focus. In other words, bokeh (optical background blur) is a phenomenon that occurs when a focused area has a small circle of confusion (CoC) and an out-of-focus area has a gradually larger circle of confusion with distance in a camera with a large-diameter lens, and is widely used as a technique for aesthetically separating a subject and a background in a photograph. For example, in images with bokeh (hereinafter referred to as "bokeh images"), light from out-of-focus areas can appear as circular, wispy circles. Because the size of the circle of confusion is related to the distance from the in-focus area and the intensity of the light, these factors must be well reflected to produce a beautiful bokeh image.

[0032] According to one embodiment of the present disclosure, in order to implement the image quality effect of a large-diameter lens such as a DSLR (Digital Single-Lens Reflex) camera (100) in an electronic device (1000) using a small optical system, a computer vision technology or a deep learning-based AI image processing algorithm may be applied. For example, the electronic device (1000) according to one embodiment of the present disclosure can emulate the effect of a large-diameter lens (e.g., a bokeh effect) by using a bokeh operation in which light can cover darkness, unlike the existing blur operation in which gray is produced when black and white are calculated. In general, a blurred image (11) obtained through a blur operation for an input area (10) becomes gray when black and white overlap, but an actual characteristic of light is that when a bright light and a dark part overlap, the bright light covers the dark part. Therefore, according to one embodiment of the present disclosure, the electronic device (1000) can generate a bokeh image (12) that implements diffusion by focusing on the intensity of light in accordance with the characteristics of a large-diameter lens by minimizing or excluding the influence of pixels darker than the current pixel during calculation.

[0033] Hereinafter, a method for an electronic device (1000) to provide a bokeh image corresponding to an input image will be specifically described with reference to FIG. 2.

[0034] FIG. 2 is a flowchart illustrating a method for providing a bokeh image by an electronic device (1000) according to one embodiment of the present disclosure.

[0035] Referring to FIG. 2, a method for providing a bokeh image by an electronic device (1000) may include steps S210 to S260. In one embodiment of the present disclosure, steps S210 to S260 may be executed by at least one processor included in the electronic device (1000). The method for providing a bokeh image by the electronic device (1000) is not limited to that illustrated in FIG. 2, and in one or more embodiments, steps not illustrated in FIG. 2 may be further included, or some steps may be omitted.

[0036] In step S210, an electronic device (1000) according to an embodiment of the present disclosure can obtain an input image.

[0037] According to one embodiment of the present disclosure, an electronic device (1000) may acquire an input image for generating a bokeh image based on original image data acquired through a camera. The input image may be a raster image composed of pixels. Each pixel of the input image has a color depth value that can be displayed on the electronic device (1000). At this time, the bit depth of each pixel of the input image may be 8 bits (about 16.8 million colors), but is not limited thereto. For example, the bit depth of the input image may be 10 bits (about 1 billion colors) or 12 bits (about 68 billion colors). The bit depth refers to the number of bits used to indicate the color of one pixel in a bitmap image or a video frame buffer, or the number of bits used for each color element of one pixel. The bit depth may also be expressed as a bit length.

[0038] According to one embodiment of the present disclosure, the input image may be a still image (e.g., a portrait, a landscape, a night view, etc.) or a video (e.g., a cinematic video, a short form (short-length video content), etc.). The video may include a plurality of frame images.

[0039] According to one embodiment of the present disclosure, the input image may be a black-and-white image (e.g., a black-and-white photograph) or a color image (e.g., an RGB image). Furthermore, the input image may be a preview image (e.g., a live image, a viewfinder image) or an image previously stored in memory. For example, if a user wishes to generate a bokeh image for a real-time captured image, the preview image may be used as the input image. Furthermore, if a user wishes to edit a portion of a previously stored image into a bokeh image, an image selected by the user from a list of previously stored images may be used as the input image.

[0040] In step S220, the electronic device (1000) according to one embodiment of the present disclosure can obtain a luminance map corresponding to an input image having a higher bit depth than the input image.

[0041] According to one embodiment of the present disclosure, in order for an electronic device (1000) having a small optical system to operate similarly to an actual optical lens, luminance information with a high bit depth (bit length) is required. However, in the process of saving the original image data as an input image in the JPEG or TIFF format, there may be cases where the luminance information of the original image data cannot be fully contained in the input image due to limitations in the bit depth. Therefore, the electronic device (1000) may generate a luminance map with a higher bit depth than the input image for reference when generating a bokeh image. For example, when the bit depth of the input image is 8 bits, the electronic device (1000) may generate a luminance map with 10 to 16 bits.

[0042] According to one embodiment of the present disclosure, the luminance map may be a one-dimensional map in which only the luminance component is extracted by preprocessing the raw image data. The luminance map may be the same scale as the original image data, but in this case, a memory loading issue may easily occur. Therefore, the electronic device (1000) may generate a luminance map with a smaller scale than the original image data and then refer to the luminance map when performing a bokeh operation on each pixel. The bokeh operation may mean performing a convolution operation that multiplies the element values ​​(weights) of the kernel to pixels overlapping with the kernel and adds them all together.

[0043] According to one embodiment of the present disclosure, the electronic device (1000) can obtain a downscaled luminance map by applying a max pooling method to luminance data extracted from preprocessed original image data. Pooling may mean lowering the resolution of the feature map by reducing the height and width of the feature map, but maintaining the features of the map necessary for classification. Pooling may also be expressed as downsampling. Max pooling means a method of extracting only the maximum value within an area overlapping a filter (kernel). For example, the electronic device (1000) can generate a downscaled luminance map by extracting the maximum luminance value of each area overlapping a filter (kernel) using the max pooling method.

[0044] According to one embodiment of the present disclosure, the electronic device (1000) can generate a luminance map using original image data before gamma is applied or original image data that has been degamma-processed. For example, if gamma is applied to preprocessed original image data corresponding to an input image, the electronic device (1000) can perform degamma processing on the preprocessed original image data. In addition, the electronic device (1000) can obtain a downscaled luminance map by applying a max pooling method to luminance data extracted from the degamma-processed original image data.

[0045] Gamma can mean nonlinearly transforming a light intensity signal using a nonlinear transfer function. Since human vision responds nonlinearly to brightness according to Weber's law, if light brightness is recorded linearly within a limited bit depth, such as 8 bits per channel, the phenomenon of posterization occurs when the brightness changes in dark areas, where the human eye is sensitive, rather than feeling smooth. Therefore, in order to show optimal image quality within the limits of a given bit depth, it is necessary to encode nonlinearly to record dark areas in more detail.

[0046] Degamma may refer to linearly transforming a nonlinearly transformed light intensity signal. Since the intensity (strength) of light must be accurately reflected in order for the optical background blur (Bokeh) to occur in a form similar to an actual optical effect, the electronic device (1000) may generate a luminance map using the original image data before gamma processing or the original image data after degamma processing.

[0047] In step S230, the electronic device (1000) according to one embodiment of the present disclosure can determine the distance difference between a reference pixel in focus and a current pixel within the input image by using a depth map corresponding to the input image.

[0048] According to one embodiment of the present disclosure, the electronic device (1000) can generate a depth map corresponding to an input image. For example, the electronic device (1000) can generate a depth map by estimating the depth of the input image. Depth estimation may refer to the task of extracting three-dimensional depth (or distance) information from the image.

[0049] According to one embodiment of the present disclosure, the electronic device (1000) can generate a depth map corresponding to an input image using monocular depth estimation, stereo depth estimation, or depth estimation using an active sensor (e.g., a depth sensor, a ToF sensor).

[0050] Monocular depth estimation is the task of estimating depth information from images from a typical single camera. For example, the electronic device (1000) can estimate depth by utilizing a triangulation method to pair images from a monocular camera with images from a previous timestamp and images from a current timestamp. Monocular depth estimation can be performed using an artificial intelligence (AI) model based on deep learning. Representative deep learning algorithms include Monodepth, DenseDepth, and FastDepth.

[0051] Stereo depth estimation can refer to estimating a depth map using the disparity between two cameras (stereo cameras). Disparity can be calculated by calculating the distance between identical points in the images captured by the two cameras. Closer objects have a higher disparity value, while farther away objects have a lower disparity value.

[0052] The electronic device (1000) can also generate a depth map using a separate depth sensor (depth camera). For example, a ToF (Time of Flight) sensor can measure the distance to an object using signals (near-infrared, ultrasound, laser, etc.), and a CCD (Charge Coupled Device) camera can obtain depth information in units of image pixels.

[0053] According to one embodiment of the present disclosure, the electronic device (1000) can determine a reference pixel that is in focus within an input image. For example, when a focused region is selected from a depth map, the electronic device (1000) can determine the region as a reference pixel. There may be multiple reference pixels. When there are multiple reference pixels, the electronic device (1000) can determine a reference area that is in focus within the input image.

[0054] According to one embodiment of the present disclosure, the electronic device (1000) may determine a distance difference between a focused reference pixel and a current pixel (corresponding pixel) on which a bokeh operation is performed using a depth map. At this time, the distance difference between the reference pixel and the current pixel (corresponding pixel) may be determined by considering a depth difference. That is, the distance difference between the reference pixel and the current pixel may mean a depth difference on the depth map. For example, if the current pixel on which the bokeh operation is performed is a focused reference pixel, the distance difference between the reference pixel and the current pixel may be 0 (or a value close to 0). In addition, even if the current pixel is located near the reference pixel, if the depth difference is large, the distance difference between the reference pixel and the current pixel may be determined to be large.

[0055] In step S240, the electronic device (1000) according to one embodiment of the present disclosure may determine the size of a geometric kernel corresponding to a current pixel in an input image by using the distance difference between a reference pixel and a current pixel. The geometric kernel may be expressed as a filter or an area for bokeh operation of the corresponding pixel.

[0056] According to one embodiment of the present disclosure, the electronic device (1000) may determine a different size of a geometric kernel for each pixel to implement a bokeh effect. For example, the electronic device (1000) may determine a larger size of a geometric kernel as the distance difference between a reference pixel and a current pixel increases. In this case, the reference pixel appears sharp because it is an area in focus, and the bokeh effect may be greater for pixels with a larger distance difference from the reference pixel.

[0057] In addition, according to one embodiment of the present disclosure, the electronic device (1000) may determine the size of the geometric kernel corresponding to the current pixel in the input image by using the aperture value of the target lens in addition to the distance difference between the reference pixel and the current pixel. For example, the electronic device (1000) may determine the size of the geometric kernel to be larger as the F number of the target lens is smaller (the size of the aperture is larger). The electronic device (1000) may determine the size of the geometric kernel to be larger as the focal length of the target lens is longer. Here, the target lens may be a large-diameter lens. The target lens may be a lens set by default for the bokeh effect, or may be a lens selected by the user. For example, the electronic device (1000) may use a basic large-diameter lens as the target lens if there is no separate input from the user, and may use the specific lens selected by the user as the target lens if the user selects a specific lens for implementing the bokeh effect. The operation of the electronic device (1000) to determine the size of the geometric kernel corresponding to the current pixel will be examined in more detail later with reference to FIG. 5.

[0058] According to one embodiment of the present disclosure, the electronic device (1000) can determine the shape of the geometric kernel based on the characteristics of the target lens. For example, the electronic device (1000) can determine the shape of the geometric kernel as one of, but not limited to, a circle, an ellipse, a polygon (e.g., a pentagon, a hexagon, etc.), a spiral shape, or a donut shape.

[0059] In step S250, the electronic device (1000) according to one embodiment of the present disclosure can generate a bokeh image corresponding to an input image by using a luminance difference between a current pixel and surrounding pixels within a geometric kernel and a luminance weight based on a luminance map.

[0060] In general, blur becomes gray when black and white overlap, but since the characteristic of light is that bright light covers dark parts when they overlap, the electronic device (1000) can perform a bokeh operation on the current pixel by using the difference in luminance between the current pixel and surrounding pixels within the geometric kernel having the size determined in step S240. That is, unlike using all pixels in a general kernel operation, the electronic device (1000) can minimize or exclude the influence of surrounding pixels that are darker than the current pixel to be operated on within the kernel on the operation of the current pixel. For example, the electronic device (1000) can determine the weight of surrounding pixels that have lower luminance than the current pixel within the geometric kernel as 0 (or a small value close to 0). On the other hand, the electronic device (1000) can determine the weight of surrounding pixels that have higher or equal luminance than the current pixel within the geometric kernel as 1 (or a large value close to 1). Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) may determine the weight of the current pixel to be a small value close to 0 when the current pixel is relatively dark compared to the surrounding pixels.

[0061] According to one embodiment of the present disclosure, the electronic device (1000) may perform a convolution operation that multiplies the image values ​​of pixels within a geometric kernel by element values ​​(weights) of the geometric kernel and adds them all together to generate a bokeh operation value of the current pixel. At this time, since surrounding pixels with lower luminance than the current pixel are multiplied by a weight of 0, surrounding pixels with lower luminance than the current pixel may have little influence on the bokeh operation of the current pixel.

[0062] Therefore, if there are surrounding pixels with higher luminance than the current pixel within the geometric kernel, the bokeh operation value of the current pixel may be determined to be a brighter value than the image value of the current pixel. On the other hand, if there are no surrounding pixels with higher luminance than the current pixel within the geometric kernel, the bokeh operation value of the current pixel may be the same as (or similar to) the image value of the current pixel.

[0063] According to one embodiment of the present disclosure, when gamma is applied to an input image, the electronic device (1000) may perform de-gamma processing on the input image to obtain a de-gamma input image. In addition, the electronic device (1000) may perform a bokeh operation on the de-gamma input image. Performing the bokeh operation may mean calculating a bokeh operation value of each pixel through a convolution operation. According to one embodiment of the present disclosure, the electronic device (1000) uses the difference in luminance between a current pixel and surrounding pixels to minimize or exclude the influence of pixels darker than the current pixel on the operation of the current pixel, so that the input image must have linear characteristics as they are in the input of the image sensor. Accordingly, since the input image for generating the bokeh image must be data to which gamma is not applied, the electronic device (1000) must apply de-gamma, which is an inverse function, again when gamma is applied to the input image so that the values ​​of the input image can be proportional to the intensity of light. Degamma processing can be applied to each pixel or to the entire input image.

[0064] According to one embodiment of the present disclosure, the electronic device (1000) may refer to a luminance map when generating a bokeh image corresponding to an input image. For example, if the luminance value of at least one pixel within a geometric kernel is a maximum luminance value, the electronic device (1000) may determine a luminance weight of at least one pixel using the luminance map. In addition, the electronic device (1000) may determine a bokeh operation value of the current pixel by reflecting the luminance weight of at least one pixel.

[0065] If the bit length or bit depth of the input image is 8 bits, the brightness value can only be expressed as a maximum of 255. Therefore, the electronic device (1000) can refer to the brightness map to distinguish whether the current pixel has a high brightness (e.g., the sky) or an ultra-high brightness (e.g., the sun) when the brightness value of the current pixel is 255. Since the brightness map has a higher bit number (e.g., 12 to 16 bits) than the input image, when the electronic device (1000) refers to the brightness map, it can distinguish whether the corresponding pixel is a light source or a reflector. For example, in the input image, a fluorescent light and a white wall may both have a brightness value of 255, but when referring to the brightness map, the brightness value of the fluorescent light may be 45000, and the brightness value of the white wall may be 320. Accordingly, when the electronic device (1000) determines the bokeh operation value of the current pixel by reflecting the luminance weight based on the luminance map, the bokeh operation value of the current pixel may be determined to be larger than before referring to the luminance map. That is, when a light source exists in the background, the brightness (intensity of light) of the light source strongly affects the surrounding pixels, so the area around the light source may appear bright and diffused.

[0066] In step S260, the electronic device (1000) according to one embodiment of the present disclosure can output a bokeh image corresponding to the input image.

[0067] According to one embodiment of the present disclosure, the electronic device (1000) can output a bokeh image in real time on a preview screen (or live screen, viewfinder). At this time, the user can check the bokeh image displayed on the preview screen and select a capture button. When the electronic device (1000) receives an input from the user to select the capture button, the electronic device (1000) can capture the bokeh image currently displayed on the preview screen and store the bokeh image in memory.

[0068] According to one embodiment of the present disclosure, the electronic device (1000) may display a bokeh image corresponding to an input image previously stored in memory on an editing screen. The user may check the bokeh image through the editing screen, and if the edited bokeh image is satisfactory, the user may complete the editing. At this time, the electronic device (1000) may store the edited bokeh image corresponding to the input image in the memory.

[0069] According to one embodiment of the present disclosure, when an input image is a video, the electronic device (1000) can generate bokeh images corresponding to each image frame included in the video, and play the video with the bokeh effect applied through a shooting screen or an editing screen.

[0070] According to one embodiment of the present disclosure, as the distance from the focused portion increases, the geometric kernel becomes larger, and the bokeh operation is performed mainly on pixels brighter than the current pixel within the geometric kernel, so that a bokeh effect can be produced in which brighter portions of the background portion of the input image appear to be widely spread out. In particular, an effect similar to the spread of light in an actual optical system can be implemented. Therefore, by reproducing (emulating) the effects of a large-diameter lens even in a smartphone with a small optical system, a user can easily shoot movie videos or short-form content using a smartphone without a large and heavy camera system.

[0071] Hereinafter, with reference to FIGS. 3 to 10, the operation of the electronic device (1000) providing a bokeh image will be examined in more detail.

[0072] FIG. 3 is a drawing for explaining a luminance map according to one embodiment of the present disclosure.

[0073] Referring to FIG. 3, the electronic device (1000) can acquire an input image (200) and generate a luminance map (300) corresponding to the input image (200) having a higher bit depth than the input image (200). For example, when the electronic device (1000) acquires an 8-bit input image (200), it can generate a 10-16-bit luminance map (300).

[0074] The electronic device (1000) can preprocess raw image data and extract a luminance component from the preprocessed raw image data to generate a luminance map (300) having a higher bit count than the input image (200). Since the luminance map (300) has a higher bit count than the input image (200), its capacity may be large. Accordingly, the electronic device (1000) can downscale the luminance map (300) to have a smaller size than the input image (200). For example, the electronic device (1000) can obtain a downscaled luminance map (300) by applying a max pooling method to luminance data extracted from the preprocessed raw image data.

[0075] Referring to 301 of FIG. 3, when the electronic device (1000) generates a downscaled luminance map (300) using a max pooling method using a 2X2 filter, only the maximum luminance value can be extracted within the area overlapping with the filter (see mathematical equation 1). Since a 2X2 filter is used, the luminance map (300) can be scaled down to 1 / 4 compared to the input image (200).

[0076]

[0077] [Mathematical Formula 1]

[0078]

[0079] Thereafter, the electronic device (1000) can refer to the luminance map (300) when performing bokeh calculations on each pixel of the input image (200). For example, if the luminance value of the current pixel or surrounding pixels in an 8-bit input image (200) exceeds 255, the electronic device (1000) can refer to the luminance map (300) to distinguish whether the luminance is ultra-high (e.g., the sun) or high (e.g., the sky) even when it is 255 or higher.

[0080] FIG. 4 is a drawing for explaining a depth map (400) according to one embodiment of the present disclosure.

[0081] Referring to FIG. 4, the electronic device (1000) can generate a depth map (400) corresponding to the input image (200) through depth estimation that extracts three-dimensional depth (or distance) information from the input image (200). The depth estimation may include, but is not limited to, monocular depth estimation, stereo depth estimation, or depth estimation using an active sensor (e.g., depth sensor, ToF sensor).

[0082] According to one embodiment of the present disclosure, when an input image (200) is acquired through a front camera (e.g., a single camera) of an electronic device (1000), the electronic device (1000) can generate a depth map (400) corresponding to the input image (200) using deep learning-based AI technology.

[0083] According to one embodiment of the present disclosure, when an input image (200) is acquired through a rear camera (e.g., wide-angle, ultra-wide-angle, telephoto) of an electronic device (1000), the electronic device (1000) can acquire a depth map (400) corresponding to the input image (200) through binocular depth estimation. For example, the electronic device (1000) can acquire a depth map (400) corresponding to the input image (200) by using a wide-angle camera and an ultra-wide-angle camera, or can acquire a depth map (400) corresponding to the input image (200) by using a telephoto camera and a wide-angle camera.

[0084] According to one embodiment of the present disclosure, when an active sensor is mounted on an electronic device (1000), the electronic device (1000) can generate a depth map (400) corresponding to an input image (200) using the active sensor.

[0085] According to one embodiment of the present disclosure, the electronic device (1000) can generate a depth map (400) so that it can be matched 1:1 with the input image (200). In addition, when the size of the depth map (400) is small, the electronic device (1000) can upscale and use the depth map (400) so that it corresponds to the size of the input image (200). The depth map (400) can be used to determine the size of the geometric kernel (area for bokeh operation) of each pixel included in the input image (200). Hereinafter, the size of the geometric kernel will be described with reference to FIG. 5.

[0086] FIG. 5 is a drawing for explaining the size of a geometric kernel according to one embodiment of the present disclosure.

[0087] According to one embodiment of the present disclosure, the electronic device (1000) can determine the size of a geometric kernel for performing a bokeh operation for each pixel using the following mathematical expression 2.

[0088] [Equation 2]

[0089]

[0090] F(i,j)-F target means the distance difference between the focused part and each pixel. When the focused part is selected in the depth map (400), the selected part is F target This becomes the electronic device (1000) uses the depth map (400) to determine the focused part (F target ) and the distance difference between each pixel (F(i,j)) can be determined. β is a value tuned to correspond to the actual aperture value of the target lens, and can be pre-calculated as a value corresponding to a given focal length and F number.

[0091] Therefore, according to mathematical expression 2, the distance difference between the reference pixel (the focused pixel) and the current pixel (F(i,j)-F target ) can be determined to be large, the size of the geometric kernel (R(i,j)) can be determined to be large. In this case, the reference pixel is a focused area, so it appears sharp, and the bokeh effect can be large for pixels with a large distance difference from the reference pixel. The electronic device (1000) can determine the size of the geometric kernel to be large as the F number of the target lens is small (the size of the aperture is large). The electronic device (1000) can determine the size of the geometric kernel to be large as the focal length of the target lens is long.

[0092] Referring to FIG. 5, when a person's face is in focus in an input image (200), the size of the first geometric kernel (510) for pixels of the person's face portion may be relatively small. On the other hand, the size of the second geometric kernel (520) for pixels of the background behind which the distance from the person's face is large may be relatively large. The geometric kernel may be an area for performing a bokeh operation on the corresponding pixel. That is, the electronic device (1000) performs a bokeh operation on the corresponding pixel based on the luminance difference between pixels within the geometric kernel. Therefore, when performing a bokeh operation, the larger the geometric kernel, the more it may be influenced by the luminance of surrounding pixels.

[0093] FIG. 6 is a drawing for explaining the shape of a geometric kernel according to one embodiment of the present disclosure.

[0094] Referring to FIG. 6, the shape of the geometric kernel for the bokeh operation may be, but is not limited to, a circle (610), an ellipse (620), a pentagon (630), etc. For example, the shape of the geometric kernel may be a donut shape, a tornado shape, a hexagon, etc. In the mathematical expression 3 below, when the geometric kernel is a circle (610), k and l may be the same (e.g., k=15, l=15), and when the geometric kernel is an ellipse (620), k and l may be different (e.g., k=16, l=10).

[0095] According to one embodiment of the present disclosure, bokeh calculations can be performed on pixels within a geometric kernel using their luminance values, as described in Equation 3. That is, only pixels within the geometric kernel can be masked. Pixels outside the geometric kernel are not considered in the bokeh calculations.

[0096] [Equation 3]

[0097]

[0098] 1, if

[0099] 0, otherwise

[0100] Meanwhile, according to one embodiment of the present disclosure, among pixels within a geometric kernel, pixels darker than the current pixel can have their influence on the bokeh operation of the current pixel minimized or excluded. Refer to FIGS. 7A and 7B.

[0101] FIG. 7a is an operation illustrating an operation of determining weights of surrounding pixels according to one embodiment of the present disclosure.

[0102] According to the characteristics of light, when a bright part and a dark part overlap, the bright light covers the dark part. Therefore, according to one embodiment of the present disclosure, the electronic device (1000) can emulate the characteristics of actual light by determining the weights of surrounding pixels having lower brightness than the current pixel (700) in the input image (710) as 0 (or a small number close to 0) and determining the weights of surrounding pixels having higher brightness than the current pixel (700) as 1 (or a large number close to 1). That is, referring to mathematical expression 4, the electronic device (1000) can generate a masking image (720) by masking only pixels that are brighter than the current pixel (700) on which the bokeh operation is to be performed.

[0103] [Equation 4]

[0104] M (i, j, k, l)=

[0105] 1, if I(i+k, j+l) ≥ I(i, j)

[0106] 0, otherwise

[0107] Meanwhile, in order to determine the weights of surrounding pixels based on the luminance value, the data of the input image must have linearity as is, as is, as a sensor input without gamma applied to match the human eye. Therefore, the data of the input image must be data that has not been subjected to gamma. Accordingly, when the data of the input image is subjected to gamma, the electronic device (1000) applies degamma, which is an inverse function, so that the luminance value of each pixel can be proportional to the intensity of light. Meanwhile, the input image may have various color spaces.

[0108] FIG. 7b is a diagram illustrating an operation of determining weights for pixels within a geometric kernel according to one embodiment of the present disclosure.

[0109] Referring to FIG. 7b, among the pixels within the geometric kernel (701) corresponding to the current pixel (700), the weights of pixels brighter than the current pixel (700) are determined as 1 (or a number close to 1), and the weights of pixels darker than the current pixel (700) are determined as 0 (or a number close to 0), thereby generating a masking image (730). That is, even if a pixel is brighter than the current pixel (700), a pixel that falls outside the area corresponding to the geometric kernel (701) can be excluded when performing a bokeh operation on the current pixel (700). Meanwhile, if the current pixel (700) is darker than the surrounding pixels, the weight of the current pixel (700) can be determined as 0 (or a number close to 0). Additionally, if the current pixel is not much darker than the surrounding pixels, the weight of the current pixel (700) can also be determined as 1 (or a number close to 1).

[0110] In Fig. 7b, the case where the geometric kernel (701) is circular is described as an example, but it is not limited thereto.

[0111] FIG. 8 is a diagram for explaining a luminance weight based on a luminance map (300) according to one embodiment of the present disclosure.

[0112] Referring to FIG. 8, there may be five pixels with a higher luminance value than the current pixel within the geometric kernel. At this time, if at least one pixel value within the geometric kernel is the maximum luminance value, the electronic device (1000) may determine the luminance weight of at least one pixel using the luminance map (300). For example, if the number of bits of the input image is 8 bits and the luminance value of the fifth pixel (800) is 255, the electronic device (1000) may refer to the luminance map (300) to accurately determine the luminance value of the fifth pixel (800). As a result of referring to the luminance map (300), if the luminance weight of the fifth pixel (800) is 1023, the luminance weight may be reflected for the fifth pixel (800) when performing the bokeh operation of the current pixel.

[0113] According to one embodiment of the present disclosure, the electronic device (1000) can perform a bokeh operation on each pixel included in the input image using the input image, the luminance map (300), and the depth map (400). With reference to FIG. 9, the operation of the electronic device (1000) performing the bokeh operation will be described in more detail.

[0114] FIG. 9 is a diagram for explaining an operation of performing a bokeh operation on an input image according to one embodiment of the present disclosure.

[0115] According to one embodiment of the present disclosure, the electronic device (1000) can perform a bokeh operation on each pixel in an input image using mathematical expression 5. That is, the electronic device (1000) can perform a convolution operation that multiplies the image values ​​of pixels brighter than the current pixel within the geometric kernel by element values ​​(weights) of the geometric kernel and adds them. In addition, the electronic device (1000) can generate a result value of performing the convolution operation as a bokeh operation value of the current pixel.

[0116] [Equation 5]

[0117]

[0118] Here, I(i+k, j+l) is the image value of the pixel, and M(i, j, k, l) and w(i+k, j+l) may mean weights using luminance. That is, M(i, j, k, l) may mean that the weight of a pixel brighter than the current pixel is 1, and the weight of a pixel darker than the current pixel is 0, and w(i+k, j+l) may mean a luminance weight referring to a luminance map (300).

[0119] For example, let's assume that among the pixels in the geometric kernel, the first to fifth pixels have a higher luminance value than the current pixel, the brightness value of the current pixel is 100, the brightness value of the first pixel is 200, the brightness value of the second pixel is 210, the brightness value of the third pixel is 120, the brightness value of the fourth pixel is 120, and the brightness value of the fifth pixel is 200, and the weight is 1. At this time, if the bokeh operation value of the current pixel is calculated using mathematical expression 5, it becomes (200+210+120+120+200) / 5 = 170. That is, the original brightness value of the current pixel was 100, but it is expressed as bright as 170 in the bokeh image due to the influence of the surrounding pixels. Meanwhile, if the current pixel and the surrounding pixels are green, the current pixel may be expressed as a brighter green than the input image in the bokeh image. Since none of the first to fifth pixels have a brightness value of 255, the luminance weight (w) is not applied when calculating the bokeh of the current pixel. While the example described above assumes that all weights are 1, the weights may be set differently for each pixel, taking into account the distance between the current pixel and surrounding pixels, depending on the characteristics of the target lens.

[0120] Referring to 910 of FIG. 9, according to an embodiment of the present disclosure, the sizes of geometric kernels for performing bokeh operations on each pixel may be different. For example, when the lower right is the focused portion, the size of the geometric kernel (903) corresponding to the lower right pixel may be small, and the size of the geometric kernel (901) corresponding to the center pixel may be large. Meanwhile, the shapes of the geometric kernels may also vary. For example, the shape of the geometric kernel (902) corresponding to the lower left pixel may be an ellipse, and the shape of the geometric kernel (901) corresponding to the center pixel may be a circle. The electronic device (1000) according to an embodiment of the present disclosure determines the shape and size of the geometric kernel corresponding to each pixel, and performs a convolution operation on the image values ​​of pixels brighter than the current pixel among the pixels within the geometric kernel, thereby obtaining the bokeh operation value of each pixel. According to an embodiment of the present disclosure, the electronic device (1000) may generate a bokeh image corresponding to an input image using the bokeh operation value of each pixel.

[0121] Referring to 930 of FIG. 9, an electronic device (1000) according to an embodiment of the present disclosure can generate a bokeh image that implements an effect (bokeh effect) similar to the spread of light in an actual optical system by performing a bokeh operation considering the difference in brightness between a current pixel and surrounding pixels. On the other hand, referring to 920 of FIG. 9, in a Gaussian blur image, black and white are calculated and expressed as gray, so it is difficult to implement an effect similar to the spread of light in an actual optical system.

[0122] FIG. 10 is a diagram for explaining an operation of outputting a bokeh image according to one embodiment of the present disclosure.

[0123] According to one embodiment of the present disclosure, when an electronic device (1000) acquires an input image (200), it can generate a luminance map (300) having a higher bit count than the input image (200) and a depth map (400) corresponding to the input image, and generate a bokeh image (1010) corresponding to the input image (200) by referring to the luminance map (300) and the depth map (400). For example, the electronic device (1000) can distinguish whether a pixel having the highest luminance value is ultra-high luminance (e.g., the sun) or high luminance (e.g., the sky) by referring to the luminance map (300) when performing a bokeh operation on each pixel. In addition, the electronic device (1000) can set a different size of an area (geometric kernel) for a bokeh operation on each pixel by referring to the depth map (400). At this time, the electronic device (1000) can generate a bokeh image (1010) that implements diffusion by focusing on the intensity of light in accordance with the characteristics of a large-diameter lens by minimizing or excluding the influence of pixels darker than the current pixel during calculation. Accordingly, the bokeh image (1010) can implement an effect in which brighter parts of the background area are more diffused than darker parts.

[0124] According to one embodiment of the present disclosure, the electronic device (1000) can output a bokeh image (1010) through a display unit. If the input image (200) is a photograph, the electronic device (1000) can display a bokeh image (1010) corresponding to the photograph on the screen, and if the input image (200) is a video, the electronic device (1000) can play a bokeh video including bokeh images. Therefore, even in an electronic device (1000) having a small optical system (e.g., a smartphone), by reproducing (emulating) large-diameter lens effects, a user can easily shoot movie images or short-form content through the electronic device (1000) without a large and heavy camera system.

[0125] Hereinafter, with reference to FIGS. 11 and 12, the operation of the electronic device (1000) providing a GUI (Graphical User Interface) related to bokeh image generation will be described.

[0126] FIG. 11 is a drawing for explaining an operation of an electronic device (1000) according to one embodiment of the present disclosure to provide a bokeh image corresponding to a preview image.

[0127] Referring to screen 1110 of FIG. 11, the electronic device (1000) can execute a camera application according to user input. At this time, the user can select the portrait mode (1101) in the execution window of the camera application. If the user selects the portrait mode (1101), the electronic device (1000) can adjust the camera setting values ​​to default values ​​corresponding to the portrait mode (1101).

[0128] Meanwhile, referring to screen 1120 of FIG. 11, the electronic device (1000) may display an icon (1102) for setting an effect related to the portrait mode on the preview screen. When the user selects the icon (1102), the electronic device (1000) may provide a list of various effects related to the portrait mode. For example, the electronic device (1000) may provide a list including bokeh (blur) (applying a blur effect to the background), studio (applying a lighting effect to the person and applying a blur effect to the background), high-key mono (emphasizing the person through a lighting effect in a black-and-white photo with a white background), low-key mono (emphasizing the person through a lighting effect in a black-and-white photo with a black background), color background (changing the background color and applying a lighting effect to the person, automatically setting the background color to a color that matches the photo by detecting the color of the person's clothes), color point (maintaining the color of the person and converting the background to black and white), etc.

[0129] Referring to screen 1130 of FIG. 11, a user can select an icon (1103) representing a bokeh effect from the effect list. The user can also select the intensity of the bokeh effect. In response to an input selecting the icon (1103), the electronic device (1000) can generate a bokeh image with the bokeh effect applied to the preview image and output the bokeh image to the display unit. The user can check the bokeh image and select a capture button. When the user selects the capture button, the electronic device (1000) can capture the bokeh image with the bokeh effect applied to the preview image and store it in memory.

[0130] In Fig. 11, an example is described in which a bokeh image with a bokeh effect applied to a preview image is generated when a user selects a portrait mode and then selects a bokeh effect, but the present invention is not limited thereto. When a user selects a portrait mode, the electronic device (1000) may automatically perform a bokeh operation on the preview image and display a bokeh image corresponding to the preview image.

[0131] FIG. 12 is a diagram for explaining an operation of an electronic device (1000) according to one embodiment of the present disclosure to provide a bokeh image corresponding to a stored image.

[0132] Referring to screen 1210 of FIG. 12, a user can select one from a list of previously stored images. At this time, the electronic device (1000) can display the stored image selected by the user on the screen. The user can check the stored image displayed on the screen and select an icon (1201) for changing the background effect. The electronic device (1000) can provide a list of effects in response to a user input of selecting the icon (1201) for changing the background effect. When the user selects an icon (1202) indicating a bokeh effect from the list of effects, the electronic device (1000) can generate a bokeh image by applying the bokeh effect to the stored image selected by the user. For example, the electronic device (1000) can generate a luminance map having a higher bit number than the stored image based on preprocessed original image data stored together with the stored image selected by the user. In addition, the electronic device (1000) can obtain a depth map corresponding to the stored image. The electronic device (1000) can determine the size of a geometric kernel for each pixel of a stored image using a depth map, and perform a bokeh operation on each pixel using a luminance difference between the current pixel and surrounding pixels within the geometric kernel and a luminance weight based on a luminance map, thereby generating a bokeh image corresponding to the stored image. The electronic device (1000) can degamma process the stored image during the bokeh operation.

[0133] Referring to screen 1220 of FIG. 12, the electronic device (1000) can display a bokeh image with a bokeh effect applied to a stored image selected by the user on the display unit. Accordingly, the electronic device (1000) can easily provide a bokeh image that implements an effect similar to the spread of light in an actual optical system, in response to a user's editing request for a previously stored image.

[0134] Meanwhile, the electronic device (1000) may also generate a bokeh image by adjusting the scale of the input image to increase the processing speed for generating a bokeh image. Referring to FIG. 13, the operation of the electronic device (1000) generating a bokeh image by adjusting the scale of the input image will be examined in detail.

[0135] FIG. 13 is a flowchart illustrating a method for providing a bokeh image by adjusting the scale of an input image by an electronic device (1000) according to one embodiment of the present disclosure.

[0136] Referring to FIG. 13, a method for providing a bokeh image by adjusting the scale of an input image by an electronic device (1000) may include steps S1310 to S1370. In one embodiment of the present disclosure, steps S1310 to S1370 may be executed by at least one processor included in the electronic device (1000). The method for providing a bokeh image by adjusting the scale of an input image by an electronic device (1000) is not limited to that illustrated in FIG. 13, and in one or more embodiments, steps not illustrated in FIG. 13 may be further included, or some steps may be omitted.

[0137] In step S1310, an electronic device (1000) according to an embodiment of the present disclosure can obtain an input image.

[0138] According to one embodiment of the present disclosure, the input image may be a still image (e.g., a portrait, a landscape, a night view, etc.) or a video (e.g., a cinematic video, a short form (short-form video content), etc.). According to one embodiment of the present disclosure, the input image may be a black-and-white image (e.g., a black-and-white photo) or a color image (e.g., an RGB image). Furthermore, the input image may be a preview image (e.g., a live image, a viewfinder image) or an image previously stored in memory.

[0139] Step S1310 corresponds to step S210 of Fig. 2, so redundant description will be omitted.

[0140] In step S1320, the electronic device (1000) according to one embodiment of the present disclosure can perform downscaling on the input image to obtain a low-resolution input image.

[0141] According to one embodiment of the present disclosure, the electronic device (1000) may perform downscaling on an input image using a general algorithm, or may perform downscaling on an input image using an AI model.

[0142] In step S1330, the electronic device (1000) according to one embodiment of the present disclosure can perform a bokeh operation on a low-resolution input image to generate a low-resolution bokeh image corresponding to the low-resolution input image.

[0143] An electronic device (1000) of the present disclosure can obtain a low-resolution bokeh image by performing a bokeh operation on a low-resolution input image according to the method described in FIG. 2. For example, the electronic device (1000) can determine a distance difference between a reference pixel in focus and a current pixel within the low-resolution input image using a depth map corresponding to the low-resolution input image, and can determine the size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel and the aperture value of a target lens. The electronic device (1000) can perform a bokeh operation on the current pixel using a luminance difference between the current pixel and surrounding pixels within the geometric kernel and a luminance weighting based on a luminance map. The electronic device (1000) can generate a low-resolution bokeh image corresponding to the low-resolution input image using a bokeh operation value for each pixel of the low-resolution input image.

[0144] In step S1340, the electronic device (1000) according to one embodiment of the present disclosure can upscale a low-resolution bokeh image.

[0145] According to one embodiment of the present disclosure, the electronic device (1000) may upscale a low-resolution bokeh image using a general algorithm or may upscale a low-resolution bokeh image using an AI model.

[0146] In step S1350, the electronic device (1000) according to one embodiment of the present disclosure can extract an image of a first area in focus from an input image, and extract an image of a second area excluding the first area from an upscaled low-resolution bokeh image.

[0147] For example, the electronic device (1000) can extract an image of a first area in focus from an original input image that has not been downscaled. In addition, the electronic device (1000) can extract an image of a second area, excluding the first area, from an upscaled low-resolution bokeh image.

[0148] In step S1360, the electronic device (1000) according to one embodiment of the present disclosure can generate a bokeh image corresponding to the input image by blending an image of a first area extracted from an input image and an image of a second area extracted from an upscaled low-resolution bokeh image.

[0149] Since the electronic device (1000) performs bokeh operations on a low-resolution input image that is smaller in size than the final image (bokeh image), the bokeh operation speed can be improved. In addition, since the electronic device (1000) extracts the in-focus area from the original input image that has not been downscaled, artifacts caused by resizing the in-focus area can be minimized.

[0150] In step S1370, the electronic device (1000) according to one embodiment of the present disclosure can output a bokeh image.

[0151] According to one embodiment of the present disclosure, the electronic device (1000) can output a bokeh image in real time on a preview screen (or live screen, viewfinder). According to one embodiment of the present disclosure, the electronic device (1000) can also display a bokeh image corresponding to an input image previously stored in a memory on an editing screen. When the input image is a video, the electronic device (1000) can generate bokeh images corresponding to each image frame included in the video and play the video with the bokeh effect applied through a shooting screen or an editing screen.

[0152] FIG. 14 is a block diagram illustrating the function of an electronic device (1000) according to one embodiment of the present disclosure.

[0153] As illustrated in FIG. 14, an electronic device (1000) according to an embodiment of the present disclosure may include an output unit (1100), a sensor unit (1200), a processor (1300), a communication interface (1400), an A / V input unit (1500), a user input unit (1600), and a memory (1700). However, not all of the components illustrated in FIG. 14 are essential components. The electronic device (1000) may be implemented with more components than the illustrated components, or may be implemented with fewer components. For example, the electronic device (1000) may be implemented with a memory (1700) and a processor (1300).

[0154] Below, we will look at the above components in turn.

[0155] The output unit (1100) is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit (1111), an audio output unit (1112), a vibration motor (1113), etc.

[0156] The audio output unit (1112) outputs audio data received from the communication interface (1400) or stored in the memory (1700). In addition, the audio output unit (1112) outputs audio signals related to functions performed in the electronic device (1000) (e.g., call signal reception sound, message reception sound, notification sound). The audio output unit (1112) may include a speaker, a buzzer, or the like.

[0157] The vibration motor (1113) can output a vibration signal. For example, the vibration motor (1113) can output a vibration signal corresponding to the output of audio data or video data (e.g., a call signal reception sound, a message reception sound, etc.). In addition, the vibration motor (1113) can also output a vibration signal when a touch is input to the touch screen.

[0158] The output unit (1100) can output an image of a 3D space. At this time, the image of the 3D space can include an image indicator displayed on the screen in a predetermined size. The image indicator can be a UI (User Interface) element for controlling the IoT device (3000). For example, the image indicator can be an icon image, but is not limited thereto.

[0159] The sensor unit (1200) may include at least one of a magnetic sensor (1211), an acceleration sensor (1212), a tilt sensor (1213), an infrared sensor (1214), a gyroscope sensor (1215), a position sensor (e.g., GPS) (1216), a temperature and humidity sensor (1217), a proximity sensor (1218), and a light sensor (1219), but is not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description thereof will be omitted.

[0160] The processor (1300) typically controls the overall operation of the electronic device (1000). For example, the processor (1300) can control the output unit (1100), the sensor unit (1200), the communication interface (1400), the A / V input unit (1500), the user input unit (1600), the memory (1700), etc., by executing programs stored in the memory (1700).

[0161] The processor (1300) may be composed of one or more processors. The one or more processors included in the processor (1300) may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. The one or more processors included in the processor (1300) may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). When the one or more processors included in the processor (1300) are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The processor (1300) may be implemented as a single core processor or as a multicore processor.

[0162] The processor (1300) can write data to the memory (1700), read data stored in the memory (1700), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program or at least one instruction stored in the memory (1700).

[0163] The communication interface (1400) may include one or more components that enable communication between the electronic device (1000) and the IoT device (3000) or between the electronic device (1000) and the server (2000). For example, the communication interface (1400) may include a short-range communication unit (1411), a mobile communication unit (1412), and a broadcast receiving unit (1413).

[0164] The short-range wireless communication unit (221) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, a UWB (ultra wideband) communication unit, an Ant+ communication unit, etc.

[0165] The mobile communication unit (1412) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signals may include various types of data, such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0166] The broadcast receiver (1413) receives broadcast signals and / or broadcast-related information from an external source via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. Depending on the implementation example, the electronic device (1000) may not include the broadcast receiver (1413).

[0167] The A / V (Audio / Video) input unit (1500) is for inputting audio signals or video signals, and may include a camera (1511), a microphone (1512), etc. The camera (1511) can obtain image frames such as still images or moving images through an image sensor in a video call mode or a shooting mode. The images captured through the image sensor can be processed through a processor (1300) or a separate image processing unit (not shown). The image frames processed by the camera (1511) can be stored in a memory (1700) or transmitted to the outside through a communication interface (1400). According to one embodiment of the present disclosure, the camera (1511) may include at least one of a telephoto camera, a wide-angle camera, and a general camera, but is not limited thereto.

[0168] The microphone (1512) receives external audio signals and processes them into electrical voice data. For example, the microphone (1512) can receive audio signals from an external device or a speaker. The microphone (1512) can utilize various noise removal algorithms to remove noise generated during the process of receiving external audio signals.

[0169] The user input unit (1600) refers to a means for a user to input data for controlling the electronic device (1000). For example, the user input unit (1600) may include, but is not limited to, a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, a jog switch, etc.

[0170] The memory (1700) may store a program for processing and controlling the processor (1300), and may also store input / output data (e.g., voice data, photo images, memo data, user's biometric information, etc.).

[0171] The memory (1700) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0172] The memory (1700) may not exist separately and may be configured to be included in the processor (1300). The memory (1700) may be configured as volatile memory, non-volatile memory, or a combination of volatile memory and non-volatile memory. The memory (1700) may store a program or at least one instruction for performing operations according to an embodiment of the present disclosure. The memory (1700) may also provide stored data to the processor (1300) upon request from the processor (1300).

[0173] According to one embodiment of the present disclosure, by performing a bokeh operation by considering the difference in brightness between a current pixel and surrounding pixels in an image device (e.g., an electronic device (1000)) having a short focal length, a bokeh image that implements an effect similar to the spread of light in an actual optical system can be provided.

[0174] A method for providing a bokeh image by an electronic device (1000) according to an embodiment of the present disclosure may include: acquiring an input image; acquiring a luminance map corresponding to the input image having a higher bit depth than the input image; determining a distance difference between a reference pixel in focus and a current pixel in the input image using the depth map corresponding to the input image; determining a size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel; generating a bokeh image corresponding to the input image using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map; and outputting the bokeh image corresponding to the input image.

[0175] The step of obtaining a luminance map according to one embodiment of the present disclosure may include a step of obtaining a downscaled luminance map while maintaining a higher number of bits than the input image by applying a max pooling method to luminance data extracted from preprocessed original image data corresponding to the input image.

[0176] The step of obtaining a downscaled luminance map while maintaining a higher bit count than an input image according to one embodiment of the present disclosure may include the step of performing de-gamma processing on preprocessed original image data corresponding to the input image when gamma is applied to the preprocessed original image data; and the step of applying a max pooling method to luminance data extracted from the de-gamma-processed original image data to obtain a downscaled luminance map while maintaining a higher bit count than the input image.

[0177] The step of determining the size of a geometric kernel according to one embodiment of the present disclosure may include the step of determining the size of a geometric kernel corresponding to a current pixel using a distance difference between a reference pixel and a current pixel and an aperture value of a target lens.

[0178] The shape of the geometric kernel according to one embodiment of the present disclosure is determined according to the characteristics of the target lens, and may include at least one of a circle, an ellipse, a polygon, a vortex shape, or a donut shape.

[0179] The step of determining the size of the geometric kernel according to one embodiment of the present disclosure may include the step of determining the size of the geometric kernel to be larger when the distance difference between the reference pixel and the current pixel is large, the F number of the target lens is small, and the focal length of the target lens is long.

[0180] The step of generating a bokeh image corresponding to an input image according to one embodiment of the present disclosure may include the step of determining weights of surrounding pixels having lower luminance than the current pixel within a geometric kernel as 0; and the step of performing a convolution operation that multiplies image values ​​of pixels within the geometric kernel by each weight and adds them all together, and generating a result value as a bokeh operation value of the current pixel.

[0181] The step of generating a bokeh image corresponding to an input image according to one embodiment of the present disclosure may include: determining a luminance weight of at least one pixel using a luminance map when the luminance value of at least one pixel within a geometric kernel is a maximum luminance value; and determining a bokeh operation value of a current pixel by reflecting the luminance weight of the at least one pixel.

[0182] According to one embodiment of the present disclosure, a bokeh operation value for a current pixel may be determined to be a brighter value than an image value of the current pixel if there are surrounding pixels with higher luminance than the current pixel within the geometric kernel.

[0183] The step of generating the bokeh image corresponding to the input image according to one embodiment of the present disclosure may include the step of obtaining a degamma input image by performing degamma processing on the input image when gamma is applied to the input image; and the step of performing a bokeh operation on the degamma input image.

[0184] The step of generating a bokeh image according to one embodiment of the present disclosure may include the steps of performing downscaling on an input image to generate a low-resolution input image; performing a bokeh operation on the low-resolution input image to generate a low-resolution bokeh image corresponding to the low-resolution input image; upscaling the low-resolution bokeh image; extracting an image of a first region in focus from the input image and extracting an image of a second region excluding the first region from the upscaled low-resolution bokeh image; and synthesizing an image of the first region extracted from the input image and an image of the second region extracted from the upscaled low-resolution bokeh image to generate a bokeh image corresponding to the input image.

[0185] An electronic device (1000) according to an embodiment of the present disclosure may include a camera (1511); a display unit (1111); a memory (1700) storing a program or one or more instructions; and at least one processor (1300). The electronic device (1000) may acquire an input image through the camera (1511) by having the at least one processor (1300) execute the program or one or more instructions stored in the memory (1700). The electronic device (1000) may acquire a luminance map corresponding to an input image having a higher bit depth than the input image. The electronic device (1000) may determine a distance difference between a reference pixel in focus and a current pixel within the input image using a depth map corresponding to the input image. The electronic device (1000) may determine a size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel. The electronic device (1000) can generate a bokeh image corresponding to an input image by using a luminance difference between a current pixel and surrounding pixels within a geometric kernel having a determined size and a luminance weight based on a luminance map. The electronic device (1000) can output the bokeh image corresponding to the input image through a display unit.

[0186] An electronic device (1000) according to one embodiment of the present disclosure can obtain a downscaled luminance map while maintaining a higher bit count than the input image by applying a max pooling method to luminance data extracted from preprocessed original image data corresponding to an input image.

[0187] An electronic device (1000) according to one embodiment of the present disclosure may perform de-gamma processing on pre-processed original image data corresponding to an input image when gamma is applied to the pre-processed original image data. The electronic device (1000) may obtain a downscaled luminance map while maintaining a higher bit count than the input image by applying a max pooling method to luminance data extracted from the de-gamma-processed original image data.

[0188] An electronic device (1000) according to one embodiment of the present disclosure can determine the size of a geometric kernel corresponding to a current pixel by using a distance difference between a reference pixel and a current pixel and an aperture value of a target lens.

[0189] An electronic device (1000) according to one embodiment of the present disclosure can determine a larger size of a geometric kernel as the distance difference between a reference pixel and a current pixel is larger, the F number of the target lens is smaller, and the focal length of the target lens is longer.

[0190] An electronic device (1000) according to one embodiment of the present disclosure may determine the weight of surrounding pixels having lower luminance than the current pixel within a geometric kernel as 0. The electronic device (1000) may generate a result value of a convolution operation that multiplies each weight by pixels within the geometric kernel as a bokeh operation value of the current pixel.

[0191] An electronic device (1000) according to one embodiment of the present disclosure may determine a luminance weight of at least one pixel using a luminance map when the luminance value of at least one pixel within a geometric kernel is a maximum luminance value. The electronic device (1000) may determine a bokeh operation value of a current pixel by reflecting the luminance weight of at least one pixel.

[0192] An electronic device (1000) according to an embodiment of the present disclosure can generate a low-resolution input image by performing downscaling on an input image. The electronic device (1000) can generate a low-resolution bokeh image corresponding to the low-resolution input image by performing a bokeh operation on the low-resolution input image. The electronic device (1000) can upscale the low-resolution bokeh image. The electronic device (1000) can extract an image of a first region in focus from the input image, and extract an image of a second region excluding the first region from the upscaled low-resolution bokeh image. The electronic device (1000) can generate a bokeh image corresponding to the input image by synthesizing an image of the first region extracted from the input image and an image of the second region extracted from the upscaled low-resolution bokeh image.

[0193] A computer-readable recording medium according to one embodiment of the present disclosure may record a program for causing an electronic device (1000) to perform a method for providing a bokeh image, the method comprising: obtaining an input image; obtaining a luminance map corresponding to the input image having a higher bit depth than the input image; determining a distance difference between a reference pixel in focus and a current pixel in the input image using the depth map corresponding to the input image; determining a size of a geometric kernel corresponding to the current pixel using the distance difference between the reference pixel and the current pixel; generating a bokeh image corresponding to the input image using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map; and outputting the bokeh image corresponding to the input image.

[0194] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0195] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In a method for an electronic device (1000) to provide a bokeh image, Step of acquiring an input image (S210); A step (S220) of obtaining a luminance map corresponding to the input image having a higher bit depth than the input image; A step (S230) of determining the distance difference between a reference pixel in focus and a current pixel within the input image by using a depth map corresponding to the input image; A step (S240) of determining the size of a geometric kernel corresponding to the current pixel by using the distance difference between the reference pixel and the current pixel; A step (S250) of generating a bokeh image corresponding to the input image by using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map; and A method comprising a step (S260) of outputting a bokeh image corresponding to the input image.

2. In the first paragraph, the step of obtaining the luminance map comprises: A method comprising the step of obtaining a downscaled luminance map while maintaining a higher bit count than the input image by applying a max pooling method to luminance data extracted from preprocessed original image data corresponding to the input image.

3. In the second paragraph, the step of obtaining a downscaled luminance map while maintaining a higher bit number than the input image is as follows: When gamma is applied to the preprocessed original image data corresponding to the input image, a step of performing de-gamma processing on the preprocessed original image data; and A method comprising a step of obtaining a downscaled luminance map while maintaining a higher bit count than the input image by applying a max pooling method to luminance data extracted from the degamma-processed original image data.

4. In any one of paragraphs 1 to 3, the step of determining the size of the geometric kernel comprises: A method comprising the step of determining the size of a geometric kernel corresponding to the current pixel by using the distance difference between the reference pixel and the current pixel and the aperture value of the target lens.

5. In the fourth paragraph, the shape of the geometric kernel is A method, wherein the target lens is determined based on its characteristics and includes at least one of a circular shape, an elliptical shape, a polygonal shape, a spiral shape, or a donut shape.

6. In the fourth paragraph, the step of determining the size of the geometric kernel is: A method comprising a step of determining a size of the geometric kernel to be larger as the distance difference between the reference pixel and the current pixel is large, the F number of the target lens is small, and the focal length of the target lens is long.

7. In any one of the first to sixth paragraphs, the step of generating a bokeh image corresponding to the input image comprises: A step of determining the weight of surrounding pixels having a lower luminance than the current pixel within the geometric kernel as 0; and A method comprising a step of generating a result value of a convolution operation that multiplies the image values ​​of pixels within the geometric kernel by each weight and adds them all together as a bokeh operation value of the current pixel.

8. In any one of the first to seventh paragraphs, the step of generating the bokeh image corresponding to the input image comprises: determining a luminance weight of at least one pixel using the luminance map when the luminance value of at least one pixel within the geometric kernel is the maximum luminance value; and A method comprising the step of determining a bokeh operation value of the current pixel by reflecting the luminance weight of at least one pixel.

9. In the 8th paragraph, the bokeh operation value for the current pixel is A method in which, if there is a surrounding pixel with a higher luminance than the current pixel within the geometric kernel, the image value of the current pixel is determined to be brighter.

10. In any one of paragraphs 1 to 9, the step of generating the bokeh image corresponding to the input image comprises: When gamma is applied to the input image, a step of performing de-gamma processing on the input image to obtain a de-gamma input image; and A method comprising the step of performing a bokeh operation on the above degamma input image.

11. In any one of paragraphs 1 to 10, the step of generating the bokeh image comprises: A step of generating a low-resolution input image by performing downscaling on the above input image; A step of performing a bokeh operation on the low-resolution input image to generate a low-resolution bokeh image corresponding to the low-resolution input image; A step of upscaling the above low-resolution bokeh image; A step of extracting an image of a first area in focus from the input image, and extracting an image of a second area excluding the first area from the upscaled low-resolution bokeh image; and A method comprising the step of generating the bokeh image corresponding to the input image by synthesizing the image of the first area extracted from the input image and the image of the second area extracted from the upscaled low-resolution bokeh image.

12. Camera (1511); Display section (1111); A memory (1700) storing a program or one or more instructions; and comprising at least one processor (1300), By having at least one processor (1300) execute a program or one or more instructions stored in the memory (1700), the electronic device (1000) Obtaining input images through the above camera, Obtain a luminance map corresponding to the input image having a higher bit depth than the input image, Using a depth map corresponding to the input image, the distance difference between the reference pixel in focus and the current pixel in the input image is determined, Using the distance difference between the reference pixel and the current pixel, the size of the geometric kernel corresponding to the current pixel is determined, Generating a bokeh image corresponding to the input image by using a luminance difference between the current pixel and surrounding pixels within the geometric kernel having the determined size and a luminance weight based on the luminance map, An electronic device that outputs a bokeh image corresponding to the input image through the display unit.

13. In the 12th paragraph, the electronic device, When gamma is applied to the preprocessed original image data corresponding to the input image, de-gamma processing is performed on the preprocessed original image data, An electronic device that obtains a downscaled luminance map while maintaining a higher bit count than the input image by applying a max pooling method to luminance data extracted from the degamma-processed original image data.

14. In the 12th or 13th paragraph, the electronic device, Using the distance difference between the reference pixel and the current pixel and the aperture value of the target lens, the size of the geometric kernel corresponding to the current pixel is determined, Within the geometric kernel, the weights of surrounding pixels with lower luminance than the current pixel are determined to be 0, An electronic device that generates a result value of a convolution operation that multiplies each weight on pixels within the geometric kernel as a bokeh operation value of the current pixel.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of clauses 1 to 11 on a computer.

Images