Electronic device and method of processing image in electronic device

Abstract

An electronic device and method are disclosed. An electronic device includes a memory and a processor that implements the method, the method including obtaining image data of an image, setting a region of interest and a background region in the image using depth information, both including one or more sub-regions, the method further includes determining respective bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions based on the first characteristic criterion and the second characteristic criterion, respectively, and processing the image to apply a bokeh effect to the plurality of pixels based on the respective bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions.

Description

[0001] This application is a divisional application of the invention patent application filed on February 18, 2020, with application number 202080030231.0 and entitled "An electronic device for applying bokeh effect to an image and a control method thereof". Technical Field

[0002] This invention relates to a technique for processing images using an electronic device. Background Technology

[0003] Modern electronic devices are capable of performing a variety of complex functions, such as still image and video capture, playback of audio, video and other multimedia, image editing, program downloading, gaming and broadcasting.

[0004] During playback and replay functions, the electronic device not only generates the displayed image itself but also applies visual effects to it. An example of a visual effect (such as a photographic "filter" or effect) is the "bokeh" effect. Bokeh can also be called a defocusing effect and refers to the blurring of a simulated background object through visual processing when the camera is focused on a foreground object.

[0005] In some cases, bokeh effects are not created during image capture. However, electronic devices can create bokeh effects within an image by performing image processing on the image.

[0006] The above information is provided as background information only to aid in understanding the present invention. No decision has been made, nor any assertion, has been made regarding whether any of the above content can be considered as prior art application of this disclosure. Summary of the Invention

[0007] Technical issues

[0008] When electronic devices use image processing to generate bokeh effects, the resulting bokeh may differ from the true bokeh produced by the camera's actual focus. For example, when an electronic device applies the generated bokeh effect to a subject, the subject may become unexpectedly blurred.

[0009] The present invention aims to at least address the aforementioned problems and / or disadvantages, and to provide at least the following advantages. Therefore, one aspect of the present invention is to provide a method for satisfying the need to apply various photographic effects to a user's image.

[0010] Technical solution

[0011] According to one aspect of the invention, the electronic device may include: a memory; and a processor electrically connected to the memory. The memory may store instructions executable by the processor to cause the electronic device to: acquire image data of an image comprising a plurality of pixels; set a region of interest and a background region within the image using depth information associated with each of the plurality of pixels, wherein the region of interest comprises one or more first sub-regions and the background region comprises one or more second sub-regions; determine corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions based on a first characteristic criterion and a second characteristic criterion, respectively; and process the image based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions to apply a bokeh effect to the plurality of pixels.

[0012] According to one aspect of the present invention, the method may include: obtaining image data of an image comprising a plurality of pixels; obtaining depth information of each of the plurality of pixels; using the depth information to set a region of interest and a background region within the image, wherein the region of interest includes one or more first sub-regions and the background region includes one or more second sub-regions; determining corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions based on a first characteristic criterion and a second characteristic criterion, respectively; and processing the image based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions to apply a bokeh effect to the plurality of pixels.

[0013] According to another aspect of the invention, the storage medium may include enabling a program to obtain image data of an image, obtain depth information associated with a plurality of pixels, use the depth information to determine a region of interest and a background region in the image, determine the bokeh characteristics of each first image region of the region of interest based on a first characteristic criterion, determine the bokeh characteristics of each second image region of the background region based on a second characteristic criterion distinct from the first characteristic criterion, and apply a bokeh effect to the plurality of pixels based on the bokeh characteristics of each first image region and the bokeh characteristics of each second image region.

[0014] According to another aspect of the invention, an electronic device may include: a memory; and a processor electrically connected to the memory, wherein the memory stores instructions executable by the processor to cause the electronic device to: acquire image data of an image comprising a plurality of pixels; set a region of interest and a background region within the image using depth information associated with each of the plurality of pixels, wherein the region of interest includes one or more first sub-regions and the background region includes one or more second sub-regions; determine a starting position of a bokeh effect within the region of interest or the background region; determine corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions based on distances between the one or more first sub-regions and the starting position and distances between the one or more second sub-regions and the starting position, respectively; and process the image to apply the bokeh effect to the plurality of pixels based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions.

[0015] According to another aspect of the present invention, a method for processing an image in an electronic device may include: obtaining image data of an image comprising a plurality of pixels; obtaining depth information of each of the plurality of pixels; using the depth information to define a region of interest and a background region within the image, wherein the region of interest includes one or more first sub-regions and the background region includes one or more second sub-regions; determining a starting position of a bokeh effect within the region of interest or the background region; determining corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions based on the distances between the one or more first sub-regions and the starting position and the distances between the one or more second sub-regions and the starting position, respectively; and processing the image to apply the bokeh effect to the plurality of pixels based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions.

[0016] Other aspects, advantages and distinctive features of the invention will become apparent to those skilled in the art from the following detailed description, which discloses certain embodiments of the invention in conjunction with the accompanying drawings.

[0017] Beneficial effects

[0018] According to the embodiments disclosed in this invention, an electronic device or method is provided that can effectively apply bokeh effects to images.

[0019] Furthermore, according to the embodiments disclosed in this invention, an electronic device or method is provided that can apply various image effects desired by the user to an image.

[0020] Furthermore, according to the embodiments disclosed in this document, an electronic device or method is provided that can easily apply advanced photographic techniques using a camera to images.

[0021] In addition, this document can provide a variety of effects, either directly or indirectly. Attached Figure Description

[0022] The above and other aspects, features, and advantages of certain embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 This is a block diagram of an electronic device in a network environment according to an embodiment of the present invention;

[0024] Figure 2 This is a block diagram illustrating the configuration of an electronic device according to an embodiment;

[0025] Figure 3 This is a flowchart illustrating the process of an electronic device processing an image according to an embodiment;

[0026] Figure 4 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, determines the bokeh characteristics of each image region associated with a region of interest in an image;

[0027] Figure 5 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, applies a bokeh effect to an image to create a radial blur effect.

[0028] Figure 6 The concept of a region of interest and a background region, divided by electronic device and image height according to an embodiment, is illustrated.

[0029] Figure 7 A conceptual example illustrating the bokeh characteristics of each image region determined by regions divided by electronic devices and image height, according to an embodiment;

[0030] Figure 8 An example is shown of an electronic device according to an embodiment applying a bokeh effect to an image with a radial blur effect;

[0031] Figure 9 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, applies a bokeh effect to an image to produce a zoom blur effect;

[0032] Figure 10 The following is a conceptual illustration of an example of the starting position of a bokeh effect determined by an electronic device according to an embodiment, and an example of the bokeh characteristics of each image region based on the starting position of the bokeh effect.

[0033] Figure 11 An example is shown of an electronic device according to an embodiment applying a bokeh effect to an image with a zoom blur effect;

[0034] Figure 12 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, applies a bokeh effect to achieve a motion blur effect;

[0035] Figure 13 A conceptual example is shown of an electronic device according to an embodiment determining the bokeh characteristics of each image region having a motion blur effect;

[0036] Figure 14 An example is shown of an electronic device according to an embodiment applying a bokeh effect to an image with motion blur.

[0037] Figure 15 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, applies a bokeh effect based on a path input by a user.

[0038] Figure 16 A conceptual illustration shows an example of an electronic device according to an embodiment determining the bokeh characteristics of each image region based on an input path;

[0039] Figure 17 This is a diagram illustrating an example of an electronic device applying a bokeh effect to an image based on an input path according to an embodiment;

[0040] Figure 18 This is a flowchart illustrating the process by which an electronic device, according to an embodiment, applies a bokeh effect to an image to produce a tilt-shift blur effect;

[0041] Figure 19 This illustrates the concept of an image divided into multiple shift regions by an electronic device according to an embodiment, and the bokeh characteristics of each image region; and

[0042] Figure 20 This is a flowchart illustrating the process by which an electronic device applies a bokeh effect to an image based on a depth value.

[0043] The same reference numerals are used to indicate the same elements throughout the entire figure. Detailed Implementation

[0044] The following description, with reference to the accompanying drawings, is provided to aid in a comprehensive understanding of certain embodiments of the invention as defined by the claims and their equivalents. It includes numerous specific details to aid understanding, but these are merely examples. Therefore, those skilled in the art will recognize that various modifications, equivalents, and / or substitutions can be made to certain embodiments described herein without departing from the invention. Furthermore, for clarity and brevity, descriptions of well-known functions and implementations may be omitted.

[0045] The terms and words used in the following description and claims are not limited to their literal meanings, but are used solely by the inventors to ensure a clear and consistent understanding of the invention. Therefore, those skilled in the art will understand that the following description of certain embodiments of the invention is for illustrative purposes only and is not intended to limit the invention as defined in the appended claims and their equivalents.

[0046] It should be understood that, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” include plural references. Thus, for example, referring to “component surface” includes referring to one or more such surfaces.

[0047] Figure 1 This is a block diagram illustrating an electronic device in a network environment according to an embodiment of the present invention.

[0048] Reference Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include at least one processor 120, memory 130, input device 150, sound output device 155, display device 160, audio module 170, sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module (SIM) 196, and / or antenna module 197. In some embodiments, at least one of the components (e.g., display device 160 or camera module 180) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) can be implemented as an embedded component in the display device 160 (e.g., a display).

[0049] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) coupled to electronic device 101 and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, auxiliary processor 123 may be adapted to consume less power than the main processor 121 or to be specifically adapted for a given function. The auxiliary processor 123 can be implemented separately from the main processor 121, or it can be implemented as part of the main processor 121.

[0050] The auxiliary processor 123 can control at least some functions or states associated with at least one component of the electronic device 101 (other than the main processor 121) (e.g., display device 160, sensor module 176, or communication module 190) in place of the main processor 121 when the main processor 121 is inactive (e.g., in a sleep state), or it can control the main processor 123 together with the main processor 121 when the main processor 121 is active (e.g., running an application). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) can be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally associated with the auxiliary processor 123.

[0051] Memory 130 may store various types of data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various types of data may include, for example, software (e.g., program 140) and input or output data of commands associated therewith. Memory 130 may include volatile memory 132 and / or non-volatile memory 134. Non-volatile memory 134 may include internal memory 136 and / or external memory 138.

[0052] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0053] Input device 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by other components of electronic device 101 (e.g., processor 120). Input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (e.g., stylus).

[0054] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. According to an embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0055] Display device 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display device 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a corresponding one of the display, holographic device, and projector. According to an embodiment, display device 160 may include touch circuitry adapted to detect touch or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of the force caused by touch.

[0056] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or an external electronic device (e.g., electronic device 102) (e.g., headphone speaker) that is directly (e.g., wired) coupled or wirelessly coupled to the electronic device 101.

[0057] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0058] Interface 177 may support one or more specific protocols used to enable electronic device 101 to be directly (e.g., wired) or wirelessly coupled to external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0059] Terminal block 178 may include a connector via which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, terminal block 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0060] The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that can be recognized by a user through his touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0061] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0062] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0063] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0064] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors that are independent of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as the first network 198 or the second network 199) using user information (e.g., the International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0065] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element implemented using a conductive material or conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, antenna module 197 may include multiple antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0066] At least some of the aforementioned components may be coupled to each other via a peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)) and communicate signals (e.g., commands or data) between them.

[0067] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 coupled to a second network 199. Each of electronic devices 102 and 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations to be performed on electronic device 101 can be performed on one or more of external electronic devices 102, 104, or 108. For example, when electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service, rather than running the function or service, or electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. The one or more external electronic devices receiving the request may perform the requested at least portion of the function or service, or perform additional functions or services associated with the request, and transmit the results of the performance to electronic device 101. The electronic device 101 can provide the results as at least part of a response to a request, with or without further processing of the results. For this purpose, technologies such as cloud computing, distributed computing, or client-server computing can be used.

[0068] Figure 2 This is a block diagram illustrating the configuration of an electronic device 200 according to an embodiment.

[0069] According to an embodiment, electronic device 200 (e.g., Figure 1 The electronic device 101 may include a processor 220 (e.g., Figure 1 The processor 120) and memory 260 (e.g., Figure 1 The memory 260 can store instructions that can be executed by the processor 220. The processor 220 can execute the instructions stored in the memory 260 to process data or control components of the electronic device 200. Operation of the electronic device or processor disclosed in this invention can be performed so that the processor executes the instructions stored in the memory.

[0070] According to an embodiment, processor 220 can obtain image data including an image having multiple pixels. According to an embodiment, electronic device 200 may further include camera module 280 (e.g., Figure 1The camera module 280 is used to acquire image data. The processor 220 can capture images using the image sensor included in the camera module 280. According to another embodiment, in order to acquire image data, the processor 220 can read image data stored in the memory 260.

[0071] According to an embodiment, processor 220 can apply a "bokeh" effect to the acquired image. Applying a bokeh effect may include changing the values ​​of some pixels in the image to blur portions of the image. According to an embodiment, processor 220 may determine the bokeh characteristics of each image region based on depth information of the image. The bokeh characteristics of each image region may indicate how the bokeh effect should be applied to each corresponding region. For example, the bokeh characteristics of each image region may indicate a pattern of bokeh effect for a specified portion of the image. The pattern of the bokeh effect may include, for example, the intensity and / or direction of the bokeh effect. The intensity of the bokeh effect may include, for example, the degree of blurring to be applied to a corresponding region or object within the image. The direction of the bokeh effect may indicate the direction or shape (e.g., radial, linear, zoom, or other) of the blur to be applied within the image.

[0072] In embodiments, depth information may include information indicating the distance or depth of a subject photographed in the image, or information associated with the distance or depth of a subject photographed in the image. However, depth information should be understood to be limited to the absolute or actual value of the distance of an object in the image relative to the camera that captured the image. As an example, depth information may include a depth map that includes the depth value of each pixel included in the captured image. As another example, depth information may include information that is not the depth value of a pixel itself. Depth information may include the results of analyzing the image. For example, depth information may include information that divides the captured image into regions of interest (e.g., a subject region) and background regions. A region of interest may refer to the area where the subject is located (on which the camera's focus typically rests). For example, a region of interest may refer to the area depicting a person in a portrait. Background and foreground regions may be different focus areas from the subject region. According to embodiments, a region of interest may include a foreground region closer to the camera than the subject, and / or a background region farther away from the camera than the subject.

[0073] According to an embodiment, depth information can indicate the relative depth between pixels in a first region and pixels in another region (e.g., one having a greater depth than the other). Processor 220 can determine the depth information of peripheral pixels based on the difference between a distance value measured by a center pixel corresponding to a specified center in the image sensor included in camera module 280 and distance values ​​measured by peripheral pixels other than the center pixel. For example, when the distance value of the center pixel sensed by camera module 280 is 15 and the distance value of the peripheral pixel is 18, processor 220 can determine the depth value of the peripheral pixel as 3, which is the difference between the two distance values.

[0074] According to an embodiment, when an image including images captured by camera module 280 is acquired, processor 220 can obtain depth information from camera module 280. For example, camera module 280 may include a sensor camera capable of detecting the distance from the camera to the subject (hereinafter referred to as "shooting distance"). The sensor camera may include, for example, a TOF camera with a time-of-flight (TOF) sensor or a structured light type 3D camera. Processor 220 may determine the depth values ​​of the pixels in the captured image based on the values ​​sensed by the sensor camera of camera module 280.

[0075] As another example, camera module 280 may include a multi-camera system with multiple cameras. Processor 220 may determine the depth of the subject corresponding to the pixels captured by the cameras based on visible light detected by the image sensors of each of the multiple cameras.

[0076] As yet another example, processor 220 may calculate depth information about an image obtained by a camera included in camera module 280. For example, processor 220 may use an image classification model that learns the shapes corresponding to the shapes of a subject to be photographed by the user to classify shapes included in the image as either a subject of interest or background. Processor 220 may obtain depth information including information about the region where the shape classified as a subject of interest is located.

[0077] Figure 3 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 Flowchart 300 of the process of processing images by the electronic device 200.

[0078] According to an embodiment, in operation 310, the electronic device can acquire image data of the image. For example, the electronic device can... Figure 2 The camera module 280 captures images. In another example, the electronic device may receive image data from another device (e.g., Figure 1(Electronic device 102, electronic device 104, or server 108).

[0079] According to an embodiment, in operation 320, the electronic device can obtain depth information about the image. For example, the electronic device can obtain depth information during image capture, such as... Figure 2 The depth is detected by the camera module 280. As another example, the electronic device can obtain the depth by analyzing the image obtained after capture. As another example, the electronic device can receive depth information via transmission from another device (e.g., electronic device 102, electronic device 104, or server 108).

[0080] In operation 330, the electronic device may determine the bokeh characteristics of each image region based on depth information. According to an embodiment, the electronic device may determine the bokeh characteristics of each image region from the depth information. For example, the value of the bokeh characteristics of each image region (e.g., the intensity of the bokeh effect) can be calculated using the depth value of each included pixel indicated by the depth information. According to another embodiment, the electronic device may divide an image into multiple regions based on depth information and determine the bokeh characteristics of each image region.

[0081] In operation 340, a bokeh effect based on the bokeh characteristics of each image region can be applied to the image. As described above, the bokeh effect can be applied to the image based on the bokeh characteristics of each image region determined in operation 330, so that a bokeh effect suitable for the characteristics of the image can be applied to the image.

[0082] Figure 4 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 A flowchart 301 shows the process by which an electronic device 200 determines the bokeh characteristics of each image region relative to the region of interest in the image.

[0083] In operation 311, the electronic device can acquire image data of the image. In operation 321, the electronic device can use the depth information of the acquired image to determine a region of interest in the image. According to an embodiment, in addition to the region of interest (e.g., the focal region), the electronic device can identify a portion of the image to be set as a background region.

[0084] According to an embodiment, in operation 331, the electronic device can determine the bokeh characteristics of each first image region relative to a region of interest (e.g., for each sub-region within one or more sub-regions of the region of interest) and the bokeh characteristics of each second image region relative to a background region (e.g., for each sub-region within one or more sub-regions of the background region). When the electronic device determines the bokeh characteristics of each first image region, it can determine the bokeh characteristics of each first image region based on a first characteristic criterion. Furthermore, the electronic device can determine the bokeh characteristics of each second image region based on a second characteristic criterion distinct from the first characteristic criterion. The first and second characteristic criteria may refer to conditions used to determine the bokeh characteristics of each image region. For example, the first and second characteristic criteria may define the intensity range of the bokeh effect included in the bokeh characteristics of each image region. Specifically, the first characteristic criterion may set the intensity of the bokeh effect in the range of "0" to "2", and the second characteristic criterion may set the intensity of the bokeh effect in the range of "0" to "10".

[0085] In operation 341, the electronic device can apply the bokeh effect to the regions of the image (i.e., the regions of the region of interest, the background region, and the corresponding sub-regions) according to the bokeh characteristics of each image region and sub-region.

[0086] Figure 5 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 A flowchart 500 describes the process by which an electronic device 200 applies a bokeh effect to an image to create a radial blur effect.

[0087] According to an embodiment, the electronic device can use an input device (e.g., Figure 1 The input device 150 receives user input for selecting a bokeh application mode. (See reference) Figure 5 In operation 510, the electronic device may receive user input for selecting a radial blur effect operation mode, one of the bokeh application modes. For example, the user input may include touch input received via a touchscreen. According to another embodiment, the form of the user input can be modified in various ways. For example, the electronic device may adjust the input based on a motion sensor (e.g., ...). Figure 1The sensor module 176 detects motion values ​​to select a bokeh application mode. Specifically, as an example, when rotation of the electronic device is detected during image capture, the electronic device can select a radial blur effect operation mode. As another example, the electronic device can select a bokeh application mode based on its operating state. A radial blur effect can refer to a photographic effect that appears in a captured image when the camera capturing the image is rotating around the normal axis passing through the center of the image sensor of the camera or the optical axis of the lens included in the camera module. Optionally, when using a lens with severe aberrations (e.g., astigmatism or coma), a radial blur effect, represented by a rotating background, may appear in the captured image.

[0088] Subsequently, in operation 520, the electronic device acquires image data including the image. In operation 530, the electronic device acquires depth information of the acquired image data. Then, in operation 540, the electronic device divides the image into background regions and regions of interest based on the depth information. For example, the electronic device can group pixels with similar depth values ​​in the depth map included in the depth information into a group, and set the region containing the pixels in the group as either a background region or a region of interest.

[0089] For example, refer to Figure 6 Image 600 includes depictions of people and trees. The electronic device can classify the area including the people as a region of interest 610, and the remaining area as a background area 620.

[0090] In operation 550, the electronic device can acquire image height information. This image height information may be referred to as "image field information." Image height information can include values ​​indicating the distance from the center point of the image to a specific pixel. For example, referencing... Figure 6 The image height information 630 can include a value 635, which increases with distance from the center point of the image. Figure 5 In this embodiment, operation 550 is shown to be performed after operation 540, but the order of operations 540 and 550 can be changed according to embodiments. According to an embodiment, because depth information is obtained in operation 530, the electronic device may not need to execute operation 550 separately.

[0091] Subsequently, in operation 560, the electronic device can determine the bokeh characteristics of each image region based on image height information of the background region and the region of interest. Therefore, according to an embodiment, the electronic device can determine the bokeh characteristics of each image region such that the intensity of the bokeh effect increases with increasing distance from the center point of the image. The center point of the image can be referred to as the starting position of the bokeh effect. Furthermore, the electronic device can determine that the bokeh effect is not applied to the region of interest, or that the region of interest has a weaker bokeh effect than the background region, to provide bokeh characteristics relative to each image area of ​​the region of interest. For example, the electronic device can set the intensity of the bokeh effect in the background region from "0" to "10" and the intensity of the bokeh effect in the region of interest from "0" to "2". However, the invention is not limited thereto, and the range of values ​​*?* can be set differently according to embodiments. According to another embodiment, the electronic device can determine the bokeh characteristics of the image region of a pixel based on the difference between the image height value (e.g., "0") of the starting position of the bokeh effect and the image height value of the pixel, thereby determining the characteristics of the bokeh effect. For example, electronic devices can determine the bokeh characteristics of each image region, so that as the difference in the value of image height information becomes larger, the intensity of the bokeh effect is applied more strongly.

[0092] For example, refer to Figure 7 Example 640 of the bokeh characteristics of each image region shown, the electronic device can be positioned relative to the region of interest in region 632 having an image height value of "2" ( Figure 6 The region 641, included in the region of interest 610, determines the intensity of the bokeh effect included in the bokeh characteristics of each image region as "1". The electronic device can then define the intensity of the bokeh effect relative to the background region in region 632, which has an image height value of "2". Figure 6 The background area 620) includes area 642, which determines the intensity of the bokeh effect included in the bokeh characteristics of each image area as "5". The electronic device can... Figure 6 In region 633 of the image with a height value of "3", relative to the region of interest ( Figure 6 The region 644, included in the region of interest 610, determines the intensity of the bokeh effect included in the bokeh characteristics of each image region as "2". The electronic device can... Figure 6 In region 633 of the image, which has a height value of "3", the area relative to the background region is ( Figure 6 The intensity of the bokeh effect included in the bokeh characteristics of each image region is determined as "8" in region 645, which is included in the background region 620. Figure 6 and Figure 7 The embodiments are described conceptually and are not intended to limit the embodiments by form or numerical values.

[0093] According to another embodiment, in operation 560, the electronic device can determine the bokeh characteristics of each image region from the depth values ​​included in the depth information. In this case, because the depth information is obtained in operation 530, the electronic device does not need to perform operation 550 separately. However, the reference value of the depth used in operation 560 to determine the bokeh characteristics of each image region may be different from the reference value of the depth used to divide the region in operation 540. For example, the reference value used in operation 560 may be a value smaller than the reference value used in operation 540.

[0094] Furthermore, in operation 560, the electronic device can determine the direction of the bokeh effect included in the bokeh characteristics of each image region as dependent on the direction of the radial blur effect operation mode. For example, the electronic device can determine the direction of the bokeh effect as the direction of the arc of a circle centered on the center point of the image.

[0095] Then, in operation 570, the electronic device can apply a bokeh effect to the image based on the bokeh characteristics determined for each image region. For example, Figure 8 An example is shown of an electronic device applying a bokeh effect to an image 800 with a radial blur effect. (See reference...) Figure 8 For areas including the shape of people in the image, the blurring effect is small. Furthermore, the further away from the image center, the more blurred the background image becomes. Additionally, it can be seen that the direction 801 of the bokeh effect applied to the background area makes it appear as if the image has been rotated.

[0096] Figure 9 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 A flowchart 1000 of the process by which an electronic device 200 applies a bokeh effect to an image to produce a zoom blur effect.

[0097] According to an embodiment, the electronic device can use an input device (e.g., Figure 1 The input device 150 receives user input for selecting a bokeh application mode. For example, the user input may include touch input received via a touchscreen. According to another embodiment, the form of the user input can be modified in various ways. As another example, the electronic device can select a bokeh application mode based on its operating state. For example, when the electronic device is in a state of performing zoom-in or zoom-out operations while taking a picture, it can select a bokeh application mode as a zoom blur effect mode. When the electronic device automatically selects a bokeh application mode based on the sensor or its operating state, it can automatically implement advanced photographic techniques for capturing images using a camera. (Reference) Figure 9In operation 910, the electronic device can receive user input selecting the "zoom" blur effect operation mode, which is one of the bokeh application modes. Zoom blur effect refers to the photographic effect generated during image capture when the camera captures an image while performing a zoom-in or zoom-out operation.

[0098] Subsequently, in operation 920, the electronic device can acquire image data including the image. In operation 930, the electronic device can acquire depth information of the acquired image data. Then, in operation 940, the electronic device can divide the image into background regions and regions of interest (e.g., focal regions) based on the depth information. For example, the electronic device can classify pixels with similar depth values ​​in the depth map included in the depth information into a group, and can set the region containing the pixels included in the classification group as either a background region or a region of interest.

[0099] In operation 950, the electronic device can set the starting position of the bokeh effect in the image. According to an embodiment, the electronic device can perform image recognition processing on the image. As a result of performing the image recognition processing, the electronic device can determine where the subject of interest is located within the image. The electronic device can determine the starting position of the bokeh effect based on the region where the subject of interest is located. For example, when a face is the object of interest, the electronic device can determine the region where the face is located in the image and can determine the center point (e.g., centroid) of the region as the starting position of the bokeh effect. For example, refer to... Figure 10 Electronic devices can determine the image ( Figure 10 The area of ​​the human face displayed within 600 ( Figure 10 The center point of (1010) Figure 10 (1025) is used as the starting position for the bokeh effect. Figure 9 In this embodiment, operation 950 is shown to be performed after operation 940, but according to the embodiment, the order of operations 940 and 950 can be changed.

[0100] Subsequently, in operation 960, the electronic device can determine the bokeh characteristics of each image region based on the inter-pixel distance from the starting position of the bokeh effect to each corresponding pixel. Here, the inter-pixel distance can indicate the distance between two pixels in the image. According to an embodiment, the electronic device can determine the bokeh characteristics of each image region such that the farther the starting position of the bokeh effect is, the greater the intensity of the bokeh effect. Furthermore, the electronic device can determine that the bokeh effect is not applied to the region of interest (ROI) or that the ROI has a weaker bokeh effect than the background region, to provide bokeh characteristics per image area relative to the ROI. For example, refer to... Figure 10 Example 1030 of the bokeh characteristics of each image region shows that the intensity of the bokeh effect increases with distance from the center point 1025. (Reference) Figure 10The bokeh effect intensity of region 1032 is set to "5", and the bokeh effect intensity of region 1033, which is farther from the center point 1025 than region 1032, is set to "8". Furthermore, although the distance between pixels and the center point 1025 is the same, the intensity of the bokeh effect can vary depending on the region to which the pixel belongs. (Reference) Figure 10 Although region 1031 included in region of interest 610 has the same pixel distance from center point 1025 as region 1032, the intensity of bokeh effect of region 1031 included in region of interest 610 is set to "1".

[0101] According to another embodiment, in operation 960, the electronic device can determine the bokeh characteristics of each image region based on the difference between the depth value of the depth information at the starting position of the bokeh effect and the depth value of the depth information on the pixel where the bokeh characteristics of each image region are to be determined. For example, the electronic device can determine the bokeh characteristics of each image region such that the bokeh effect is applied more strongly when the depth value of the pixel is greater than the depth value at the starting position of the bokeh effect.

[0102] Then, in operation 970, the electronic device can apply a bokeh effect to the image based on the bokeh characteristics determined for each image region. For example, Figure 11 An example is shown of an electronic device according to an embodiment applying a bokeh effect to an image 1100 with a zoom blur effect. (Refer to...) Figure 11 For areas including the shape of people in the image, the blurring effect is small. Furthermore, the further away from the image center, the more blurred the background image becomes. Additionally, the direction 1110 of the bokeh effect applied to the background area appears in a straight line extending from the center position towards the pixels.

[0103] Figure 12 It shows an electronic device (e.g., Figure 1 Electronic device 101 or Figure 2 Flowchart 1200 of the process of applying bokeh effect to an electronic device 200 to achieve motion blur effect.

[0104] According to an embodiment, the electronic device can use an input device (e.g., Figure 1 The input device 150 receives user input for selecting a bokeh application mode. (See reference) Figure 12 Operation 1210 allows the electronic device to receive user input selecting a motion blur effect operation mode, which is a bokeh application mode. The motion blur effect operation mode refers to a photographic effect generated during image capture when a moving subject is captured in focus while the camera tracks the subject, typically resulting in a linearly blurred background.

[0105] Subsequently, in operation 1220, the electronic device can acquire image data including the image. In operation 1230, the electronic device can acquire depth information about the acquired image. Then, in operation 1240, the electronic device can divide the image into background regions and regions of interest based on the depth information. For example, the electronic device can classify pixels with similar depth values ​​in the depth map included in the depth information into a group, and can set the region containing the pixels included in the classification group as either a background region or a region of interest.

[0106] Then, in operation 1250, the electronic device can analyze the image included in the region of interest. As a result of analyzing the image included in the region of interest, the electronic device can determine the direction of movement of the subject being photographed in the region of interest. For example, the processor can analyze the features of a human body or vehicle and, using image recognition technology, determine the "front" side of the human body or vehicle depicted in the image. Then, based on the estimated "front" side of the depicted object (e.g., a user running forward or a car moving forward, such as...), the direction of movement can be determined. Figure 13 (As shown) Estimate the direction of movement. Subsequently, in operation 1260, the electronic device can determine the bokeh characteristics of each image region based on the determined direction of movement. For example, reference is shown... Figure 13 The conceptual diagram 1300 illustrates the bokeh characteristics of each image region for a motion blur effect. An electronic device can analyze the region of interest 1330 capturing a moving subject to determine the direction of movement 1340. The electronic device can determine the bokeh characteristics of each image region such that the direction of the bokeh effect corresponding to the direction of movement 1340 determined relative to the background region 1320 is applied. According to an embodiment, the electronic device can determine the bokeh characteristics of each image region of a boundary region 1350 such that the bokeh effect is applied to the boundary region 1350, which includes a boundary line 1355 adjacent to the region of interest 1330 and the background region 1320. After determining the direction of movement, then in operation 1270, the bokeh effect can be applied to the background to simulate motion blur of the image background, thereby giving the impression that the subject is moving.

[0107] Figure 14 This is an example of an electronic device applying bokeh effects to generate a motion blur effect in image 1400. (Reference) Figure 14 The system identifies regions of interest 1410, including the motorcycle and the person. A bokeh effect is not applied to the motorcycle and the person. Instead, a bokeh effect is applied to the background region 1420 and the boundary region 1450, thus blurring the background of the image and creating a greater illusion that the motorcycle and the user are moving in the detected direction.

[0108] Figure 15 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 A flowchart 1500 shows the process by which an electronic device 200 applies a bokeh effect based on a path input by the user.

[0109] According to an embodiment, the electronic device can use an input device (e.g., Figure 1 The input device 150 receives user input for selecting a bokeh application mode. (See reference) Figure 15 In operation 1510, the electronic device can receive user input selecting a path selection blur effect operation mode, which is a bokeh application mode. The path selection blur effect can indicate the bokeh effect, wherein the area and direction of the bokeh effect are determined based on the selected path (e.g., the desired path that the user follows or otherwise inputs the effect for).

[0110] Subsequently, in operation 1520, the electronic device can acquire image data including the image. In operation 1530, the electronic device can acquire depth information of the acquired image data. Then, in operation 1540, the electronic device can divide the image into background regions and regions of interest based on the depth information. For example, the electronic device can classify pixels with similar depth values ​​in the depth map included in the depth information into a group, and can set the region containing the pixels included in the classification group as either a background region or a region of interest. For example, refer to... Figure 16 The electronic device can divide the image 1600 into a region of interest 1610 that displays the main subject and a background region 1620.

[0111] In operation 1550, the electronic device can be accessed via an input device (e.g., Figure 1 The input device 150 receives user input selecting a path. For example, the user input may include drag input on a touchscreen displaying an image, starting from a first position and ending at a second position. Alternatively, the electronic device may receive user input from another type of input device. According to an embodiment, the electronic device may perform operation 1550 after displaying an image.

[0112] Furthermore, in operation 1560, the electronic device can determine the bokeh characteristics of each image region based on the selected path. Here, the electronic device can determine the bokeh characteristics of each image region by further considering the regions to which pixels belong in the background region or region of interest classified based on depth information.

[0113] refer to Figure 16 When the user inputs a path 1630, the electronic device can determine a path region 1635 that includes path 1630. The electronic device can determine the bokeh characteristics of each image region, so that a large bokeh effect is applied to the path region. For example, in areas not included in path region 1635... Figure 16In region 1641 of the region of interest 1610, the intensity of the bokeh effect can be set to "0", and in region 1642 included in the path region 1635, the intensity of the bokeh effect can be set to "3". Furthermore, the electronic device can determine the bokeh characteristics of each image region, so that a large bokeh effect is applied to the background region. For example, in the path region 1635... Figure 16 In the background area 1620, within area 1643, the bokeh effect can be set to "10".

[0114] Then, in operation 1570, the electronic device can apply a bokeh effect to the image based on the determined bokeh characteristics of each image region. For example, refer to Figure 17 Electronic device 1700 (e.g., Figure 1 Electronic device 101 or Figure 2 The electronic device 200 can display the original image 1710 on a touchscreen. According to an embodiment, the original image 1710 may be an image without bokeh (e.g., an image that clearly shows the subject). User 1 can select a path 1720 on the touchscreen using a touch input interface. The electronic device 1700, which receives user input for path selection, can output an image 1730 with bokeh effects applied. A weak bokeh effect is applied to the subject corresponding to the region of interest 1740 on image 1730. Furthermore, a relatively strong bokeh effect is applied to the area on the path 1720 selected by the user.

[0115] Figure 18 This illustrates an electronic device according to an embodiment (e.g., Figure 1 Electronic device 101 or Figure 2 Flowchart 1800 of the process of applying bokeh effect to an electronic device 200 to have a tilt-shift blur effect.

[0116] According to an embodiment, the electronic device can use an input device (e.g., Figure 1 The input device 150 receives user input for selecting a bokeh application mode. (See reference) Figure 18 In operation 1810, the electronic device can receive user input for selecting a "tilt" shift blur effect mode, which is a bokeh application mode. The tilt-shift blur effect refers to the photographic effect that occurs in the captured image when the camera is tilted while taking the picture.

[0117] Then, in operation 1820, the electronic device can acquire image data including the image. In operation 1830, the electronic device can acquire depth information about the acquired image data. Subsequently, in operation 1840, the electronic device can divide the image into background regions and regions of interest based on the depth information. For example, the electronic device can classify pixels with similar depth values ​​in the depth map included in the depth information into a group, and can set the region containing the pixels included in the classification group as either a background region or a region of interest.

[0118] In Operation 1850, the electronic device can divide an image into multiple shifted regions. For example, reference... Figure 19 The electronic device can divide the image 1910 into a clear region 1911, a faded region 1912, and a blurred region 1913.

[0119] Subsequently, in operation 1860, the electronic device can determine the bokeh characteristics of each image region based on the shifted region and the region of interest / background region. For example, the electronic device can determine the bokeh characteristics such that the region of interest is included in the sharp region (e.g., Figure 19 Region 1921), and the region of interest included in the faded region (e.g., Figure 19 Region 1922) and regions of interest included in the fuzzy regions (e.g., Figure 19 The 1923 area), and the background area included in the faded area (e.g., Figure 19 The 1925 area) and the background area included in the blurred area (e.g., Figure 19 The high intensity of the bokeh effect is applied sequentially in the 1926 area. Figure 19 In the context of a clear area, the background area included (e.g., Figure 19 In region 1924), the bokeh effect intensity is set to "5". However, the bokeh effect may not be applied (or the bokeh effect intensity may be 0) to the background area included in the sharp area (e.g., Figure 19 (Region 1924), or electronic devices can be applied to regions of interest included in a clearly defined region (e.g., Figure 19 Region 1921 has the same intensity of bokeh effect. Finally, in operation 1870, the bokeh effect can be applied to the image according to the characteristics of each image region.

[0120] Figure 20 It shows an electronic device (e.g., Figure 1 Electronic device 101 or Figure 2 Flowchart 2000 of the process of applying bokeh effect to an image based on depth values ​​in an electronic device 200.

[0121] According to an embodiment, in operation 2010, the electronic device can acquire image data including an image. In operation 2020, the electronic device can acquire depth information about the image.

[0122] Then, in operation 2030, the electronic device can calculate the value of the bokeh characteristic of each image region from the depth values ​​of the pixels included in the depth information. For example, the electronic device can determine the value obtained by multiplying the depth value of the pixel by a weight specified as the intensity of the bokeh effect to be applied to the pixel.

[0123] According to an embodiment, in operation 2030, the electronic device can input a depth value into a depth-to-feature conversion function to calculate a value for the bokeh characteristic of each image region. When a depth value is input, the depth-to-feature conversion function can return a value for the bokeh characteristic corresponding to the depth value. The electronic device can determine the value returned by the depth-to-feature conversion function as the bokeh characteristic of each image region relative to each pixel. According to an embodiment, as the depth value becomes larger, the electronic device determines that the intensity of the bokeh effect has a larger value.

[0124] Then, in operation 2040, the electronic device can apply a bokeh effect to the image based on the bokeh characteristics of each determined image region.

[0125] Figure 20 It shows an electronic device (e.g., Figure 1 Electronic device 101 or Figure 2 Flowchart 2000 of the process by which an electronic device (200) applies a bokeh effect to an image based on a depth map.

[0126] According to an embodiment, in operation 2010, the electronic device can acquire image data including an image. In operation 2020, the electronic device can acquire depth information including a depth map of the image.

[0127] Then, in operation 2030, the electronic device can configure a bokeh characteristic map corresponding to the depth values ​​of the depth map. Subsequently, in operation 2040, the electronic device can apply a bokeh effect to the image based on the bokeh characteristic map included in the bokeh characteristics of each image region.

[0128] Electronic devices according to embodiments (e.g., Figure 1 Electronic device 101 or Figure 2 The electronic device 200 may include a memory (e.g., Figure 1 memory 130 or Figure 2 The memory 260) and the processor electrically connected to the memory (e.g., Figure 1 Processor 120 or Figure 2The processor 220. The memory can store instructions that the processor can execute. The processor executes instructions to obtain image data including an image comprising multiple pixels, obtain depth information associated with the multiple pixels, use the depth information to determine the bokeh characteristics of each image region in the image, and apply bokeh effects to the multiple pixels based on the bokeh characteristics of each image region.

[0129] According to an embodiment, the processor can execute instructions to obtain image data of an image comprising multiple pixels. The processor can use depth information associated with the multiple pixels to determine regions of interest and background regions in the image. The processor can determine the bokeh characteristics of each first image region of the region of interest based on a first characteristic criterion, determine the bokeh characteristics of each second image region of the background region based on a second characteristic criterion distinct from the first characteristic criterion, and apply a bokeh effect to the multiple pixels based on the bokeh characteristics of each first image region and each second image region.

[0130] According to an embodiment, the bokeh characteristics of each first image region and each second image region may include the intensity of the bokeh effect to be applied to the pixels included in the region. As the depth value included in the depth information increases, the processor may determine the intensity of the bokeh effect to be a large value relative to the bokeh characteristics of each first image region and each second image region.

[0131] According to an embodiment, the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) may include the intensity of the bokeh effect to be applied to the pixels included in the region where the bokeh characteristics of each image region are set. According to an embodiment, when the depth value included in the depth information becomes larger, the processor may determine that the intensity of the bokeh effect has a larger value.

[0132] According to embodiments, the electronic device may include a camera module comprising an image sensor for capturing images (e.g., Figure 1 Camera module 180 or Figure 2 (Camera module 280). The processor can determine the depth information of the peripheral pixels based on the difference between the distance value measured by the center pixel corresponding to a specified center position in the image sensor and the distance values ​​measured by the peripheral pixels other than the center pixel.

[0133] According to an embodiment, based on the region to which each of the multiple pixels in the region of interest or the background region belongs, the processor can use depth information to divide the image into regions of interest and background regions, and determine the bokeh characteristics of each image region differently (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region).

[0134] According to an embodiment, the processor can determine the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) based on the image height information of the image and the region to which each of the multiple pixels belongs.

[0135] According to an embodiment, the electronic device may include a camera module capable of capturing images with the camera and obtaining depth values ​​based on the distance to a subject corresponding to a region in the image. The depth information may include a depth map including the depth values, and the processor may use the depth map to determine the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region).

[0136] According to an embodiment, the camera module may include a multi-camera system comprising multiple cameras or a sensor capable of detecting depth.

[0137] According to an embodiment, the depth information may include a depth value for each of a plurality of pixels. The processor may input the depth values ​​to a depth-to-feature conversion function and determine the value returned by the depth-to-feature conversion function as a bokeh characteristic for each image region relative to each of the plurality of pixels.

[0138] According to an embodiment, the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) may include the direction of the bokeh effect to be applied to the pixels included in the region.

[0139] The electronic device according to the embodiment can be based on an input device (e.g., Figure 1 The direction of the bokeh effect is determined by the user input input through the input device 150.

[0140] According to an embodiment, user input may include input for selecting a bokeh application mode. When the bokeh application mode selected based on user input is a first mode (e.g., a radial blur effect operation mode), the processor may determine the direction of a tangential vector of a circle having a specified center position in the image and a radius from the center position in the image to the pixel, as the direction of the pixel's bokeh effect. Optionally, when the bokeh application mode selected based on user input is a second mode (e.g., a zoom blur effect operation mode), the processor may determine the direction of a straight line extending from the center position toward the pixel as the direction of the pixel's bokeh effect.

[0141] According to an embodiment, the electronic device can receive user input, including input for selecting a path located on an image, via an input device. The processor can determine the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) based on the selected path and depth information.

[0142] According to an embodiment, the processor can use depth information to divide an image into regions of interest and background regions, and set the starting position of the bokeh effect within the regions of interest. The processor can determine the bokeh characteristics of each image region of multiple pixels (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) based on the inter-pixel distance from the starting position of the bokeh effect to the pixel.

[0143] According to an embodiment, the processor can detect a subject of interest included in the region of interest and determine the starting position of the bokeh effect in the region where the subject of interest is displayed.

[0144] According to an embodiment, the processor can determine the center point (e.g., the center of gravity) of the area where the subject of interest is displayed as the starting point of the bokeh effect.

[0145] According to an embodiment, the processor can use depth information to divide an image into regions of interest (ROIs) and background regions. The processor can analyze the movement direction of a subject within the divided ROIs. The processor can determine the bokeh characteristics of each image region relative to the background region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) based on the movement direction of the subject.

[0146] According to an embodiment, the processor can apply a bokeh effect to a boundary region that includes the boundary lines adjacent to the region of interest and the background region.

[0147] According to an embodiment, the processor can divide a region in an image into multiple shift regions (e.g., blurred regions, faded regions, or sharp regions) with shift steps in a specified direction. The processor can determine the bokeh characteristics of each image region (e.g., the bokeh characteristics of each first image region and the bokeh characteristics of each second image region) based on the divided region of interest or using the depth information of the shift region to which the pixel belongs and the regions divided by the shift steps.

[0148] According to an embodiment, the processor can perform image recognition processing on the region of interest to classify the subjects included in the region of interest. The processor can determine the bokeh characteristics of each image region within the region of interest based on the classification results of the subjects. For example, when the subject is a person, the electronic device may not apply a bokeh effect, and when the subject is an object such as a building or tree, a bokeh effect may be applied.

[0149] According to an embodiment, an electronic device can be controlled to obtain image data including an image, obtain depth information associated with multiple pixels, use the depth information to determine the bokeh characteristics of each image region in the image, and apply the bokeh effect based on the bokeh characteristics of each image region to the multiple pixels.

[0150] According to an embodiment, an electronic device can be controlled to obtain image data of an image, obtain depth information associated with multiple pixels, determine the image as a region of interest and a background region based on the depth information, determine the bokeh characteristics of each first image region of the region of interest based on a first characteristic criterion, determine the bokeh characteristics of each second image region of the background region based on a second characteristic criterion different from the first characteristic criterion, and apply a bokeh effect to multiple pixels based on the bokeh characteristics of each first image region and each second image region.

[0151] It should be understood that certain embodiments of this disclosure and the terminology used therein are not intended to limit the technology described herein to the particular embodiments, but rather to include various changes, equivalents, and / or substitutions to the embodiments. For the description of the drawings, similar components may be labeled by similar reference numerals. As used herein, singular forms may also include plural forms unless the context clearly indicates otherwise. In this invention, expressions such as “A or B,” “at least one of A and / or B,” “A, B, or C,” “at least one of A, B, and / or C,” etc., may include any and all combinations of one or more associated listed items. Terms such as “first,” “second,” etc., used in this invention may be used to refer to multiple components regardless of order and / or priority, and are used to distinguish related components from other components, but are not limiting. It should be understood that when a component (e.g., a first component) is referred to as being (operationally or communicatively) coupled / coupled to another component (e.g., a second component), or “connected” to another component (e.g., a second component), it may be directly coupled / coupled to another component or connected to another component, or there may be an intermediate component (e.g., a third component).

[0152] Depending on the context, the expression “suitable for or configured to” as used herein may be used interchangeably with expressions such as “suitable for,” “capable,” “designed for,” “suitable for,” “manufactured for,” or “capable of.” The expression “device configured to” may mean that the device is “capable” of operating with another device or other component. For example, “processor configured (or suitable for) performing A, B, and C” may refer to a dedicated processor (e.g., an embedded processor) or a general-purpose processor (e.g., a central processing unit (CPU) or application processor (AP)) for performing the corresponding operations, which can perform the corresponding operations by running one or more programs stored in a memory device (e.g., memory 130).

[0153] The term "module" as used in this invention can refer to a unit, for example, comprising one or more combinations of hardware, software, and firmware. The term "module" is used interchangeably with the terms "unit," "logic," "logic block," "component," and "circuit." A "module" can be the smallest unit of an integrated component or a part thereof. A "module" can be implemented mechanically or electronically. For example, a "module" can include at least one of an application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), and a programmable logic device for performing known or to be developed operations.

[0154] At least a portion of a device (e.g., its modules or functions) or method (e.g., operation) according to certain embodiments can be implemented, for example, by instructions stored in a computer-readable storage medium (e.g., memory 130) as program modules. When executed by a processor (e.g., processor 120), the instructions cause one or more processors to perform the function corresponding to the instructions. The computer-readable recording medium may include a hard disk, floppy disk, magnetic media (e.g., magnetic tape), optical media (e.g., optical disc read-only memory (CD-ROM) and digital versatile disc (DVD), magneto-optical media (e.g., magneto-optical floppy disk)), internal memory, etc. The instructions may contain code generated by a compiler or code executable by an interpreter.

[0155] Each component (e.g., a module or program module) according to certain embodiments may be implemented using one or more entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be further included. Optionally, or additionally, some components (e.g., modules or program modules) may be integrated into one entity to perform the same or similar functions as those performed by the respective components prior to integration. According to certain embodiments, operations performed by modules, program modules, or other components may be run sequentially, in parallel, repeatedly, or heuristically. Furthermore, some operations may be run in a different order, or may be omitted. Alternatively, other operations may be added.

[0156] While the invention has been shown and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the invention as defined by the appended claims and their equivalents.

Claims

1. An electronic device comprising: Memory; as well as The processor is electrically connected to the memory. The memory stores instructions that can be executed by the processor to enable the electronic device to: Obtain image data that includes multiple pixels. A region of interest and a background region are set within the image using depth information associated with each of the plurality of pixels, wherein the region of interest includes one or more first sub-regions and the background region includes one or more second sub-regions; Determine the starting position of the bokeh effect within the region of interest or background area; Based on the distances between the one or more first sub-regions and the starting position, and the distances between the one or more second sub-regions and the starting position, the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions are determined, and... Based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions, the image is processed to apply the bokeh effect to the plurality of pixels.

2. The electronic device according to claim 1, wherein, The corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions indicate the corresponding intensity of the bokeh effect, and The intensity of the indicated bokeh effect increases based on the increase in depth indicated by the depth information.

3. The electronic device according to claim 1, wherein, The instructions can also be executed by a processor to enable electronic devices: The height of the image is obtained, wherein the corresponding bokeh characteristics are determined at least in part based on the obtained height, and whether the sub-region is included in the region of interest or the background region.

4. The electronic device according to claim 1, further comprising: A camera module for capturing images, the camera including at least one of a plurality of image sensors and a depth detection sensor.

5. The electronic device according to claim 1, wherein, The depth information includes the depth value of each pixel among the plurality of pixels, and The instructions can be executed by a processor to enable the electronic device to: The corresponding bokeh characteristics for each pixel are generated by processing the depth value of each pixel using a depth-to-feature conversion function.

6. The electronic device according to claim 1, wherein, At least one of the corresponding bokeh characteristics includes the direction of the bokeh effect to be applied to each of the plurality of pixels.

7. The electronic device according to claim 6, further comprising: Input device, The instructions can be executed by the processor to determine the direction of the bokeh effect based on user input received through the input device.

8. The electronic device according to claim 7, wherein, The user selects the bokeh application mode. The instructions can also be executed by a processor to enable the electronic device to: When the selected bokeh application mode is the first mode, the direction of the tangential vector of a circle including pixels is detected. This circle is centered on the image center and has a radius defined as the inter-pixel distance from the center to the pixel. The direction of the detected tangential vector is set as the direction of the pixel's bokeh effect. When the selected bokeh application is the second mode, the direction of the bokeh effect of the pixel is set to the direction of the straight line extending from the center position towards the pixel.

9. The electronic device according to claim 7, wherein, The user inputs the input path on the tracking image. In addition to depth information, bokeh characteristics are also based at least on the tracked input path.

10. The electronic device according to claim 1, wherein, The instructions can be executed by a processor to enable electronic devices to: Analyze the image to detect the subject of interest within the region of interest. The starting position is set within a sub-region of the area of ​​interest where the subject of interest is located.

11. The electronic device according to claim 1, wherein, The instructions can be executed by a processor to enable electronic devices to: Determine the direction of movement of the subject within the region of interest; and The bokeh characteristics of each image region relative to the background region are determined based on the direction of movement.

12. The electronic device according to claim 11, wherein, The instructions can be executed by a processor to enable electronic devices to: A bokeh effect is applied to a boundary region that includes a boundary line, the boundary region being arranged adjacent to both the region of interest and the background region.

13. The electronic device according to claim 1, wherein, The instructions can be executed by a processor to enable electronic devices to: The image is divided into multiple shift regions, each of which includes a shift step value; as well as The bokeh characteristics of each of the plurality of shift regions are determined by applying a baseline bokeh value modified by the corresponding shift step value to each shift region.

14. A method for processing images in an electronic device, the method comprising: Obtain image data that includes multiple pixels; Obtain the depth information of each of the plurality of pixels; The depth information is used to set a region of interest and a background region within the image, wherein the region of interest includes one or more first sub-regions and the background region includes one or more second sub-regions; Determine the starting position of the bokeh effect within the region of interest or background area; Based on the distances between the one or more first sub-regions and the starting position, and the distances between the one or more second sub-regions and the starting position, the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions are determined, and... Based on the corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions, the image is processed to apply the bokeh effect to the plurality of pixels.

15. The method according to claim 14, wherein, The corresponding bokeh characteristics of the one or more first sub-regions and the one or more second sub-regions indicate the corresponding intensity of the bokeh effect, and The intensity of the indicated bokeh effect increases based on the increase in depth indicated by the depth information.

Images