Electronic device and control method for image processing of ghost object in live-view image

Abstract

This electronic device may comprise a camera, a display, a sensor, and at least one processor which: controls the display to display a plurality of first live-view images acquired through the camera; identifies, in the plurality of first live-view images, a first region and a second region each having a predetermined luminance or higher; compares a first motion of the first region and a second motion of the second region with a third motion of the electronic device acquired through the sensor so as to identify one of the first region and the second region as a region to be removed; performs image processing of a region corresponding to the region to be removed in a plurality of second live-view images acquired after the plurality of first live-view images; and displays the plurality of image-processed second live-view images through the display.

Description

Electronic device for image processing of ghost objects in live view images and control method thereof

[0001] One or more embodiments of the present disclosure relate to an electronic device and a control method thereof, and more particularly, to an electronic device for image processing a ghost object in a live view image and a control method thereof.

[0002] Advances in electronic devices and multimedia technology have led to the use of smartphones for taking photos and videos. In particular, smartphone cameras, powered by AI models and high-performance processors, can capture photos and / or videos in a wide range of scenarios and conditions. Furthermore, AI models can further enhance the quality of photos and / or videos through one-click post-processing.

[0003] However, smartphones can experience lens flare phenomena such as camera ghosting or lens ghosting, which reduces the quality of the results and requires delicate post-processing to manually remove ghost objects.

[0004] According to one embodiment of the present disclosure to achieve the above object, an electronic device may include a camera, a display, a sensor, and at least one processor configured to control the display to display a plurality of first live view images acquired through the camera, identify a first area and a second area each having a preset brightness or higher in the plurality of first live view images, compare a first motion of the first area and a second motion of the second area with a third motion of the electronic device acquired through the sensor, identify one area of ​​the first area and the second area as a removal target area, image-process an area corresponding to the removal target area in a plurality of second live view images acquired after the plurality of first live view images through the camera, and display the image-processed plurality of second live view images through the display.

[0005] Meanwhile, according to one embodiment of the present disclosure, a control method of an electronic device may include the steps of displaying a plurality of first live view images acquired through a camera included in the electronic device, identifying a first area and a second area each having a preset brightness or higher in the plurality of first live view images, comparing a first motion of the first area and a second motion of the second area with a third motion of the electronic device to identify one of the first area and the second area as a removal target area, performing image processing on an area corresponding to the removal target area in a plurality of second live view images acquired after the plurality of first live view images, and displaying the image-processed plurality of second live view images.

[0006] The above and other aspects, features and advantages of the disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0007] Figures 1a to 1c are drawings for explaining examples of camera ghosting.

[0008] FIG. 2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.

[0009] FIG. 3 is a block diagram showing a detailed configuration of an electronic device according to an embodiment of the present disclosure.

[0010] FIG. 4 is a drawing for explaining the motion of a ghost object according to the motion of an electronic device according to an embodiment of the present disclosure.

[0011] FIGS. 5 to 10 are drawings for explaining a method of tracking a ghost area according to one embodiment of the present disclosure.

[0012] FIG. 11 is a diagram illustrating a method for identifying an additional ghost area according to an embodiment of the present disclosure.

[0013] FIG. 12 is a diagram for explaining an operation when a ghost area is removed according to an embodiment of the present disclosure.

[0014] FIG. 13 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

[0015] Hereinafter, some exemplary embodiments are illustrated in the drawings and described in detail in the detailed description. However, it should be understood that the exemplary embodiments of the present disclosure are subject to various modifications. In other words, the present disclosure is not limited to the specific exemplary embodiments, but encompasses all modifications, equivalents, and alternatives that do not depart from the scope and spirit of the present disclosure. Furthermore, well-known functions or configurations are not described in detail as unnecessary details may obscure the disclosure.

[0016] An object of the present disclosure is to provide an electronic device and a control method thereof for identifying a ghost object in a live view image and providing a live view image from which the identified ghost object has been removed.

[0017] It should be understood that the various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments.

[0018] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0019] The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0020] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or any possible combination of them. For example, the expression "at least one of A and B" should be understood to include A alone, B alone, or both A and B.

[0021] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0022] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0023] The terms “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in this document, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0024] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0025] When we say that a component is “on” another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0026] The term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0027] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0028] Figures 1a to 1c are drawings for explaining examples of camera ghosting.

[0029] Camera ghosting, also known as lens ghosting, is a type of flare that occurs when light or rays from a bright object are repeatedly reflected off the lens surface. For example, the live view image in Fig. 1a may include a lit traffic light (real object) and a ghost object resembling a lit traffic light. As illustrated in Fig. 1b, the ghost object included in the live view image may be generated by light rays emanating from the surface of the cover glass that contacts the lens.

[0030] The occurrence of ghost objects is explained in more detail with reference to Fig. 1c. First, when light from a bright light source strikes a lens, some of the light may pass through the lens and reach the image sensor, while others may be reflected by the lens. The reflected light may then strike the cover glass, and may be reflected by the cover glass again and head toward the lens. Some of the light reflected by the cover glass may pass through the lens and reach the image sensor. Accordingly, the image sensor can detect two similarly shaped objects. Here, one of the two similarly shaped objects may be caused by light from the bright light source, and the other may be caused by light from a ghost source formed on the cover glass when light from the bright light source is reflected more than twice.

[0031] Ghost objects can be manually removed through image post-processing. For example, a smartphone can use a neural network model to remove a user-selected ghost object and fill in the area of ​​the removed ghost object with related background. However, this process requires user input, and skipping it without user input can lead to problems.

[0032] Additionally, when there are multiple light sources, it is difficult to identify multiple ghost objects corresponding to the multiple light sources, and the technique for removing ghost objects may be applied only to individual corresponding frames and may not be applicable to the video.

[0033] Furthermore, because ghost objects are an optical phenomenon, image sensors cannot identify them in live view images or videos. This means that automatic ghost object detection or real-time ghost object removal is necessary.

[0034] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0035] The electronic device (100) is a device that provides an image captured by a camera as a live view image, and may be implemented as a smartphone, tablet PC, digital camera, TV, desktop PC, laptop, etc. In particular, the electronic device (100) may be a device equipped with a camera, removing a ghost object from an image captured by the camera, and providing an image from which the ghost object has been removed as a live view image.

[0036] However, the present invention is not limited thereto, and the electronic device (100) may be a device that provides an image captured by a camera as a live view image. Alternatively, the electronic device (100) may receive an image from an external camera and provide the received image as a live view image. Alternatively, the electronic device (100) may be a device that provides an image captured by a camera as a live view image to an external display device.

[0037] According to FIG. 2, the electronic device (100) includes a camera (110), a display (120), a sensor (130), and a processor (140). However, the present invention is not limited thereto, and the electronic device (100) may be implemented in a form in which some components are omitted or additional components are added.

[0038] The camera (110) may be implemented to capture still images and / or moving images. The camera (110) may capture still images at a specific point in time, but may also capture still images continuously.

[0039] The camera (110) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). In addition, the camera (110) may further include a cover glass that is in contact with the lens. The shutter controls the time at which light reflected from a subject enters the camera (110), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. The solid-state image sensor outputs an image generated by the photocharges as an electrical signal when light reflected from a subject is accumulated as a photocharge. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0040] The display (120) is a configuration that displays content and can be implemented as a variety of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (120) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (120) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0041] The sensor (130) can obtain motion information of the electronic device (100) through at least one sensor. For example, the sensor (130) includes at least one of a gyro sensor, an acceleration sensor, or a magnetometer sensor, and the processor (140) can obtain motion information of the electronic device (100) based on information received from one or more sensors included in the sensor (130).

[0042] A gyro sensor is a sensor that detects the rotation angle of an electronic device (100) by measuring angular velocity, and can measure changes in the orientation of an object based on maintaining a constant initially set direction with high accuracy regardless of the rotation of the Earth. A gyro sensor is also called a gyroscope, and can be implemented mechanically or optically using light.

[0043] An acceleration sensor is a sensor that measures the acceleration or impact intensity of an electronic device (100), and is also called an accelerometer. An acceleration sensor detects dynamic forces such as acceleration, vibration, and impact, and can be implemented as an inertial type, a gyro type, a silicon semiconductor type, etc. depending on the detection method. In other words, an acceleration sensor is a sensor that senses the degree of inclination of an electronic device (100) using gravitational acceleration, and can typically be formed of a two-axis or three-axis fluxgate.

[0044] A magnetometer sensor generally refers to a sensor that measures the strength and direction of the Earth's magnetism, but in a broader sense, it also includes a sensor that measures the strength of an object's magnetization, and is also called a magnetometer. A magnetometer sensor can be implemented to measure the strength of a magnetic field by suspending a magnet horizontally in a magnetic field and measuring the direction of the magnet's movement, or by rotating a coil in a magnetic field and measuring the induced electromotive force generated in the coil.

[0045] The processor (140) can obtain motion information of the electronic device (100) using one or more sensors as described above. For example, the processor (140) can obtain the movement of the electronic device (100) as information of a three-dimensional space. Alternatively, the processor (140) can obtain the movement of the electronic device (100) as information of a three-dimensional space and project the information of the three-dimensional space into a two-dimensional space to obtain information of a two-dimensional space.

[0046] For convenience of explanation, the sensor (130) is described as including at least one of a gyro sensor, an acceleration sensor, or a magnetometer sensor. However, the present invention is not limited thereto, and the sensor (130) may be any sensor as long as the information obtained by the sensor can be used to obtain motion information of the electronic device (100).

[0047] The processor (140) controls the overall operation of the electronic device (100). Specifically, the processor (140) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100). For example, the processor (140) is connected to components such as a camera (110), a display (120), a sensor (110), and the like and can control the operation of the electronic device (100).

[0048] The processor (140) may be implemented as one or more processors. For example, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or more combinations of components of the electronic device (100) and perform operations related to communication and / or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0049] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-specific processor). For example, a process of quantizing a neural network model according to an embodiment of the present disclosure may be performed by a general-purpose processor, and a process of learning or inferring the quantized neural network model may be performed by an AI-specific processor.

[0050] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0051] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0052] In embodiments of the present disclosure, one or more processors may refer to a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression processor (140).

[0053] The processor (140) can display a plurality of first live view images acquired through the camera (110) through the display (120), and identify a first area and a second area having a preset brightness or higher in the plurality of first live view images.

[0054] For example, the processor (140) may display a plurality of first live view images acquired through the camera (110) through the display (120), identify a first area and a second area having a brightness higher than a preset level in a first image among the plurality of first live view images, and track the first area and the second area identified in at least one second image after the first image among the plurality of first live view images.

[0055] The processor (140) may identify a target region to be removed among the first region and the second region based on the shape of at least one of the first region and the second region. For example, if the processor (140) identifies a first region and a second region having a brightness higher than a preset value in a plurality of first live view images, the processor (140) may compare the shape of the first region with the shape of the second region, and if the similarity is less than a threshold value, the processor (140) may identify the first region and the second region as being related. For example, if the similarity between the shape of the first region and the shape of the second region is less than a threshold value, the processor (140) may identify one of the first region and the second region as a light source and the other as a ghost object.

[0056] The processor (140) can track a first region and a second region in a plurality of first live view images to identify a first motion in the first region and a second motion in the second region. In addition, the processor (140) can obtain a third motion of the electronic device through the sensor (130).

[0057] The processor (140) can compare the first motion and the second motion with the third motion and identify one of the first and second regions as a region to be removed. For example, the processor (140) can identify a region among the first and second motions in which the motion direction is the same as the third motion or the movement distance over time is proportional to the third motion as a region to be removed.

[0058] Alternatively, the processor (140) may identify a first distance to a first object corresponding to a first area and a second distance to a second object corresponding to a second area, and may further consider a distance between a minimum distance from a lens to a cover glass and a maximum distance from a lens to a cover glass among the first distances and the second distances to identify a region to be removed. For example, one of the first object and the second object is a light source outside the electronic device (100), and the other of the first object and the second object is an area where light emitted from the light source reaches the cover glass after being reflected by the lens, and the light that reaches the cover glass after being reflected by the lens is reflected by the cover glass and reaches the lens, and the processor (140) may identify the other of the first object and the second object as the region to be removed.

[0059] Alternatively, the processor (140) may further consider the luminance of the first region and the luminance of the second region to identify the target region for removal. For example, the processor (140) may identify a region, among the first and second motions, that corresponds to a motion whose movement direction is the same as that of the third motion or whose distance by time is proportional to the distance of the third motion, and has relatively low luminance, as the target region for removal. Since the ghost source is generated on the cover glass as the light source's rays are reflected multiple times, the luminance of the ghost source may be lower than the luminance of the light source.

[0060] The processor (140) may change the method for identifying the target area for removal based on at least one of the hardware performance or resources of the electronic device (100). For example, if the hardware performance of the electronic device (100) is high and the resources are sufficient, the processor (140) may identify the target area for removal by performing all or part of the motion comparison, distance (movement distance and / or distance to an object) comparison, and luminance comparison mentioned above. Alternatively, if the hardware performance of the electronic device (100) is low and the resources are not sufficient, the processor (140) may identify the target area for removal by comparing only the movement direction of each area with the movement direction of the electronic device (100). In addition, the processor (140) may identify the target area for removal by comparing only the movement direction of each area with the movement direction of the electronic device (100), and then sequentially perform the proportionality between motions, distance comparison, luminance comparison, etc. to change the target area for removal.

[0061] The third motion is three-dimensional motion data, and the processor (140) may convert the third motion into two-dimensional motion data, and compare the first motion and the second motion with the converted third motion to identify one of the first area and the second area as a target area for removal.

[0062] The processor (140) can image-process an area corresponding to a target area to be removed in a plurality of second live view images after a plurality of first live view images, and display the image-processed plurality of second live view images through the display (120).

[0063] For example, the processor (140) may image process an area corresponding to a target area for removal in a plurality of second live view images following a plurality of first live view images based on at least one surrounding pixel value of the area corresponding to the target area for removal. In this case, the plurality of second live view images may be image processed as if there were no ghost objects included in the target area for removal.

[0064] However, the present invention is not limited thereto, and the processor (140) may also image process the target area for removal using a neural network model. For example, the processor (140) may apply a generative neural network model to the target area for removal to image process multiple second live view images as if the ghost object did not exist.

[0065] The processor (140) may determine the location of the area corresponding to the area to be removed in the plurality of second live view images based on the third motion. That is, when the area to be removed is identified, the processor (140) may determine the location of the area to be removed in subsequent live view images based on the third motion of the electronic device (100), and may omit the operation of identifying an area having a brightness higher than a preset value and the operation of identifying the area to be removed in the plurality of second live view images. The location of the area corresponding to the area to be removed in the plurality of second live view images may differ from the location of the area to be removed (the first area or the second area) determined in the plurality of first live view images due to the third motion of the electronic device (100).

[0066] However, the present invention is not limited thereto, and when a target area for removal is identified, the processor (140) may perform an operation of additionally identifying an area having a brightness higher than a preset level. When additional identification is made, the processor (140) may additionally identify the target area for removal, and when no additional identification is made, the operation of identifying the target area for removal may be omitted. Alternatively, the processor (140) may determine the location of an area corresponding to the target area for removal in a plurality of second live view images based on a third motion, and when the determined location is outside the plurality of second live view images, identify a third area having a brightness higher than a preset level in the plurality of second live view images. When the third area is identified, the processor (140) may additionally identify the target area for removal.

[0067] The electronic device (100) further includes a user interface and a memory, and when a shooting command is received through the user interface, the processor (140) can image-process an area corresponding to a target area to be removed from a live view image corresponding to the shooting command, and store the image-processed live view image in the memory.

[0068] By doing the above, you can solve the problem of ghost objects appearing in the live view image or being included in the captured image.

[0069] The functions related to artificial intelligence according to the present disclosure can be operated through a processor (140) and memory.

[0070] The processor (140) may be composed of one or more processors. The one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0071] One or more processors are controlled to process input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are dedicated AI processors, the dedicated AI processors may be designed with a hardware structure specialized for processing a specific AI model. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0072] Here, "created through learning" means that a basic artificial intelligence model is learned using a learning algorithm using a plurality of learning data, thereby creating a predefined set of operating rules or an artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0073] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0074] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0075] FIG. 3 is a block diagram showing a detailed configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0076] The electronic device (100) may include a camera (110), a display (120), a sensor (130), and a processor (140). In addition, according to FIG. 3, the electronic device (100) may further include a user interface (150), a memory (160), a communication interface (170), a microphone (180), and a speaker (190). Among the components illustrated in FIG. 3, a detailed description of the overlapping parts with the components illustrated in FIG. 2 will be omitted.

[0077] The user interface (150) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc., formed on any area of ​​the front, side, or back of the main body of the electronic device (100).

[0078] Memory (160) may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor (140) or the like can access it. To this end, the memory (160) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0079] At least one instruction related to the operation of the electronic device (100) or processor (140) may be stored in the memory (160). Here, the instruction is a code unit that instructs the operation of the electronic device (100) or processor (140), and may be written in machine language, which is a language that a computer can understand.

[0080] The memory (160) may store data in bit or byte units that can represent characters, numbers, images, etc. For example, a neural network model may be stored in the memory (160).

[0081] The memory (160) is accessed by the processor (140), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (140).

[0082] The communication interface (170) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device (100) can perform communication with a server through the communication interface (170).

[0083] The communication interface (170) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0084] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0085] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0086] Alternatively, the communication interface (170) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0087] In addition, the communication interface (170) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0088] The microphone (180) is configured to receive sound and convert it into an audio signal. The microphone (180) is electrically connected to the processor (140) and can receive sound under the control of the processor (140).

[0089] For example, the microphone (180) may be formed as an integral part integrated into the electronic device (100) in the upper direction, the front direction, the side direction, etc. of the electronic device (100). Alternatively, the microphone (180) may be provided in a remote control, etc., separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (180) and provide the received sound to the electronic device (100).

[0090] The microphone (180) may include various configurations such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0091] The microphone (180) may be implemented in the form of a sound sensor, and any configuration capable of collecting sound may be used.

[0092] The speaker (190) is a component that outputs various audio data processed by the processor (140) as well as various notification sounds and voice messages.

[0093] As described above, the electronic device (100) can provide a live view image from which a ghost object has been removed by performing image processing on a removal target area based on motion of an area having a preset brightness or higher and motion of the electronic device (100).

[0094] Additionally, the electronic device (100) can remove ghost objects from videos as well because it tracks the target removal area based on the motion of the electronic device (100).

[0095] Hereinafter, the operation of the electronic device (100) will be described in more detail with reference to FIGS. 4 to 12. For convenience of explanation, individual embodiments are described in FIGS. 4 to 12. However, the individual embodiments of FIGS. 4 to 12 may be implemented in any combination.

[0096] FIG. 4 is a drawing for explaining the motion of a ghost object according to the motion of an electronic device according to an embodiment of the present disclosure.

[0097] As illustrated in FIG. 4, when the electronic device (100) moves to the right, the actual object at position 410-1 and the actual object at position 420-1 in the live view image may move to positions 410-2 and 420-2, respectively. That is, when the electronic device (100) moves to the right, the actual object may move to the left, which is the opposite direction to the movement direction of the electronic device (100).

[0098] On the other hand, when the electronic device (100) moves to the right, the ghost object at position 430-1 in the live view image may move to position 430-2. That is, when the electronic device (100) moves to the right, the ghost object may move to the right in the same direction as the movement direction of the electronic device (100).

[0099] Accordingly, the processor (140) can identify a ghost object by comparing the motion of the electronic device (100) with the motion of each object.

[0100] Additionally, the processor (140) may provide a guidance message to the user to identify ghost objects. For example, if an area with a brightness higher than a preset value is identified in a plurality of first live view images, the processor (140) may provide a message such as "Please shake the electronic device" (or "Please move the electronic device").

[0101] Alternatively, the processor (140) may display a plurality of first live view images through the display (120) and provide a message such as “Shake the electronic device.”

[0102] FIGS. 5 to 10 are drawings for explaining a method of tracking a ghost area according to one embodiment of the present disclosure.

[0103] First, the processor (140) can identify a light source (S510). For example, the processor (140) can display a plurality of first live view images acquired through the camera (110) through the display (120) and identify an area having a brightness higher than a preset value among the plurality of first live view images.

[0104] For example, the processor (140) can capture a light source (610) and a ghost source (620) as shown in the upper left of FIG. 6 and display a live view image as shown in the upper right of FIG. 6 through the display (120).

[0105] The processor (140) can analyze the live view image through an object identification model as illustrated in the lower left of FIG. 6 to identify a first region (630) corresponding to the light source (610) and a second region (640) corresponding to the ghost source (620) in the live view image as illustrated in the lower right of FIG. 6. Here, the object identification model is a model that identifies a region with a preset brightness or higher in the image, and may be a rule-based model or a neural network model. The object identification model may be a model that identifies two regions with similar shapes. For example, the object identification model may be a model that identifies a first region (630) with a preset brightness or higher, and performs at least one of rotation and scaling on the first region (630) to identify a second region (640) corresponding thereto. Alternatively, the object identification model may identify a plurality of regions with a preset brightness or higher, and compare the shapes of two regions among the plurality of regions to identify the first region (630) and the second region (640).

[0106] However, it is not limited thereto, and the processor (140) may identify the first area (630) and the second area (640) through various methods.

[0107] The processor (140) can identify the first region (630) and the second region (640) as rectangular regions in the live view image. For example, the light source (610) and the ghost source (620) in the live view image may each be circular, but the processor (140) can identify the first region (630) and the second region (640) as rectangles that include the respective circular regions. That is, the first region (630) and the second region (640) may not only include the light source (610) and the ghost source (620), but may also further include a surrounding region of the light source (610) and a surrounding region of the ghost source (620). Thereafter, the processor (140) removes the ghost source (620) of the second area (640) through a mask corresponding to the second area (640), wherein the currently used mask may be a mask that removes the ghost source (620) through the surrounding pixel values ​​of the ghost source (620). The mask may be generated based on the surrounding pixel values ​​of the ghost source (620).

[0108] However, the present invention is not limited thereto, and the processor (140) may identify a first region (630) and a second region (640) corresponding to the light source (610) and the ghost source (620), respectively. That is, the first region (630) and the second region (640) may include only the light source (610) and the ghost source (620), respectively. In this case, the processor (140) removes the ghost source (620) of the second region (640) through a mask corresponding to an area larger than the second region (640), and the mask used here may be a mask that removes the ghost source (620) through the surrounding pixel values ​​of the ghost source (620). The mask may be generated based on the surrounding pixel values ​​of the ghost source (620).

[0109] When a light source is identified, the processor (140) can obtain motion data (S520). For example, as illustrated in FIG. 7, the processor (140) can identify a first motion (710) of a first region and a second motion (720) of a second region in multiple live view images according to the movement of the electronic device (100). For example, the processor (140) can identify a pixel-wise displacement of the first region in two live view images as the first motion (710), and can identify a pixel-wise displacement of the second region in two live view images as the second motion (720). As described above, the movement direction of the light source and the movement direction of the ghost source may be opposite to each other depending on the movement of the electronic device (100).

[0110] In addition, the processor (140) can identify a third motion of the electronic device (100) according to the movement of the electronic device (100). Here, the third motion is three-dimensional motion data, and the processor (140) can convert the third motion into two-dimensional motion data and compare the first motion and the second motion with the converted third motion. For example, the processor (140) can convert the three-dimensional motion data into two-dimensional motion data by projecting the third motion onto a plane parallel to the display (120) of the electronic device (100). However, the present invention is not limited thereto, and the processor (140) can convert the three-dimensional motion data into two-dimensional motion data in any number of ways.

[0111] The processor (140) can identify a ghost area based on each motion, i.e., a first motion of the first area and a motion of the second area (S530). Here, the ghost area may be a region to be removed. For example, the processor (140) can identify one of the first area and the second area as a region to be removed based on the movement direction of the electronic device (100). For example, the processor (140) can identify an area having a motion in the same direction as the movement direction of the electronic device (100) as a region to be removed.

[0112] Alternatively, the processor (140) may identify the target area for removal by considering not only the direction of movement of each motion but also whether the movement distance by time zone is proportional to the motion of the electronic device. For example, the processor (140) may compare one of the motions of the first and second regions with a third motion through regression analysis, and identify the area where a proportional relationship is established as a ghost area and the target area for removal, as illustrated in FIG. 8.

[0113] Alternatively, the processor (140) may further consider the distance to the object to identify the target area for removal. For example, as illustrated in FIG. 9, the processor (140) may identify a first distance (D1) to a light source corresponding to the first area and a second distance (D2) to a ghost source corresponding to the second area, and among the first distance (D1) and the second distance (D2), identify a second distance (D2) between a minimum distance (Rmin) from the lens to the cover glass and a maximum distance (Rmax) from the lens to the cover glass, and identify a second area corresponding to the second distance (D2) as the target area for removal.

[0114] Alternatively, the processor (140) may identify the target area for removal based on at least one of the brightness or size of the first area and the second area. For example, the processor (140) may identify a region with a lower brightness among the first area and the second area as the target area for removal. Alternatively, the processor (140) may identify a region with a smaller size among the first area and the second area as the target area for removal.

[0115] The above describes various methods for identifying target areas for removal, and each method can be combined in various ways.

[0116] When the target area for removal is identified, the processor (140) can mask the target area for removal (ghost area) (S540). For example, when the target area for removal is identified, the processor (140) can image-process the area corresponding to the target area for removal in a plurality of second live view images after the identification of the target area for removal is completed, based on the surrounding pixel values. For example, as illustrated in FIG. 10, when the target area for removal is identified in the first live view image, the processor (140) can identify the area corresponding to the target area for removal in the second live view image immediately following the first live view image, based on the third motion of the electronic device (100). That is, the processor (140) can predict the degree of movement of the target area for removal based on the third motion of the electronic device (100) during the time from the first live view image to the second live view image, and identify the area corresponding to the target area for removal in the second live view image. When the processor (140) identifies an area (1010) corresponding to the area to be removed in the second live view image, it generates an object removal mask (1020) corresponding thereto, and can remove a ghost object from the area (1010) corresponding to the area to be removed using the object removal mask (1020).

[0117] However, the present invention is not limited thereto, and the processor (140) may also guide the user to the ghost area before removing the ghost area. For example, if the ghost area is identified as a target area for removal, the processor (140) may display a focus indicating the ghost area. In addition, the processor (140) may provide a guidance message indicating that the ghost area can be removed through the displayed focus. When the user selects the focus, the processor (140) may also perform a ghost area masking operation.

[0118] When a plurality of second live view images are continuously displayed, the processor (140) can track an area (or ghost area) corresponding to a removal target area in each second live view image based on the third motion of the electronic device (100) (S550). The processor (140) can track an area corresponding to a removal target area in each second live view image and perform image processing to remove a ghost object in the removal target area in each second live view image.

[0119] FIG. 11 is a diagram illustrating a method for identifying an additional ghost area according to an embodiment of the present disclosure.

[0120] The processor (140) can perform image processing to remove ghost objects from a live view image. For example, the processor (140) can identify a first light source (1110-1) and a first ghost region corresponding to the ghost source (caused by light emitted from the first light source (1110-1)), and remove the first ghost region through a first object removal mask (1110-2), as illustrated on the left side of FIG. 11.

[0121] The processor (140) can predict the location of the target area to be removed in a subsequent live view image based on the third motion of the electronic device (100), and can continuously remove the first ghost area in the subsequent live view image through the first object removal mask (1110-2).

[0122] Additionally, the processor (140) can identify an area having a brightness higher than a preset level in a subsequent live view image. For example, the processor (140) can identify a second ghost area (1120-2) corresponding to a second light source (1120-1) and a ghost source (due to light emitted by the second light source (1120-1)), as shown on the left side of FIG. 11. The identification method of the second light source (1120-1) and the second ghost area (1120-2) may be the same as the identification method of the first light source (1110-1) and the first ghost area.

[0123] The processor (140) can remove the second ghost area through the second object removal mask (1120-3) using the same method as the method for removing the first ghost area (1110-2), as shown on the right side of FIG. 11.

[0124] FIG. 12 is a diagram for explaining an operation when a ghost area is removed according to an embodiment of the present disclosure.

[0125] The processor (140) can predict the location of the target area to be removed in the live view image based on the third motion of the electronic device (100). In addition, the processor (140) can predict the location of the light source in the live view image based on the third motion of the electronic device (100). As illustrated in FIG. 12, if the processor (140) identifies that the light source (1210) and the target area to be removed (1220) are outside the live view image based on the third motion of the electronic device (100), the processor (140) can stop the prediction operation.

[0126] FIG. 13 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

[0127] First, a plurality of first live view images acquired through a camera included in an electronic device are displayed (S1310). Then, a first region and a second region having a preset brightness or higher are identified from the plurality of first live view images (S1320). Then, a first motion of the first region and a second motion of the second region are compared with a third motion of the electronic device to identify one of the first region and the second region as a region to be removed (S1330). Then, an region corresponding to the region to be removed from a plurality of second live view images acquired after the plurality of first live view images is image-processed (S1340). Then, a plurality of second live view images that have been image-processed are displayed (S1350).

[0128] In addition, the step (S1330) of identifying the area to be removed may identify an area corresponding to a motion in which the direction of the motion is the same as the direction of the third motion and the distance by time is proportional to the distance by time of the third motion, among the first motion and the second motion, as the area to be removed.

[0129] And, the camera includes a lens and a cover glass in contact with the lens, and the step of identifying the area to be removed (S1330) identifies a first distance to a first object corresponding to the first area and a second distance to a second object corresponding to the second area, and the area to be removed can be identified based on the existence of a first distance or a second distance between a minimum distance from the lens to the cover glass and a maximum distance from the lens to the cover glass among the first area and the second area.

[0130] In addition, one of the first object and the second object is a light source outside the electronic device, and the other of the first object and the second object corresponds to an area where light emitted from the light source and reflected by the lens reaches the cover glass and is reflected by the cover glass and reaches the lens, and the step (S1330) of identifying the other of the first object and the second object as the area to be removed can identify the other of the first object and the second object as the area to be removed.

[0131] And, the image processing step (S1340) can process an area corresponding to the area to be removed in a plurality of second live view images obtained after (or acquired after) the plurality of first live view images based on at least one surrounding pixel value of the area corresponding to the area to be removed.

[0132] Additionally, the method may further include a step of determining a location of an area corresponding to a target area for removal in a plurality of second live view images based on the third motion.

[0133] And, if the determined location is outside the plurality of second live view images, the step of identifying a third area having a preset brightness or higher in the plurality of second live view images may be further included.

[0134] Additionally, the step of identifying the first region and the second region (S1320) can identify one of the first region and the second region as a target region for removal based on the shape of at least one region among the first region and the second region.

[0135] And, the third motion is three-dimensional motion data, and the step of identifying the area to be removed (S1330) converts the third motion into two-dimensional motion data, and compares the first motion and the second motion with the converted third motion to identify one area among the first area and the second area as the area to be removed.

[0136] In addition, when a shooting command is received, the method may further include a step of processing an area corresponding to a target area to be removed from a live view image corresponding to the shooting command and a step of storing the processed live view image.

[0137] According to various embodiments of the present disclosure as described above, an electronic device can provide a live view image from which a ghost object has been removed by performing image processing on a removal target area based on motion of an area having a predetermined brightness or higher and motion of the electronic device.

[0138] Additionally, ghost objects can be removed from videos as well, since the electronic device tracks the area to be removed based on the motion of the electronic device.

[0139] According to an exemplary embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., electronic device A) according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, and does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0140] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0141] Furthermore, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a computer-readable recording medium or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0142] Meanwhile, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by at least one processor of a specific device, cause the specific device to perform processing operations of the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0143] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0144] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, camera; display; sensor; and Controlling the display to display a plurality of first live view images acquired through the camera; Identifying a first area and a second area each having a preset brightness or higher in the plurality of first live view images, By comparing the first motion of the first area and the second motion of the second area with the third motion of the electronic device obtained through the sensor, one of the first area and the second area is identified as a removal target area, Processing an area corresponding to the removal target area in a plurality of second live view images acquired after the plurality of first live view images through the camera, An electronic device comprising at least one processor for displaying the plurality of second live view images processed through the display.

2. In paragraph 1, At least one processor, An electronic device that identifies, among the first and second regions, a region in which the direction of motion is the same as the direction of the third motion and the movement distance by time zone is proportional to the movement distance of the third motion as the removal target region.

3. In paragraph 2, The above camera, lens; and Includes a cover glass in contact with the above lens; At least one processor, Identify a first distance to a first object corresponding to the first area and a second distance to a second object corresponding to the second area, An electronic device that identifies the target area for removal based on whether a first distance or a second distance exists between a minimum distance from the lens to the cover glass and a maximum distance from the lens to the cover glass among the first area and the second area.

4. In paragraph 3, One of the first object and the second object, A light source external to the above electronic device, The other object among the first object and the second object, The light emitted from the light source and reflected by the lens reaches the cover glass and corresponds to an area where the light is reflected by the cover glass and reaches the lens. At least one processor, An electronic device that identifies one of the first object and the second object as the removal target area.

5. In paragraph 1, At least one processor, An electronic device that processes the image based on at least one surrounding pixel value of the area corresponding to the removal target area in the plurality of second live view images.

6. In paragraph 5, At least one processor, An electronic device that determines the location of an area corresponding to the removal target area in the plurality of second live view images based on the third motion.

7. In paragraph 6, At least one processor, An electronic device that identifies a third area having the preset brightness in the plurality of second live view images when the determined location is outside the plurality of second live view images.

8. In paragraph 1, At least one processor, An electronic device that identifies one of the first area and the second area as the area to be removed based on the shape of at least one of the first area and the second area.

9. In paragraph 1, The third motion above is, It is 3D motion data, At least one processor, Convert the above third motion into two-dimensional motion data, An electronic device that compares the first motion and the second motion with the converted third motion to identify one of the first area and the second area as the area to be removed.

10. In paragraph 1, User interface; and memory; including more, At least one processor, When a command is received through the user interface, an area corresponding to the removal target area is image-processed in a live view image captured based on the command, An electronic device that stores the processed live view image in the memory.

11. In a method for controlling an electronic device, A step of displaying a plurality of first live view images acquired through a camera included in the electronic device; A step of identifying a first area and a second area, each of which has a preset brightness or higher, in the plurality of first live view images; A step of comparing a first motion of the first region and a second motion of the second region with a third motion of the electronic device to identify one of the first region and the second region as a region to be removed; A step of image processing an area corresponding to the removal target area in a plurality of second live view images acquired after the plurality of first live view images; and A control method comprising: a step of displaying a plurality of second live view images processed as described above; 12. In paragraph 11, The step of identifying the above target area for removal is: A control method for identifying, among the first and second regions, a region in which the direction of motion is the same as the direction of the third motion and the movement distance by time zone is proportional to the movement distance of the third motion as the removal target region.

13. In paragraph 12, The above camera, lens; and Includes a cover glass in contact with the above lens; The step of identifying the above target area for removal is: Identify a first distance to a first object corresponding to the first area and a second distance to a second object corresponding to the second area, A control method for identifying the removal target area based on the existence of a first distance or a second distance between the minimum distance from the lens to the cover glass and the maximum distance from the lens to the cover glass among the first area and the second area.

14. In paragraph 13, One of the first object and the second object, A light source external to the above electronic device, The other of the first object and the second object, The light emitted from the light source and reflected by the lens reaches the cover glass and corresponds to an area where the light is reflected by the cover glass and reaches the lens. The step of identifying the above target area for removal is: A control method for identifying one of the first object and the second object as the removal target area.

15. In paragraph 11, The above image processing step is: A control method for processing an image corresponding to the target area for removal in a plurality of second live view images after the plurality of first live view images based on at least one surrounding pixel value of the area corresponding to the target area for removal.

Images