Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises: a memory for storing at least one instruction; and one or more processors that individually and / or collectively include processing circuits, identify a foreground region and a background region included in an input image on the basis of a depth map corresponding to the input image, and convert a viewpoint on the basis of the foreground region to generate a new viewpoint image. The one or more processors: identify, on the basis of the depth map, side effect prediction information including at least one of whether objects of at most a specific thickness are included in the input image or how densely concentrated the objects are; and generate the new viewpoint image by controlling a viewpoint movement path on the basis of the identified side effect prediction information.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for generating a new viewpoint image and a method for controlling the same.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, display devices, used in a variety of settings, including homes, offices, and public spaces, have been continuously evolving in recent years.

[0003] Novel view synthesis (novel view synthesis) technology generates images from different viewpoints from a two-dimensional image acquired through a monocular camera. This technology can be used to create a video, a sequence of new viewpoint images that follow the movement of a viewpoint trajectory within a single image.

[0004] An electronic device according to the above-described embodiment includes a memory storing at least one command and a processing circuit individually or collectively, and includes one or more processors for identifying a foreground area and a background area included in an input image based on a depth map corresponding to the input image, and converting a viewpoint based on the foreground area to generate a new viewpoint image. The one or more processors identify, based on the depth map, side effect prediction information including at least one of whether an object having a thickness less than a specific thickness or a degree of object density in the input image, and control a viewpoint movement path based on the identified side effect prediction information to generate the new viewpoint image.

[0005] According to one embodiment, the one or more processors, individually or collectively, can generate the new viewpoint image by controlling the viewpoint movement path to be reduced to less than a threshold range when the input image is identified as containing an object of the specific thickness or less based on the depth map or the object density is identified as being greater than a threshold value.

[0006] According to one embodiment, the one or more processors, individually or collectively, if it is identified that the input image includes an object of the specific thickness or less based on the depth map, can control the viewpoint movement path to be reduced in proportion to the ratio of the area occupied by the object of the specific thickness or less in the input image.

[0007] According to one embodiment, the one or more processors can apply an opening operation to the depth map to obtain a depth map to which the opening operation has been applied, and identify an area containing an object having a thickness less than the specific thickness based on difference information between the depth map and the depth map to which the opening operation has been applied.

[0008] According to one embodiment, the one or more processors may apply a window to the depth map to calculate a standard deviation of depth values ​​of pixels excluding a depth boundary area within the window, and perform depth map refinement if the calculated standard deviation is greater than or equal to a threshold value.

[0009] According to one embodiment, the one or more processors can generate the new viewpoint image by controlling the viewpoint movement path to be reduced to less than a threshold range when the object density is identified as being greater than a threshold value based on the depth map.

[0010] According to one embodiment, the one or more processors adjust an occlusion area and a context area based on a boundary complexity representing the object density information, wherein the occlusion area is an area that is not exposed to the foreground area at the current point in time and becomes exposed when the point in time moves, and the context area may be an area adjacent to the occlusion area.

[0011] According to one embodiment, the one or more processors can identify the boundary complexity based on the number of neighboring boundaries included in the context area, and reduce the width of at least one of the occlusion area and the context area when the number of neighboring boundaries is greater than or equal to a threshold number.

[0012] According to one embodiment, the one or more processors can refine the depth map by replacing the depth values ​​of areas corresponding to objects having a specific thickness or less with depth values ​​of surrounding pixels having color information most similar to the target pixels.

[0013] In one embodiment, the one or more processors can refine the depth map by replacing depth values ​​outside an area corresponding to objects having a thickness less than the specific thickness with a median value of depth values ​​of pixels within the window.

[0014] A method for controlling an electronic device according to one embodiment includes a step of identifying a foreground area and a background area included in an input image based on a depth map corresponding to the input image, and a step of generating a new viewpoint image by converting a viewpoint based on the foreground area, wherein the step of generating the new viewpoint image may include a step of identifying side effect prediction information including at least one of whether an object having a thickness less than a specific thickness or a degree of object density in the input image based on the depth map, and a step of generating the new viewpoint image by controlling a viewpoint movement path based on the identified side effect prediction information.

[0015] In one embodiment, a non-transitory computer-readable medium storing computer instructions that, when individually and / or collectively executed by one or more processors of an electronic device, cause the electronic device to perform an operation, the operation includes: a step of identifying a foreground area and a background area included in an input image based on a depth map corresponding to the input image; and a step of converting a viewpoint based on the foreground area to generate a new viewpoint image; wherein the step of generating the new viewpoint image may include a step of identifying, based on the depth map, side effect prediction information including at least one of whether an object having a thickness less than a specific thickness or a degree of object density in the input image; and a step of controlling a viewpoint movement path based on the identified side effect prediction information to generate the new viewpoint image.

[0016] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0017] FIG. 1 is a drawing for explaining a new viewpoint image generation technology according to various embodiments.

[0018] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0019] FIG. 2b is a block diagram specifically showing the configuration of an electronic device according to various embodiments.

[0020] FIG. 3 is a flowchart for explaining a method of controlling an electronic device according to various embodiments.

[0021] FIGS. 4 and 5 are drawings for explaining a control method of an electronic device according to various embodiments.

[0022] FIG. 6 is a drawing for explaining a method for identifying a depth boundary area according to various embodiments.

[0023] FIGS. 7A and 7B are drawings for explaining a depth map refinement method according to various embodiments.

[0024] FIG. 8 illustrates an example of a process for identifying thin objects within a depth map using an opening operation according to various embodiments.

[0025] FIGS. 9A and 9B are drawings for explaining the depth boundary area of ​​a general object and the depth boundary area of ​​a thin object according to various embodiments.

[0026] FIG. 10 is a diagram illustrating examples of occlusion areas and context areas according to various embodiments.

[0027] Figure 11 is a drawing for explaining a side effect that occurs when an occlusion area is excessively set according to an example.

[0028] FIG. 12 is a drawing for explaining a method for identifying an occluded area according to various embodiments.

[0029] Figures 13a and 13b illustrate boundary complexity according to an example.

[0030] FIG. 14 is a drawing for explaining a method for adjusting an occlusion area according to various embodiments.

[0031] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0032] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions or circumstances of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0033] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0034] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0035] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0036] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0037] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0038] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0039] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0040] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0041] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0042] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0043] FIG. 1 is a drawing for explaining a new viewpoint image generation technology according to various embodiments.

[0044] Novel view synthesis (novel view synthesis) technology generates new perspectives from two-dimensional images acquired through a monocular camera. Using this technology, a video can be generated, consisting of a series of new viewpoint images generated by moving the viewpoint trajectory of a single image (https: / / shihmengli.github.io / 3D-Photo-Inpainting / content.sniklaus.com / kenburns / video.mp4).

[0045] In general, a new viewpoint image (10) generated based on depth information has the characteristic that the background object (11) moves less (13) and the foreground object (12) moves more (14), as shown in Fig. 1.

[0046] For example, in the case of foreground-background separation-based technology, the foreground and background are separated based on a depth map, the viewpoint is transformed based on the foreground, and the occlusion area that occurs due to the change in viewpoint is filled in with the background, thereby generating an image viewed from an arbitrary viewpoint. In this case, accurate separation of the foreground and background is required to obtain natural rendering results. However, depth maps estimated using monocular or stereo images are predicted to have continuous values ​​near the boundary between the foreground and background due to the nature of regression technology. Therefore, accurately determining the boundary during foreground and background separation can be a key technology for generating high-quality images.

[0047] Deep learning-based alpha matting techniques can be used to accurately distinguish between foreground and background boundaries. However, since these techniques operate under the assumption that there is only one foreground object in the image, they can cause various side effects as the object becomes more complex. Furthermore, techniques that identify ambiguous depth boundary regions and apply a median filter to determine the boundary can, due to the nature of the median filter, assign background regions if the foreground area within the filter is small, which can cause thin foreground objects to disappear.

[0048] Problems such as inaccurate depth estimation and occlusion filling lead to various side effects, and these problems tend to be more pronounced as the newly generated point's movement path becomes longer. For example, problems such as incorrect hole filling or failure to properly segment thin objects can occur.

[0049] To minimize or reduce potential artifacts during image generation, existing novel viewpoint image generation techniques either use fixed-range viewpoint movement paths for all images or control the viewpoint movement paths within small occlusion regions. However, using fixed viewpoint movement paths cannot avoid artifacts occurring within the movement path. Furthermore, since only images from a limited number of viewpoints are generated, they provide users with limited stereoscopic images. Furthermore, if the viewpoint is moved only to areas with small occlusion regions, the number of viewpoints that can be generated is limited.

[0050] For example, object boundaries are often not accurately segmented at depth estimates from monocular images, resulting in warped or stretched object boundaries. To address this, depth map refinement is performed as a post-processing step. However, existing depth map refinement techniques cannot adequately handle thin objects, potentially resulting in various side effects in the resulting new viewpoint image.

[0051] Accordingly, below, various embodiments are described to avoid or minimize side effects by predicting various side effects that may occur in generated images in advance and controlling the movement path of a newly generated point in time.

[0052] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0053] According to FIG. 2a, an electronic device (100) includes a memory (110) and one or more processors (120) (e.g., including a processing circuit).

[0054] The memory (110) can store data required for various embodiments. Depending on the purpose of data storage, the memory (110) may be implemented in the form of memory embedded in the electronic device (100) or may be implemented in the form of memory that can be attached or detached from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0055] As an example, the memory (110) may store a computer program including at least one instruction or instructions for controlling the electronic device (100).

[0056] According to another example, the memory (110) may store an image received from an external device (e.g., a source device), an external storage medium (e.g., USB), an external server (e.g., web hard), etc., i.e., an input image. Alternatively, the memory (110) may store an image acquired through a camera (not shown) provided in the electronic device (100). Here, the image may be a 2D video, but is not limited thereto.

[0057] As another example, the memory (110) may store various information required for image quality processing, such as information, algorithms, image quality parameters, etc. for performing at least one of Noise Reduction, Detail Enhancement, Tone Mapping, Contrast Enhancement, Color Enhancement, or Frame Rate Conversion. In addition, the memory (110) may also store an intermediate image generated by image processing and an image generated based on depth information.

[0058] According to one embodiment, the memory (110) may be implemented as a single memory that stores data generated from various operations according to the present disclosure. However, according to another embodiment, the memory (110) may be implemented to include multiple memories that each store different types of data or each store data generated at different stages.

[0059] In the above-described embodiment, it has been described that various data are stored in the external memory (110) of the processor (120), but at least some of the above-described data may be stored in the internal memory of the processor (120) according to an implementation example of at least one of the electronic device (100) or the processor (120).

[0060] The one or more processors (120) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits comprising one or more processors, one or more of which may be configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, where "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms may encompass, for example, without limitation, a single processor performing the functions. Some of the recited functions may be performed by different processor(s) and other functions may be performed by different processor(s), and further, a single processor may perform all of the recited functions. Additionally, the one or more processors may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. The one or more processors (120) generally control the operation of the electronic device (100) by executing program instructions to achieve or perform various functions. Specifically, one or more processors (120) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (120) may be electrically connected to the display (130) and the memory (110) to control the overall operation of the electronic device (100). One or more processors (120) may be configured as one or more processors.

[0061] One or more processors (120) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (110).

[0062] The one or more processors (120) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (120) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (120) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform methods according to various embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0063] When a method according to various embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to various embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0064] One or more processors (120) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (120) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to various embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to various embodiments of the present disclosure.

[0065] When a method according to various embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to various embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0066] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (120) will be referred to as a processor (120).

[0067] The processor (120) can obtain depth information from an input image. Here, the input image may include a still image, a plurality of consecutive still images (or frames), or a video. For example, the input image may be a 2D image. Here, the depth information may be in the form of a depth map. A depth map refers to a table containing depth information for each region of the image. The region may be divided into pixel units or may be defined as a preset region larger than a pixel unit. For example, the depth map may be in the form of a grayscale value from 0 to 255, with 127 or 128 as a reference value, that is, 0 (or focal plane), and a value smaller than 127 or 128 is represented as a - value, and a larger value is represented as a + value. The reference value of the focal plane may be arbitrarily selected from 0 to 255. Here, a - value means subsidence, and a + value means protrusion. However, this is just an example, and a depth map can express depth with different values ​​depending on different criteria.

[0068] According to an example, the processor (120) may process an input image and then acquire depth information based on the processed image. Here, the image processing may be digital image processing including at least one of image enhancement, image restoration, image transformation, image analysis, image understanding, image compression, image decoding, or scaling.

[0069] For example, various preprocessing may be performed before acquiring depth information for an input image. However, for convenience of explanation, the input image and the preprocessed image will not be distinguished and will be referred to as an input image.

[0070] The processor (120) may store depth information corresponding to the input image in the memory (110). For example, when a first image frame and a second image frame are sequentially input, the processor (120) may sequentially acquire first depth information and second depth information corresponding to the first image frame and the second image frame while applying preprocessing and / or postprocessing to the first image frame and the second image frame, and store the acquired first depth information and second depth information in the memory (110). Here, the first image frame and the second image frame may be 2D monocular image frames.

[0071] For example, the processor (120) can obtain depth information of the first image frame and the second image frame based on various image processing methods, such as algorithms, formulas, artificial intelligence models, etc.

[0072] FIG. 2b is a block diagram specifically showing the configuration of an electronic device according to various embodiments.

[0073] According to FIG. 2B, the electronic device (100') may include a memory (110), one or more processors (120), a display (130), a camera (140), a user interface (150) (e.g., including a circuit), a communication interface (160) (e.g., including a communication circuit), and a speaker (170). A detailed description of the configurations illustrated in FIG. 2B that overlap with the configuration illustrated in FIG. 2A may not be repeated.

[0074] The display (130) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (130) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to an example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (130) so as to be implemented so as to detect various types of touch inputs. For example, the display (130) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (130) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0075] The camera (140) can be turned on and take pictures according to a preset event. The camera (140) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera (140) can include at least one of a general (or basic) camera and an ultra-wide-angle camera.

[0076] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0077] It goes without saying that the communication interface (160) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. In one example, the communication interface (160) can communicate with another electronic device, an external server, and / or a remote control device.

[0078] The speaker (170) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (120) may control the speaker (170) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0079] In addition, the electronic device (100') may include sensors, microphones, etc., depending on the implementation example.

[0080] Sensors may include various types of sensors, such as touch sensors, proximity sensors, acceleration sensors (or gravity sensors), geomagnetic sensors, gyro sensors, pressure sensors, position sensors, distance sensors, light sensors, etc.

[0081] A microphone is a device configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input via an external device through a communication interface (160).

[0082] FIG. 3 is a flowchart illustrating an exemplary method of an electronic device according to various embodiments.

[0083] According to FIG. 3, the electronic device (100) can obtain a depth map corresponding to the input image (S310). The depth map (or depth information) represents three-dimensional distance information of an object existing in the image, and can be assigned to each pixel of the image. For example, an 8-bit depth can have a grayscale value from 0 to 255. For example, when expressed based on black / white, black (low value) can represent a place far from the viewer, and white (high value) can represent a place close to the viewer. According to an example, a depth map can be obtained from a 2D input image based on an existing depth estimation algorithm, a mathematical formula, a learned artificial intelligence model, etc. For example, the artificial intelligence model can be implemented as a neural network including a plurality of neural network layers. Artificial intelligence models can be implemented using, but are not limited to, Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Bidirectional Recurrent Deep Neural Networks (BRDNN), or Deep Q-Networks.

[0084] The electronic device (100) can identify the foreground area and background area included in the input image based on the depth map (S320).

[0085] The electronic device (100) can identify side effect prediction information including at least one of whether an object with a preset (e.g., specific) thickness or less is included in the input image based on the depth map (S330). For example, when generating a new viewpoint image, if the input image includes a thin object, if multiple objects are densely packed, etc., various side effects described in FIG. 1 may occur, and thus, these can be defined as side effect prediction information and the side effect prediction information can be identified in the input image.

[0086] The electronic device (100) can control the viewpoint movement path based on the identified side effect prediction information, transform the viewpoint based on the foreground area, and generate a new viewpoint image (S340). For example, the viewpoint movement path can be controlled based on the level of the side effect prediction information. This is to minimize or reduce the side effect by moving the viewpoint within a relatively limited range when the side effect is predicted to be severe.

[0087] According to one example, if the electronic device (100) identifies that an object with a thickness less than or equal to the preset thickness is included in the input image based on the depth map or if the object density is identified as being greater than or equal to a threshold value, the electronic device (100) can generate a new viewpoint image by controlling the viewpoint movement path to be reduced to less than the threshold range.

[0088] According to one example, if the electronic device (100) identifies that an object with a preset thickness or less is included in the input image based on the depth map, the electronic device can control the viewpoint movement path to be reduced in proportion to the ratio of the area occupied by the object with a preset thickness or less in the input image.

[0089] According to an example, the electronic device (100) can obtain a depth map to which the opening operation has been applied by applying an opening operation to a depth map, and identify an area containing an object having a thickness less than a preset thickness based on difference information between the depth map and the depth map to which the opening operation has been applied.

[0090] For example, the electronic device (100) can refine the depth map by replacing the depth value of an area corresponding to an object having a preset thickness or less with the depth value of a surrounding pixel having color information most similar to the target pixel.

[0091] According to one example, if the electronic device (100) identifies that the object density is greater than a threshold value based on the depth map, it can generate a new viewpoint image by controlling the viewpoint movement path to be reduced to less than the threshold range.

[0092] According to one example, the electronic device (100) can refine the depth map by replacing the depth value outside the area corresponding to an object having a preset thickness or less with the central value of the depth values ​​of pixels within the window.

[0093] According to an example, the electronic device (100) can apply a preset window to a depth map, calculate a standard deviation of depth values ​​of pixels excluding a depth boundary area within the window, and perform depth map refinement if the calculated standard deviation is greater than a threshold value.

[0094] For example, the electronic device (100) can adjust occlusion areas and context areas based on boundary complexity representing object density information. For example, the occlusion area may be an area that is not exposed to the foreground area at the current viewpoint but becomes exposed when the viewpoint moves. For example, the context area may be an area adjacent to the occlusion area.

[0095] In one example, the electronic device (100) may identify boundary complexity based on the number of neighboring boundaries included within a context area. If the number of neighboring boundaries is greater than or equal to a threshold number, the electronic device (100) may reduce the width of at least one of the occlusion area and the context area.

[0096] FIGS. 4 and 5 are drawings for explaining a control method of an electronic device according to various embodiments.

[0097] According to FIG. 4, the processor (120) can estimate a depth map from an input image (410).

[0098] Next, the processor (120) may refine the depth map based on the side effect prediction information (460) (420). For example, as illustrated in FIG. 5, the processor (120) may identify whether the input image includes a foreground object having a preset thickness or less based on the depth map (S510). If the processor (120) determines that the input image includes a foreground object having a preset thickness or less, the processor (120) may refine the depth map by replacing the depth value of the area corresponding to the object having the preset thickness or less with the depth value of the surrounding pixel having the color information most similar to the target pixel (S520). For example, if the processor (120) predicts a side effect due to the presence of a thin object in the input image, the processor (120) may perform depth map refinement based on color information comparison. In addition, the processor (120) may refine the depth map by replacing the depth value of an area other than the area corresponding to the object having the preset thickness or less with the median value of the depth values ​​of the pixels within the window (S530).

[0099] For example, the processor (120) can refine the boundary area of ​​the predicted depth map object to precisely separate the foreground area and the background area. To this end, after identifying the boundary area of ​​the object within the depth map, the size of the object can be identified through a morphological operation. For example, the processor (120) can identify the object area included in the input image through at least one of object recognition, object detection, object tracking, or image segmentation. For example, the processor (120) can identify the object area by using a technique such as semantic segmentation that classifies and extracts objects included in the input image by type, instance segmentation that recognizes objects by classifying them by object even if they are of the same type, and a bounding box in the shape of a rectangle that includes the detected object when detecting an object included in the image.

[0100] For example, the processor (120) may refine the depth map by selectively applying median filter-based refinement and color information comparison-based refinement based on the thickness (or size) of the identified object region. For example, the same process may be repeated several times (e.g., five times) to obtain more accurate refinement results.

[0101] For example, the processor (120) can use the difference in depth values ​​from neighboring pixels to determine the boundary area between an object and a background or two objects in a depth map. For example, the processor (120) can define a pixel whose depth difference from a center pixel among the four neighboring pixels above, below, left, and right is greater than a certain value as a depth boundary area. For example, as illustrated in FIG. 6, in the depth map (610), an area (611) including a pixel whose depth difference from a center pixel among the four neighboring pixels above, below, left, and right is greater than a certain value can be identified as a depth boundary area (620).

[0102] For example, the processor (120) may perform depth map refinement on the identified depth boundary region several times (e.g., five times). The processor (120) may adaptively utilize either a median filter-based refinement or a generation information comparison-based refinement method depending on the presence or absence of thin objects during refinement. Each refinement method is performed within a window of a preset size, and refinement may not be performed if the depth boundary region is not included within the window.

[0103] FIGS. 7A and 7B are drawings for explaining a depth map refinement method according to various embodiments.

[0104] Figure 7a illustrates a median filter-based refinement process according to an example.

[0105] Median filter-based refinement assigns the median depth value of pixels within a window as the output value. During this process, pixels within the depth boundary region may be excluded. Median filter-based refinement can obtain stable depth boundaries and is effective in mitigating the depth value inversion phenomenon that occurs at the depth boundary. However, for thin objects, since most of the object is within the depth boundary region, applying the median filter can result in the output being the background depth value, causing the thin object to disappear.

[0106] Figure 7b illustrates a color information comparison-based refinement process according to an example.

[0107] A color information comparison-based refinement method assigns the depth value of the pixel within a window with the most similar color value to the target pixel as the output value. During this process, pixels within the depth boundary area may be excluded from the comparison.

[0108] For example, the degree of color value similarity can be calculated as the Euclidean distance between the target pixel and the comparison pixel. Color information comparison-based refinement methods enable precise depth map refinement even for thin objects. However, if similar pixels do not exist within the window during the refinement process, unstable depth boundaries can be generated.

[0109] Accordingly, the processor (120) can adaptively select a refinement method based on the presence or absence of a thin object in the input depth map. For example, if the depth boundary area is determined to be a thin object, refinement based on color information comparison can be performed, and if not, refinement based on a median filter can be performed. The processor (120) can use an opening operation, which is a morphological operation according to an example, to determine a thin object. The opening operation is an operation that applies a dilation operation following an erosion operation, and the opening, erosion, and dilation operations can be expressed as in the following mathematical expression 1.

[0110]

[0111] The size of the structuring element S used for the opening operation may be the same as or similar to the window size for refinement. The processor (120) may determine that the object is thin if the difference between the result obtained by applying the opening operation and the input depth map is greater than a threshold value.

[0112] FIG. 8 illustrates an example of a process for identifying thin objects within a depth map using an opening operation according to various embodiments.

[0113] For example, the processor (120) may apply an opening operation to the depth map (810) as illustrated in FIG. 8 to obtain an image (820) to which the opening operation has been applied. Subsequently, the processor (120) may identify a thin object region based on an image (830) showing a depth difference before and after the application of the opening operation. Thereafter, the processor (120) may perform color information comparison-based refinement on the region including the thin object in the depth map (840) and median filter-based refinement on the remaining region.

[0114] FIGS. 9A and 9B are drawings for explaining the depth boundary area of ​​a general object and the depth boundary area of ​​a thin object according to various embodiments.

[0115] As an example, FIG. 9a shows the depth boundary region of a general object, and FIG. 9b shows the depth boundary region of a thin object.

[0116] As illustrated in Fig. 9b, when two or more depth boundaries are adjacent, such as the depth boundary area of ​​a thin object, there is a possibility that the entire area between the two depth boundaries may be set as the depth boundary area, and thus the entire area may be refined with background depth values. Accordingly, during the first few refinement rounds (e.g., rounds 1 and 2) of several refinement rounds (e.g., 5 rounds), the standard deviation of the depth values ​​of pixels excluding the depth boundary area within the window may be calculated, and depth map refinement may be performed only if the standard deviation is greater than a threshold value. For example, around a typical depth boundary area, the depth of the foreground object and the depth of the background exist together, so the standard deviation of the depth values ​​within the kernel is large. On the other hand, if most of the foreground object is included in the depth boundary area, the standard deviation is small. Therefore, if the standard deviation is small, the presence of a thin object can be assumed, and side effects that may occur in a median filter-based refinement method can be avoided.

[0117] Referring to FIG. 4, the processor (120) can identify occluded areas in the depth map (430) based on side effect prediction information (460). For example, the processor (120) can predict side effects that may occur depending on the density of objects and identify occluded areas based on the predicted side effects. For example, the processor (120) can adjust the sizes of the occluded area and the context area based on the density of objects.

[0118] For example, the processor (120) may set an occlusion area and a context area based on the degree of object density, and adjust the size of each area based on the boundary complexity indicating the degree of object density.

[0119] An occluded area may refer to an area that is currently hidden from the foreground but will be revealed when the viewpoint moves. For example, the processor (120) may set an occluded area based on the boundary between objects extracted from the depth map.

[0120] The context area can be used as a hint to fill in the occluded area with the area adjacent to the occluded area.

[0121] FIG. 10 is a diagram illustrating examples of occlusion areas and context areas according to various embodiments.

[0122] According to FIG. 10, the sizes of the occlusion area (1011) and the context area (1012) can be set according to the boundary complexity of the first boundary in the depth map (1010). In addition, the sizes of the occlusion area (1021) and the context area (1022) can be set according to the boundary complexity of the second boundary in the depth map (1020). In addition, the sizes of the occlusion area (1031) and the context area (1032) can be set according to the boundary complexity of the third boundary in the depth map (1030).

[0123] For example, when multiple objects are mixed around a boundary, the boundaries between objects become dense. This may increase the probability of unstable results when filling in the occlusion area (440) for generating a new viewpoint image (450) due to excessive occlusion areas or inclusion of unnecessary information within the context area.

[0124] Figure 11 is a drawing for explaining a side effect that occurs when an occlusion area is excessively set according to an example.

[0125] For example, if a target boundary is close to a neighboring boundary, the occlusion region of the target boundary may encroach on the occlusion region of the neighboring boundary, potentially producing undesirable results when generating images from other viewpoints. This can be resolved by reducing the size of the occlusion region of the target boundary. Accordingly, boundary complexity, which represents the density of objects, can be used to adjust the sizes of the occlusion region and context region, thereby minimizing or reducing the side effects.

[0126] FIG. 12 is a drawing for explaining a method for identifying an occluded area according to various embodiments.

[0127] Referring to FIG. 12, the electronic device (100) can identify an occlusion area and a context area based on a depth map (1210) and identify a boundary within the context area (1220). The electronic device (100) can calculate the boundary complexity of the identified boundary (1230) and adjust the occlusion area and the context area based on the boundary complexity (1240).

[0128] For example, boundary complexity can be defined as the number of boundaries surrounding a target boundary. For example, occlusion regions and corresponding context regions within a defined range based on the target boundary can be initialized, and the number of boundaries existing within the initial context region can be defined as boundary complexity.

[0129] Figures 13a and 13b illustrate boundary complexity according to an example.

[0130] In one example, the electronic device (100) can identify boundary complexity based on the number of neighboring boundaries included within a context area.

[0131] For example, as illustrated in FIG. 13A, a context area can be identified (1320) based on an image (1310) representing a target boundary, and two boundaries within the context area can be identified based on an image (1330) representing a boundary within the context area. Accordingly, the boundary complexity can be identified as 2.

[0132] For example, as illustrated in FIG. 13b, a context area can be identified (1340) based on a target boundary (1340), and five boundaries within the context area can be identified (1360) based on an image (1350) representing the boundaries within the context area. Accordingly, the boundary complexity can be identified as 5.

[0133] For example, if the boundary complexity is above a certain level (e.g., a certain number), the widths of the occlusion region and the context region can be reduced to prevent excessive allocation of the occlusion region or inclusion of unnecessary information in the context region. For example, as illustrated in FIG. 14, the electronic device (100) can identify a boundary (1420) within the context region based on the context region (1410). In the example illustrated in FIG. 14, if the complexity of the boundary (1420) within the context region is as high as 5, at least one of the size and position of the identified occlusion region (1430) can be adjusted to identify an adjusted occlusion region (1440).

[0134] However, the number of boundary complexities and the adjustment of the location / size of the occlusion area shown in FIGS. 13a, 13b, and 14 are only examples and are not limited thereto.

[0135] As described above, according to various embodiments of the present disclosure, a new viewpoint image can be generated by reducing the viewpoint movement range in proportion to the area ratio of thin objects and / or boundary complexity relative to the entire image based on an estimated depth map. Furthermore, a new viewpoint image can be generated by applying different depth map refinement methods to the thin object area and the remaining area. Accordingly, side effects that may occur based on thin objects and / or the density of objects can be minimized and / or reduced.

[0136] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing display device or an existing electronic device.

[0137] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0138] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0139] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0140] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0141] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure. It will also be understood that any embodiment(s) described in the present disclosure may be used in conjunction with any other embodiment(s) described in the present disclosure.

Claims

1. In electronic devices, a memory storing at least one instruction; and Identifying the foreground area and background area included in the input image based on the depth map corresponding to the input image, A new viewpoint image is generated by transforming the viewpoint based on the foreground area, and includes one or more processors, individually and / or collectively including processing circuits; The one or more processors, individually and / or collectively, Based on the depth map, the side effect prediction information including at least one of whether the input image contains an object with a thickness less than a specific thickness or the degree of object density is identified, An electronic device that generates a new viewpoint image by controlling a viewpoint movement path based on the identified side effect prediction information.

2. In paragraph 1, The one or more processors, individually and / or collectively, An electronic device that generates a new viewpoint image by controlling the viewpoint movement path to be reduced to less than a threshold range when the input image is identified as containing an object less than or equal to a specific thickness based on the depth map or when the object density is identified as being greater than a threshold value.

3. In paragraph 2, The one or more processors, individually and / or collectively, An electronic device that, if it is identified that an object of a specific thickness or less is included in the input image based on the depth map, controls the viewpoint movement path to be reduced in proportion to the ratio of the area occupied by the object of a specific thickness or less in the input image.

4. In paragraph 2, The one or more processors, individually and / or collectively, apply an opening operation to the depth map to obtain a depth map to which the opening operation has been applied; An electronic device that identifies an area containing an object less than a specific thickness based on difference information between the depth map and the depth map to which the opening operation is applied.

5. In paragraph 2, The one or more processors, individually and / or collectively, apply a preset window to the depth map to calculate a standard deviation of depth values ​​of pixels excluding a depth boundary area within the specific window, An electronic device that performs depth map refinement when the above-described standard deviation is greater than a threshold value.

6. In paragraph 2, An electronic device in which the one or more processors, individually and / or collectively, generate the new viewpoint image by controlling the viewpoint movement path to be reduced to less than a threshold range when the object density is identified to be greater than a threshold value based on the depth map.

7. In paragraph 2, The one or more processors, individually and / or collectively, adjust the occlusion region and the context region based on the boundary complexity representing the object density information, The above occluded area is an area that is not exposed to the foreground area at the current point in time and becomes exposed when the point in time moves. The above context area is, An electronic device, which is adjacent to the above occluded area.

8. In paragraph 7, The one or more processors, individually and / or collectively, Identify the boundary complexity based on the number of neighboring boundaries included within the context area, An electronic device that reduces the width of at least one of the occlusion area and the context area when the number of the neighboring boundaries is greater than or equal to a threshold number.

9. In paragraph 1, An electronic device wherein said one or more processors, individually and / or collectively, refine the depth map by replacing depth values ​​of areas corresponding to objects less than the specific thickness with depth values ​​of surrounding pixels having color information most similar to the target pixel.

10. In paragraph 1, An electronic device wherein said one or more processors, individually and / or collectively, refine said depth map by replacing depth values ​​outside an area corresponding to objects below said particular thickness with a median value of depth values ​​of pixels within a particular window.

11. In a method for controlling an electronic device, A step of identifying a foreground area and a background area included in an input image based on a depth map corresponding to the input image; and A step of generating a new viewpoint image by transforming the viewpoint based on the foreground area; The step of generating the above new viewpoint image is: A step of identifying side effect prediction information including at least one of whether the input image contains an object of a specific thickness or less or the degree of object density based on the depth map; and A control method, comprising: a step of generating a new viewpoint image by controlling a viewpoint movement path based on the identified side effect prediction information.

12. In paragraph 11, The step of generating the above new viewpoint image is: A control method, comprising: a step of generating a new viewpoint image by controlling the viewpoint movement path to be reduced to less than a threshold range when the input image is identified as containing an object less than the specific thickness based on the depth map or when the object density is identified as being greater than a threshold value.

13. In paragraph 12, The step of generating the above new viewpoint image is: A control method, comprising: a step of reducing a viewpoint movement path in proportion to a ratio of an area occupied by an object less than or equal to a specific thickness in the input image, if the input image is identified as containing an object less than or equal to a specific thickness based on the depth map; 14. In paragraph 12, The step of generating the above new viewpoint image is: A step of applying an opening operation to the depth map to obtain a depth map to which the opening operation has been applied; and A control method, comprising: a step of identifying an area containing an object less than a specific thickness based on difference information between the depth map and the depth map to which the opening operation is applied.

15. A non-transitory computer-readable medium storing computer instructions that, when individually and / or collectively executed by one or more processors of an electronic device, cause the electronic device to perform an operation, The above actions are, A step of identifying a foreground area and a background area included in an input image based on a depth map corresponding to the input image; and A step of generating a new viewpoint image by transforming the viewpoint based on the foreground area; The step of generating the above new viewpoint image is: A step of identifying side effect prediction information including at least one of whether the input image contains an object less than a specific thickness or the degree of object density based on the depth map; and A non-transitory computer-readable medium, comprising: a step of generating a new time point image by controlling a time point movement path based on the identified side effect prediction information.

Images