Image processing device and operation method thereof

Abstract

An image processing device and an operation method thereof are provided. The image processing device comprises: at least one processor including a processing circuit; and a memory including one or more storage media for storing one or more instructions, wherein the at least one processor individually or collectively executes the one or more instructions so that the image processing device acquires an input depth map from a two-dimensional input image, the input depth map including a boundary area and a non-boundary area of an object, and, in order to acquire a first filtered depth map, first filtering is performed by applying different weights to the boundary area and the non-boundary area for a first frame and a previous frame of the input depth map, and, on the basis of the first filtered depth map and the two-dimensional input image, a three-dimensional image is generated.

Description

Image processing device and its operating method

[0001] The present disclosure relates to an image processing device and an operating method thereof. More specifically, the present disclosure relates to an image processing device and an operating method thereof for acquiring depth information from a two-dimensional image and for acquiring a three-dimensional image.

[0002] 3D imaging is a technology that adds depth information to 2D images to create more realistic images. By providing physical elements, it stimulates the human visual senses in the same way as real objects, allowing them to be perceived as three-dimensional. In 3D images, distance information (or depth information) to a subject (or object) is added to 2D images, and the depth information can be stored in the form of a depth map consisting of depth values ​​for each pixel or block.

[0003] To generate 3D images from 2D images, image-based depth estimation techniques are being studied. Image-based depth estimation techniques analyze 2D images, measure the distance between the camera and the subject, and generate a depth map based on the measured distance. Recent advances in deep learning have made it possible to measure the distance from a single camera image of an object and generate a depth map based on deep learning. This can be referred to as single image depth estimation.

[0004] One way to express a three-dimensional (or 3D) image using a depth map is Depth Image-Based Rendering (DIBR). Depth Image-Based Rendering is a technology that receives a color image and a depth image as input and generates multiple color images with different viewpoints (e.g., left eye and right eye).

[0005] Meanwhile, while depth estimation using deep learning has the advantage of being able to generate depth maps using only images captured by a single camera, it offers lower precision compared to distance measurement methods using sensors such as 3D cameras or LiDAR (Light Detection And Ranging) sensors. This limits the ability to capture subtle depth changes in objects within the image. Consequently, if a 3D image is generated using a depth map that does not reflect the depth changes of an object, the 3D image's three-dimensionality may be reduced.

[0006] In one embodiment of the present disclosure, an image processing device may be provided. The image processing device may include a memory including at least one processor and one or more storage media storing one or more commands. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to obtain an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary region and a non-boundary region of an object. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to perform a first filtering by applying different weights to the boundary region and the non-boundary region for a first frame and a previous frame of the input depth map, in order to obtain a first filtered depth map. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to generate a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

[0007] In one embodiment of the present disclosure, an operating method of an image processing device is provided, including the steps of: obtaining an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary region and a non-boundary region of an object; performing first filtering by applying different weights to the boundary region and the non-boundary region for a first frame and a previous frame of the input depth map to obtain a first filtered depth map; and generating a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

[0008] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing the method on a computer is provided.

[0009] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a drawing for explaining an operation of an image processing device according to one embodiment of the present disclosure to generate a three-dimensional image from a two-dimensional image.

[0011] FIG. 2 is a diagram illustrating a two-dimensional image, an input depth map corresponding to the two-dimensional image, and a filtered depth map according to one embodiment of the present disclosure.

[0012] FIG. 3 is a block diagram of a configuration of an image processing device according to one embodiment of the present disclosure for generating a three-dimensional image using a depth map of a two-dimensional image.

[0013] FIG. 4 is a detailed block diagram of a first filtering module of an image processing device according to one embodiment of the present disclosure.

[0014] FIG. 5 is an example of a depth map including a moving object in neighboring frames according to one embodiment of the present disclosure.

[0015] FIG. 6 is an example of depth information acquired by an image processing device through a boundary area detection module according to one embodiment of the present disclosure.

[0016] FIG. 7 is a graph showing weights used for first filtering obtained through a weight generation module by an image processing device according to one embodiment of the present disclosure.

[0017] Figure 8a is an example of a depth map with global IIR filtering applied.

[0018] Figure 8b is an example of a depth map with local IIR filtering applied.

[0019] FIG. 9 is a diagram for explaining an operation of an image processing device according to one embodiment of the present disclosure to obtain an output depth map from an input depth map through a first filtering module.

[0020] FIG. 10 is a detailed block diagram of a second filtering module of an image processing device according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram for explaining a blur effect applied to a foreground area and a background area of ​​depth maps used in a maximum value calculation module according to one embodiment of the present disclosure.

[0022] FIG. 12 is a graph showing depth values ​​for each region of depth maps used in a maximum value calculation module according to one embodiment of the present disclosure.

[0023] FIG. 13 is a detailed block diagram of a second filtering module of an image processing device according to one embodiment of the present disclosure.

[0024] FIG. 14 is a diagram for explaining an operation of an image processing device according to one embodiment of the present disclosure to obtain an output depth map from an input depth map through a second filtering module.

[0025] FIG. 15 is a diagram for explaining a depth map blur processing module according to one embodiment of the present disclosure.

[0026] FIG. 16 is a flowchart illustrating a method for generating a three-dimensional image by performing first filtering on a depth map using an image processing device according to one embodiment of the present disclosure.

[0027] FIG. 17 is a flowchart illustrating a method for generating a three-dimensional image by performing second filtering on a depth map using an image processing device according to one embodiment of the present disclosure.

[0028] FIG. 18 is a flowchart for explaining an operation method of an image processing device according to one embodiment of the present disclosure.

[0029] FIG. 19 is a diagram for explaining an operation of an image processing device according to one embodiment of the present disclosure to filter an input depth map through a first filtering module and a second filtering module.

[0030] FIG. 20 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0031] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0032] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0033] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0034] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0035] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0036] As used herein, and particularly in the claims, the terms "above" and "above" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0037] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0038] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0039] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0040] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0041] In this disclosure, a "processor" may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0042] In this disclosure, the term "user" refers to a person who uses a display device, and may include a consumer, evaluator, viewer, administrator, or installer. Furthermore, the term "manufacturer" or "provider" in this specification may refer to a manufacturer that manufactures the display device and / or components included in the display device.

[0043] In the present disclosure, an 'image' may include a still image, a graphic, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0044] In this disclosure, "neural network" is a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0045] The present disclosure will be described in detail with reference to the attached drawings below.

[0046] FIG. 1 is a drawing for explaining an operation of an image processing device according to one embodiment of the present disclosure to generate a three-dimensional image from a two-dimensional image.

[0047] Referring to FIG. 1, an image processing device (100) according to one embodiment of the present disclosure may be a device that converts a two-dimensional image into a three-dimensional image (or stereoscopic image).

[0048] An image processing device (100) according to one embodiment of the present disclosure may be a device that receives a two-dimensional image signal received from various image input sources and converts the two-dimensional image signal into a three-dimensional image before displaying it. Here, the image processing device (100) may be implemented in a form that is mounted on a display device such as a TV or a monitor, or may be implemented as a separate device such as a set-top box and implemented in a form that is linked to the display device. When the image processing device (100) is implemented in a form that is mounted on a display device, the image processing device (100) may display the converted three-dimensional image in a stereoscopic or auto-stereoscopic manner.

[0049] An image processing device (100) according to one embodiment of the present disclosure may include a depth map generation module (101) and a depth image-based rendering module (102). However, not all of the illustrated components are essential components. The image processing device (100) may be implemented with more components than the illustrated components, or may be implemented with fewer components. Specific components of the image processing device (100) will be described in detail with reference to FIG. 20. The depth map generation module (101) and the depth image-based rendering module (102) may be included in a memory of the image processing device (100) and may be implemented as software such as commands, algorithms, data structures, or program codes executed by a processor of the image processing device (100).

[0050] An image processing device (100) according to one embodiment of the present disclosure can generate a depth map (20) from a two-dimensional image (10) through a depth map generation module (101). The image processing device (100) can analyze the two-dimensional image (10), measure the distance between an imaging device and a subject (or, object), and generate a depth map (20) based on the measured distance. The image processing device (100) can estimate a depth value for each pixel existing in each frame of an input two-dimensional image (10). The depth map (20) can include a depth value between a subject located in 3D space and an imaging device that captured the subject. The depth map (20) can be composed of a depth value for each pixel existing in each frame of the two-dimensional image (10). Each frame of the depth map (20) can correspond to each frame of the two-dimensional image (10). A depth map (20) can be expressed in the form of a depth image that represents depth values ​​for an object in units of black and white. For example, the depth map (20) can express objects at a close distance in dark colors and objects at a far distance in bright colors.

[0051] A depth map generation module (101) of an image processing device (100) according to one embodiment of the present disclosure may include a depth map estimation model implemented by deep learning, and the depth map estimation model may measure a distance from an image of a subject captured by a single camera and generate a depth map (20).

[0052] An image processing device (100) according to one embodiment of the present disclosure can receive a two-dimensional image (10) and a depth map (20) through a depth image-based rendering module (102) and generate a plurality of three-dimensional images having different viewpoints. For example, the plurality of three-dimensional images can include a three-dimensional image (30) for the left eye and a three-dimensional image (40) for the right eye. The image processing device (100) can generate a virtual viewpoint image by projecting or warping an image of one viewpoint onto an image of another viewpoint using the two-dimensional image (10) and the depth map (20).

[0053] An image processing device (100) according to one embodiment of the present disclosure can generate a 3D image (50) in which the left and right eyes are combined (i.e., a binocular 3D image) based on a 3D image (30) for the left eye and a 3D image (40) for the right eye. For example, the 3D image (50) has been exemplified as having a side-by-side format in which the 3D image (30) for the left eye and the 3D image (40) for the right eye are arranged side by side in one frame, but is not limited thereto. For example, in a 3D image (50) of a side-by-side format, the 3D image (30) for the left eye and the 3D image (40) for the right eye, which have an original resolution of 1920x1080, can be reduced to 960x1080 and arranged left and right, respectively.

[0054] As described above, the image processing device (100) according to one embodiment of the present disclosure can use the depth map (20) to generate a three-dimensional image (e.g., 30, 40, 50). The depth map (20) can include a moving object, and in this case, the less afterimages there are at the boundary of the moving object existing in the depth map (20), the more the three-dimensional effect of the three-dimensional image can be improved. In addition, the more accurately the subtle depth changes inside the object included in the depth map (20) are reflected, the more the three-dimensional effect of the three-dimensional image can be improved.

[0055] An image processing device (100) according to one embodiment of the present disclosure can perform filtering on a depth map to improve the three-dimensional effect of a three-dimensional image. For example, the image processing device (100) can generate an input depth map from a two-dimensional image (10) and perform filtering on the input depth map. A filtered depth map according to one embodiment of the present disclosure can have less afterimages at object boundaries due to object movement, and accurately reflect changes in depth within the object.

[0056] Below, an operation of an image processing device (100) performing filtering on an input depth map to generate a depth map to improve the three-dimensional effect of a three-dimensional image is described.

[0057] FIG. 2 is a diagram for explaining a two-dimensional image, a depth map corresponding to the two-dimensional image, and a filtered depth map according to one embodiment of the present disclosure.

[0058] Referring to 201 of FIG. 2, a two-dimensional image (210), a comparison depth map (220), and a first filtered depth map (230) are described.

[0059] A two-dimensional image (210) may include an object with a lot of movement. If the object has a lot of movement, an input depth map generated from the two-dimensional image (210) may have inter-frame shaking. For example, if an input depth map is generated from the two-dimensional image (210) using a depth map estimation model, the input depth map may have non-uniform depth values ​​for each frame depending on the accuracy of the depth map estimation model. In this case, inter-frame shaking or frame flickering may occur in the input depth map. To reduce inter-frame shaking, inter-frame smoothing filtering may be applied. For example, inter-frame smoothing filtering may be a method of eliminating inter-frame shaking through a weighted average of the current frame and the previous frame. In the present disclosure, inter-frame smoothing filtering may correspond to IIR (Infinite Impulse Response) filtering.

[0060] However, in the weighted average comparison depth map (220) using global weights (e.g., fixed weights in the entire area), there is a problem that afterimages remain at the boundaries of objects, so the three-dimensionality of moving objects is reduced.

[0061] In one embodiment of the present disclosure, the image processing device (100) may perform filtering on an input depth map to generate a first filtered depth map (230). The first filtered depth map (230) may have reduced inter-frame shaking and reduced afterimages at the boundary of a moving object. Here, filtering for reducing inter-frame shaking in the input depth map and reducing afterimages at the boundary of a moving object may be referred to as first filtering or inter-frame three-dimensional effect enhancement filtering.

[0062] In one embodiment of the present disclosure, the image processing device (100) can generate a depth map having uniform depth values ​​between frames through first filtering. For example, the image processing device (100) can perform the first filtering using variable weights (e.g., weights having different values ​​in the boundary area and the non-boundary area), thereby applying different weighted averages to the boundary area and the non-boundary area of ​​an object, which will be described later.

[0063] Referring to 202 of FIG. 2, a two-dimensional image (250), an input depth map (260) corresponding to the two-dimensional image (250), and a second filtered depth map (270) are described.

[0064] A two-dimensional image (250) includes a foreground and a background, and an object is located in the foreground, and the background can be located further away than the object in the foreground.

[0065] An input depth map (260) generated from a two-dimensional image (250) may have small differences between depth values ​​constituting an object (e.g., a human face) located in the foreground. If local features on the surface of an object have similar depth values ​​overall, the object's three-dimensionality is reduced and it may appear flat, resulting in an unnatural appearance.

[0066] In one embodiment of the present disclosure, the image processing device (100) may perform filtering on the input depth map (260) to generate a second filtered depth map (270) in which subtle depth changes inside an object are reflected and the boundary between the inside of the object and the outside of the object is clearly distinguished. Here, the filtering for reflecting subtle depth changes inside the object in the input depth map and distinguishing the boundary between the inside of the object and the outside of the object may be referred to as second filtering or object boundary three-dimensionality enhancement filtering.

[0067] In one embodiment of the present disclosure, the image processing device (100) can generate a depth map that reflects subtle depth changes within an object and accurately reflects the difference in depth values ​​based on the boundary between the inside and outside of the object through second filtering. For example, the image processing device (100) can apply different blur processing to the foreground and background areas through second filtering, which will be described later.

[0068] An operation of performing filtering on an input depth map according to one embodiment of the present disclosure can overcome limitations of a depth map estimation model that estimates a depth map using only an image of a subject captured by a single camera (e.g., a limitation that the precision of the depth map is lower than that of a 3D camera or LiDAR method). Accordingly, the image processing device (100) can reduce afterimages of object boundaries in the input depth map through first filtering, and can increase depth map accuracy by accurately reflecting changes in depth within the object through second filtering.

[0069] FIG. 3 is a block diagram of a configuration of an image processing device according to one embodiment of the present disclosure for generating a three-dimensional image using a depth map of a two-dimensional image.

[0070] Referring to FIG. 3, an image processing device (100) according to an embodiment of the present disclosure may include a depth map estimation module (310), a depth map filtering module (e.g., a first filtering module (320), a second filtering module (330)), a binocular view generation module (340), a hole filling module (350), and a binocular view combining module (360). However, not all of the illustrated components are essential components. The image processing device (100) may be implemented with more components than the illustrated components, or may be implemented with fewer components. The depth map estimation module (310), the first filtering module (320), the second filtering module (330), the binocular view generation module (340), the hole filling module (350), and the binocular view combination module (360) are included in the memory of the image processing device (100) and can be implemented as software such as commands, algorithms, data structures, or program codes executed by the processor of the image processing device (100).

[0071] Here, the depth map generation module (101) of FIG. 1 may include a depth map estimation module (310), a first filtering module (320), and a second filtering module (330). The depth image-based rendering module (102) of FIG. 1 may include a binocular view generation module (340), a hole filling module (350), and a binocular view combination module (360). However, the present invention is not limited thereto.

[0072] The depth map estimation module (310) can generate a depth map from a two-dimensional image. In one embodiment of the present disclosure, the depth map estimation module (310) may be a neural network trained to output a depth map from a two-dimensional image. The neural network may input a two-dimensional color image (e.g., an RGB image) and output a depth map. The depth map may be in the form of a one-channel image in which depth values ​​corresponding to each pixel of the two-dimensional color image are stored. The depth map may be configured with a depth value of 0.0 to 1.0 through a normalization process. For example, the closest distance may be represented as 0.0 and may be represented by a dark color in the depth map. For example, the farthest distance may be represented as 1.0 and may be represented by a bright color in the depth map. The depth map estimation module (310) may output an input depth map from a two-dimensional image input to the image processing device (100). Here, the neural network may be referred to as a depth map estimation model. However, it is not limited thereto, and the depth map estimation module (310) may be configured with commands or program codes related to an operation or function of generating a depth map based on a two-dimensional image.

[0073] The image processing device (100) can input the input depth map output from the depth map estimation module (310) to at least one of the first filtering module (320) or the second filtering module (330), or input it to both the first filtering module (320) and the second filtering module (330). For example, the image processing device (100) can perform first filtering on an image with a lot of movement. For example, the image processing device (100) can perform second filtering on an image for expressing a change in depth inside an object (e.g., a human face).

[0074] The first filtering module (320) can generate a first filtered depth map from the depth map input to the first filtering module (320). Here, the depth map input to the first filtering module (320) is exemplified as an input depth map output from the depth map estimation module (310), but is not limited thereto. The first filtering module (320) can perform IIR (Infinite Impulse Response) filtering to reduce inter-frame shaking for an image including a moving object. For example, the first filtering module (320) can perform local IIR (Local IIR) filtering that performs a weighted average using variable weights having different weights for each boundary area and non-boundary area. By reducing afterimages in the boundary area of ​​the depth map through the first filtering module (320), the accuracy of the depth map in the boundary area can be improved. The first filtered depth map can be a depth map with reduced inter-frame shaking and less afterimages at object boundaries, thereby enhancing a three-dimensional effect.

[0075] For example, the first filtering module (320) may include a boundary region detection module (410 in FIG. 4) for detecting a boundary region of a moving object. The first filtering module (320) may include a weight acquisition module (420 in FIG. 4) for obtaining weights for performing a weighted average on a pixel-by-pixel basis differently for each region. The first filtering module (320) may include a weighted average application module (430 in FIG. 4) for performing a weighted average on a pixel-by-pixel basis between the current frame and the previous frame of the input depth map. This will be described in FIG. 4.

[0076] The second filtering module (330) can generate a second filtered depth map from the depth map input to the second filtering module (330). Here, the depth map input to the second filtering module (330) may be an input depth map output from the depth map estimation module (310) or a first filtered depth map output from the first filtering module (320). Hereinafter, in order to clearly explain the operation of the second filtering module (330), the depth map input to the second filtering module (330) is exemplified as an input depth map. The second filtering module (330) can implement a depth change inside an object through blur processing for the foreground area. The second filtering module (330) can generate a depth map to which blur processing is applied differently for the foreground area and the background area, thereby implementing a depth change inside an object and distinguishing the boundary between the foreground area and the background area. The second filtered depth map can be a depth map with enhanced three-dimensionality by implementing depth changes within an object and distinguishing boundaries between foreground and background areas.

[0077] For example, the second filtering module (330) may include a depth map blur processing module (1010 of FIG. 10) for blurring an input depth map. A smooth boundary can be formed between the foreground area and the background area of ​​the input depth map through the depth map blur processing module (1010). The second filtering module (330) may include a maximum operation module (1020 of FIG. 10) for performing a maximum operation to prevent the boundary between the foreground area and the background area from disappearing due to a decrease in the depth value of the background after blur processing. This will be described in FIG. 10. The second filtering module (330) may include a weighted average application module (1330 of FIG. 13) (or a mixed operation module) for performing a weighted average on a pixel basis using the input depth map as a weight to reflect less blur processing in the background area. The weighted average application module (1330) can prevent the boundary of another object located in the background area from disappearing during blur processing. This is explained in Fig. 13.

[0078] The binocular viewpoint generation module (340) can generate a 3D image of a virtual viewpoint based on the depth map and 2D image input to the binocular viewpoint generation module (340). The 3D image of the virtual viewpoint may include, for example, a 3D image for the left eye and a 3D image for the right eye. The depth map input to the binocular viewpoint generation module (340) may be at least one of a first filtered depth map output from the first filtering module (320) or a second filtered depth map output from the second filtering module (330). For example, the binocular viewpoint generation module (340) can generate a 3D image for the left eye by shifting pixels included in a 2D image to the left by a depth value. For example, the binocular viewpoint generation module (340) can generate a 3D image for the right eye by shifting pixels included in a 2D image to the right by a depth value.

[0079] The hole filling module (350) can form an interpolation pixel within the hole using pixels adjacent to the hole when a hole occurs within the frame as a pixel moves. Here, a hole may refer to a pixel for which a pixel value (e.g., RGB data) is not input. A hole may occur in a dis-occlusion region that was covered by an object and then becomes visible again as the pixel moves. The dis-occlusion region may not have information that can be obtained from the original image when generating an image of a virtual viewpoint. Since the dis-occlusion region must display non-existent information, it appears as an empty pixel or hole.

[0080] The binocular viewpoint combining module (360) can create a 3D image by combining a left-eye 3D image and a right-eye 3D image. Depending on the method by which the left and right images are combined into a 3D image, it is divided into a top-down method in which the left-eye image and the right-eye image are arranged vertically within one frame, an L-to-R (left-to-right) (or side-by-side) method in which the left-eye image and the right-eye image are arranged left-to-right within one frame, a checker board method in which pieces of the left-eye image and the right-eye image are arranged in a tile shape, an interlaced method in which the left-eye image and the right image are arranged alternately in columns or rows, and a time-sequential (frame by frame) method in which the left-eye image and the right-eye image are alternately displayed over time. For example, the binocular viewpoint combining module (360) can combine multiple input signals (e.g., left-eye images and right-eye images) into a single output signal via a multiplexer. The generated 3D image can be displayed on a 3D display.

[0081] Hereinafter, the first filtering will be described with reference to FIGS. 4 to 9. In addition, the second filtering will be described with reference to FIGS. 10 to 15. Subsequently, the operating method of the image processing device (100) for processing an input depth map will be described with reference to FIGS. 16 to 19.

[0082] FIG. 4 is a detailed block diagram of a first filtering module of an image processing device according to an embodiment of the present disclosure. FIG. 5 is an example of a depth map including a moving object in neighboring frames according to an embodiment of the present disclosure.

[0083] Referring to FIG. 4, the first filtering module (320) may include a boundary area detection module (410), a weight acquisition module (420), and a weighted average application module (430). However, not all of the illustrated components are essential components. The first filtering module (320) may be implemented with more components than the illustrated components, or may be implemented with fewer components. The boundary area detection module (410), the weight acquisition module (420), and the weighted average application module (430) may be included in the memory of the image processing device (100) and may be implemented as software such as instructions, algorithms, data structures, or program codes executed by the processor of the image processing device (100).

[0084] Referring to FIG. 5, the input depth map input to the first filtering module (320) includes a moving object, and the position of the moving object may be different for each frame. For example, the input depth map may include a previous frame (510) of a first time point (t-1) and a current frame (520) of a second time point (t). The object included in the input depth map may move over time, and FIG. 5 exemplifies an object moving to the right over time.

[0085] In the present disclosure, the current frame (520) may be divided into a first region (501), a second region (502), a third region (503), and a fourth region (504) according to the movement of the object. Here, the first region (501) may represent an object region common to the current frame (520) and the previous frame (510). That is, the first region (501) may be a region where the depth information of the object in the current frame (520) and the previous frame (510) overlaps. The second region (502) may represent a dis-occlusion region that is behind the object in the previous frame (510) but newly appears due to the movement of the object. The third region (503) may represent an occlusion region that is visible in the previous frame (510) but is obscured due to the movement of the object. The fourth region (504) may represent a background region other than the objects common to the previous frame (510) and the current frame (520).

[0086] For example, in the current frame (520), since the object in the previous frame (510) moves to the right, the dis-occlusion region that was occluded by the object may be located to the left of the object. Additionally, in the current frame (520), the occlusion region that was present in the previous frame (510) but becomes occluded as the object moves to the right may be located to the right of the object.

[0087] In the present disclosure, the boundary region of an object is a region that changes as the object moves, and may include a second region (502) and a third region (503). The non-boundary region of the object is a region other than the boundary region, and may include a first region (501) and a fourth region (504).

[0088] As described in 201 of FIG. 2, in one embodiment of the present disclosure, the input depth map may have non-uniform depth values ​​for each frame depending on the accuracy (or performance) of the depth map estimation model. For example, if the accuracy of the depth map estimation model is low, the depth values ​​may be different even in the same area in the previous frame (510) and the current frame (520). For example, if the depth value of the background area of ​​the previous frame (510) is estimated to be 1.0 and the depth value of the background area of ​​the current frame (520) is estimated to be 0.9, a shaking phenomenon may appear between consecutive frames. Therefore, the image processing device (100) may perform IIR filtering (or infinite impulse response filtering) to weightedly average the depth maps between frames to alleviate the shaking phenomenon between frames for the input depth map.

[0089] IIR filtering can perform an operation of adjusting the current output by considering the past input and past output. For example, the first filtering module (320) can adjust the depth value of the current frame (520) by using the previous frame (510) and the current frame (520), which are consecutive neighboring frames. Here, the previous frame may be a previously filtered frame. Since the first filtering module (320) adjusts the depth value for each pixel in the frame, it can be referred to as pixel-by-pixel filtering.

[0090] According to one embodiment of the present disclosure, the first filtering module (320) performs a weighted average using variable weights, which may be referred to as local IIR filtering. The variable weights used in local IIR filtering are different weights for each region and may be referred to as local weights. For example, when performing a weighted average of the previous frame (510) and the current frame (520) of the depth map, the first filtering module (320) may apply different weighted averages to the boundary region and non-boundary region of the object. Local filtering is more effective in reducing afterimages at the boundary of a moving object than global IIR filtering using global weights (e.g., fixed weights in the entire region). This will be further described with reference to FIGS. 8A and 8B.

[0091] According to one embodiment of the present disclosure, the first filtering module (320) can detect a boundary area included in an input depth map through a boundary area detection module (410). The first filtering module (320) can generate variable weights having different values ​​in boundary areas and non-boundary areas through a weight acquisition module (420). The first filtering module (320) can perform a weighted average between a previous frame (510) and a current frame (520) of the input depth map based on the variable weights through a weighted average application module (430).

[0092] The operation of detecting a boundary area included in an input depth map through a boundary area detection module (410) is described in FIG. 6, the operation of obtaining a variable weight through a weight obtaining module (420) is described in FIG. 7, and the operation of generating a first filtered depth map through a weighted average application module (430) is described in FIGS. 8a and 8b.

[0093] FIG. 6 is an example of depth information acquired by an image processing device according to an embodiment of the present disclosure through a boundary region detection module. Here, the depth information (e.g., 610, 620, 630) acquired through the boundary region detection module (410) may be expressed in the form of a depth image. In the depth image, large depth values ​​may be expressed in a bright color, and small depth values ​​may be expressed in a dark color.

[0094] The boundary area detection module (410) can detect the boundary area of ​​a moving object. The boundary area detection module (410) can detect the depth information of the boundary area and the depth information of the non-boundary area separately.

[0095] The boundary area detection module (410) can obtain first depth information (610) based on the previous frame (510) and the current frame (520). The first depth information (610) can include depth information of the boundary area and the background area common to the previous frame (510) and the current frame (520). For example, the first depth information (610) can have a large depth value in the second area (602), the third area (603), and the fourth area (604) excluding the first area (601).

[0096] The first depth information (610) can be obtained through mathematical expression 1.

[0097] [Mathematical Formula 1]

[0098] b[x,y] = 1 - (1 - z[x,y,t]) * (1 - z[x,y,t-1])

[0099] In mathematical expression 1, z[x, y, t] represents depth information at the (x, y) position of the current frame (520) of the depth map, and z[x, y, t-1] represents depth information at the (x, y) position of the previous frame (510) of the depth map.

[0100] Mathematical expression 1 can produce results having large values ​​in the second region (602), the third region (603), and the fourth region (604) through the operation of z[x,y,t-1] corresponding to the previous frame (510) and z[x,y,t] corresponding to the current frame (520) at the (x,y) location.

[0101] In mathematical expression 1, 1- z[x, y, t] may have a large value in the foreground of the (x, y) position of the current frame (520). For example, in 1- z[x, y, t], the depth values ​​of the first region (601) and the third region (603) may be larger than the depth values ​​of the second region (602) and the fourth region (604). The depth values ​​of the first region (601) and the third region (603) may be similar to each other, and the depth values ​​of the second region (602) and the fourth region (604) may be similar to each other. (i.e., ①=③ > ②=④)

[0102] In mathematical expression 1, 1- z[x, y, t-1] may have a large value in the foreground of the (x, y) location of the previous frame (510). For example, in 1- z[x, y, t-1], the depth values ​​of the first region (601) and the second region (602) may be larger than the depth values ​​of the third region (603) and the fourth region (604). The depth values ​​of the first region (601) and the second region (602) may be similar to each other, and the depth values ​​of the third region (603) and the fourth region (604) may be similar to each other. (i.e., ①=② > ③=④)

[0103] In (1-z[x,y,t])*( 1-z[x,y,t-1]) of mathematical expression 1, the depth value of the first region (601) may be the largest, the depth values ​​of the second region (602) and the third region (603) may be similar to each other, and the depth value of the fourth region (604) may be the smallest. (i.e., ① > ② = ③ > ④)

[0104] In 1 - (1 - z[x,y,t]) * (1 - z[x,y,t-1]) of mathematical expression 1, the depth value of the fourth region (604) may be the largest, the depth values ​​of the second region (602) and the third region (603) may be similar to each other, and the depth value of the first region (601) may be the smallest. (i.e., ④ > ② = ③ >> ①)

[0105] The boundary area detection module (410) can obtain second depth information (620) based on the previous frame (510) and the current frame (520). The second depth information (620) can have depth values ​​of the foreground area of ​​the previous frame (510) and the foreground area of ​​the current frame (520). For example, the second depth information (620) can have large depth values ​​in the first area (601), the second area (602), and the third area (603).

[0106] The second depth information (620) can be obtained through mathematical expression 2.

[0107] [Equation 2]

[0108] o[x,y] = 1 - min(z[x,y,t-1] , z[x,y,t])

[0109] Mathematical expression 2 can produce results having large values ​​in the first region (601), the second region (602), and the third region (603) through the operation of z[x,y,t-1] representing the depth information of the (x,y) position of the previous frame (510) and z[x,y,t] representing the depth information of the (x,y) position of the current frame (520) at the (x,y) position.

[0110] In the min(z[x,y,t-1], z[x,y,t]) of mathematical expression 2, it can have a large value in the fourth region (604). (i.e., ④ >> ① = ② = ③)

[0111] In Equation 2, 1 - min(z[x,y,t-1], z[x,y,t]) can have large values ​​in the first region (601), the second region (602), and the third region (603). (i.e., ① = ② = ③ >> ④)

[0112] The boundary area detection module (410) can obtain third depth information (630) based on the first depth information (610) and the second depth information (620). The third depth information (630) can have a depth value of the boundary area. For example, the third depth information (630) can have large depth values ​​in the second area (602) and the third area (603), and small depth values ​​in the first area (601) and the fourth area (604).

[0113] The third depth information (630) can be obtained through mathematical expression 3.

[0114] [Equation 3]

[0115] m[x,y] = b[x,y] * o[x, y]

[0116] Mathematical expression 3 can produce a result value with a large value in the boundary region by multiplying the result of Mathematical expression 1 and the result of Mathematical expression 2. (i.e., ② = ③ >> ① > ④)

[0117] The boundary area detection module (410) can transmit the third depth information (630), which is the boundary area detection result, to the weight acquisition module (420).

[0118] Referring to FIG. 7, the operation of the weight acquisition module (420) is further described. FIG. 7 is a graph showing weights used for first filtering obtained through a weight generation module by an image processing device according to an embodiment of the present disclosure. The graph (700) represents a boundary area detection value m[x, y] on the x-axis and a variable weight w[x, y] on the y-axis. The graph (700) illustrates changes in variable weights according to boundary areas and non-boundary areas.

[0119] The weight acquisition module (420) may receive third depth information (730) from the boundary region detection module (410). The weight acquisition module (420) may acquire weights for weighted averaging differently for each region. The weights may be variable weights having different values ​​for each region. The variable weights may be calculated on a pixel-by-pixel basis.

[0120] The variable weight can be obtained through mathematical expression 4.

[0121] [Equation 4]

[0122] w[x,y] = clamp(slope* m[x,y], wmin, wmax)

[0123] In mathematical expression 4, clamp(x, minVal, maxVal) can be interpreted as min(max(x, minVal), maxVal), and can produce a result value that limits the upper and lower limits so that x becomes minVal ≤ x ≤ maxVal. Here, slope*m[x,y] is a result value obtained by multiplying a set slope by the third depth information (630), and m[x,y] can be scaled according to the set slope. Depending on the size of the slope, it can be determined whether the influence of the weight during the weighted average is greatly affected. w[x,y] represents a variable weight at the (x, y) location.

[0124] Referring to mathematical expression 4 and graph (700), the variable weight w[x,y] is a value obtained by scaling the third depth information with a predetermined slope, and the upper and lower limits may be limited by wmax and wmin, respectively. The variable weight w[x,y] may have a large value in a boundary area (e.g., the second area and the third area) and a small value in a non-boundary area (e.g., the first area and the fourth area). For example, in the boundary area, w[x,y] > (1-w[x,y]), and in the non-boundary area, w[x,y]≤(1-w[x,y]).

[0125] The weight acquisition module (420) can transmit the calculated variable weights to the weighted average application module (430).

[0126] Referring to FIGS. 8A and 8B, the operation of the weighted average application module (430) is further described. FIG. 8A is an example of a depth map (810) to which global IIR filtering is applied, and FIG. 8B is an example of a depth map (820) to which local IIR filtering is applied.

[0127] The weighted average application module (430) can receive variable weights from the weight acquisition module (420) and receive the current frame (520) and previous frame (510) of the input depth map.

[0128] The weighted average application module (430) can perform a weighted average between the current frame (520) and the previous frame (510) of the input depth map based on variable weights. The weighted average can be applied on a pixel basis. The weighted average application module (430) can generate a depth map (820) of the first filtered current frame by performing a first filtering on the current frame (520). The depth map (820) of the first filtered current frame can correspond to a depth map to which local IIR filtering is applied.

[0129] Local IIR filtering can be performed according to the following mathematical expression 5.

[0130] [Equation 5]

[0131] z[x, y, t] = clamp(z[x, y, t-1] * (1 - w[xy]) + z[x, y, t] * w[x,y], 0.0, 1.0)

[0132] In mathematical expression 5, the depth map of the current frame to which local IIR filtering is applied may be a weighted average result value such that the depth map of the current frame (520) is more reflected in the boundary area, and the depth map of the previous frame (510) is more reflected in the non-boundary area. The upper and lower limits of the depth map may be 1 and 0, respectively.

[0133] In the boundary region, since w[x,y] > (1-w[x,y]), the depth value z[x,y,t] at the (x,y) position of the current frame (520) may be reflected more than the depth value z[x,y,t-1] at the (x,y) position of the previous frame (510).

[0134] In the non-boundary region, since w[x,y]≤(1-w[x,y]), the depth value z[x,y,t-1] at the (x,y) position of the previous frame (510) may be reflected more or similarly than the depth value z[x,y,t] at the (x,y) position of the current frame (520).

[0135] The depth map of the current frame with local IIR filtering can be updated according to the following mathematical expression (6).

[0136] [Equation 6]

[0137] z[x, y, t-1] = z[x, y, t]

[0138] Through mathematical expression 6, the current frame information that has been locally IIR filtered can be updated to the previous frame. Accordingly, the current frame that has been locally IIR filtered at the second time point (t) can be used as the previous frame for the local IIR filtering of the next frame at the third time point (t+1).

[0139] The first filtering module (320) can generate a first filtered input depth map composed of a plurality of first filtered frames by performing a first filtering for each frame of the input depth map. The first filtering module (320) can output the first filtered input depth map.

[0140] An image processing device (100) according to one embodiment of the present disclosure can perform variable weighted averaging so that the current frame is reflected more in occluded areas and open areas, and the previous frame is reflected more in other areas, when weighting the previous frame and the current frame of a depth map.

[0141] Meanwhile, referring to FIG. 8A, the depth map (810) to which global IIR filtering is applied may be a result of performing a weighted average by applying a 0.5:0.5 weight between the previous frame (510) and the current frame (520) (e.g., w[x, y]=0.5). Since the depth values ​​of the previous frame (510) and the current frame (520) are reflected in the same ratio in the depth map (810), an afterimage may remain in the boundary area of ​​the object (e.g., the second area (812) and the third area (813)). For example, the second area (812) may not be updated to the background of the current frame (520) according to the movement of the object, but may be weighted and combined with the depth value corresponding to the foreground area of ​​the previous frame (510) to remain as an afterimage. In addition, for example, the third region (813) may not be updated to the foreground of the current frame (520) according to the movement of the object, but may remain as an afterimage by being weighted with the depth value corresponding to the background region of the previous frame (510). For example, the second region (812) may be the background of the current frame (520), and the third region (813) may be the foreground of the current frame (520), but may be configured with the same or similar depth values. The depth map (810) to which global IIR filtering is applied may have an afterimage remaining in the boundary region of the object, so the three-dimensional stereoscopic expression may be degraded.

[0142] The depth map (820) with local IIR filtering applied in FIG. 8B may be a result of performing a weighted average by applying variable weights between the previous frame (510) and the current frame (520). The depth map (820) may perform a weighted average between the previous frame (510) and the current frame (520) in the boundary region of the object (e.g., the second region (822) and the third region (823)), but may apply a larger weight to the current frame (520). For example, the third region (823) may be updated to the foreground of the current frame (520) according to the movement of the object, and the second region (822) may be updated to the background of the current frame (520) according to the movement of the object. For example, the depth value of the third region (823) corresponding to the background of the current frame (520) may be smaller than the depth value of the second region (822) corresponding to the foreground of the current frame (520). That is, the third area (823) can be expressed darker. Accordingly, the depth map (820) to which local IIR filtering is applied can have residual images in the boundary area removed, and the three-dimensional stereoscopic expression can be enhanced.

[0143] Meanwhile, the depth map (820) performs a weighted average between the previous frame (510) and the current frame (520) in the non-boundary area of ​​the object (e.g., the first area (821) and the fourth area (824)), but may apply a greater weight or a similar weight to the previous frame (510). Accordingly, rapid depth changes between frames can be reduced, thereby preventing judder between frames.

[0144] 9 is a diagram illustrating an operation of an image processing device according to one embodiment of the present disclosure to obtain an output depth map from an input depth map through a first filtering module. Descriptions of each term are provided in FIGS. 4 to 8b.

[0145] Referring to FIG. 9, the image processing device (100) can input the current frame (520) of the input depth map and the previous frame (510) of the input depth map into the boundary area detection module (410) to obtain third depth information (630) corresponding to the boundary area detection result. The image processing device (100) can input the third depth information (630) into the weight acquisition module (420) to obtain a weight. The weight may be a variable weight that is different in the boundary area and the non-boundary area, as shown in the graph (700). The image processing device (100) can input the current frame (520) of the input depth map, the previous frame (510) of the input depth map, and the variable weight into the weighted average application module (430) to obtain a depth map (820) of the first filtered current frame. The image processing device (100) can update the previous frame to use the depth map (820) of the first filtered current frame as the previous frame in the first filtering process for the next frame. The image processing device (100) can generate a first filtered depth map including a plurality of first filtered frames.

[0146] FIG. 10 is a detailed block diagram of a second filtering module of an image processing device according to one embodiment of the present disclosure.

[0147] Referring to FIG. 10, the second filtering module (330) may include a depth map blur processing module (1010) and a maximum value calculation module (1020). However, not all of the illustrated components are essential components. The second filtering module (330) may be implemented with more components than the illustrated components, or may be implemented with fewer components. The depth map blur processing module (1010) and the maximum value calculation module (1020) are included in the memory of the image processing device (100) and may be implemented as software such as instructions, algorithms, data structures, or program codes executed by the processor of the image processing device (100).

[0148] Hereinafter, the operation of the depth map blur processing module (1010) and the maximum value calculation module (1020) will be described in detail with reference to the depth maps illustrated in FIGS. 10 and 11. FIG. 11 is a diagram for explaining a blur effect applied to the foreground and background areas of depth maps used in the maximum value calculation module according to one embodiment of the present disclosure.

[0149] Referring to FIGS. 10 and 11, the image processing device (100) can input an input depth map (1110) into a depth map blur processing module (1010) to obtain a blurred depth map (1120). The blurred depth map (1120) may be referred to as a first depth map (1120). The image processing device (100) can input the input depth map (1110) and the first depth map (1120) into a maximum value calculation module (1020) to obtain a second depth map (1130). Here, the input depth map (1110) may be an input depth map output from the depth map estimation module (310) or may be a first filtered depth map output from the first filtering module (320). Here, the second depth map (1130) may be a second filtered depth map.

[0150] The depth map blur processing module (1010) can perform blur processing on the input depth map (1110) to generate a first depth map (1120). Blur processing refers to making the image blurry by reducing the sharpness of the image and applying a soft blur effect. For example, when the depth map is blurred, the depth value of the background area may decrease and the depth value of the foreground area may increase. A soft boundary may be formed between the foreground area and the background area of ​​the first depth map (1120).

[0151] Meanwhile, the depth map blur processing module (1010) may use a Gaussian filter, an average filter, a separable filter, etc., but is not limited thereto. The detailed operation of the depth map blur processing module (1010) is described in FIG. 15.

[0152] The maximum value operation module (1020) can perform a maximum value operation between the input depth map (1110) and the first depth map (1120) to generate a second depth map (1130). The maximum value operation module (1020) can obtain the second depth map (1130) through mathematical expression 7.

[0153] [Equation 7]

[0154] e[x,y] = max( z[x,y], blur( z[x,y] )

[0155] In mathematical expression 7, z[x,y] may represent a depth value of an input depth map (1110), and blur(z[x,y]) may represent a first depth map (1120), which is a blurred depth map. In mathematical expression 7, the depth values ​​of the input depth map (1110) and the first depth map (1120) may be compared at the (x,y) location, and a larger value may be selectively reflected for each location (x,y).

[0156] For example, in the foreground area, the depth value of the first depth map (1120) may be greater than the depth value of the input depth map (1110). In the background area, the depth value of the input depth map (1110) may be greater than the depth value of the first depth map (1120). Accordingly, the second depth map (1130) may have a foreground area in which the depth value of the first depth map (1120) is reflected and a background area in which the depth value of the input depth map (1110) is reflected. For example, the foreground area of ​​the second depth map (1130) may have a depth value after blurring, and the background area may have a depth value before blurring.

[0157] Since the second depth map (1130) has a blurred depth value in the foreground area, the three-dimensionality of the foreground area can be emphasized. In addition, since the second depth map (1130) has a blurred depth value only in the foreground area, the depth value of the background can be reduced after blurring, preventing the boundary between the foreground area and the background area from disappearing.

[0158] Hereinafter, with further reference to FIGS. 11 and 12, a description will be given of a change in the boundary between the foreground area and the background area. FIG. 12 is a graph showing depth values ​​for each area of ​​depth maps used in a maximum value calculation module according to an embodiment of the present disclosure. The specific depth values ​​illustrated in FIG. 11 are merely examples for convenience of explanation and are not limited thereto. Graphs (1210) and (1220) represent x-coordinate positions on the x-axis and depth values ​​on the y-axis. Graph (1210) shows a change in the depth value of an input depth map (1110) and a change in the depth value of a first depth map (1120) according to a background area and a foreground area. Graph (1220) shows a change in the depth value of a second depth map (1130) according to a background area and a foreground area.

[0159] Referring to the graph (1210) of FIGS. 11 and 12, in the input depth map (1110), the boundary between the background area and the foreground area can be clearly distinguished. For example, at the boundary, the depth value of the background area may be 0.9, and the depth value of the foreground area may be 0.1.

[0160] The depth map blur processing module (1010) can receive an input depth map (1110) and output a first depth map (1120). When the depth map is blurred, the depth value of the foreground area of ​​the first depth map (1120) increases, so that the three-dimensionality of the object can be improved. However, since the first depth map (1120) is blurred, the boundary between the background area and the foreground area can disappear. In particular, with respect to the object, the boundary between the inside and the outside of the object may not be distinguished, and no depth change may appear. For example, at the boundary, the depth value of the background area may be 0.55, and the depth value of the foreground area may be 0.45.

[0161] Referring to the graph (1220) of FIGS. 11 and 12, the maximum value operation module (1020) can generate a second depth map (1130) including a background area reflecting the depth value of the input depth map (1110) and a foreground area reflecting the depth value of the first depth map (1120). The foreground area of ​​the second depth map (1130) may be identical to the depth value of the first depth map (1120). The background area of ​​the second depth map (1130) may be identical to the depth value of the input depth map (1110).

[0162] The second depth map (1130) can clearly distinguish the boundary between the background and foreground areas through maximum value calculation. In particular, the boundary between the inside and outside of the object can be distinguished based on the object. For example, at the boundary, the depth value of the background area may be 0.9, and the depth value of the foreground area may be 0.45. Accordingly, background distortion due to blurring can be minimized.

[0163] The second depth map (1130) may include a blurred foreground area and an unblurred background area. By blurring the foreground area, the three-dimensionality within the object can be enhanced. By not blurring the background area, the boundary between the inside and outside of the object can be clearly distinguished, and background distortion can be minimized.

[0164] FIG. 13 is a detailed block diagram of a second filtering module of an image processing device according to an embodiment of the present disclosure. FIG. 14 is a diagram for explaining an operation of an image processing device according to an embodiment of the present disclosure for obtaining an output depth map from an input depth map through a second filtering module.

[0165] Referring to FIGS. 13 and 14, the second filtering module (330a) may further include a weighted average application module (1330). The weighted average application module (1330) is included in the memory of the image processing device (100) and may be implemented as software such as commands, algorithms, data structures, or program codes executed by the processor of the image processing device (100).

[0166] The image processing device (100) can input an input depth map (1110) and a second depth map (1130) into a weighted average application module (1330) to obtain a third depth map (1410). Here, the second depth map (1130) may be an intermediate depth map, and the third depth map (1410) may be a second filtered depth map (i.e., an output depth map).

[0167] The weighted average application module (1330) can perform a mixing operation by applying a mixing ratio to the input depth map (1110). The mixing operation may be identical to a process of performing a weighted average on the input depth map (1110) and the second depth map (1130) by applying the input depth map (1110) as a weight.

[0168] The weighted average application module (1330) can obtain the third depth map (1410) through mathematical expression 8.

[0169] [Equation 8]

[0170] o[x, y] = mix( e[x,y], z[x,y], z[x,y] )

[0171] = e[x,y] * (1-z[x,y]) + z[x, y] * z[x, y]

[0172] In mathematical expression 8, a mixing operation between the input depth map (1110) and the second depth map (1130) can be performed at a mixing ratio. Mathematical expression 8 can be interpreted as performing a weighted average on the input depth map (1110) and the second depth map (1130), but applying the input depth map (1110) as a weight.

[0173] The input depth map (1110) used as a variable weight may have a large value in the foreground area and a small value in the background area.

[0174] In the foreground area, since z[x,y] < (1-z[x,y]), the depth value e[x,y] at the (x,y) position of the second depth map (1130) may be reflected more than the depth value z[x,y] at the (x,y) position of the input depth map (1110).

[0175] In the background area, since z[x,y] ≥ (1-z[x,y]), the depth value z[x,y] at the (x,y) position of the input depth map (1110) can be reflected more or similarly than the depth value e[x,y] at the (x,y) position of the second depth map (1130).

[0176] The third depth map (1410) may be a weighted average result value such that the second depth map (1130) is reflected more in the foreground area, and the input depth map (1110) is reflected more in the background area.

[0177] For example, since the input depth map (1110) used as a weight has a large depth value (e.g., 1.0 or 0.9) in the background area, the background area of ​​the third depth map (1410) may be similar to the background area of ​​the input depth map (1110). That is, since the background area of ​​the third depth map (1410) reflects more of the input depth map (1110), blur processing may not be reflected in the background area like the input depth map (1110).

[0178] The third depth map (1410) may reflect blurring in the foreground area and not in the background area. Accordingly, when an object other than the object of interest (e.g., another object located behind a human face) is located in the background area, the problem of the boundary disappearing due to blurring can be prevented.

[0179] FIG. 15 is a diagram for explaining a depth map blur processing module according to one embodiment of the present disclosure.

[0180] Referring to FIG. 15, a depth map blur processing module (1010) according to one embodiment of the present disclosure can apply a blur effect by mixing the depth value of a target pixel of an input depth map with the depth values ​​of surrounding pixels of the target pixel.

[0181] The depth map blur processing module (1010) may include average filtering, Gaussian filtering, etc. Average filtering may perform an operation of calculating an average of depth values ​​corresponding to a target pixel and surrounding pixels determined according to a sampling range. Gaussian filtering may perform an operation of assigning a high weight to pixels close to the target pixel and a low weight to pixels far from the target pixel.

[0182] Here, the surrounding pixels referenced when performing blur processing on the target pixel can be referred to as sample pixels. The sampling range indicates the range of surrounding pixels referenced when performing blur processing, and can vary, such as 3x3 or 5x5, depending on the size of the blur filter. The number of sampled pixels can indicate the number of pixels actually referenced within the sampling range. For example, in the case of a 5x5 sampling range, the number of sampleable pixels can be 24.

[0183] Meanwhile, as illustrated in the depth map (1510), there are cases where the number of sampleable pixels is small compared to the sampling range. For example, if there are few surrounding pixels with depth values ​​around a target pixel requiring blur processing, natural blur processing may be difficult.

[0184] In this case, as illustrated in the depth map (1520), the depth map blur processing module (1010) can generate a mipmap in the form of an image pyramid of the depth map. The mipmap can be a block unit composed of a plurality of pixels. For example, the mipmap can include a sample pixel and n pixels adjacent to the sample pixel. For example, the depth map blur processing module (1010) can calculate the depth value of the mipmap by averaging the depth values ​​of the plurality of pixels constituting the mipmap.

[0185] As illustrated in the depth map (1530), the depth map blur processing module (1010) may instead sample a mipmap that is a higher level than a pixel. For example, the depth map blur processing module (1010) may perform blur processing on a target pixel by calculating an average of the depth values ​​of the target pixel and each of a plurality of mipmaps through average filtering. The image processing device (100) may perform blur processing using a mipmap, thereby applying a natural blur effect using a small number of sample pixels.

[0186] When blurring is applied to a depth map, the depth near the boundary within an object becomes deeper than the center within the object, which may cause the object to appear convex. However, since the boundary between the foreground and background areas becomes ambiguous due to the blurring effect, a maximum value calculation process may be additionally performed to improve this. The blurred depth map output from the depth map blur processing module (1010) may be passed to the maximum value calculation module (1020 of FIG. 10).

[0187] FIG. 16 is a flowchart illustrating a method for generating a three-dimensional image by performing first filtering on a depth map by an image processing device according to an embodiment of the present disclosure. For convenience of explanation, FIG. 16 is described with reference to FIGS. 4 to 9. In an embodiment of the present disclosure, operations 1610 to 1630 may be performed by the image processing device (100) or a processor of the image processing device (100).

[0188] In operation 1610, the image processing device (100) can obtain an input depth map from a two-dimensional input image.

[0189] For example, the image processing device (100) can obtain an input depth map from a two-dimensional input image through a depth map estimation module (310 of FIG. 3). If the depth map estimation module (310 of FIG. 3) is implemented as a neural network trained to output a depth map from a two-dimensional image, the neural network can receive a two-dimensional color image (e.g., an RGB image) as input and estimate a depth map.

[0190] In operation 1620, the image processing device (100) may perform first filtering on the input depth map. For example, the image processing device (100) may perform first filtering on the input depth map through the first filtering module (320 of FIG. 3). The first filtering may correspond to IIR (Infinite Impulse Response) filtering for reducing inter-frame shaking for an image including a moving object. IIR filtering may filter the depth map of the current frame using the filtered depth map of the previous frame.

[0191] More specifically, the first filtering may correspond to local IIR filtering that performs a weighted averaging using variable weights that have different weights for each boundary region and non-boundary region of the input depth map. For example, local IIR filtering may apply different weighted averages to boundary regions and non-boundary regions of the object when weighting the previous frame and the current frame of the depth map. Local IIR filtering is more effective in reducing ghosting at the boundary of a moving object than global IIR filtering that uses global weights (e.g., fixed weights in the entire region).

[0192] Here, the boundary region of the object may include, for successive frames, a disjoint region that was behind the object in the previous frame but newly appears due to the object's movement, and an occluded region that was visible in the previous frame but is obscured due to the object's movement. The disjoint region may be referred to as a second region, and the occluded region may be referred to as a third region.

[0193] Here, the non-boundary region of the object may include, for consecutive frames, a first region, which is an object region common to the current frame and the previous frame, and a fourth region, which is a background region other than the object common to the previous frame and the current frame.

[0194] Action 1620 may include actions 1623, 1625, and 1627.

[0195] In operation 1623, the image processing device (100) can detect a boundary area included in the input depth map.

[0196] For example, the image processing device (100) can detect the boundary area of ​​a moving object in the input depth map through the boundary area detection module (410 of FIG. 4). For example, the image processing device (100) can obtain first depth information (610 of FIG. 6) having depth values ​​of the boundary area (i.e., the second area and the third area) and the background area (i.e., the fourth area) based on the previous frame and the current frame through mathematical expression 1. The image processing device (100) can obtain second depth information (620 of FIG. 6) having depth values ​​of the boundary area (i.e., the second area and the third area) and the foreground area (i.e., the first area) based on the previous frame and the current frame through mathematical expression 2. The image processing device (100) can obtain third depth information (630 in FIG. 6) having a depth value of a boundary area (i.e., the second area and the third area) based on the first depth information (610 in FIG. 6) and the second depth information (620 in FIG. 6) through mathematical expression 3.

[0197] In operation 1625, the image processing device (100) can generate variable weights having different values ​​in the boundary area and the non-boundary area.

[0198] For example, the image processing device (100) can obtain variable weights having different values ​​for each area through a weight acquisition module (420 in FIG. 4).

[0199] For example, the image processing device (100) can obtain variable weights having different values ​​in the boundary area and the non-boundary area by scaling the third depth information (630 in FIG. 6) with a predetermined slope using mathematical expression 4 and limiting the upper and lower limits. The variable weights can have large values ​​in the boundary area and small values ​​in the non-boundary area. The variable weights are illustrated in the graph (700) of FIG. 7.

[0200] In operation 1627, the image processing device (100) can perform a weighted average between the previous frame and the current frame of the input depth map based on variable weights.

[0201] For example, the image processing device (100) can perform a weighted average between the current frame and the previous frame through the weighted average application module (430 of FIG. 4). The image processing device (100) can apply a larger weight to the current frame than to the previous frame in a boundary area and apply a larger weight to the previous frame than to the current frame in a non-boundary area by using a variable weight. For example, the image processing device (100) can obtain a first filtered depth map through mathematical expression 5. The image processing device (100) can update the first filtered depth map to the previous frame and perform first filtering on the next frame through mathematical expression 6. The image processing device (100) can generate a first filtered input depth map composed of a plurality of first filtered frames by performing the first filtering for each frame of the input depth map.

[0202] The first filtered depth map may include a boundary area with a high reflection ratio of the current frame and a non-boundary area with a high reflection ratio of the previous frame. In other words, the first filtered depth map may have the current frame reflected more significantly in the boundary area and the previous frame reflected more significantly in the non-boundary area.

[0203] An image processing device (100) according to one embodiment of the present disclosure may perform a weighted average of a current frame and a previous frame to reduce inter-frame shaking occurring in a depth map including a moving object. During the weighted average, the image processing device (100) may perform a weighted average using variable weights having different weights for each boundary area and non-boundary area to reduce afterimages occurring in the boundary area of ​​the object.

[0204] In operation 1630, the image processing device (100) can generate a three-dimensional image based on the first filtered depth map and the input image.

[0205] For example, the image processing device (100) can generate a three-dimensional image through at least one of a binocular view generation module (340 in FIG. 3), a hole filling module (350 in FIG. 3), or a binocular view combination module (360 in FIG. 3).

[0206] For example, the image processing device (100) can generate a 3D image for the left eye and a 3D image for the right eye based on the first filtered depth map and the 2D image through the binocular view generation module (340 in FIG. 3).

[0207] For example, when a hole occurs in a frame as a pixel moves through a hole filling module (350 in FIG. 3), the image processing device (100) can perform an operation of interpolating the hole using pixels adjacent to the hole.

[0208] For example, the image processing device (100) can generate a 3D image by combining a 3D image for the left eye and a 3D image for the right eye through a binocular viewpoint combining module (360 in FIG. 3).

[0209] An image processing device (100) according to one embodiment of the present disclosure can generate a depth map in which inter-frame shaking is reduced and afterimages occurring in the boundary area of ​​an object are reduced through first filtering. Since the image processing device (100) generates a three-dimensional image using the first filtered depth map, the three-dimensional effect between frames of the three-dimensional image can be improved.

[0210] FIG. 17 is a flowchart illustrating a method for generating a three-dimensional image by performing a second filtering on a depth map by an image processing device according to an embodiment of the present disclosure. For convenience of explanation, FIG. 16 is described with reference to FIGS. 10 to 15. In an embodiment of the present disclosure, operations 1710 to 1730 may be performed by the image processing device (100) or a processor of the image processing device (100).

[0211] In operation 1710, the image processing device (100) can obtain an input depth map from a two-dimensional input image. Operation 1710 may correspond to operation 1610 of FIG. 16.

[0212] At operation 1720, the image processing device (100) may perform second filtering on the input depth map.

[0213] For example, the image processing device (100) can perform second filtering on the input depth map through the second filtering module (330 of FIG. 3). The second filtering can implement a change in depth inside an object by blurring the foreground area. The second filtering module (330) can implement a change in depth inside an object and distinguish the boundary between the foreground area and the background area by generating a depth map to which different blurring processes are applied to the foreground area and the background area. More specifically, the second filtering can improve the three-dimensionality inside an object by applying blurring to the foreground area including the object. The second filtering can prevent the boundary between the inside of the object (i.e., the foreground area) and the outside of the object (i.e., the background area) from disappearing due to blurring by not applying blurring to the background area. Here, the foreground area can represent an area including an object (or an object of interest) in the depth map. Here, the background area can represent an area other than the object in the depth map.

[0214] Action 1720 may include actions 1723, 1725, and 1727.

[0215] In operation 1723, the image processing device (100) can generate a first depth map by performing blur processing on the input depth map.

[0216] For example, the image processing device (100) can perform blur processing on an input depth map through a depth map blur processing module (1010 of FIG. 10). The image processing device (100) can generate a first depth map (1110 of FIG. 11), which is a blurred depth map.

[0217] In operation 1725, the image processing device (100) can generate a second depth map by performing a maximum value operation on the first depth map and the input depth map.

[0218] For example, the image processing device (100) can perform a maximum value operation on the first depth map and the input depth map through the maximum value operation module (1020 of FIG. 10). For example, the image processing device (100) can compare the depth value before blur processing and the depth value after blur processing through mathematical expression 7, and selectively reflect a larger value for each area. For example, in the foreground area, the depth value of the first depth map (1120) may be greater than the depth value of the input depth map (1110). In the background area, the depth value of the input depth map (1110) may be greater than the depth value of the first depth map (1120). Accordingly, the second depth map (1130 of FIG. 11) may have a foreground area in which the depth value of the first depth map (1120) is reflected, and a background area in which the depth value of the input depth map (1110) is reflected. For example, the second depth map (1130) may include a foreground area having a depth value after blurring and a background area having a depth value before blurring.

[0219] The second depth map (1130) can implement depth changes within an object through a foreground area having depth values ​​after blurring. The second depth map (1130) can distinguish the boundary between the inside and outside of an object through a background area having depth values ​​before blurring.

[0220] The depth values ​​for each region of the input depth map (1110), the first depth map (1120), and the second depth map (1130) are shown in the graphs (1210) and (1220) of FIG. 12.

[0221] In operation 1727, the image processing device (100) can perform a weighted average on the second depth map and the input depth map by applying the input depth map as a weight.

[0222] For example, the image processing device (100) can perform a weighted average on the second depth map (1130) and the input depth map (1110) by applying the input depth map (1110) as a weight through a weighted average application module (1330 of FIG. 13). The input depth map (1110) used as a variable weight can have a large value in a foreground area and a small value in a background area. The image processing device (100) can apply a large weight to the second depth map (1130) in the foreground area and apply a large weight to the input depth map (1110) in the background area.

[0223] In the third depth map (1410 of FIG. 14), which is the second filtered depth map, blurring may be reflected in the foreground area, while blurring may not be reflected in the background area. Accordingly, when an object other than the object of interest (e.g., another object located behind a human face) is located in the background area, the problem of the boundary disappearing due to blurring can be prevented.

[0224] In operation 1730, the image processing device (100) can generate a three-dimensional image based on the second filtered depth map and the input image. For example, the image processing device (100) can generate a left-eye three-dimensional image and a right-eye three-dimensional image based on the second filtered depth map and the two-dimensional image through a binocular view generation module (340 of FIG. 3). The image processing device (100) can interpolate holes through a hole filling module (350 of FIG. 3). The image processing device (100) can generate a three-dimensional image by combining the left-eye three-dimensional image and the right-eye three-dimensional image through a binocular view combining module (360 of FIG. 3).

[0225] An image processing device (100) according to one embodiment of the present disclosure can implement a change in depth within an object through second filtering and generate a depth map that distinguishes the boundary between the inside of the object and the outside of the object. Since the image processing device (100) generates a three-dimensional image using the second filtered depth map, the three-dimensional effect between the object boundaries of the three-dimensional image can be improved.

[0226] FIG. 18 is a flowchart for explaining an operation method of an image processing device according to an embodiment of the present disclosure. FIG. 19 is a diagram for explaining an operation of an image processing device according to an embodiment of the present disclosure for filtering an input depth map through a first filtering module and a second filtering module. In an embodiment of the present disclosure, operations 1810 to 1840 may be performed by the image processing device (100) or a processor of the image processing device (100). However, the operation method of the image processing device (100) is not limited to that illustrated in FIG. 18, and any one of the steps illustrated in FIG. 18 may be omitted, or steps not illustrated in FIG. 18 may be further included. The depth map filtering module (1900) illustrated in FIG. 19 may be implemented as software such as commands, an algorithm, a data structure, or a program code for sequentially applying the first filtering module (320) and the second filtering module (330). The second filtering module (330) illustrated in FIG. 19 can be applied identically to the second filtering module (330) of FIG. 10 or the second filtering module (330a) of FIG. 13, except that the depth map input to the second filtering module (330) is a first filtered depth map.

[0227] Referring to FIG. 18, in operation 1810, the image processing device (100) can obtain an input depth map from a two-dimensional input image. Operation 1810 may correspond to operation 1610 of FIG. 16.

[0228] In operation 1820, the image processing device (100) may perform first filtering on the input depth map.

[0229] The image processing device (100) can perform first filtering on the input depth map through the first filtering module (320) to generate a first filtered depth map.

[0230] For example, the image processing device (100) can detect a boundary area included in an input depth map through a boundary area detection module (410 in FIG. 4). For example, the image processing device (100) can generate variable weights having different values ​​in boundary areas and non-boundary areas through a weight acquisition module (420 in FIG. 4). For example, the image processing device (100) can perform a weighted average between a previous frame and a current frame of the input depth map based on the variable weights through a weighted average application module (430 in FIG. 4).

[0231] The first filtered depth map can be passed to the second filtering module (330).

[0232] In operation 1830, the image processing device (100) can perform second filtering on the first filtered depth map.

[0233] The image processing device (100) can perform second filtering on the first filtered depth map through the second filtering module (330) to generate a second filtered depth map.

[0234] For example, the image processing device (100) can generate a first depth map by performing blur processing on the first filtered depth map through a depth map blur processing module (1010 of FIG. 10). For example, the image processing device (100) can generate a second depth map by performing an optional maximum value operation on the first depth map and the first filtered depth map through a maximum value operation module (1020 of FIG. 10). For example, the image processing device (100) can perform a weighted average on the second depth map and the first filtered depth map by applying the first filtered depth map as a weight through a weighted average application module (1330 of FIG. 13).

[0235] The second filtered depth map can be passed as an output depth map to the binocular view generation module (340 in FIG. 3).

[0236] In operation 1840, the image processing device (100) can generate a three-dimensional image based on the second filtered depth map and the input image. The image processing device (100) can generate the three-dimensional image through at least one of a binocular view generation module (340 of FIG. 3), a hole filling module (350 of FIG. 3), or a binocular view combination module (360 of FIG. 3).

[0237] An image processing device (100) according to one embodiment of the present disclosure can generate a depth map in which inter-frame shaking is reduced and afterimages occurring in the boundary area of ​​an object are reduced through first filtering. Since the image processing device (100) generates a three-dimensional image using the first filtered depth map, the three-dimensional effect between frames of the three-dimensional image can be improved.

[0238] An image processing device (100) according to one embodiment of the present disclosure can implement a change in depth within an object through second filtering and generate a depth map that distinguishes the boundary between the inside of the object and the outside of the object. Since the image processing device (100) generates a three-dimensional image using the second filtered depth map, the three-dimensional effect between the object boundaries of the three-dimensional image can be improved.

[0239] FIG. 20 is a block diagram of an image processing device according to one embodiment of the present disclosure.

[0240] Referring to FIG. 20, an image processing device (100) according to an embodiment of the present disclosure may include a processor (110), a camera (110), a display (130), and a memory (140). However, not all of the components illustrated in FIG. 20 are essential components. The image processing device (100) may be implemented with more components than the components illustrated in FIG. 20, or may be implemented with fewer components. In addition, each component of the image processing device (100) may be configured as an internal component or may be configured as an external component operatively connected to the image processing device.

[0241] The processor (110) controls the overall operation of the image processing device (100). For example, the processor (110) may perform the functions of the image processing device (100) described in the present disclosure by executing one or more instructions stored in the memory (140). In this case, the memory (140) may store one or more instructions executable by the processor (110). In addition, the processor (110) may store one or more instructions in an internally provided memory, and execute one or more instructions stored in the internally provided memory to control the above-described operations to be performed. That is, the processor (110) may perform a predetermined operation by executing at least one instruction or program stored in the internal memory provided in the processor (110) or the memory (140).

[0242] The processor (110) may be configured with at least one of a central processing unit, a microprocessor, a graphic processing unit, an application processor (AP), an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), a field programmable gate array (FPGAs), and a neural processing unit or an artificial intelligence (AI) processor designed with a hardware structure specialized for learning and processing an artificial intelligence (AI) model, but is not limited thereto.

[0243] The camera (120) can track the user's gaze under the control of the processor (110). The camera (120) can track the user's eye movement and detect the user's gaze information, such as the direction of the user's eyes and the position of the user's pupils.

[0244] The display (130) can display information or images according to the received image data under the control of the processor (110). For example, the display (130) can display execution screen information of an application program driven by the image processing device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to the execution screen information.

[0245] The display (130) may include a 3D display for displaying a 3D image. The display (130) may display the 3D image in a stereoscopic, auto-stereoscopic, projection, or holographic manner. The stereoscopic method is a method of filtering a desired image through polarization-based division, time division, wavelength division with different wavelengths of primary colors, etc. The auto-stereoscopic method is a method of making an image visible at a specific viewpoint in space by using 3D optical elements such as a parallax barrier, a lenticular lens, or a directional backlight unit. The display (130) displays color images of different viewpoints to both eyes of the user through the above-described method, and the user can perceive the three-dimensional effect of the 3D image.

[0246] The memory (140) can store instructions, algorithms, data structures, program codes, and application programs for processing and controlling the processor (110), and can store data input to or output from the image processing device (100). The memory (140) can include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, a flash ROM, a hard disk drive (HDD), or a solid state drive (SSD). A program (one or more instructions) or application stored in memory (140) can be executed by the processor (110).

[0247] In one embodiment of the present disclosure, the memory (140) may store various types of modules that can be used to generate a three-dimensional image based on a two-dimensional image. The memory (140) may store a depth map estimation module (310), a first filtering module (320), a second filtering module (330), a binocular viewpoint generation module (340), a hole filling module (350), and a binocular image combination module (360). The memory (140) may further store a boundary area detection module (410), a weight acquisition module (420), and a weighted average application module (430) that constitute the first filtering module (320). The memory (140) may further store a depth map blur processing module (1010), a maximum value calculation module (1020), and a weighted average application module (1330) that constitute the second filtering module (330). A 'module' included in the memory (140) means a unit that processes a function or operation performed by the processor (110), and this can be implemented as software such as commands, algorithms, data structures, or program codes.

[0248] In one embodiment of the present disclosure, the processor (110) can obtain an input depth map from a two-dimensional input image by executing one or more instructions included in the depth map estimation module (310).

[0249] In one embodiment of the present disclosure, the processor (110) may perform first filtering on an input depth map by executing one or more instructions included in the first filtering module (320). The processor (110) may perform weighted averaging using variable weights having different weights for each boundary area and non-boundary area of ​​the input depth map by executing one or more instructions included in the first filtering module (320).

[0250] In one embodiment of the present disclosure, the processor (110) can detect a boundary region of an object in an input depth map by executing one or more instructions included in a boundary region detection module (410).

[0251] In one embodiment of the present disclosure, the processor (110) may obtain variable weights having different values ​​for each region by executing one or more instructions included in the weight acquisition module (420). The variable weights may have large values ​​in boundary regions and small values ​​in non-boundary regions.

[0252] In one embodiment of the present disclosure, the processor (110) may perform a weighted average between a current frame and a previous frame by executing one or more instructions included in the weighted average application module (430). The processor (110) may apply a large weight to the current frame in a boundary region and apply a large weight to the previous frame in a non-boundary region by executing one or more instructions included in the weighted average application module (430) using variable weights.

[0253] In one embodiment of the present disclosure, the processor (110) may perform second filtering on an input depth map by executing one or more instructions included in the second filtering module (330). The processor (110) may generate a depth map to which blur processing is applied differently to a foreground area and a background area by executing one or more instructions included in the second filtering module (330).

[0254] In one embodiment of the present disclosure, the processor (110) can generate a first depth map by performing blur processing on an input depth map by executing one or more instructions included in a depth map blur processing module (1010).

[0255] In one embodiment of the present disclosure, the processor (110) can generate a second depth map by performing a maximum operation on the first depth map and the input depth map by executing one or more instructions included in the maximum operation module (1020).

[0256] In one embodiment of the present disclosure, the processor (110) can perform a weighted average on the second depth map and the input depth map by applying a weight to the input depth map by executing one or more instructions included in the weighted average application module (1330).

[0257] In one embodiment of the present disclosure, the processor (110) can generate a left-eye 3D image and a right-eye 3D image based on a first filtered or second filtered depth map and a 2D image by executing one or more instructions included in the binocular view generation module (340).

[0258] In one embodiment of the present disclosure, the processor (110) may perform an operation of interpolating a hole using pixels adjacent to the hole when a hole occurs within a frame as a pixel moves by executing one or more instructions included in the hole filling module (350).

[0259] In one embodiment of the present disclosure, the processor (110) can generate a 3D image by combining a 3D image for the left eye and a 3D image for the right eye by executing one or more instructions included in the binocular image combining module (360).

[0260] In one embodiment of the present disclosure, an image processing device may be provided. The image processing device may include a memory including at least one processor and one or more storage media storing one or more commands. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to obtain an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary region and a non-boundary region of an object. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to perform a first filtering by applying different weights to the boundary region and the non-boundary region for a first frame and a previous frame of the input depth map, in order to obtain a first filtered depth map. The at least one processor individually or in combination executes the one or more commands, thereby causing the image processing device to generate a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

[0261] In one embodiment of the present disclosure, when performing the first filtering, the image processing device can detect the boundary area included in the input depth map by having the at least one processor individually or in combination execute the one or more commands. The image processing device can perform a weighted average between the previous frame and the first frame by having the at least one processor individually or in combination execute the one or more commands, and can apply a greater weight to the first frame than to the previous frame in the boundary area, and can apply a greater weight to the previous frame than to the first frame in the non-boundary area.

[0262] In one embodiment of the present disclosure, in detecting the boundary area included in the input depth map, the image processing device may obtain first depth information having depth values ​​of the boundary area and the background area based on the previous frame and the first frame, obtain second depth information having depth values ​​of the boundary area and the foreground area based on the previous frame and the first frame, and obtain third depth information having depth values ​​of the boundary area based on the first depth information and the second depth information, by having the at least one processor individually or in combination execute the one or more commands.

[0263] In one embodiment of the present disclosure, in performing the first filtering, the image processing device scales the third depth information with a predetermined slope by the at least one processor individually or in combination executing the one or more commands, and obtains a variable weight having a first weight value in the boundary area and a second weight value in the non-boundary area by limiting an upper limit value and a lower limit value, and the first weight value in the boundary area may be greater than the second weight value in the non-boundary area.

[0264] In one embodiment of the present disclosure, the boundary region of the first filtered depth map may be characterized in that the first frame is reflected to a greater extent than the previous frame, and the non-boundary region of the first filtered depth map may be characterized in that the previous frame is reflected to a greater extent than the first frame.

[0265] In one embodiment of the present disclosure, the first filtered depth map includes a foreground area and a background area, and the image processing device can perform second filtering such that the foreground area of ​​the first filtered depth map and the background area of ​​the first filtered depth map are differently blurred by having the at least one processor individually or in combination execute the one or more commands.

[0266] In one embodiment of the present disclosure, in performing the second filtering, the image processing device may generate a first depth map through blur processing on the first filtered depth map by having the at least one processor individually or in combination execute the one or more commands, and generate a second depth map having a foreground area corresponding to the first depth map and a background area corresponding to the first filtered depth map based on a maximum value operation between the first depth map and the first filtered depth map.

[0267] In one embodiment of the present disclosure, in performing the second filtering, the image processing device may perform a weighted average between the second depth map and the first filtered depth map by having the at least one processor individually or in combination execute the one or more commands, and may apply a greater weight to the second depth map than to the first filtered depth map in the foreground area and may apply a greater weight to the first filtered depth map than to the second depth map in the background area.

[0268] In one embodiment of the present disclosure, in generating a first depth map through blur processing on the first filtered depth map, the image processing device may generate a mipmap, which is a block unit composed of a sample pixel referenced for blur processing on a target pixel and pixels adjacent to the sample pixel, by having the at least one processor individually or in combination execute the one or more commands, and perform depth map blur processing using the target pixel and the mipmap.

[0269] In one embodiment of the present disclosure, in generating the three-dimensional image, the image processing device can generate a left-eye three-dimensional image and a right-eye three-dimensional image based on the first filtered depth map and the two-dimensional input image by having the at least one processor individually or in combination execute the one or more commands, perform hole filling on the left-eye three-dimensional image and the right-eye three-dimensional image, and combine the left-eye three-dimensional image and the right-eye three-dimensional image, thereby generating a binocular three-dimensional image.

[0270] In one embodiment of the present disclosure, an operating method of an image processing device is provided, including the steps of: obtaining an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary region and a non-boundary region of an object; performing first filtering by applying different weights to the boundary region and the non-boundary region for a first frame and a previous frame of the input depth map to obtain a first filtered depth map; and generating a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

[0271] In one embodiment of the present disclosure, the step of performing the first filtering may include the step of detecting the boundary area included in the input depth map, and the step of performing a weighted average between the previous frame and the first frame, applying a greater weight to the first frame than to the previous frame in the boundary area, and applying a greater weight to the previous frame than to the first frame in the non-boundary area.

[0272] In one embodiment of the present disclosure, the step of detecting the boundary area included in the input depth map may include the step of obtaining first depth information having depth values ​​of the boundary area and the background area based on the previous frame and the first frame, the step of obtaining second depth information having depth values ​​of the boundary area and the foreground area based on the previous frame and the first frame, and the step of obtaining third depth information having depth values ​​of the boundary area based on the first depth information and the second depth information.

[0273] In one embodiment of the present disclosure, the step of performing the first filtering further includes the step of obtaining a variable weight having a first weight value in the boundary area and a second weight value in the non-boundary area by scaling the third depth information with a predetermined slope and limiting an upper limit value and a lower limit value, wherein the first weight value in the boundary area may be greater than the second weight value in the non-boundary area.

[0274] In one embodiment of the present disclosure, the boundary area of ​​the first filtered depth map may reflect the first frame to a greater extent than the previous frame, and the non-boundary area of ​​the first filtered depth map may reflect the previous frame to a greater extent than the first frame.

[0275] In one embodiment of the present disclosure, the first filtered depth map includes a foreground area and a background area, and the method may further include a step of performing a second filtering such that the foreground area of ​​the first filtered depth map and the background area of ​​the first filtered depth map are blurred differently.

[0276] In one embodiment of the present disclosure, the step of performing the second filtering may include the step of generating a first depth map by blurring the first filtered depth map, and the step of generating a second depth map having a foreground area corresponding to the first depth map and a background area corresponding to the first filtered depth map based on a maximum value operation between the first depth map and the first filtered depth map.

[0277] In one embodiment of the present disclosure, the step of performing the second filtering may include the step of performing a weighted average between the second depth map and the first filtered depth map, applying a greater weight to the second depth map than to the first filtered depth map in the foreground area, and applying a greater weight to the first filtered depth map than to the second depth map in the background area.

[0278] In one embodiment of the present disclosure, the step of generating a first depth map through blur processing on the first filtered depth map may include the step of generating a mipmap, which is a block unit composed of a sample pixel referenced for blur processing on a target pixel and pixels adjacent to the sample pixel, and the step of performing depth map blur processing using the target pixel and the mipmap.

[0279] In one embodiment of the present disclosure, in generating the three-dimensional image, the image processing device can generate a left-eye three-dimensional image and a right-eye three-dimensional image based on the first filtered depth map and the two-dimensional input image by having the at least one processor individually or in combination execute the one or more commands, perform hole filling on the left-eye three-dimensional image and the right-eye three-dimensional image, and combine the left-eye three-dimensional image and the right-eye three-dimensional image, thereby generating a binocular three-dimensional image.

[0280] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing the method on a computer may be provided.

[0281] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0282] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the image processing device, At least one processor comprising a processing circuit; and A memory comprising one or more storage media storing one or more instructions, The image processing device, wherein the at least one processor individually or collectively executes the one or more instructions, Obtaining an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary area and a non-boundary area of ​​an object, To obtain a first filtered depth map, a first filtering is performed by applying different weights to the boundary area and the non-boundary area for the first frame and the previous frame of the input depth map, An image processing device that generates a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

2. In paragraph 1, In performing the above first filtering, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, Detecting the boundary area included in the input depth map, An image processing device that performs a weighted average between the previous frame and the first frame, applying a greater weight to the first frame than to the previous frame in the boundary area, and applying a greater weight to the previous frame than to the first frame in the non-boundary area.

3. In paragraph 2, In detecting the boundary area included in the input depth map, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, Based on the previous frame and the first frame, first depth information having depth values ​​of the boundary area and the background area is obtained, Based on the previous frame and the first frame, second depth information having depth values ​​of the boundary area and the foreground area is obtained, An image processing device that obtains third depth information having a depth value of the boundary area based on the first depth information and the second depth information.

4. In paragraph 3, In performing the above first filtering, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, By scaling the third depth information with a predetermined slope and limiting the upper and lower limits, a variable weight having a first weight value in the boundary area and a second weight value in the non-boundary area is obtained. An image processing device, wherein the first weight value in the boundary area is greater than the second weight value in the non-boundary area.

5. In any one of paragraphs 1 to 4, An image processing device, characterized in that the boundary area of ​​the first filtered depth map reflects the first frame to a greater extent than the previous frame, and the non-boundary area of ​​the first filtered depth map reflects the previous frame to a greater extent than the first frame.

6. In any one of paragraphs 1 to 5, The above first filtered depth map includes a foreground area and a background area, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, An image processing device that performs second filtering so that the foreground area of ​​the first filtered depth map and the background area of ​​the first filtered depth map are blurred differently.

7. In paragraph 6, In performing the above second filtering, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, Generating a first depth map by blurring the first filtered depth map, An image processing device that generates a second depth map having a foreground area corresponding to the first depth map and a background area corresponding to the first filtered depth map based on a maximum value operation between the first depth map and the first filtered depth map.

8. In paragraph 7, In performing the above second filtering, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, An image processing device that performs a weighted average between the second depth map and the first filtered depth map, applying a greater weight to the second depth map than to the first filtered depth map in the foreground area, and applying a greater weight to the first filtered depth map than to the second depth map in the background area.

9. In paragraph 7, In generating a first depth map by blurring the first filtered depth map, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, Generate a mipmap, which is a block unit composed of a sample pixel referenced for blur processing for a target pixel and pixels adjacent to the sample pixel, An image processing device that performs depth map blur processing using the target pixel and the mipmap.

10. In any one of paragraphs 1 to 9, In creating the above three-dimensional image, The image processing device, wherein the at least one processor individually or in combination executes the one or more commands, Based on the first filtered depth map and the two-dimensional input image, a three-dimensional image for the left eye and a three-dimensional image for the right eye are generated, Perform hole filling for the 3D image for the left eye and the 3D image for the right eye, An image processing device that generates a binocular 3D image by combining the 3D image for the left eye and the 3D image for the right eye.

11. In the operating method of the image processing device, A step of obtaining an input depth map from a two-dimensional input image, wherein the input depth map includes a boundary area and a non-boundary area of ​​an object; To obtain a first filtered depth map, a step of performing a first filtering by applying different weights to the boundary area and the non-boundary area for the first frame and the previous frame of the input depth map; and A method comprising the step of generating a three-dimensional image based on the first filtered depth map and the two-dimensional input image.

12. In paragraph 11, The step of performing the above first filtering is: A step of detecting the boundary area included in the input depth map; and A method comprising the step of performing a weighted average between the previous frame and the first frame, applying a greater weight to the first frame than to the previous frame in the boundary area, and applying a greater weight to the previous frame than to the first frame in the non-boundary area.

13. In paragraph 12, The step of detecting the boundary area included in the input depth map is: A step of obtaining first depth information having depth values ​​of the boundary area and the background area based on the previous frame and the first frame; A step of obtaining second depth information having depth values ​​of the boundary area and the foreground area based on the previous frame and the first frame; and A method comprising the step of obtaining third depth information having a depth value of the boundary area based on the first depth information and the second depth information.

14. In paragraph 13, The step of performing the above first filtering is: Further comprising a step of obtaining a variable weight having a first weight value in the boundary area and a second weight value in the non-boundary area by scaling the third depth information with a predetermined slope and limiting the upper and lower limits, A method wherein the first weight value in the boundary region is greater than the second weight value in the non-boundary region.

15. A computer-readable recording medium having recorded thereon a program for performing the method of Article 11 on a computer.

Images