Method and apparatus of improving depth map

By forming a point cloud and applying feature embedding vectors to improve depth maps, the method addresses inconsistencies across viewpoints, enhancing depth map accuracy for accurate 3D reconstruction.

KR103046790B1Active Publication Date: 2026-11-11SAMSUNG ELECTRONICS CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
KR1020210130985
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-10-01
Publication Date
2026-11-11
Estimated Expiration
2041-10-01

AI Technical Summary

Technical Problem

Neural networks often produce inconsistent depth maps across different viewpoints due to independent estimation of depth values, leading to distorted object shapes.

Method used

A method involving forming a point cloud from multiple viewpoint depth maps, extracting feature embedding vectors, generating a two-dimensional feature map, obtaining a residual depth map through a depth improvement network, and combining it with the original depth map to improve consistency across viewpoints.

Benefits of technology

The method enhances depth map consistency and accuracy by adjusting shift, scale, and image quality, allowing for accurate 3D reconstruction of objects from multiple viewpoints.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112021113325215-PAT00033_ABST
    Figure 112021113325215-PAT00033_ABST
Patent Text Reader

Abstract

According to one embodiment, a method for improving a depth map involves back-projecting multiple depth maps into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the multiple depth maps, extracting feature embedding vectors corresponding to points in the point cloud, generating a two-dimensional feature map of a specific viewpoint based on the feature embedding vectors of points corresponding to a specific viewpoint among the points in the point cloud, obtaining a residual depth map by applying the two-dimensional feature map to a depth improvement network, and outputting an improved depth map by combining the residual depth map with the depth map of a specific viewpoint among the multiple depth maps.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The following disclosure relates to a method and apparatus for improving a depth map. Background Technology

[0002] Neural networks can be used to extract low-dimensional feature vectors of images containing the intensities of individual raw pixels, thereby identifying the objects contained within the image. For example, when using a network that takes an image from a single viewpoint as input to estimate a depth map, inconsistent results may occur between viewpoints because depth values ​​are estimated independently for each image. For instance, if there is a discrepancy in depth information between the depth map of a specific viewpoint where an arbitrary object was captured and the depth map of another viewpoint, the shape of the object may be distorted. means of solving the problem

[0003] According to one embodiment, a method for improving a depth map includes: forming a point cloud corresponding to the multiple viewpoint depth maps by un-projecting the multiple viewpoint depth maps into a three-dimensional space using camera parameters corresponding to each viewpoint; extracting feature embedding vectors corresponding to points of the point cloud; generating a two-dimensional feature map of a specific viewpoint based on feature embedding vectors of points corresponding to a specific viewpoint among the points of the point cloud; obtaining a residual depth map by applying the two-dimensional feature map to a depth improvement network; and outputting an improved depth map by combining the residual depth map with the depth map of the specific viewpoint among the multiple viewpoint depth maps.

[0004] The method for improving the depth map may further include the operation of estimating the depth maps of multiple viewpoints by applying an input image containing multiple frames of the same object captured at different viewpoints to a network that estimates the depth map of a single viewpoint.

[0005] The above input image may include RGB images from multiple viewpoints.

[0006] The operation of extracting the above feature embedding vectors may include the operation of fetching features corresponding to the location of each point in the point cloud from the feature map of the last layer of the network that estimates the depth map at the single time point, and the operation of combining the fetched features with each point at the corresponding location in the point cloud to extract the above feature embedding vectors.

[0007] The operation of generating the above two-dimensional feature map may include the operation of generating the two-dimensional feature map of the specific point in time by projecting the feature embedding vectors onto the image plane of the specific point in time.

[0008] The operation of acquiring the above residual depth map may include the operation of acquiring the above residual depth map by applying the above input image and the above 2D feature map to a network that improves the depth.

[0009] The operation of acquiring the above residual depth map may include the operation of acquiring the above residual depth map by adding the channels of the input image to the channels of the above 2D feature map and applying them to a network that improves the depth.

[0010] The operation of outputting the improved depth map may include the operation of combining the residual depth map with the depth map of the specific point in time among the depth maps of the multiple point in time, and the operation of outputting the improved depth map by adjusting at least one of the shift, scale, and image quality of the combined depth maps.

[0011] The method for improving the depth map may further include, for the improved depth map, the operation of forming the point cloud, the operation of extracting the feature embedding vectors, the operation of generating the two-dimensional feature map, the operation of obtaining the residual depth map, and the operation of outputting the operation repeatedly according to a set number of repetitions.

[0012] The method for improving the depth map may further include the operation of reconstructing a 2D object corresponding to the depth maps of multiple viewpoints into a 3D object based on the improved depth map.

[0013] The operation of outputting the improved depth map may include the operation of combining the residual depth map with the depth map of the specific point in time among the depth maps of multiple point in time, and the operation of outputting the improved depth map by adjusting at least one of the shift, scale, and image quality of the combined depth maps.

[0014] The method for improving the depth map may further include, for the improved depth map, the operation of forming the point cloud, the operation of extracting the feature embedding vectors, the operation of generating the two-dimensional feature map, the operation of obtaining the residual depth map, and the operation of outputting the operation repeatedly according to a set number of repetitions.

[0015] The method for improving the depth map may further include the operation of reconstructing a 2D object corresponding to the depth maps of multiple viewpoints into a 3D object based on the improved depth map.

[0016] According to one embodiment, a device for improving a depth map includes a sensor for acquiring depth maps of multiple viewpoints, a processor for back-projecting the depth maps of multiple viewpoints into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the depth maps of multiple viewpoints, extracting feature embedding vectors corresponding to points of the point cloud, generating a two-dimensional feature map of a specific viewpoint based on the feature embedding vectors of points corresponding to a specific viewpoint among the points of the point cloud, acquiring a residual depth map by applying the two-dimensional feature map to a depth improvement network, and generating an improved depth map by combining the residual depth map with the depth map of the specific viewpoint among the depth maps of multiple viewpoints, and a communication interface for outputting the improved depth map.

[0017] The communication interface receives an input image containing multiple frames of the same object captured at different viewpoints, and the processor can estimate the depth maps at multiple viewpoints by applying the input image to a network that estimates a depth map at a single viewpoint.

[0018] The processor can patch features corresponding to the location of each point in the point cloud from the feature map of the last layer of the network that estimates the depth map of the single time point, and combine the patched features with each point at the corresponding location in the point cloud to extract the feature embedding vectors.

[0019] The processor can obtain the residual depth map by applying the input image and the 2D feature map to a depth-improving network.

[0020] The processor can repeatedly perform the operation of forming the point cloud, the operation of extracting the feature embedding vectors, the operation of generating the 2D feature map, the operation of obtaining the residual depth map, and the operation of outputting the improved depth map according to a set number of repetitions.

[0021] Based on the improved depth map, the processor can reconstruct a 2D object corresponding to the depth maps of multiple viewpoints into a 3D object. Brief explanation of the drawing

[0022] FIG. 1 is a flowchart illustrating a method for improving a depth map according to one embodiment. FIG. 2 is a diagram illustrating the structure of a device for improving a depth map according to one embodiment. FIG. 3 is a diagram illustrating the iterative operation of a device for improving the depth map shown in FIG. 2. FIG. 4 is a diagram illustrating the structure of a device for improving a depth map according to another embodiment. FIG. 5 is a diagram illustrating the iterative operation of a device for improving the depth map shown in FIG. 4. FIG. 6 is a diagram illustrating the operation when the number of iterations is set to 2 in a device for improving a depth map shown in FIG. 4. FIG. 7 is a diagram illustrating the structure of a device for improving a depth map according to another embodiment. FIG. 8 is a flowchart illustrating a method for improving a depth map according to another embodiment. FIG. 9 is a block diagram of a device for improving a depth map according to one embodiment. Specific details for implementing the invention

[0023] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0024] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0025] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0026] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, actions, functions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, actions, functions, components, parts, or combinations thereof.

[0027] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0028] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0030] FIG. 1 is a flowchart illustrating a method for improving a depth map according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0031] Referring to FIG. 1, a device for improving a depth map according to one embodiment (hereinafter referred to as the 'improvement device') can output an improved depth map through operations 110 to 150.

[0032] The improvement device may be, for example, any one of the improvement device (200) of FIG. 2, the improvement device (400) of FIG. 4, the improvement device (700) of FIG. 7, and the improvement device (900) of FIG. 9, but is not necessarily limited thereto.

[0033] In operation 110, the improvement device un-projects the depth maps of multiple viewpoints into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the depth maps of multiple viewpoints.

[0034] Multi-viewpoint depth maps may correspond, for example, to depth maps of the same object captured from different viewpoints, that is, to a set of single-viewpoint depth maps captured from different viewpoints. Multi-viewpoint depth maps may be acquired, for example, by a depth sensor, a LiDAR sensor, or a depth camera, but are not necessarily limited thereto. Camera parameters may be parameters of the sensor or camera that captured each single-viewpoint depth map. Camera parameters may include, for example, at least one of intrinsic parameters and extrinsic parameters.

[0035] According to an embodiment, the improvement device may estimate depth maps of multiple viewpoints by applying an input image containing multiple frames of the same object captured at different viewpoints to a network that estimates a depth map of a single viewpoint (e.g., the network that estimates a depth map of a single viewpoint (405) of FIG. 4). The input image may include, for example, RGB images of multiple viewpoints. The input image may be a two-dimensional image. The network that estimates a depth map of a single viewpoint may, for example, estimate a depth map of a first viewpoint corresponding to a video frame when a video frame of object A captured at a first viewpoint is applied, and estimate a depth map of a second viewpoint corresponding to a video frame when a video frame of object A captured at a second viewpoint is applied. In this way, multiple depth maps estimated by the network that estimates a depth map of a single viewpoint for multiple video frames of object A at multiple viewpoints may be combined to form depth maps of multiple viewpoints. The network that estimates a depth map of a single viewpoint may be, for example, a U-Net, but is not necessarily limited thereto.

[0036] In operation 110, the improvement device can calculate the locations of 3D points corresponding to pixels of 2D objects in an image by back-projecting multi-view depth maps into 3D space using, for example, camera parameters. The improvement device can form a point cloud corresponding to multi-view depth maps based on the locations of the 3D points. At this time, the points (P) included in the point cloud may include 3D location information such as, for example, P(x,y,z).

[0037] In operation 120, the improvement device extracts feature embedding vectors corresponding to the points of the point cloud formed in operation 110. The improvement device may extract feature embedding vectors using a network including, for example, Multi-Layer Perceptrons (MLPs) and a pooling layer (e.g., Point Net (225) of FIG. 2 or Point Net (425) of FIG. 4). For example, if multiple viewpoint depth maps are estimated from an input image (e.g., input image (401) of FIG. 4) in operation 110, the improvement device may fetch features corresponding to the location of each point of the point cloud (e.g., Point Cloud (420) of FIG. 4) from the feature map of the last layer of the network (405) that estimates a single viewpoint depth map, such as feature fetching (427) of FIG. 4. In one embodiment, obtaining features corresponding to the location of each point in the point cloud from the feature map of the last layer of a network that estimates a depth map at a single point in time can be called 'feature fetching'. At this time, the last layer of the network may correspond, for example, to the layer immediately before the network (405) outputs the depth map, but is not necessarily limited thereto. The improvement device can extract feature embedding vectors by combining the patched features with each point at the corresponding location in the point cloud.

[0038] In operation 130, the enhancement device generates a 2D feature map of a specific point in time based on feature embedding vectors of points in the point cloud that correspond to a specific point in time. The enhancement device can generate a 2D feature map of a specific point in time by projecting the feature embedding vectors onto an image plane of the specific point in time.

[0039] The enhancement device can generate a 2D feature map containing features equal to the dimension of the feature vectors by reprojecting the acquired vector-shaped features—that is, feature embedding vectors—obtained by applying feature embeddings to points in a point cloud that correspond to a specific point in time, onto the camera plane of a specific point in time. At this time, the 2D feature map can include information not only from the corresponding point in time but also from other points in time.

[0040] In operation 140, the enhancement device obtains a residual depth map by applying the 2D feature map generated in operation 130 to a depth enhancement network. For example, if depth maps of multiple viewpoints are estimated from an input image in operation 110, the enhancement device can obtain a residual depth map by applying the input image and the 2D feature map generated in operation 130 to a depth enhancement network. The enhancement device can obtain a residual depth map by adding the channels of the input image (e.g., three channels of R, G, and B) to the channels of the 2D feature map (e.g., C channels) and applying them to a depth enhancement network.

[0041] In operation 150, the improvement device combines the residual depth map obtained in operation 140 with the depth map of a specific time point among multiple depth maps to output an improved depth map.

[0042] The enhancement device can combine the residual depth map obtained in operation 140 with the depth map of a specific point in time among multiple depth maps. The enhancement device can maintain consistency between points in time by adjusting at least one of the shift, scale, and image quality of the combined feature maps.

[0043] According to an embodiment, the enhancement device can reconstruct a 2D object corresponding to multiple viewpoint depth maps into a 3D object based on the enhanced depth map output through operation 150. Through this reconstruction, the enhancement device can estimate and / or obtain a 3D model of the object from an RGB image.

[0044] In one embodiment, during the reconstruction process for obtaining a 3D model of an object from a 2D image, the depth map can be improved through the aforementioned process to maintain consistency between viewpoints in depth maps of different viewpoints, thereby avoiding distortion caused by inconsistency in depth information and improving the structure and texture of the depth map.

[0046] FIG. 2 is a diagram illustrating the structure of a device for improving depth maps according to one embodiment. Referring to FIG. 2, the structure of an improvement device (200) is illustrated, which can improve the image quality of a length map while maintaining view consistency between multiple depth maps according to one embodiment.

[0047] For example, the improvement device (200) obtains multiple depth maps from the sensor (205). (210) can be obtained. Here, D represents initial depth maps, and i may be an index representing a viewpoint. The sensor (205) may be, for example, a LiDAR sensor that detects the depth value of an object, or a depth sensor, but is not necessarily limited thereto.

[0048] For example, in the case of a depth map obtained using a single viewpoint image, since information from other viewpoints is not used, it may show a significant difference in shift and scale compared to a depth map from any other viewpoint. When the depth map of each viewpoint is back-projected into 3D space using camera pose information from each viewpoint, if the depth values ​​between viewpoints are inaccurate, parts of the point cloud back-projected into 3D space may overlap and fail to form a single object. In one embodiment, an improved depth map can be obtained through the depth refinement process described below.

[0049] The improvement device (200) is depth maps of multiple viewpoints (210) can be un-projected (215) into a three-dimensional space to form a point cloud (220) containing K points. Each of the K points is three-dimensional information (R 3 It may include ). The enhancement device (200) can back-project (215) the depth map of each viewpoint into a three-dimensional space using camera parameters or camera pose information of each viewpoint. The enhancement device (200) can obtain the position of P in the image space by back-projecting (215) any point P included in the point cloud (220) into the corresponding image space using camera parameters. The enhancement device (200) can obtain a feature vector at the corresponding position of P.

[0050] For example, if the point cloud (220) is represented based on a single viewpoint, the object may not be accurately formed because the correlation between viewpoints or the information between viewpoints does not match. The improvement device (200) can obtain new features for each point by applying feature embedding to K points of the point cloud (220). In one embodiment, depth maps of multiple viewpoints are obtained through a point network (Point Net) (225) that includes various layers such as Multi-Layer Perceptrons (MLPs) and pooling layers. Features for all points of the point cloud (220) obtained from (210) can be embedded.

[0051] The improvement device (200) can embed features for each point of the point cloud (220) through the point network (225) to extract a feature embedding vector (230) for each point. For example, if the number of points is K, the K feature vectors can generate K feature embedding vectors with a dimension of C x 1, that is, feature embedding vectors (230) for K x C points.

[0052] The enhancement device (200) can generate a two-dimensional feature map (240) for depth enhancement by projecting (235) feature embedding vectors (230) onto a camera plane at a specific point in time. The two-dimensional feature map (240) may have a size equal to the dimension size of the feature embedding vectors (230). The two-dimensional feature map (240) may include information not only at that point in time but also at other points in time.

[0053] The improvement device (200) can generate a two-dimensional feature map (240) of a specific point in time based on feature embedding vectors of points corresponding to a specific point in time among the points of the point cloud (220).

[0054] The refinement device (200) can obtain a residual depth map by applying a two-dimensional feature map (240) to a depth refinement network (245). The depth refinement network (245) may be, for example, a U-Net pre-trained to perform segmentation on an image, but is not necessarily limited thereto.

[0055] The improvement device (200) is a residual depth map, depth maps of multiple viewpoints (210) An improved depth map combined with a depth map at a specific point in time. (250) can be output. The improvement device (200) can output a residual depth map, multiple viewpoint depth maps (210) An improved depth map that maintains consistency between viewpoints by adjusting at least one of shift, scale, and image quality in combination with a depth map of a specific viewpoint. (250) can be printed.

[0057] FIG. 3 is a diagram illustrating the iterative operation of a device for improving a depth map illustrated in FIG. 2. Referring to FIG. 3, the structure of a device for improving a depth map that performs an iterative scheme according to one embodiment is illustrated.

[0058] The improvement device obtains depth maps of multiple viewpoints through the process described above via FIG. 2 A depth map improved by performing a depth refinement process (310) on (210). (330) can be obtained. Here, the depth improvement process (310) may include the operation of forming the point cloud (220) described above through FIG. 2, the operation of extracting feature embedding vectors (230), the operation of generating a two-dimensional feature map (240), the operation of obtaining a residual depth map, and the operation of outputting an improved depth map (250).

[0059] The improvement device is an improved depth map output through the depth improvement process (310). A more improved depth map can be obtained by repeatedly performing the depth improvement process (310) using (330) as input. The improvement device can repeatedly perform the depth improvement process (310) according to, for example, a number of repetitions set by the user or a preset number of repetitions.

[0061] FIG. 4 is a diagram illustrating the structure of a device for improving a depth map according to another embodiment. Referring to FIG. 4, the structure of an improvement device (400) according to one embodiment is illustrated.

[0062] The improvement device (400) is an input image By applying (401) to a Single-View Depth Estimation Network (405) to estimate a single-view depth map, multiple-view depth maps corresponding to the input image (401) (410) can be estimated. Here, represents a set of RGB images, and D can represent initial depth maps.

[0063] Input image (401) may be two-dimensional RGB images containing multiple frames of the same object (e.g., a car) taken at N different viewpoints. Input image (401) may include three channels of R (Red), G (Green), and B (Blue). The enhancement device (400) can apply input images (401) of N time points one by one to the network (405) to estimate a depth map corresponding to each of the N time points.

[0064] The improvement device (400) is a depth map of multiple viewpoints (410) can be back-projected (415) into a three-dimensional space to form a point cloud (420) containing K points. The K points are three-dimensional information (R 3It may include ). The improvement device (400) can back-project the depth map of each viewpoint into a three-dimensional space using camera parameters or camera pose information of each viewpoint.

[0065] The improvement device (400) is, for example, a camera parameter corresponding to each input image at each point in time. depth map using By un-projecting it into 3D space, the depth map It can be converted into a point cloud (420) containing three-dimensional information. Here, 'camera parameters corresponding to each input image at each viewpoint' are the camera parameters of the camera that captured the two-dimensional input image at each viewpoint. It can be understood as meaning... Camera parameters It may include intrinsic parameters and extrinsic parameters. Extrinsic parameters may correspond, for example, to a rotation / translation transformation matrix for converting a 3D world coordinate system to a camera coordinate system.

[0066] The improvement device (400) can obtain three-dimensional information, that is, a shape expressed in the form of a point cloud (420) in three-dimensional space by integrating the depth maps estimated for all two-dimensional input images.

[0067] The enhancement device (400) can obtain the position of an arbitrary point P included in the point cloud (420) by back-projecting (415) the point P into the corresponding image space using camera parameters. The enhancement device (400) can obtain a feature vector at the corresponding position of P. At this time, the enhancement device (400) can perform feature fetching (427) by using the feature map in the last layer of the network (405) to retrieve and attach the feature vector corresponding to the position of each point back-projected into the 3D space. By performing feature fetching (427) for all viewpoints, the enhancement device (400) can form a point cloud (420) having a feature vector for each point's position.

[0068] The enhancement device (400) can convert a new feature z'(x, z) created by attaching the position coordinates x(x, y, z) for each point of the point cloud (420) or the feature z of each point obtained through feature patching (427) together with the position coordinates into a point network (425), which is a point-based feature embedding network composed of Multi-Layer Perceptrons (MLPs) and a pooling layer, into a new feature.

[0069] For example, if the point cloud (420) is represented based on a single point in time, the correlation between points in time or the information between points in time may not match, so the object may not be formed accurately. The improvement device (400) can obtain new features for each point by applying feature embedding to K points of the point cloud (420). In one embodiment, a feature embedding vector for each point can be extracted by embedding features for each point through a point network (Point Net) (425). For example, if the number of points is K, the K feature vectors can generate K feature embedding vectors with a dimension of C x 1, that is, feature embedding vectors (430) for K x C points.

[0070] The enhancement device (400) can generate a two-dimensional feature map (440) for depth enhancement by projecting (435) feature embedding vectors (430) onto a plane at a specific point in time. The enhancement device (400) can generate a two-dimensional feature map (440) at a specific point in time based on the feature embedding vectors of points corresponding to a specific point in time among the points of the point cloud (420). At this time, since the two-dimensional feature map (440) at a specific point in time is obtained by projecting points in three-dimensional space, holes in which no values ​​exist may exist. Therefore, in one embodiment, the input image (401) can be used to supplement holes where no values ​​exist in the 2D feature map.

[0071] The improvement device (400) is an input image A residual depth map can be obtained by applying (401) and a 2D feature map (440) to a depth refinement network (445). More specifically, the refinement device (400) can obtain a residual depth map by concatenating the channels of an input image consisting of a total of three R, G, and B channels with the channels of a 2D feature map and applying them to the depth refinement network (445). Here, the depth refinement network (445) may be a network trained to perform segmentation on an image similar to the network (405) that estimates a depth map at a single viewpoint.

[0072] The improvement device (400) is a residual depth map, depth maps of multiple viewpoints (410) Improved depth map combined with depth map at a specific point in time. (450) can be printed.

[0074] FIG. 5 is a diagram for explaining the iterative operation of a depth map improvement device illustrated in FIG. 4, and FIG. 6 is a diagram for explaining the operation of a depth map improvement device illustrated in FIG. 4 when the number of iterations is set to 2.

[0075] Referring to FIG. 5, the structure of an improvement device that performs an iterative scheme according to one embodiment is illustrated.

[0076] The improvement device, through the process described above via FIG. 4, input image (401) is applied to a network (405) that estimates a single-point depth map to obtain multiple-point depth maps (410) can be estimated. The improvement device is depth maps of multiple viewpoints A depth map improved by performing a depth refinement process (510) on (410). (530) can be obtained. Here, the depth refinement process (510) may include the operation of forming the point cloud (420) described above through FIG. 4, the operation of extracting feature embedding vectors (430), the operation of generating a two-dimensional feature map (440), the operation of obtaining a residual depth map, and the operation of outputting the improved depth map.

[0077] The improvement device is an improved depth map A more improved depth map can be obtained by repeatedly performing the depth improvement process (510) using (530) as input. The improvement device can repeatedly perform the depth improvement process (510) according to the number of repetitions set by the user.

[0078] For example, if the number of repetitions is set to 2, the operation of the improvement device (400) that repeats the improvement process (510) 2 times according to the number of repetitions can be referenced in FIG. 6.

[0080] FIG. 7 is a diagram illustrating the structure of a device for improving a depth map according to another embodiment. Referring to FIG. 7, the structure of an improvement device (700) is illustrated, which further includes a reconstruction network (760) for three-dimensional modeling in addition to the improvement device (400) described above through FIG. 4. Since the description of 701 to 750 in FIG. 7 is consistent with the description of 401 to 450 described above through FIG. 4, configurations and operations different from FIG. 4 will be described below.

[0081] The improvement device (700) can apply the improved depth map (750) to the reconstruction network (760) to reconstruct a 2D object corresponding to multiple depth maps into a 3D object (770).

[0082] Or the improvement device (700) may, for example, have a sufficient number of arbitrary 3D points in world coordinates to reconstruct the shape of a 3D object (770). Along with this, feature patching (727) can be re-performed using the feature maps obtained from intermediate layers within the network (705 or 745). As a result, the enhancement device (700) can obtain feature vectors for each point. The enhancement device (700) combines the obtained feature vectors with the position coordinates (x,y,z) for each point and uses them as input to a non-linear function expressed in various forms, such as a Multi-Perceptron Layer (MPL), a Graphic Convolutional Network (GCN), etc., thereby 3D points The probability value of can be obtained. The improvement device (700) is three-dimensional points A three-dimensional object (770) can be reconstructed from the probability value.

[0084] FIG. 8 is a flowchart illustrating a method for improving a depth map according to another embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0085] Referring to FIG. 8, an improved device according to one embodiment can output an improved depth map through operations 810 to 870.

[0086] In operation 810, the enhancement device can acquire multiple viewpoint depth maps. The enhancement device can acquire multiple viewpoint depth maps estimated from input images (e.g., 2D RGB images). Alternatively, the enhancement device can acquire multiple viewpoint depth maps from a LiDAR sensor or a depth sensor.

[0087] In operation 820, the improved device can back-project the multi-view depth map obtained in operation 810 into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the multi-view depth maps.

[0088] In operation 830, the improvement device can apply feature embeddings to the data represented as a point cloud in operation 820. The improvement device can obtain features in the form of vectors, that is, feature embedding vectors, by applying feature embeddings to points among the points of the point cloud that correspond to a specific time point.

[0089] In operation 840, the improvement device can reproject the feature embedding vectors obtained in operation 830 onto the camera plane at a specific point in time.

[0090] In operation 850, the improvement device can generate a 2D feature map containing features equal to the dimension of the feature embedding vectors re-projected onto the camera plane at a specific point in time in operation 840.

[0091] In operation 860, the enhancement device can obtain a residual depth map by applying the 2D feature map generated in operation 850 to a depth enhancement network. At this time, the enhancement device can obtain a residual depth map by applying the input image and the 2D feature map generated in operation 850 to a depth enhancement network.

[0092] In operation 870, the improvement device can output an improved depth map by combining the residual depth map obtained in operation 860 with the initial depth map, that is, the depth map of a specific point in time among multiple depth maps, and adjusting at least one of shift, scale, and image quality.

[0094] FIG. 9 is a block diagram of a device for improving a depth map according to one embodiment. Referring to FIG. 9, a device for improving a depth map ('improvement device') (900) according to one embodiment may include a sensor (910), a processor (920), a communication interface (930), a display (940), and a memory (950). The sensor (910), the processor (920), the communication interface (930), the display (940), and the memory (950) may be connected to each other via a communication bus (905).

[0095] The improvement device (900) may be implemented as at least part of, for example, a mobile device such as a mobile phone, smartphone, PDA, netbook, tablet computer, laptop computer, etc., a wearable device such as a smart watch, smart band, smart glasses, etc., a computing device such as a desktop, server, etc., a home appliance such as a television, smart television, refrigerator, etc., a security device such as a door lock, etc., a medical device, or a smart vehicle, but is not necessarily limited thereto.

[0096] The sensor (910) acquires depth maps at multiple viewpoints. The sensor (910) may be, for example, a depth sensor or a lidar sensor, but is not necessarily limited thereto.

[0097] The processor (920) back-projects the depth maps of multiple viewpoints into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the depth maps of multiple viewpoints. The processor (920) extracts feature embedding vectors corresponding to the points of the point cloud. The processor (920) generates a two-dimensional feature map of a specific viewpoint based on the feature embedding vectors of the points of the point cloud that correspond to a specific viewpoint. The processor (920) obtains a residual depth map by applying the two-dimensional feature map to a depth improvement network stored in memory (950). The processor (920) generates an improved depth map by combining the residual depth map with the depth map of a specific viewpoint among the depth maps of multiple viewpoints.

[0098] The communication interface (930) outputs an enhanced depth map generated by the processor (920). Additionally, the communication interface (930) can receive an input image containing multiple frames of the same object captured at different viewpoints. In this case, the processor (920) can estimate depth maps at multiple viewpoints by applying the input image received through the communication interface (930) to a network that estimates a depth map at a single viewpoint.

[0099] Additionally, the processor (920) can reconstruct a 2D object corresponding to multiple viewpoint depth maps into a 3D object based on the improved depth map. In this case, the display (940) can display the 3D object reconstructed by the processor (920).

[0100] Memory (950) can store a network that improves depth. Memory (950) can store a network that estimates a depth map at a single time point. Additionally, memory (950) can store a network containing multiple Multi-Layer Perceptrons (MLPs).

[0101] The memory (950) can store various information generated during the processing of the processor (920). In addition, the memory (950) can store various data and programs. The memory (950) may include volatile memory or non-volatile memory. The memory (950) can store various data by being equipped with a large-capacity storage medium such as a hard disk.

[0102] The processor (920) can execute executable instructions contained in memory (950). The processor (920) can execute a program and control the improvement device (900). Program code executed by the processor (920) can be stored in memory (950).

[0103] Additionally, the processor (920) may perform at least one method or a technique corresponding to at least one method described above through FIGS. 1 to 8. The processor (920) may be an enhancement device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program. For example, the enhancement device (900) implemented in hardware may include a microprocessor, a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Neural Processing Unit (NPU), etc.

[0105] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0106] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0107] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0108] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0109] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0110] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for improving a depth map, comprising: an operation of forming a point cloud corresponding to the multiple viewpoint depth maps by un-projecting the multiple viewpoint depth maps into a three-dimensional space using camera parameters corresponding to each viewpoint; an operation of extracting feature embedding vectors corresponding to the points of the point cloud; an operation of generating a two-dimensional feature map of a specific viewpoint based on the feature embedding vectors of the points of the point cloud corresponding to the specific viewpoint; an operation of obtaining a residual depth map by applying the two-dimensional feature map to a depth-improving network; and an operation of outputting an improved depth map by combining the residual depth map with the depth map of the specific viewpoint among the multiple viewpoint depth maps. Claim 2 A method for improving a depth map according to claim 1, further comprising the operation of estimating multiple depth maps by applying an input image containing multiple frames of the same object captured at different viewpoints to a network that estimates a depth map at a single viewpoint. Claim 3 A method for improving a depth map according to paragraph 2, wherein the input image includes RGB images of multiple viewpoints. Claim 4 A method for improving a depth map according to claim 2, wherein the operation of extracting the feature embedding vectors comprises: fetching features corresponding to the location of each point in the point cloud from the feature map of the last layer of the network estimating the depth map at a single time point; and combining the fetched features with each point at the corresponding location in the point cloud to extract the feature embedding vectors. Claim 5 A method for improving a depth map according to claim 2, wherein the operation of generating the two-dimensional feature map includes the operation of generating the two-dimensional feature map at a specific point in time by projecting the feature embedding vectors onto the image plane at the specific point in time. Claim 6 A method for improving a depth map according to claim 2, wherein the operation of acquiring the residual depth map includes the operation of acquiring the residual depth map by applying the input image and the two-dimensional feature map to a depth-improving network. Claim 7 A method for improving a depth map according to claim 6, wherein the operation of acquiring the residual depth map includes the operation of acquiring the residual depth map by adding the channels of the input image to the channels of the two-dimensional feature map and applying them to a network that improves the depth. Claim 8 A method for improving a depth map according to claim 2, wherein the operation of outputting the improved depth map comprises: combining the residual depth map with the depth map of a specific point in time among the depth maps of multiple point in time; and outputting the improved depth map by adjusting at least one of the shift, scale, and image quality of the combined depth maps. Claim 9 A method for improving a depth map according to claim 2, further comprising, for the improved depth map, the operation of forming a point cloud, the operation of extracting feature embedding vectors, the operation of generating a two-dimensional feature map, the operation of obtaining a residual depth map, and the operation of outputting the operation repeatedly according to a set number of repetitions. Claim 10 A method for improving a depth map according to claim 2, further comprising the operation of reconstructing a 2D object corresponding to the multiple viewpoint depth maps into a 3D object based on the improved depth map. Claim 11 A method for improving a depth map according to claim 1, wherein the operation of outputting the improved depth map comprises: combining the residual depth map with the depth map of a specific point in time among the depth maps of multiple point in time; and outputting the improved depth map by adjusting at least one of the shift, scale, and image quality of the combined depth maps. Claim 12 A method for improving a depth map according to claim 1, further comprising, for the improved depth map, the operation of forming a point cloud, the operation of extracting feature embedding vectors, the operation of generating a two-dimensional feature map, the operation of obtaining a residual depth map, and the operation of outputting the operation repeatedly according to a set number of repetitions. Claim 13 A method for improving a depth map according to claim 1, further comprising the operation of reconstructing a 2D object corresponding to the multiple viewpoint depth maps into a 3D object based on the improved depth map. Claim 14 A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 13. Claim 15 A device for improving a depth map, comprising: a sensor for acquiring depth maps of multiple viewpoints; a processor for back-projecting the depth maps of multiple viewpoints into a three-dimensional space using camera parameters corresponding to each viewpoint to form a point cloud corresponding to the depth maps of multiple viewpoints, extracting feature embedding vectors corresponding to points of the point cloud, generating a two-dimensional feature map of a specific viewpoint based on the feature embedding vectors of points corresponding to a specific viewpoint among the points of the point cloud, acquiring a residual depth map by applying the two-dimensional feature map to a depth improvement network, and generating an improved depth map by combining the residual depth map with the depth map of the specific viewpoint among the depth maps of multiple viewpoints; and a communication interface for outputting the improved depth map. Claim 16 In claim 15, the communication interface receives an input image comprising a plurality of frames capturing the same object at different viewpoints, and the processor estimates the depth maps at multiple viewpoints by applying the input image to a network that estimates a depth map at a single viewpoint. Claim 17 In claim 16, the processor improves the depth map by patching features corresponding to the location of each point in the point cloud from the feature map of the last layer of the network that estimates the depth map at a single point in time, and combining the patched features with each point at the corresponding location in the point cloud to extract the feature embedding vectors. Claim 18 In claim 16, the device for improving a depth map, wherein the processor obtains the residual depth map by applying the input image and the two-dimensional feature map to the depth-improving network. Claim 19 A device for improving a depth map according to claim 16, wherein the processor repeatedly performs, for the improved depth map, the operation of forming a point cloud, the operation of extracting feature embedding vectors, the operation of generating a two-dimensional feature map, the operation of obtaining a residual depth map, and the operation of outputting, according to a set number of repetitions. Claim 20 In claim 16, the processor is a device for improving a depth map, which reconstructs a two-dimensional object corresponding to the multiple viewpoint depth maps into a three-dimensional object based on the improved depth map.

Citation Information

Patent Citations

  • Depth image supplementation method and device, computer-readable storage medium

    KR1020210027269A

  • Method for generation 3D model using single input image and apparatus using the same

    KR1020210058320A

  • Method and system for synthesizing three-dimensional data

    US20200265259A1