Method and apparatus for estimating depth of transparent object, learning method for depth estimating of transparent object

Abstract

The present invention relates to a depth estimation device for a transparent object, comprising: an acquisition unit for acquiring a depth image of a transparent object; a storage unit including a learning model trained to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture; and a processing unit for estimating the depth of the transparent object by extracting features from the depth image of the non-transparent object based on the learning model and generating a depth map of the depth image of the transparent object based on the features.

Description

Technology Field

[0001] The present invention relates to image depth estimation technology, and more specifically to object depth estimation technology using deep learning. Background Technology

[0003] With the advent of deep learning technology, there has been significant growth in various image signal processing (ISP) fields, such as object detection, image segmentation, and depth estimation. Recently, there has been a trend of universally adopting and using deep learning in most image signal processing applications.

[0004] Among these, distance measurement technologies are broadly classified into two types: monocular depth estimation models that estimate depth from a single image and stereo depth estimation models that estimate depth using multiple images, and active research is being conducted on these.

[0005] However, both of the above models are based on the assumption that the surface of objects in an image follows Lambertian reflection, without considering environments affected by light refraction and reflection, such as transparent objects or liquids. While this assumption may yield good depth estimation performance in specific environments, it can cause difficulties for the model to learn distance measurement in images containing plastics, glass, disposable containers, and liquids that are commonly seen in everyday life. Prior art literature

[0007] Published Patent Application No. 10-2022-0080696 (Published June 14, 2022) The problem to be solved

[0008] In an embodiment of the present invention, we propose an image depth estimation technique based on a convolutional neural network based on supervised learning.

[0009] In an embodiment of the present invention, we propose a deep learning network-based image depth estimation technique suitable for extracting features of transparent objects and estimating their depth.

[0010] The problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned problems to be solved will be clearly understood by those skilled in the art to which the present invention pertains from the descriptions below. means of solving the problem

[0012] According to an embodiment of the present invention, a depth estimation device for a transparent object may be provided, comprising: an acquisition unit for acquiring a depth image of a transparent object; a storage unit including a learning model learned to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture; and a processing unit for extracting features from the depth image of the non-transparent object based on the learning model and generating a depth map of the depth image of the transparent object based on the features to estimate the depth of the transparent object.

[0013] Here, the learning model may include a first learning model learned to estimate the depth of the non-transparent object; and a second learning model learned to estimate the depth of the transparent object using the first learning model.

[0014] In addition, the first learning model is a teacher learning model, and the second learning model is a student learning model, and the teacher learning model and the student learning model can be learned based on the knowledge distillation technique.

[0015] Additionally, the processing unit may include: a data set configuration unit that configures a training data set for the depth image of the transparent object; an encoding unit that extracts features from the training data set and the depth image of the non-transparent object based on the training model; and a decoding unit that generates a depth map of the depth image of the transparent object based on the features to estimate the depth of the transparent object.

[0016] Additionally, the encoding unit may include a dense feature extraction unit that generates a feature map corresponding to a task for applying the feature; and a contextual information extraction unit that extracts a global feature map for predicting a dense image for the feature map.

[0017] Additionally, the decoding unit may include a local planar map unit that restores the resolution of the global feature map to an output resolution; and a skip connection unit that minimizes information loss by connecting the global feature map restored to the output resolution to a feature map of the same size output from the decoder layer.

[0018] According to an embodiment of the present invention, a method for estimating the depth of a transparent object performed in a depth estimation device comprises: a step of acquiring a depth image of the transparent object; a step of extracting features from the depth image based on a previously stored learning model; and a step of generating a depth map of the depth image based on the features to estimate the depth of the transparent object; wherein the previously stored learning model is a learning model trained to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture.

[0019] Here, the step of estimating the depth of the transparent object may include: a step of training a first learning model to estimate the depth of the non-transparent object; and a step of training a second learning model to estimate the depth of the transparent object using the first learning model.

[0020] In addition, the first learning model is a teacher learning model, and the second learning model is a student learning model, and the teacher learning model and the student learning model can be learned based on the knowledge distillation technique.

[0021] Additionally, the step of estimating the depth of the transparent object may include: a step of constructing a training dataset for the depth image of the transparent object; a step of extracting features from the training dataset and the depth image of the non-transparent object based on the training model; and a step of estimating the depth of the transparent object by generating a depth map of the depth image of the transparent object based on the features.

[0022] Additionally, the step of extracting the above features may include: a step of extracting dense features to generate a feature map corresponding to a task for applying the above features; and a step of extracting context information to extract a global feature map for predicting a dense image for the above feature map.

[0023] Additionally, the step of estimating the depth of the transparent object may include: a step of restoring the resolution of the global feature map to an output resolution; and a skip connection step of connecting the global feature map restored to the output resolution to a feature map of the same size output from the decoder layer to minimize information loss.

[0024] According to an embodiment of the present invention, a learning method for estimating the depth of a transparent object performed in a depth estimation device comprises: a step of acquiring a depth image of a transparent object and a depth image of a non-transparent object having the same shape as the transparent object but a different texture; a step of training a first learning model for estimating the depth of the non-transparent object based on the features of the depth image of the non-transparent object; and a step of training a second learning model for estimating the depth of the transparent object based on the features of the depth image of the transparent object using the first learning model.

[0025] Here, the first learning model is a teacher learning model, and the second learning model is a student learning model, and the teacher learning model and the student learning model can be learned based on the knowledge distillation technique.

[0026] Additionally, the method may further include the step of calculating a loss function using distillation loss with respect to the teacher learning model and the student learning model, with the output of the teacher learning model as a soft label; the step of calculating a loss function using student loss with the output of the student learning model as a hard label; and the step of defining a final loss function based on the weighted sum of the loss function using distillation loss and the loss function using student loss.

[0027] Additionally, the above method may further include the step of updating the first and second learning models by applying a backpropagation algorithm to the final loss function.

[0028] According to an embodiment of the present invention, a computer-readable recording medium storing a computer program comprises, wherein the computer program includes instructions for a processor to perform a method for estimating the depth of a transparent object performed by a depth estimation device, and comprises the steps of: acquiring a depth image of the transparent object; extracting features from the depth image based on a previously stored learning model; and generating a depth map of the depth image based on the features to estimate the depth of the transparent object; wherein the previously stored learning model may be a learning model trained to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture.

[0029] According to an embodiment of the present invention, a computer program stored on a computer-readable recording medium comprises instructions for a processor to perform a learning method for estimating the depth of a transparent object performed by a depth estimation device, and the method may include: a step of acquiring a depth image of the transparent object and a depth image of a non-transparent object having the same shape as the transparent object but a different texture; a step of training a first learning model for estimating the depth of the non-transparent object based on the features of the depth image of the non-transparent object; and a step of training a second learning model for estimating the depth of the transparent object based on the features of the depth image of the transparent object using the first learning model. Effects of the invention

[0031] According to an embodiment of the present invention, by extracting features of non-transparent objects and estimating the depth of an image, it is easy to utilize for detecting other means of transportation and surrounding objects caused by light reflection and refraction, which are difficult to detect in autonomous driving. Furthermore, since there are many issues regarding transparent or uniquely textured objects in robotic grasping systems, this invention can provide a positive effect on these problems. From a modeling perspective, applying knowledge distillation improves the generalization ability of the model. By lightweighting the depth estimation model, which is relatively heavy, a small model suitable for use in real-world environments can be constructed, enabling the building of an easily accessible model such as one used in mobile phone cameras. Brief explanation of the drawing

[0033] FIG. 1 is a block diagram schematically illustrating the function of a depth estimation device (100) for a transparent object according to an embodiment of the present invention. FIG. 2 is a conceptual diagram showing a depth estimation program (122) in a storage unit (120) of the depth estimation device (100) of FIG. 1. FIG. 3 is a specific functional block diagram of a transfer learning model (124) within the depth estimation program (122) of FIG. 2. FIG. 4 is a detailed functional block diagram of the processing unit (130) of FIG. 1. FIG. 5 is a specific functional block diagram of the encoding unit (134) within the processing unit (130) of FIG. 4. FIG. 6 is a specific functional block diagram of the decoding unit (136) within the processing unit (130) of FIG. 4. FIG. 7 is a conceptual diagram illustrating the transfer learning model (124) of FIG. 3 in an exemplary manner. FIG. 8 is a conceptual diagram illustrating the processing unit (130) of FIG. 1 as an example. FIG. 9 is a conceptual diagram illustrating the configuration of the regional planar map (136a) of FIG. 6 in an exemplary manner. Specific details for implementing the invention

[0034] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the scope of the present invention is defined only by the claims.

[0035] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted unless actually necessary for describing the embodiments of the present invention. Furthermore, the terms described below are defined in consideration of the functions in the embodiments of the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0037] The present invention aims to solve problems inherent in conventional distance measurement technologies that have not been addressed for various reasons, and relates to a deep learning-based method for generating distance estimation and object detection models in various environments by utilizing features of non-transparent object images identical to real-world transparent object images.

[0038] First, due to the difficulty of obtaining identical real-world images of transparent and opaque objects, most distance estimation techniques generate ground truth images for model training using synthetic data generators. However, since such image data is virtual rather than real, the model faces clear limitations when performing inference on actual images. Therefore, it is desirable to utilize real-world data as training data to train an accurate and reasonable distance measurement model.

[0039] Second, due to the aforementioned issues, there is a lack of research on feature analysis of real-world data that does not follow Lambertian reflection. It has been proven that for opaque objects, features such as edges and textures can be extracted from low-level features, while increasingly abstract features can be extracted as one moves toward high-level features. Transparent objects also exhibit this tendency, but because they possess properties distinctly different from ordinary objects, it is necessary to analyze the features of target images by experimenting with and devising methods to extract other meaningful features. If this is utilized effectively, it can not only improve performance in various fields such as knowledge distillation, domain adaptation, and transfer learning but also suggest new directions for specific tasks.

[0040] Therefore, there is a need for neural network-based methods and devices that address the shortcomings of distance estimation models not covered so far, and further solve such problems in various domains, such as object detection and image segmentation, rather than being limited to distance estimation.

[0041] Accordingly, an embodiment of the present invention proposes an image depth estimation technique based on a convolutional neural network based on supervised learning.

[0042] The ground truth data of the training data for a distance measurement model can be obtained through a Kinect camera. However, since this Kinect camera can only measure distance when infrared light is reflected from the surface of an object and detected by the sensor, it is difficult to measure the distance of transparent objects through which light is transmitted or refracted. Therefore, by preparing an object with the same shape as the transparent object but a different texture, fixing the camera, and separately capturing an RGB image and a 3D depth image, the ground truth depth data of the transparent object can be obtained.

[0043] Transparent objects and textured objects, while distinct from each other, possess a wealth of similar feature information. By leveraging this, a student model can be trained using a teacher model's outputs and feature maps via the knowledge distillation technique. This allows for the creation of a robust, lightweight, and high-performing model capable of handling diverse environments. Furthermore, based on an analysis of feature differences between the two models, weakly supervised learning can be applied to detect transparent or textured objects within images without coordinate labels.

[0045] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0046] Terms defined in embodiments of the present invention may be defined as follows.

[0047] "Encoding" is the part that takes an input image, extracts an "encoding feature map" using a convolutional neural network to be fed as input to the dense feature extraction and decoding sections, and outputs the result.

[0048] "Decoding" refers to the part where a "decoding feature map" is extracted into a convolutional neural network by connecting skip connections from the encoding feature map and each output of the convolution layer of the encoding part using a U-Net structure. The decoding feature map functions as a part to generate inputs for the context information estimation part and the local plane map part.

[0049] A "skip connection" is a structure used to compensate for the loss of information at the beginning as the depth of a deep learning model increases during training.

[0050] "U-Net" is an encoder-decoder structure typically used when the final output of a model performs pixel-level classification (image classification, image segmentation, depth estimation, etc.).

[0051] "Transfer learning" refers to the process of solving an image classification problem by applying a network trained on vast amounts of data to a different dataset or task.

[0052] "Contextual information" refers to the information generated through multiple convolutions and pooling steps; however, as these stages progress, detailed information decreases and characteristics become increasingly abstracted. To address this, Atrous Spatial Pyramid Pooling (ASPP) is used to provide diverse viewing angles.

[0053] "Knowledge distillation" is a technique that helps lighten a model or improve model performance by transferring knowledge from a well-trained large network (teacher network) to a small network (student network) that actually needs to use it.

[0054] First, FIG. 1 is a block diagram for schematically explaining the function of a depth estimation device (100) for a transparent object according to an embodiment of the present invention.

[0055] As illustrated in FIG. 1, the depth estimation device (100) for a transparent object may include an acquisition unit (110), a storage unit (120), and a processing unit (130).

[0056] The acquisition unit (110) can acquire a depth image of a transparent object. In addition to the depth image of a transparent object, the acquisition unit (110) can acquire a depth image of a non-transparent object that has the same shape as the transparent object but a different texture. The depth images of the transparent object and the non-transparent object acquired through the acquisition unit (110) can be provided to the processing unit (130). To this end, the depth estimation device (100) according to an embodiment of the present invention may be connected to a shooting means for capturing three-color images and a three-dimensional depth image of a transparent object and a non-transparent object. Here, the three-dimensional depth image may be acquired, for example, through a Kinect camera. A Kinect camera that measures three-dimensional depth information is composed of a color sensor, an infrared emitter, an infrared depth sensor, etc. The operating principle of the Kinect camera utilizes the time-of-flight (TOF) method, which calculates the distance between the target object and the camera by measuring the time it takes for infrared rays emitted from the infrared emitter to be reflected from the surface of the target object and detected by the infrared depth sensor of the Kinect camera. In an embodiment of the present invention, an object having the same shape as a transparent object but a different texture is prepared together with the transparent object, and after fixing the Kinect camera, an RGB image and a 3D depth image are captured separately to obtain the correct depth data of the transparent object.

[0057] The storage unit (120) may include a learning model trained to estimate the depth of a transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture. According to an embodiment of the present invention, the storage unit (120) may store a program for estimating the depth of an object and information capable of executing such a program. The storage unit (120) may include any type of recording device in which data that can be read by a computer system is stored, and is not limited to a specific recording device. The program and execution information within the storage unit (120) may be loaded by the processing unit (130) as needed.

[0058] The processing unit (130) can estimate the depth of a transparent object by extracting features from a depth image of a non-transparent object based on a learning model and generating a depth map of a depth image of a transparent object based on the extracted features. In particular, the processing unit (130) can train a learning model to output a depth estimation result for a depth image of a transparent object acquired through the acquisition unit (110) based on a depth estimation program described later, or output a depth estimation result of a transparent object through the learning model.

[0060] FIG. 2 is a conceptual diagram showing a depth estimation program (122) in a storage unit (120) of the depth estimation device (100) of FIG. 1.

[0061] The depth estimation program (122) may include a transfer learning model (124), and may load the transfer learning model (124) under the control of the processing unit (130), and execute or update the loaded transfer learning model (124). The depth estimation program (122) according to an embodiment of the present invention may refer to software that includes instructions programmed to estimate the depth of a transparent object by generating a depth map of a depth image of a transparent object when a captured image of a transparent object and a non-transparent object is acquired through the acquisition unit (110).

[0062] This transfer learning model (124) may include a learning model for estimating the depth of a non-transparent object and a learning model for estimating the depth of a transparent object.

[0064] FIG. 3 is a specific functional block diagram of a transfer learning model (124) within the depth estimation program (122) of FIG. 2.

[0065] As illustrated in FIG. 3, the transfer learning model (124) may include a non-transparent object depth estimation learning model (124a), a transparent object depth estimation learning model (124b), and a knowledge distillation unit (124c).

[0066] A learning model for estimating the depth of a nontransparent object (124a) can be trained to estimate the depth of a nontransparent object based on a depth image of a nontransparent object that has the same shape as a transparent object but a different texture. This learning model for estimating the depth of a nontransparent object (124a) is a pre-learning model for estimating the depth of a transparent object, and can be named, for example, as a teacher learning model.

[0067] The transparent object depth estimation learning model (124b) can be trained to estimate the depth of a transparent object using the non-transparent object depth estimation learning model (124a). This transparent object depth estimation learning model (124b) is a subsequent learning model for estimating the depth of a transparent object based on the results of estimating the depth of a non-transparent object, and can be named, for example, a student learning model.

[0068] The knowledge distillation unit (124c) can define a loss function for the teacher learning model and student learning model constructed from the non-transparent object depth estimation learning model (124a) and the transparent object depth estimation learning model (124b) by using the output of the teacher learning model as a soft label for the distillation loss and by using the output of the student learning model as a hard label for the student loss. This distillation loss can be expressed by [Equation 1] below, and the student loss can be expressed by [Equation 2] below.

[0069]

[0070]

[0071] Through this, the knowledge distillation unit (124c) defines a final loss function through a weighted sum of the defined loss functions, and the transfer learning model (124) proceeds with learning. The final loss function for this learning ( L t A back-propagation algorithm can be applied to ), and the transfer learning model (124) can be updated through the back-propagation algorithm. The loss function can be expressed as shown in [Equation 3] below.

[0072]

[0074] FIG. 4 is a detailed functional block diagram of the processing unit (130) of FIG. 1.

[0075] As illustrated in FIG. 4, the processing unit (130) may include a data set configuration unit (132), an encoding unit (134), and a decoding unit (136).

[0076] The data set configuration unit (132) can configure a training data set for depth images of transparent objects. It includes a data set creation process for training a deep learning network, and this process can generate one depth image per image through a single depth estimation device, such as a Kinect camera.

[0077] The encoding unit (134) can extract features from a learning dataset configured through the dataset configuration unit (132) based on the transfer learning model (124) and from a depth image of a non-transparent object.

[0078] That is, the encoding unit (134) can extract features or feature maps of an input image by freezing a pre-trained model trained for image classification in a large dataset image-net.

[0079] The decoding unit (136) can estimate the depth of the transparent object by generating a depth map of the depth image of the transparent object based on the features or feature map extracted from the encoding unit (134). That is, the decoding unit (136) can perform dense image prediction by making the feature map extracted from the encoding unit (134) into a richer feature map and simultaneously increasing the resolution to the desired final output depth map.

[0081] FIG. 5 is a specific functional block diagram of the encoding unit (134) within the processing unit (130) of FIG. 4.

[0082] As illustrated in FIG. 5, the encoding unit (134) may include a dense feature extraction unit (134a) and a contextual information extraction unit (134b).

[0083] The dense feature extraction unit (134a) can generate a feature map corresponding to a task for applying features of an input image. This dense feature extraction unit (134a) can operate organically with the context information extraction unit (134b).

[0084] The context information extraction unit (134b) can extract a global feature map for predicting a dense image of a feature map. This context information extraction unit (134b) enables the feature map to be extracted into a more global feature map by using various viewing angles.

[0086] FIG. 6 is a specific functional block diagram of the decoding unit (136) within the processing unit (130) of FIG. 4.

[0087] As illustrated in FIG. 6, the decoding unit (136) may include a local plane map unit (136a) and a skip connection unit (136b).

[0088] The local planar map unit (136a) can restore the resolution of the global feature map extracted from the context information extraction unit (134b) of the encoding unit (134) to the output resolution. Unlike conventional up-sampling techniques, this local planar map unit (136a) performs efficient resolution restoration by appropriately using 1x1 convolution and activation functions to restore the feature map to the output resolution. In this process, a total of four coefficients can be estimated, and the estimation formula can be calculated by estimating n4 through the straight-line distance between the original and the plane, along with n1, n2, and n3. The coefficients of this estimation formula can be expressed by [Equation 4] below.

[0089]

[0090] In [Equation 4], n1, n2, and n3 can be expressed as [Equation 5] below.

[0091]

[0092] The skip connection section (136b) can minimize information loss by connecting the global feature map restored to output resolution through the local planar map section (136a) to the feature map of the same size output from the decoder layer. The final depth estimation result output through the skip connection section (136b) can be expressed by [Equation 6] below.

[0093]

[0095] FIG. 7 is a conceptual diagram illustrating the transfer learning model (124) of FIG. 3 in an exemplary manner.

[0096] As exemplified in Figure 7, basically, the output of the teacher learning model can be defined as a loss function using distillation loss as a soft label, and the output of the student learning model can be defined as a loss function using student loss as a hard label.

[0098] FIG. 8 is a conceptual diagram illustrating the processing unit (130) of FIG. 1 as an example.

[0099] In FIG. 8, the dense feature extractor (DFE) corresponds to the encoding unit (134) of FIG. 4, and the configuration after ASPP (atrous spatial pyramid pooling) may correspond to the decoding unit (136) of FIG. 4.

[0100] The decoding unit (136) may include LPG (local planar guidance), ds (down-sampling), conv (convolution), upconv (up convolution), etc.

[0102] FIG. 9 is a conceptual diagram illustrating the configuration of the regional planar map (136a) of FIG. 6 in an exemplary manner.

[0103] As exemplified in FIG. 9, the local planar map section (136a) performs efficient resolution restoration by appropriately using 1x1 convolution and activation functions to restore the feature map to output resolution. In this process, a total of four coefficients can be estimated, and the estimation formula can be calculated by estimating n4 through n1, n2, and n3, and the straight-line distance between the original and the plane.

[0105] According to an embodiment of the present invention as described above, dense image prediction for plastic, glass, and objects with special textures commonly seen in daily life is performed based on dense image prediction for general (opaque) objects, which have been well-handled in the field of deep learning. Since related research on target objects has not yet been actively conducted, this study establishes a baseline and initial environment settings for the subject to conduct exploration and analysis of the new subject.

[0106] In terms of practical applications, this invention can be utilized in autonomous driving to detect other means of transportation and surrounding objects that are difficult to detect due to light reflection and refraction. Furthermore, given the significant issues regarding transparent or uniquely textured objects in robotic grasping systems, this invention is expected to have a positive effect on addressing these issues.

[0107] In terms of modeling, the effects of the invention include improving the generalization ability of the model by applying knowledge distillation techniques, and lightweighting the depth estimation model—which is relatively heavy—to construct a small model suitable for use in real-world environments, thereby enabling the construction of an easily accessible model such as a camera used in a mobile phone.

[0108] From a research perspective, the effect of this invention is that it enables a comparative analysis of the characteristics of non-transparent and transparent objects by conducting an in-depth analysis of the target object and analyzing characteristic changes regarding various techniques. Although sufficient data for related analysis has not yet been secured, the analysis of characteristic changes regarding various techniques discovered through this invention can be applied to other tasks and utilized as data to construct efficient models for other research.

[0110] Meanwhile, combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram.

[0111] Since these computer program instructions may be stored in a computer-available or computer-readable recording medium (or memory), etc., which can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, the instructions stored in the computer-available or computer-readable recording medium (or memory) may also be used to produce a manufactured item containing instruction means that perform the function described in each block of the block diagram.

[0112] And, since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on a computer or other programmable data processing equipment to create a process executed by a computer and perform the computer or other programmable data processing equipment may also provide steps for executing the functions described in each block of the block diagram.

[0113] Additionally, each block may represent a module, segment, or part of code containing at least one executable instruction for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks may occur out of order. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to the corresponding function. Explanation of the symbols

[0115] 100: Depth estimation device for transparent objects 110: Acquisition Department 120: Storage section 122: Depth Estimation Program 124: Transfer Learning Model 124a: Learning model for opaque object depth estimation 124b: Transparent Object Depth Estimation Learning Model 124c: Knowledge Distillation Department 130: Processing unit 132: Data set components 134: Encoding section 134a: Dense feature extraction unit 134b: Context Information Extraction Unit 136: Decoding section 136a: Local planar map 136b: Skip connection

Claims

Claim 1 A method comprising: an acquisition unit that acquires a depth image of a transparent object and a depth image of a non-transparent object having the same shape as the transparent object but a different texture, respectively; a storage unit including a learning model trained to estimate the depth of the transparent object based on the depth image of the non-transparent object; and a processing unit that extracts features from the depth image of the non-transparent object based on the learning model and generates a depth map of the depth image of the transparent object based on the features to estimate the depth of the transparent object; wherein the depth image of the non-transparent object is a depth image acquired from a real-world object image captured through a Kinect camera, and the learning model includes a learning model trained to estimate the depth of the transparent object using correct depth data of the transparent object based on the depth image of the non-transparent object acquired from the real-world object image, and the processing unit includes a data set configuration unit that configures a learning data set for the depth image of the transparent object; and an encoding unit that extracts features from the learning data set and the depth image of the non-transparent object based on the learning model. A device for estimating the depth of a transparent object, comprising: a decoding unit that estimates the depth of the transparent object by generating a depth map of the depth image of the transparent object based on the above features; wherein the encoding unit includes: a dense feature extraction unit that generates a feature map corresponding to a task for applying the above features; and a contextual information extraction unit that extracts a global feature map for predicting a dense image for the feature map; wherein the decoding unit includes a local planar map unit that restores the resolution of the global feature map to an output resolution, and the local planar map unit estimates four local planar coefficients to restore the resolution of the global feature map to the output resolution. Claim 2 A device for estimating the depth of a transparent object according to claim 1, wherein the learning model comprises: a first learning model learned to estimate the depth of the non-transparent object; and a second learning model learned to estimate the depth of the transparent object using the first learning model. Claim 3 In claim 2, the first learning model is a teacher learning model and the second learning model is a student learning model, and the teacher learning model and the student learning model are learned based on a knowledge distillation technique, a device for estimating the depth of a transparent object. Claim 4 delete Claim 5 delete Claim 6 A depth estimation device for a transparent object according to claim 1, wherein the decoding unit further includes a skip connection unit that minimizes information loss by connecting the global feature map restored to the output resolution to a feature map of the same size output from the decoder layer. Claim 7 A method for estimating the depth of a transparent object performed in a depth estimation device comprises: a step of acquiring a depth image of the transparent object; a step of extracting features from the depth image based on a previously stored learning model; and a step of generating a depth map of the depth image based on the features to estimate the depth of the transparent object; wherein the learning model is previously trained to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture, and is previously trained to estimate the depth of the transparent object using ground truth depth data of the transparent object based on the depth image of the non-transparent object acquired from a real-world object image, and the depth image of the non-transparent object is a depth image acquired from a real-world object image captured through a Kinect camera, and the step of estimating the depth of the transparent object comprises: a step of constructing a learning dataset for the depth image of the transparent object; and a step of extracting features from the learning dataset and the depth image of the non-transparent object based on the learning model. A method for estimating the depth of a transparent object, comprising: a step of estimating the depth of the transparent object by generating a depth map of the depth image of the transparent object based on the above features; wherein the step of extracting features includes: a step of extracting dense features to generate a feature map corresponding to a task for applying the features; and a step of extracting context information to extract a global feature map for predicting a dense image for the feature map; and wherein the step of estimating the depth of the transparent object includes a step of estimating four local plane coefficients to restore the resolution of the global feature map to an output resolution. Claim 8 A method for estimating the depth of a transparent object according to claim 7, wherein the step of estimating the depth of the transparent object comprises: a step of training a first learning model to estimate the depth of the non-transparent object; and a step of training a second learning model to estimate the depth of the transparent object using the first learning model. Claim 9 In claim 8, the first learning model is a teacher learning model and the second learning model is a student learning model, and the teacher learning model and the student learning model are learned based on a knowledge distillation technique, a method for estimating the depth of a transparent object. Claim 10 delete Claim 11 delete Claim 12 In claim 7, the step of estimating the depth of the transparent object further comprises a skip connection step that minimizes information loss by connecting the global feature map restored to the output resolution to a feature map of the same size output from a decoder layer. Claim 13 A learning method for estimating the depth of a transparent object performed in a depth estimation device, comprising: a step of acquiring a depth image of a transparent object and a depth image of a non-transparent object having the same shape as the transparent object but a different texture; a step of training a first learning model for estimating the depth of the non-transparent object based on the features of the depth image of the non-transparent object; and a step of training a second learning model for estimating the depth of the transparent object based on the features of the depth image of the transparent object using the first learning model; wherein the depth image of the non-transparent object is a depth image acquired from a real-world object image captured through a Kinect camera, and the second learning model includes a learning model trained to estimate the depth of the transparent object using ground truth depth data of the transparent object based on the depth image of the non-transparent object acquired from the real-world object image, and the depth of the transparent object is estimated using four local plane coefficients. Claim 14 In claim 13, the first learning model is a teacher learning model, the second learning model is a student learning model, and the teacher learning model and the student learning model are learned based on a knowledge distillation technique, a learning method for estimating the depth of a transparent object. Claim 15 A learning method for depth estimation of a transparent object according to claim 14, further comprising: a step of calculating a loss function using distillation loss with the output of the teacher learning model as a soft label for the teacher learning model and the student learning model; a step of calculating a loss function using student loss with the output of the student learning model as a hard label; and a step of defining a final loss function based on the weighted sum of the loss function using distillation loss and the loss function using student loss. Claim 16 A learning method for depth estimation of a transparent object according to claim 15, further comprising the step of updating the first and second learning models by applying a backpropagation algorithm to the final loss function. Claim 17 A computer-readable recording medium storing a computer program, wherein the computer program includes instructions for a processor to perform a method for estimating the depth of a transparent object performed by a depth estimation device, and comprises the steps of: acquiring a depth image of the transparent object; extracting features from the depth image based on a previously stored learning model; and generating a depth map of the depth image based on the features to estimate the depth of the transparent object; wherein the learning model is previously trained to estimate the depth of the transparent object based on a depth image of a non-transparent object having the same shape as the transparent object but a different texture, and is previously trained to estimate the depth of the transparent object using correct depth data of the transparent object based on the depth image of the non-transparent object acquired from a real-world object image, and the depth image of the non-transparent object is a depth image acquired from a real-world object image captured through a Kinect camera, and the step of estimating the depth of the transparent object comprises the steps of: constructing a learning dataset for the depth image of the transparent object; and extracting features from the learning dataset and the depth image of the non-transparent object based on the learning model. A computer-readable recording medium comprising: a step of estimating the depth of a transparent object by generating a depth map of a depth image of the transparent object based on the above features; wherein the step of extracting features includes: a step of extracting dense features to generate a feature map corresponding to a task for applying the features; and a step of extracting context information to extract a global feature map for predicting a dense image for the feature map; and wherein the step of estimating the depth of the transparent object includes a step of restoring the resolution of the global feature map to an output resolution by estimating four local plane coefficients. Claim 18 A computer program stored on a computer-readable recording medium, wherein the computer program includes instructions for a processor to perform a learning method for estimating the depth of a transparent object performed by a depth estimation device, and the method comprises: a step of acquiring a depth image of the transparent object and a depth image of a non-transparent object having the same shape as the transparent object but a different texture; a step of training a first learning model for estimating the depth of the non-transparent object based on features of the depth image of the non-transparent object; and a step of training a second learning model for estimating the depth of the transparent object based on features of the depth image of the transparent object using the first learning model; wherein the depth image of the non-transparent object is a depth image acquired from a real-world object image captured through a Kinect camera, and the second learning model includes a learning model trained to estimate the depth of the transparent object using ground truth depth data of the transparent object based on the depth image of the non-transparent object acquired from the real-world object image, and the depth of the transparent object is estimated using four local plane coefficients.

Images