Depth map generation method and apparatus

Abstract

The present invention provides a method and apparatus for generating depth maps. [Solution] The depth map generation method involves acquiring an RGB color image through a monocular camera installed in the robot system, acquiring a 3D point cloud through a LiDAR sensor installed in the robot system, generating a rare depth map from the 3D point cloud that contains depth information for only a portion of the points in a given space, inputting the RGB color image and the rare depth map into a pre-trained diffusion model, and generating a dense depth map that contains depth information for all points in the given space based on the diffusion model. The diffusion model is trained by introducing a loss function that reflects confidence, which is a numerical representation of the confidence level of the diffusion model's predictions.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for generating a depth map.

Background Art

[0002] Human-Robot Interaction (HRI) is a research field that understands, designs, and evaluates the interaction between humans and robots. The main aspects of HRI include communication and interaction, design and aesthetics, safety and trust, social interaction, adaptability and learning, etc. In particular, in the physical interaction between humans and robots, it is important that the robot behaves in a predictable and reliable manner, and that the robot recognizes the environment and the state of humans and adjusts its behavior accordingly. Depth estimation is essential when embodying such aspects. A depth map is a representation of the depth information of an object or environment in a three-dimensional space in a two-dimensional image format, and each pixel value of the depth map indicates the distance at that point. Depth maps are used not only in human-robot interaction, but also in 3D reconstruction, object detection and tracking, scene understanding, and robot navigation for a robot to recognize the surrounding environment and avoid obstacles while moving.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention has been made in view of the above prior art, and an object of the present invention is to provide a depth map generation method and apparatus that generate a refined dense depth map based on a sparse depth map and a diffusion model.

Means for Solving the Problem

[0005] A depth map generation method according to an aspect of the present invention made to achieve the above object is a depth map generation method executed by program code loaded in one or more memory devices through one or more processors, the method including: obtaining an RGB color image through a monocular camera provided in a robot system; obtaining a three-dimensional point cloud through a LiDAR (Light Detection and Ranging) sensor provided in the robot system; generating a sparse depth map including only depth information for points in a part of a given space from the three-dimensional point cloud; inputting the RGB color image and the sparse depth map into a pre-trained diffusion model; and generating a dense depth map including depth information for all points in the given space based on the diffusion model. The diffusion model is learned by introducing a loss function reflecting confidence, which numerically represents the reliability of the prediction of the diffusion model.

[0006] The depth map generation method may further include training the diffusion model to predict the dense depth map when the noise and the sparse depth map are input as training data according to a predetermined setting, using the noise and the sparse depth map. The steps for training the diffusion model may include: reading the predetermined settings; if it is determined that the predetermined settings include a first setting, normalizing the depth values ​​of the rare depth map to values ​​in the range of -1 to 1; setting one or more local regions in the rare depth map; replacing the position values ​​corresponding to the one or more local regions with the noise to the rare depth values ​​of the one or more local regions; and training the diffusion model based on the noise from which the position values ​​corresponding to the one or more local regions have been replaced and the normalized rare depth map.

[0007] The steps for training the diffusion model may include: reading the predetermined settings; if it is determined that the predetermined settings include a second setting, normalizing the depth values ​​of the rare depth map used as training data to a value in the range of -1 to 1; and training the diffusion model based on the noise and the normalized rare depth map.

[0008] The loss function can be determined by the following equation 1.

number

number

[0009] The aforementioned Confidence C dc This can be determined by the following equations 3 and 4.

number

number

number

number

[0010] The aforementioned C can be determined by the following equations 7 and 8.

number

number

[0011] The loss function can be determined by the following equation 9.

number

number

number

number

[0012] The aforementioned loss function can be determined by the following equation 13.

number

number

Equation

Equation

[0013] The loss function can be determined by the following Equation 17.

Equation

Equation

number

[0014] A depth map generation device according to one aspect of the present invention, made to achieve the above objective, is a depth map generation device that executes program code stored in one or more memory devices through one or more processors, wherein when the program code is executed, the one or more processors acquire an RGB color image through a monocular camera provided in the robot system, acquire a 3D point cloud through a LiDAR (Light Detection and Ranging) sensor provided in the robot system, generate a sparse depth map from the 3D point cloud that contains only depth information for some points in a given space, input the RGB color image and the sparse depth map into a pre-trained diffusion model, and generate a dense depth map that contains depth information for all points in the given space based on the diffusion model, wherein the diffusion model is trained by introducing a loss function that reflects confidence, which is a numerical representation of the confidence in the predictions of the diffusion model.

[0015] When the program code is executed, one or more processors may be further configured to train the diffusion model to predict the dense depth map when the noise and the rare depth map are input, using the noise and the rare depth map as training data according to predetermined settings. Training the diffusion model involves reading the predetermined settings, determining that the predetermined settings include a first setting, normalizing the depth values ​​of the rare depth map to values ​​in the range of -1 to 1, setting one or more local regions in the rare depth map, replacing the position values ​​corresponding to the one or more local regions with the noise to the rare depth values ​​of the one or more local regions, and training the diffusion model based on the noise from which the position values ​​corresponding to the one or more local regions have been replaced and the normalized rare depth map. Training the diffusion model may include reading the predetermined settings, and if it is determined that the predetermined settings include a second setting, normalizing the depth values ​​of the rare depth map used as training data to a value in the range of -1 to 1, and training the diffusion model based on the noise and the normalized rare depth map.

[0016] The loss function can be determined by the following equation 1.

number

number

[0017] The aforementioned Confidence C dc This can be determined by the following equations 3 and 4.

number

number

number

number

[0018] The aforementioned C can be determined by the following equations 7 and 8.

number

number

[0019] The loss function can be determined by the following equation 9.

number

number

number

number

[0020] The aforementioned loss function can be determined by the following equation 13.

number

number

number

number

[0021] The aforementioned loss function can be determined by the following equation 17.

number

number

number

[0022] According to the present invention, a rare depth map can be generated using data acquired from a LiDAR sensor and a monocular camera installed in a robot system, and a sophisticated dense depth map can be generated using a diffusion model. This makes it possible to obtain a sufficient amount of depth information from a rare depth map that lacks sufficient information for use in human interaction and navigation. Furthermore, both a discrete depth map corresponding to a pseudo-ground answer and a continuous depth map corresponding to the final ground answer can be obtained using the diffusion model. In addition, the accuracy of the dense depth map generated from the diffusion model can be improved by introducing noise and loss functions specifically designed for robot systems. [Brief explanation of the drawing]

[0023] [Figure 1] This is a block diagram illustrating a depth map generation device according to one embodiment. [Figure 2] This is a flowchart illustrating a depth map generation method according to one embodiment. [Figure 3] This is a flowchart illustrating a depth map generation method according to one embodiment. [Figure 4]This figure shows an example of the operation of a depth map generation device according to one embodiment. [Figure 5] This figure shows an example of the operation of a depth map generation device according to one embodiment. [Figure 6] This figure shows an example of the operation of a depth map generation device according to one embodiment. [Figure 7] This figure shows an example of the operation of a depth map generation device according to one embodiment. [Figure 8] This figure shows an example of the operation of a depth map generation device according to one embodiment. [Figure 9] This figure shows an exemplary density depth map generated by one embodiment. [Figure 10] This is a diagram illustrating a computing device according to one embodiment. [Modes for carrying out the invention]

[0024] Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the present invention can be embodied in various different forms and is not limited to the embodiments described herein. In the drawings, unnecessary parts have been omitted in order to clearly illustrate the present invention, and similar parts throughout the specification are denoted by similar reference numerals.

[0025] When the specification and claims as a whole state that a part "includes" a certain component, this does not exclude other components unless otherwise stated, and may include other components. Ordinal terms such as "first," "second," etc., are used to describe a variety of components, but components are not limited by the terminology. The terminology is used solely for the purpose of distinguishing one component from others.

[0026] The terms "~part," "~device," and "module" used in this specification refer to a unit that processes at least one function or operation as described herein, which can be embodied in hardware or circuitry, software, or a combination of hardware or circuitry and software. Furthermore, at least some of the configurations or functions of the depth map generation method and apparatus according to the embodiments described later can be embodied in a program or software, which can be stored in a computer-readable storage medium.

[0027] Figure 1 is a block diagram illustrating a depth map generation device according to one embodiment.

[0028] Referring to Figure 1, the depth map generation device 10 according to this embodiment executes program code or instructions stored in one or more memory devices through one or more processors. For example, the depth map generation device 10 is embodied in a computing device 50 as described later with respect to Figure 10. In this case, one or more processors correspond to the processor 510 of the computing device 50, and one or more memory devices correspond to the memory 530 of the computing device 50. The program code or instructions are executed by one or more processors and perform functions for generating sparse depth maps and dense depth maps from data acquired through the robot's sensors. In this specification, the term "module" is used to logically distinguish these functions performed by the program code.

[0029] The depth map generation device 10 according to this embodiment executes program code that includes an RGB image acquisition module 110, a rare depth map generation module 120, a diffusion model learning module 130, and a dense depth map generation module 140.

[0030] The RGB image acquisition module 110 acquires RGB color images through a monocular camera installed in the robot system. A monocular camera captures images through a single lens. Monocular cameras are widely used for robots to perceive their surroundings and identify or track objects because they are cost-effective, simple in configuration, and small in size. However, monocular cameras generally do not directly provide depth information.

[0031] The rare depth map generation module 120 acquires a 3D point cloud through a LiDAR (Light Detection and Ranging) sensor installed in the robot system, and generates a rare depth map from the 3D point cloud that contains only depth information for a portion of a given space, such as a specific point.

[0032] LiDAR sensors use light to measure distance to the surrounding environment. Specifically, they emit laser pulses to a target, measure the time it takes for the reflected pulses to return, calculate the distance to the target, and generate a 3D map of the surrounding environment based on this information. LiDAR sensors can measure distance with high precision, generate detailed 3D images, and not only allow for a precise understanding of the environment, but also enable use in dark environments and adverse weather conditions.

[0033] A 3D point cloud acquired through a LiDAR sensor is a collection of points in space, where each point corresponds to a specific location in the actual physical environment. In one embodiment, the 3D point cloud data includes information such as the location of the points (e.g., x, y, z coordinates), reflectivity, and color (e.g., RGB values).

[0034] A rare depth map does not include all points, but only those selected according to predetermined criteria. Some points selected from a 3D point cloud are projected onto a 2D plane to generate a 2D image, and each pixel in the 2D image is assigned the depth value (e.g., z-coordinate) of the corresponding 3D point. Areas not projected onto the 2D plane remain without depth values.

[0035] The diffusion model learning module 130 trains a diffusion model using rare depth maps generated by the rare depth map generation module 120 according to predetermined settings as training data.

[0036] A diffusion model is an algorithm designed to analogize the data generation process to the diffusion process in physics. It involves a diffusion process, which progressively degrades the actual data with noise, and an inverse process, or inverse diffusion process, which restores the original data from the noise. The diffusion process proceeds through various detailed steps, with noise being added to the data at each step, until the data is completely noisy. In the inverse process, the diffusion model learns how to restore the noise to the original data, ultimately removing the noise and recovering the original features of the data.

[0037] For example, original image x0 and image x that follows a completely random Gaussian. T Between, intermediate image x t-1 From intermediate image x t The diffusion process q(x) proceeds t |x t-1 ) starts with the original image x0 and the intermediate image x t-1 and intermediate image x t Image x T A Gaussian Markov chain is applied sequentially up to image x. TStarting from there, the reverse process p(x) returns to the original image x0. t-1 |x t The purpose of a diffusion model is to learn the intermediate image x. t From intermediate image x t-1 p(x) proceeds t-1 |x t ) and intermediate image x t-1 From intermediate image x t q(x) t |x t-1 The goal is to narrow the distance between ) and ). After the diffusion model is trained, image x follows a completely random Gaussian through sequential sampling. T Starting from, a realistic image x0 is generated. In one embodiment, p(x t-1 |x t ) and q(x t |x t-1 The distance between ) can be measured using the Kullback-Leibler divergence (KL-Divergence), and the difference can be measured using p(x t-1 |x t ) and q(x t |x t-1 Minimizing the distance between ) is equivalent to minimizing the Kullback-Leibler divergence.

[0038] The dense depth map generation module 140 generates a dense depth map containing depth information for all points in a given space by inputting the RGB color image acquired from the RGB image acquisition module 110 and the rare depth map generated from the rare depth map generation module 120 into a pre-trained diffusion model.

[0039] In one embodiment, the diffusion model learning module 130 uses noise and rare depth maps as training data according to predetermined settings to train the diffusion model to predict a dense depth map when noise and rare depth maps are input. Specifically, the diffusion model learning module 130 reads the predetermined settings, and if it determines that the predetermined settings include a first setting, it normalizes the depth values ​​of the rare depth maps used as training data to values ​​in the range of -1 to 1. For example, the actual distribution of depth values ​​in rare depth maps appears as values ​​between 0 and 80. In the case of a diffusion model, training proceeds while adding noise with values ​​between -1 and 1, so if depth values ​​between 0 and 80 are input directly to the diffusion model, training may not proceed normally due to differences in the range of values. To prevent such problems, the diffusion model learning module 130 normalizes the depth values ​​of the rare depth map used as training data to a range of -1 to 1, and then inputs the normalized values ​​into the diffusion model for training.

[0040] The diffusion model learning module 130 generates noise whose horizontal and vertical dimensions are the same as those of the rare depth map. In one embodiment, the diffusion model learning module 130 generates Gaussian noise containing random values ​​following a Gaussian distribution. The diffusion model learning module 130 then manipulates the noise by substituting its values ​​with other values. Specifically, the diffusion model learning module 130 sets one or more local regions in the rare depth map. The pixels in the set local regions have rare depth values, which are treated as a kind of ground truth, i.e., dense depth data. The diffusion model learning module 130 replaces the position values ​​corresponding to one or more local regions in the noise with the rare depth values ​​of one or more local regions. The diffusion model learning module 130 also trains a diffusion model based on the noise with the position values ​​corresponding to one or more local regions replaced and the normalized rare depth map. In other words, the diffusion model learning module 130 manipulates the noise value for a specific pixel with noise, thereby enabling a more accurate depth value to be predicted at that location.

[0041] A diffusion model is a model that finds the data distribution at the pixel level from random data. By setting a rare depth value as noise, the starting point of a given pixel location begins at that rare depth value, and the pixel location has a distribution with an even narrower deviation. On the other hand, since depth information is continuous, the depth value of a particular pixel is likely to be similar to the depth values ​​of surrounding pixels. The convolution operation takes this surrounding information into account, so more accurate depth estimation becomes possible through the influence that pixels with a rare depth value specified by noise have on their surroundings.

[0042] In one embodiment, the diffusion model learning module 130 reads a predetermined setting, and if it determines that the predetermined setting includes a second setting different from the first setting, it normalizes the depth values ​​of the rare depth map used as training data to a value in the range of -1 to 1. Next, the diffusion model learning module 130 generates noise whose horizontal and vertical dimensions are the same as the rare depth map. In one embodiment, the diffusion model learning module 130 generates Gaussian noise containing random values ​​following a Gaussian distribution. The diffusion model learning module 130 also trains a diffusion model based on the generated noise and the normalized rare depth map. That is, the diffusion model learning module 130 does not manipulate the noise value for specific pixels with the noise, and ensures that depth values ​​are predicted in an efficient manner that saves computing resources.

[0043] In one embodiment, the diffusion model is trained using a loss function that reflects confidence, which is a numerical representation of the confidence level of the diffusion model's predictions. Here, confidence is a measure of how confident the diffusion model is in a particular prediction, and is expressed as a probability between 0 and 1, for example. The closer the value is to 1, the more confident the diffusion model is in that prediction. The diffusion model training module 130 trains the diffusion model with different loss functions using multiple loss function introduction modes, taking into account specific objectives or environments.

[0044] In one embodiment, in the first loss function introduction mode, the loss function is determined by the following equation 1.

[0045]

number

[0046] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. * This is determined by the following equation 2.

[0047]

number

[0048] Here, C dc It is confidence, TIFF2026062444000048.tif6128 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual ground truth for the dense depth map, and PDDM (Predicted Dense Depth Map) is the predicted value for the dense depth map, R DxHxW This represents the set of real numbers (where D is the number of channels in the density depth map, H is the vertical length of the density depth map, and W is the horizontal length of the density depth map).

[0049] Confidence C dc This is determined by the following equations 3 and 4.

[0050]

number

number

[0051] Here, C is the difference between the output value and the correct value in the diffusion model, and C e This is determined by the following equations 5 and 6.

[0052]

number

number

[0053] Here, E is the edge map obtained after passing through the edge detector, Sobel() is a function that detects edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight. Sobel() calculates the slope of the pixel values ​​contained in the image, thereby allowing the image boundaries to be detected.

[0054] C is determined by the following equations 7 and 8.

[0055]

number

number

[0056] Here, α is a predetermined weight. The diffusion model learning module 130 trains the diffusion model by introducing such a loss function in the first loss function introduction mode.

[0057] In other embodiments, in the second loss function introduction mode, the loss function is determined by the following equation 9.

[0058]

number

[0059] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. * This is determined by the following formula 10.

[0060]

number

[0061] Here, GTDDM is the actual ground truth for the density depth map, PDDM is the predicted value for the density depth map, and R DxHxW is the set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C is determined by the following equations 11 and 12.

[0062]

number

number

[0063] Here, α is a predetermined weight. The diffusion model learning module 130 trains the diffusion model by introducing such a loss function in the second loss function introduction mode.

[0064] In other embodiments, in the mode introducing the third loss function, the loss function is determined by the following equation 13.

[0065]

number

[0066] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. * This is determined by the following formula 14.

[0067]

number

[0068] Here, TIFF2026062444000061.tif6128 The operator is the pixel-level dot product operator, GTDDM is the actual ground truth for the dense depth map, PDDM is the predicted value for the dense depth map, R DxHxWis the set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C can be determined by the following equations 15 and 16.

[0069]

number

number

[0070] Here, α is a predetermined weight. The diffusion model learning module 130 trains the diffusion model by introducing such a loss function in the third loss function introduction mode.

[0071] In other embodiments, in the fourth loss function introduction mode, the loss function is determined by the following equation 17.

[0072]

number

[0073] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. * This is determined by the following formula 18.

[0074]

number

[0075] Here, TIFF2026062444000066.tif6128 The operator is the pixel-level dot product operator, GTDDM is the actual ground truth for the dense depth map, PDDM is the predicted value for the dense depth map, R DxHxW This represents the set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and E* This is determined by the following formula 19.

[0076]

number

[0077] Here, E is the edge map obtained after passing through the edge detector, Sobel() is a function that detects edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight. The diffusion model learning module 130 trains the diffusion model by introducing such a loss function in the fourth loss function introduction mode.

[0078] Thus, the depth map generation device can improve the predictive quality and accuracy appropriate to the situation by determining different execution paths based on predetermined settings (for example, a combination of either the first or second setting and one of the first to fourth loss function introduction modes) that reflect and consider the specific implementation purpose and environment, and by training the diffusion model with the determined execution path. For example, by applying different settings to applications such as human-robot interaction, general object detection and tracking, scene understanding, and robot navigation, appropriate depth map generation can be realized, taking into account the performance required and computing resources consumed in each situation.

[0079] Figure 2 is a flowchart illustrating a depth map generation method according to one embodiment.

[0080] Referring to Figure 2, the depth map generation method according to this embodiment includes the steps of: acquiring an RGB color image through a monocular camera provided on the robot system (S201); acquiring a 3D point cloud through a LiDAR sensor provided on the robot system (S202); generating a sparse depth map from the 3D point cloud that includes only depth information for some points in a given space (S203); inputting the RGB color image and the sparse depth map into a pre-trained diffusion model (S204); and generating a dense depth map that includes depth information for all points in a given space (S205). For more detailed information on the above method, please refer to or apply the description of the embodiments described herein, and therefore, redundant explanations will be omitted here.

[0081] Figure 3 is a flowchart illustrating a depth map generation method according to one embodiment.

[0082] Referring to Figure 3, the depth map generation method according to this embodiment provides a color image acquired from a camera as a first input to the diffusion model, and a rare depth map generated from a 3D point cloud acquired through a LiDAR sensor as a second input to the diffusion model.

[0083] As described above, diffusion models perform a diffusion process that progressively degrades the actual data with noise, and a reverse or dediffusion process that restores the original data from the noise. Learning a diffusion model focuses on accurately modeling the amount of noise the neural network should predict at specific time steps, minimizing loss to guide the network to make increasingly accurate predictions, and ultimately enabling the network to mimic the actual data distribution.

[0084] Figure 4 shows an example of the operation of a depth map generation device according to one embodiment.

[0085] Referring to Figure 4, an example of how the diffusion model learning module 130 operates can be understood, where it replaces the values ​​of noise (A) with other values ​​to generate noise (C) with some values ​​replaced. One or more local regions are set in the rare depth map (B) acquired from the LiDAR sensor, as shown by red (gray in the figure) circles. Each of the set one or more local regions is assigned a rare depth value, and the noise values ​​at the locations on the noise (A) corresponding to one or more local regions in the rare depth map (B) are replaced with the corresponding rare depth values. A diffusion model trained on the noise (C) with the values ​​at the locations corresponding to one or more local regions replaced, and the normalized rare depth map (B), can predict more accurate density depth values. In one embodiment, a local region is a single pixel with a depth value acquired from the LiDAR sensor.

[0086] In one embodiment, noise (C) is generated by the following equation 20.

number

[0087] In equation 20, d s This represents the values ​​in local regions, as shown by the gray circles. The scarcity depth map (B) contains non-zero positive real values ​​in local regions and zero values ​​in the remaining regions excluding the local regions. Therefore, m i This acts as a mask indicating the locations of pixels with non-zero values ​​in the rarity depth map (B).

[0088] SDN defines noise (C). t Since the diffusion model operates through multiple iterations, t represents a specific iteration, and z t This represents the random noise in the relevant iteration. Therefore, SDN means noise obtained by replacing specific pixels in the random noise (i.e., pixels that have non-zero values ​​in the scarcity depth map (B)) with values ​​from the scarcity depth map (B). Here, The operator in TIFF2026062444000069.tif6128 is the dot product operator in pixels.

[0089] Figure 5 shows an example of operation of a depth map generation device according to one embodiment, and Figure 6 shows an example of operation of a depth map generation device according to one embodiment.

[0090] The diffusion model learning module 130 determines the conditions to be added along with noise as input to the diffusion model, concatenates the determined conditions with the noise, and trains the diffusion model based on the conditions concatenated with the noise. The conditions include one of the first to fifth conditions, where the first condition includes a rare depth map, the second condition includes an RGB color image and a rare depth map, the third condition includes an RGB color image, an edge image, and a rare depth map, the fourth condition includes a gray image and a rare depth map, and the fifth condition includes a gray image, an edge image, and a rare depth map.

[0091] Referring to Figure 5, the diffusion model learning module 130 trains the diffusion model based on an input that concatenates a first condition, which is conditioned only on the rare depth map, and noise. Here, the rare depth map corresponds to the rare depth map (B) in Figure 4, and the noise corresponds to the noise (C) in which the values ​​of the locations corresponding to one or more local regions in Figure 4 are replaced.

[0092] Referring to Figure 6, the diffusion model training module 130 trains a diffusion model based on an input that concatenates an RGB color image, an edge image, a third condition consisting of a rare depth map, and noise.

[0093] The depth map generation device runs under various conditions, determines the conditions that provide results with a high degree of similarity to the ground truth, and then trains using these determined conditions. By running and evaluating under multiple conditions in this way, it is possible to determine conditions optimized for specific objectives and environments, and then train based on these conditions, thereby improving prediction quality and accuracy.

[0094] Figures 7 and 8 show an example of the operation of a depth map generation device according to one embodiment.

[0095] Referring to Figures 7 and 8, in the process of improving depth information based on rare depth information, the rare depth information is propagated to surrounding pixels, and depth values ​​are estimated for areas where rare depth values ​​do not exist. To improve this depth information propagation method more flexibly and effectively, the depth map generation device according to this embodiment adjusts the pixel position to emphasize important parts of the input feature map through deformable convolution based on DySPN (Dynamic Spatial Propagation Network) and performs calculations. In this way, when information is propagated to the surrounding area by a pixel that has rare depth information, the relationship with surrounding pixels can be dynamically adjusted while the depth information is propagated, so that the initial depth map is improved to a more accurate depth map.

[0096] Figure 9 shows an exemplary density depth map generated by one embodiment.

[0097] Referring to Figure 9, (a) shows the ground truth image, (b) shows the image generated by a conventional diffusion model, (c) shows the image generated by a diffusion model trained with noise that replaces the position values ​​corresponding to one or more local regions with the rare depth values ​​of one or more local regions according to the first setting, and (d) shows the image generated by a diffusion model trained with noise according to the first setting and a loss function with confidence introduced by the first loss function introduction mode. The RMSE (Root Mean Square Error) is 218.04 for (b), 209.04 for (c), and 200.83 for (d), and it can be seen that (d) predicted the dense depth map most accurately.

[0098] Figure 10 is a diagram illustrating a computing device according to one embodiment.

[0099] Referring to Figure 10, the depth map generation method and apparatus according to this embodiment are implemented using a computing device 50. Such a computing device 50 can be implemented as various forms of electronic equipment, servers, or similar devices, and its functions are implemented through the combination of software and hardware.

[0100] The computing device 50 includes at least one of a processor 510, memory 530, user interface input device 540, user interface output device 550, and storage device 560, which communicate via bus 520. The computing device 50 also includes a network interface 570 which is electrically connected to network 40. The network interface 570 transmits or receives signals with other devices via network 40.

[0101] The processor 510 is a versatile computing device, such as an MCU (Micro Controller Unit), AP (Application Processor), CPU ( CeThis is embodied in a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a quantum processing unit (QPU), etc. The processor 510 is also a semiconductor device that executes instructions stored in memory 530 or storage device 560, and plays a core role in the system. The program code and data stored in memory 530 or storage device 560 instruct the processor 510 to perform specific tasks, thereby enabling the operation of the entire system. The processor 510 is configured to embody the various functions and methods described above in Figures 1 to 9.

[0102] The memory 530 and storage device 560 include various forms of volatile or non-volatile storage media for storing and accessing system data. For example, the memory 530 includes ROM (read-only memory) 531 and RAM (random access memory) 532. In one embodiment, the memory 530 is integrated into the processor 510, in which case the data transmission speed between the memory 530 and the processor 510 becomes very fast. In another embodiment, the memory 530 is located outside the processor 510, in which case the memory 530 is connected to the processor 510 through various data buses or interfaces. Such connections are made by various already known means, such as PCIe (Peripheral Component Interconnect Express) interfaces for high-speed data transmission or through memory controllers.

[0103] In one embodiment, at least some of the configurations or functions of the depth map generation method and apparatus according to this embodiment are embodied in a program or software executed on a computing device 50, and the program or software is recorded on a computer-readable recording medium. Specifically, the computer-readable recording medium in one embodiment is a recording of a program that causes a computer, including a processor 510 that executes programs or instructions stored in memory 530 or storage device 560, to perform steps included in the embodiment of the depth map generation method and apparatus according to this embodiment.

[0104] In one embodiment, at least some of the configurations or functions of the depth map generation method and apparatus according to this embodiment are implemented using the hardware or circuits of the computing device 50, or by separate hardware or circuits electrically connected to the computing device 50.

[0105] According to embodiments of the present invention, a rare depth map can be generated using data acquired from a LiDAR sensor and a monocular camera installed in a robot system, and a sophisticated dense depth map can be generated using a diffusion model. This makes it possible to obtain a sufficient amount of depth information from a rare depth map that lacks sufficient information for use in human interaction and navigation. Furthermore, using the diffusion model, both a discrete depth map corresponding to a similar ground truth and a continuous depth map corresponding to the final ground truth can be obtained. In addition, by introducing noise and loss functions specifically designed for robot systems, the accuracy of the dense depth map generated from the diffusion model can be improved.

[0106] Although embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the embodiments described above, and can be further implemented in various ways without departing from the technical spirit of the present invention. [Explanation of Symbols]

[0107] 10 Depth Map Generator 40 Networks 50 Computing Devices 110 RGB Image Acquisition Module 120 Rare Depth Map Generation Module 130 Diffusion Model Learning Module 140 Dense Depth Map Generation Module 510 Processor 520 Bus 530 memory 540 User Interface Input Device 550 User Interface Output Device 560 Storage device 570 Network Interfaces

Claims

1. A depth map generation method executed by program code stored in one or more memory devices via one or more processors, The steps include acquiring an RGB color image through a monocular camera installed in the robot system, The steps include: acquiring a three-dimensional point cloud through a LiDAR (Light Detection and Ranging) sensor provided in the robot system; The steps include generating a scarce depth map from the aforementioned three-dimensional point cloud, which contains only depth information for a portion of the points in a given space, The steps include inputting the RGB color image and the scarcity depth map into a pre-trained diffusion model, The process includes the step of generating a dense depth map containing depth information for all points in the given space based on the diffusion model, A depth map generation method characterized in that the diffusion model is trained by introducing a loss function that reflects confidence, which is a numerical representation of the reliability of the diffusion model's predictions.

2. The depth map generation method according to claim 1, further comprising the step of training the diffusion model to predict the dense depth map when the noise and the rare depth map are input, using the noise and the rare depth map as training data according to predetermined settings.

3. The step of training the diffusion model is: The step of reading the predetermined settings mentioned above, If it is determined that the predetermined setting includes the first setting, the steps include normalizing the depth value of the scarcity depth map to a value in the range of -1 to 1, The steps include setting one or more local regions in the aforementioned scarcity depth map, The steps include replacing the values ​​of the positions corresponding to one or more local regions with the noise with the rare depth values ​​of the one or more local regions, The depth map generation method according to claim 2, comprising the step of training the diffusion model based on noise in which the position values ​​corresponding to one or more local regions have been replaced and the normalized rare depth map.

4. The step of training the diffusion model is: The step of reading the predetermined settings mentioned above, If it is determined that the predetermined setting includes the second setting, the steps include normalizing the depth values ​​of the rare depth map used as training data to values ​​in the range of -1 to 1, The depth map generation method according to claim 2, comprising the step of training the diffusion model based on the noise and the normalized rare depth map.

5. The depth map generation method according to claim 1, characterized in that the loss function is determined by the following formula 1. [Math 1] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following equation 2. [Math 2] Here, C ｄｃ This is the aforementioned confidence, 【number】 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ This represents a set of real numbers (where D is the number of channels in the density depth map, H is the vertical length of the density depth map, and W is the horizontal length of the density depth map).

6. The aforementioned Confidence C ｄｃ The depth map generation method according to claim 5, characterized in that it is determined by the following formulas 3 and 4. [Math 3] [Math 4] Here, C is the difference between the output value and the correct answer in the diffusion model, and C ｅ This is determined by the following equations 5 and 6. [Math 5] 【Number 6】 Here, E is an edge map obtained by passing through an edge detector, Sobel() is a function for detecting edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight.

7. The depth map generation method according to claim 6, characterized in that C is determined by the following formulas 7 and 8. [Number 7] [Number 8] Here, α is a predetermined weight.

8. The depth map generation method according to claim 1, characterized in that the loss function is determined by the following formula 9. [Number 9] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 10. [Number 10] Here, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ is a set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C is determined by the following equations 11 and 12. [Math 11] [Math 12] Here, α is a predetermined weight.

9. The depth map generation method according to claim 1, characterized in that the loss function is determined by the following formula 13. [Number 13] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 14. [Number 14] Here, 【number】 The operator is a pixel-wise dot product operator, GTDD (Ground Truth Dense Depth Map) is the actual correct answer for the dense depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the dense depth map, and R ＤｘＨｘＷ means the set of real numbers (where D is the number of channels of the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C is determined by the following Equation 15 and the following Equation 16. [Number 15] [Number 16] Here, α is a predetermined weight.

10. The depth map generation method according to claim 1, characterized in that the loss function is determined by the following formula 17. [Number 17] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 18. [Number 18] Here, 【number】 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ This represents the set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and E ＊ This is determined by the following formula 19. [Number 19] Here, E is an edge map obtained by passing through an edge detector, Sobel() is a function for detecting edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight.

11. A depth map generation device that executes program code stored in one or more memory devices through one or more processors, When the program code is executed, one or more processors will: The robot system acquires an RGB color image through a monocular camera, A three-dimensional point cloud is acquired through a LiDAR (Light Detection and Ranging) sensor installed in the robot system. From the aforementioned 3D point cloud, a scarce depth map is generated that contains only depth information for a portion of the points in the given space. The RGB color image and the scarcity depth map are input into a pre-trained diffusion model. It is configured to generate a dense depth map containing depth information for all points in the given space based on the diffusion model, The depth map generation device is characterized in that the diffusion model is trained by introducing a loss function that reflects confidence, which is a numerical representation of the reliability of the diffusion model's predictions.

12. When the program code is executed, one or more processors will: The depth map generation device according to claim 11, further configured to train the diffusion model to predict the dense depth map when the noise and the rare depth map are input, using the noise and the rare depth map as training data according to predetermined settings.

13. Training the aforementioned diffusion model is Read the predetermined settings mentioned above, If it is determined that the predetermined setting includes the first setting, the depth value of the rare depth map is normalized to a value in the range of -1 to 1. One or more local regions are defined in the aforementioned scarcity depth map, The noise is used to replace the positional values ​​corresponding to one or more local regions with the rare depth values ​​of the one or more local regions. The depth map generation apparatus according to claim 12, characterized in that it includes training the diffusion model based on noise in which the position values ​​corresponding to one or more local regions have been replaced and the normalized rare depth map.

14. Training the aforementioned diffusion model is Read the predetermined settings mentioned above, If it is determined that the predetermined setting includes the second setting, the depth values ​​of the rare depth map used as training data are normalized to a value in the range of -1 to 1. The depth map generation apparatus according to claim 12, characterized by comprising training the diffusion model based on the noise and the normalized rare depth map.

15. The depth map generation device according to claim 11, characterized in that the loss function is determined by the following formula 1. [Math 1] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following equation 2. [Math 2] Here, C ｄｃ This is the aforementioned confidence, 【number】 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ This represents a set of real numbers (where D is the number of channels in the density depth map, H is the vertical length of the density depth map, and W is the horizontal length of the density depth map).

16. The aforementioned Confidence C ｄｃ The depth map generating device according to claim 15, characterized in that it is determined by the following formulas 3 and 4. [Math 3] [Math 4] Here, C is the difference between the output value and the correct answer in the diffusion model, and C ｅ This is determined by the following equations 5 and 6. [Math 5] [Math 6] Here, E is an edge map obtained by passing through an edge detector, Sobel() is a function for detecting edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight.

17. The depth map generation device according to claim 16, characterized in that C is determined by the following formulas 7 and 8. [Number 7] [Number 8] Here, α is a predetermined weight.

18. The depth map generation device according to claim 11, characterized in that the loss function is determined by the following formula 9. [Number 9] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 10. [Number 10] Here, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ is a set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C is determined by the following equations 11 and 12. [Math 11] [Math 12] Here, α is a predetermined weight.

19. The depth map generation device according to claim 11, characterized in that the loss function is determined by the following formula 13. [Number 13] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 14. [Number 14] Here, 【number】 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ is a set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and C is determined by the following equations 15 and 16. [Number 15] [Number 16] Here, α is a predetermined weight.

20. The depth map generation device according to claim 11, characterized in that the loss function is determined by the following formula 17. [Number 17] Here, L is the loss function, Mean() is the function that calculates the mean, and R represents the set of real numbers. ＊ This is determined by the following formula 18. [Number 18] Here, 【number】 The operator is the pixel-wise dot product operator, GTDDM (Ground Truth Dense Depth Map) is the actual correct answer for the density depth map, PDDM (Predicted Dense Depth Map) is the predicted value for the density depth map, and R ＤｘＨｘＷ This represents the set of real numbers (where D is the number of channels in the dense depth map, H is the vertical length of the dense depth map, and W is the horizontal length of the dense depth map), and E ＊ This is determined by the following formula 19. [Number 19] Here, E is an edge map obtained by passing through an edge detector, Sobel() is a function for detecting edge intensity in the edge map, γ is a predetermined reference value, and w is a predetermined weight.

Images