Apparatus for training, inference and methods therefor

Abstract

The present disclosure relates to an apparatus for training and effecting autonomous driving control of a vehicle.The apparatus may comprise at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the apparatus to obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point, obtain a depth estimation map based on an input image associated with the target time point and applied to a monocular depth estimation (MDE) model, update a plurality of weights present in the MDE model based on a loss function set applied to the MDE model, wherein the loss function set may comprise a first loss function obtained based on the depth distribution map and the depth estimation map, and output a signal indicative of the updated plurality of weights.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a training apparatus, an inference apparatus, and a training method, and more particularly to a technology for training a monocular depth estimation model. STATE OF THE ART

[0002] With the development of deep neural network-based computer vision technology in autonomous driving technology, various artificial intelligence models such as object detection, semantic segmentation, depth map estimation, and lane detection are being investigated.

[0003] In particular, the output of the monocular depth estimation model can be used for autonomous driving to detect a drivable area of ​​an autonomous vehicle. However, it is difficult to quantitatively obtain the uncertainty of the output of a monocular depth estimation model.

[0004] For example, in a regression task to estimate a specific value, a Bayesian and ensemble method can be used to quantitatively obtain the uncertainty by using multiple outputs for an input. However, it is inefficient to apply the method to an actual driving vehicle. Furthermore, in a classification task where a specific class is to be obtained, the uncertainty can be quantitatively obtained by using entropy. However, the number of cases in which depth information can be expressed can be reduced.

[0005] To solve these problems, it is necessary to develop a technology for training a monocular depth estimation model to estimate depth in pixel units and estimate the uncertainty of depth in an actual driving vehicle. DESCRIPTION OF THE INVENTION

[0006] According to the present disclosure, an apparatus may include at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point; obtain a depth estimation map based on an input image associated with the target time point and applied to a monocular depth estimation (MDE) model; update a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group may include a first loss function obtained based on the depth distribution map and the depth estimation map; and output a signal indicative of the updated plurality of weights.

[0007] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to effect autonomous driving control of a vehicle based on an updated MDE model, the updated MDE model being updated based on the updated plurality of weights.

[0008] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain the depth map by extracting pieces of depth information from the point cloud, wherein the pieces of depth information are associated with a plurality of pixels included in the depth map, obtain a depth tensor by extending a channel of the depth map, wherein the channel of the depth map is extended by a first condition based on the pieces of depth information, and obtain the depth distribution map based on the depth tensor.

[0009] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to determine, based on sensing information obtained from a sensor, a minimum discretization value and a maximum discretization value associated with channels included in a single pixel of the depth tensor, and to determine a discretization value of each of the channels, wherein the discretization value of each of the channels is determined based on an index of each of the channels included in the single pixel, the minimum discretization value, the maximum discretization value, and a number of the channels.

[0010] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to determine, based on a discretization value of an Nth channel of a single pixel of the depth tensor and a discretization value of an (N+1)th channel of the single pixel following the Nth channel, a ratio of the discretization value of the Nth channel and the discretization value of the (N+1)th channel as a representative discretization value of the Nth channel, and wherein N is a natural number and less than or equal to the total number of channels of the depth tensor.

[0011] The apparatus, wherein the depth distribution map may comprise pixels, and wherein a representative discretization value of a pixel of the pixels is associated with channels contained in the pixel of the pixels of the depth tensor, and wherein a sum of probabilities may comprise probabilities satisfying a second condition, the sum of probabilities being associated with channels contained in each of the pixels of the depth tensor.

[0012] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain a first depth estimate of a first pixel from a plurality of pixels present in the depth distribution map, obtain a second depth estimate of a second pixel from a plurality of pixels present in the depth estimate map, wherein the second pixel is related to a location corresponding to the first pixel, determine the first loss function based on the first depth estimate and the second depth estimate, and update the plurality of weights included in the MDE model based on the determined first loss function to obtain the second depth estimate.

[0013] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to determine a difference between the first estimate depth and the second estimate depth as the first loss function based on the first estimate depth satisfying a third condition, or to skip updating the plurality of weights included in the MDE model to obtain the second estimate depth based on the first estimate depth not satisfying the third condition.

[0014] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain pose change information by applying an input image at a time different from the target time and an input image at the target time to a pose estimation model, obtain a first point cloud at the target time by applying an inverse of an intrinsic parameter related to a sensor to the depth estimation map, obtain a second point cloud at a time different from the target time by applying the pose change information to the first point cloud, and determine, based on the second point cloud, a second loss function different from the first loss function, and wherein the loss function set may further comprise the second loss function.

[0015] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain a reconstruction image by applying the intrinsic parameter to the second point cloud and to determine the second loss function based on the input image and the reconstruction image.

[0016] The apparatus, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain a first factor having a mean value associated with channels of a target pixel among pixels included in the depth estimation map and a second factor having a standard deviation value associated with the channels of the target pixel, and to obtain, based on the first factor and the second factor, a relative standard deviation value indicative of the uncertainty of the target pixel.

[0017] According to the present disclosure, an apparatus may include at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: obtain a target image for testing; obtain a target depth estimation map by applying the target image to a monocular depth estimation model having updated weights, wherein the target depth estimation map may include an estimation depth of each of a plurality of pixels included in the target image; obtain a target uncertainty map, wherein the target uncertainty map may include a relative standard deviation value of each of a plurality of estimation depths included in the target depth estimation map; and output a signal indicative of the target uncertainty map.

[0018] According to the present disclosure, a method performed by a processor may include: obtaining a depth distribution map based on a depth map obtained from a point cloud at a target time point, obtaining a depth estimation map by applying an input image associated with the target time point to a monocular depth estimation (MDE) model, updating a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group may include a first loss function obtained based on the depth distribution map and the depth estimation map, and outputting a signal indicative of the updated plurality of weights.

[0019] The method, wherein obtaining the depth distribution map may comprise: obtaining the depth map by extracting pieces of depth information from the point cloud, wherein the pieces of depth information are associated with a plurality of pixels included in the depth map, obtaining a depth tensor by expanding a channel of the depth map, wherein the channel of the depth map is expanded by a first condition based on the pieces of depth information, and obtaining the depth distribution map based on the depth tensor.

[0020] The method, wherein obtaining the depth distribution map may comprise determining, based on sensing information obtained from a sensor, a minimum discretization value and a maximum discretization value associated with channels included in a single pixel of the depth tensor, and determining a discretization value of each of the channels, wherein the discretization value of each of the channels is determined based on an index of each of the channels included in the single pixel, the minimum discretization value, the maximum discretization value, and a number of the channels.

[0021] The method, wherein obtaining the depth distribution map may comprise determining, based on a discretization value of an Nth channel of a single pixel and a discretization value of an (N+1)th channel of the single pixel following the Nth channel, a ratio of the discretization value of the Nth channel and the discretization value of the (N+1)th channel as a representative discretization value of the Nth channel, and wherein N is a natural number and less than or equal to a total number of channels of the depth tensor, wherein the depth distribution map may comprise pixels, and wherein a representative discretization value of a pixel of the pixels is associated with channels included in the pixel of the pixels of the depth tensor, and wherein a sum of probabilities may comprise probabilities satisfying a second condition, wherein the sum of probabilities is associated with channels,contained in each of the pixels of the depth tensor.

[0022] The method, wherein updating the plurality of weights included in the MDE model may comprise: obtaining a first estimation depth of a first pixel among a plurality of pixels included in the depth distribution map, obtaining a second estimation depth of a second pixel among a plurality of pixels included in the depth estimation map, the second pixel being associated with a location corresponding to the first pixel, determining, based on the first estimation depth and the second estimation depth, the first loss function, and updating, based on the determined first loss function, the plurality of weights included in the MDE model to obtain the second estimation depth.

[0023] The method, wherein updating the plurality of weights included in the MDE model based on the first estimation depth satisfying a third condition may comprise determining a difference between the first estimation depth and the second estimation depth as the first loss function, or, based on the first estimation depth not satisfying the third condition, skipping updating the plurality of weights included in the MDE model to obtain the second estimation depth.

[0024] The method may further comprise: obtaining pose change information by applying an input image at a time different from the target time and an input image at the target time to a pose estimation model, obtaining a first point cloud at the target time by applying an inverse of an intrinsic parameter related to a sensor to the depth estimation map, obtaining a second point cloud at a time different from the target time by applying the pose change information to the first point cloud, and determining, based on the second point cloud, a second loss function different from the first loss function, and wherein the loss function group may further comprise the second loss function.

[0025] The method, wherein determining the second loss function may comprise obtaining a reconstruction image by applying the intrinsic parameter to the second point cloud, and determining the second loss function based on the input image and the reconstruction image.

[0026] The method may further comprise obtaining a first factor indicative of a mean value associated with channels of a target pixel among pixels included in the depth estimation map and a second factor indicative of a standard deviation value associated with the channels of the target pixel, and obtaining, based on the first factor and the second factor, a relative standard deviation value indicative of the uncertainty of the target pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Fig. 1 shows an example of a training device for a monocular depth estimation model, according to an example of the present disclosure; Fig. 2 shows an example of a flowchart for describing a method for training a monocular depth estimation model, according to an example of the present disclosure; Fig. 3 shows an example of a method for training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device, according to an example of the present disclosure; Fig. 4 shows an example of a method for obtaining a depth distribution map based on a depth map in a training device, according to an example of the present disclosure; Fig. 5 shows an example of generating a depth distribution map in a training device according to an example of the present disclosure; Fig. 6 shows an example of a method for training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device, according to an example of the present disclosure; Fig. 7 shows an example of a target image, an estimation depth of each of the pixels included in an input image, and the uncertainty of each of the pixels included in the input image in an inference apparatus according to an example of the present disclosure; Fig. 8 shows an example of a correlation between an estimation depth and an RSD in a training device according to an example of the present disclosure; and Fig. 9 shows an example of a computing system connected to a training device or method according to an example of the present disclosure.

[0028] When describing drawings, identical or similar components are identified by identical or similar reference symbols. DETAILED DESCRIPTION

[0029] Hereinafter, some examples of the present disclosure will be described in detail with reference to the accompanying drawings. When attaching reference numerals to components in each drawing, it should be noted that the same components will have the same reference numerals even if they are indicated on a different drawing. Furthermore, in describing the examples of the present disclosure, detailed descriptions related to well-known functions or configurations will be omitted if they might unnecessarily obscure the subject matter of the present disclosure. Hereinafter, various examples of the present disclosure may be described with reference to the accompanying drawings. Accordingly, those skilled in the art will recognize that modifications, equivalents, and / or alternative embodiments of the examples described herein are possible without departing from the scope and spirit of the present disclosure.With regard to the description of the drawings, similar components may be identified by similar reference numerals.

[0030] In describing elements of an example of the present disclosure, the terms first, second, A, B, (a), (b), and the like may be used herein. These terms are used only to distinguish one element from another and do not limit the corresponding elements, regardless of the type, order, or priority of the corresponding elements. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, are to be interpreted as is customary in the art to which the present disclosure belongs. It is to be understood that the terms used herein have a meaning consistent with their meaning in the context of the present disclosure and the relevant prior art, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.For example, terms used herein such as "first," "second," and the like may refer to various components of various examples of the present disclosure, but are not limiting. For example, "a first device" and "a second device" may refer to different devices, regardless of their order or priority. For example, without departing from the scope of the present disclosure, a first supplement may be referred to as a second component, and similarly, a second supplement may be referred to as a first supplement.

[0031] The terms "have", "may have", "comprise" and "have" or "may include" and "may have" used in this specification indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations, or components), but do not exclude the presence of additional features.

[0032] It should be understood that when an element (e.g., a first element) is described as being "(operatively or communicatively) connected to" or "connected to" another element (e.g., a second element), it may be directly coupled or connected to the other element, or an intervening element (e.g., a third element) may be present. On the other hand, when an element (e.g., a first element) is described as being "directly coupled to" or "directly connected to" another element (e.g., a second element), it should be understood that no intervening element (e.g., a third element) is present.

[0033] Depending on the situation, the term “adapted to” as used here can be used to mean, for example, “suitable for”, “capable of”, “designed to”, “adapted to”, “made to” or “capable of”.

[0034] The phrase "configured to" may not only mean "particularly adapted to" in hardware. Instead, the phrase "a device configured to" may also mean that the device is "capable of cooperating with" another device or other components. For example, a "processor configured (or adjusted) to perform A, B, and C" may mean a special-purpose processor (e.g., an embedded processor) for performing a corresponding operation, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that performs corresponding operations by executing one or more software programs stored in a memory device. The terms used in the description are for the purpose of describing a specific example only and are not intended to limit the scope of the present disclosure.Unless otherwise specified, terms in the singular may also have the plural. All terms used herein that have technical or scientific terms may have the same meaning as commonly understood by a person trained in the art. Further, it is to be understood that terms defined in a dictionary and commonly used should also be interpreted as is customary in the relevant related art, and not in an idealized or overly formal disclosure, unless expressly defined so herein in various examples of the present disclosure. In some cases, even if terms are terms defined in the specification, they may not be interpreted to exclude examples of the present disclosure.

[0035] In the present disclosure, the terms "A or B," "at least one of A and / or B," or "one or more of A and / or B," and the like, as used herein, may include any and all combinations of one or more of the associated listed elements. For example, the term "A or B," "at least one of A and B," or "at least one of A or B" may refer to all cases (1) where at least one A is present, the case (2) where at least one B is present, or the case (3) where both at least one A and at least one B are present.Furthermore, when describing a component of an example of the present disclosure, the terms "at least one of A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," or "at least one of A, B, or C," or any combination thereof, may include any and all combinations of one or more of the associated listed elements. In particular, the terms "at least one of A, B, or C, or any combination thereof" may include A, B, or C, or any combination thereof, such as AB, ABC, or the like.

[0036] Below, various examples of the present disclosure are described with reference to the Fig. 1 to 9 described in detail.

[0037] Fig. 1 shows an example of a training device for a monocular depth estimation model according to an example of the present disclosure.

[0038] For example, a device 100 may include a processor 110 and a memory 120 containing instructions 122.

[0039] The device 100 may indicate a device that trains a monocular depth estimation model. For example, the device 100 may update a plurality of weights included in the monocular depth estimation model by applying a loss function set to the monocular depth estimation model. In other words, the device 100 may train the monocular depth estimation model by updating the plurality of weights included in the monocular depth estimation model.

[0040] The monocular depth estimation model may be a network with a U-net structure, including an encoder and a decoder. Here, the encoder may be a ResNet model, and the decoder may be a model that converts a sigmoid output into a depth estimation map. Specifically, the device may obtain the depth estimation map from the decoder by applying an input image of a target time point to the encoder of the monocular depth estimation model. A detailed description of the monocular depth estimation model will be provided later in Fig. 2 described below.

[0041] The loss function group may include a first loss function and a second loss function. The device 100 may update the plurality of weights included in the monocular depth estimation model by applying each of the first loss function and the second loss function to the monocular depth estimation model.

[0042] The first loss function can be obtained based on a depth distribution map and a depth estimation map.

[0043] The depth distribution map may display a set of depth information (e.g., information about a distance between an object and a lidar) indicating three-dimensional (3D) distance information of an object and a background present in an image obtained when the lidar acquires an actual scene including a predetermined area based on a location of a vehicle. Specifically, the depth distribution map may include a plurality of pixels obtained by dividing the object and the background present in the above-described image using a grid having a predetermined size. The depth distribution map may include depth information about each of the plurality of pixels. Specifically, the depth distribution map may include a probability distribution of the depth information for each of the plurality of pixels.Accordingly, the depth distribution map may display a map including the probability distribution of the depth information of each of the plurality of pixels in an image in which an actual scene is displayed using the plurality of pixels.

[0044] The depth estimation map may display a set of depth information (e.g., information about a distance between an object and a camera including an image sensor) obtained by a monocular depth estimation model from 3D distance information of an object and a background present in an image obtained when a mono camera (or a monocular camera) captures an actual scene including a predetermined area based on a location of the vehicle. Specifically, the depth estimation map may include a plurality of pixels obtained by dividing the object and the background present in the above-described image using a grid having a predetermined size. The depth estimation map may include depth information about each of the plurality of pixels obtained by a monocular depth estimation model.

[0045] Accordingly, for convenience of description, in this specification, it is mainly described that the depth distribution map may include depth information of the image obtained by the lidar, and that the depth estimation map may include depth information obtained by applying an image obtained by a mono camera to a monocular depth estimation model.

[0046] A method in which the device 100 determines a first loss function is as follows. For example, the device 100 may obtain a depth distribution map based on a depth map obtained from a point cloud of a target time point. Furthermore, the device 100 may obtain a depth estimation map by applying the input image of the target time point to a monocular depth estimation model. The device 100 may determine a first loss function based on the depth distribution map and the depth estimation map. The detailed method by which the device 100 determines the first loss function will be described later in Fig. 6 described below.

[0047] The second loss function can be obtained based on a reconstruction image and the input image of the target time point.

[0048] The input image of the target time point may be generated by an image sensor at the target time point. An input image of a time point different from the target time point may be generated by an image sensor at a time point different from the target time point. In this case, the time point different from the target time point may be at least one of the other time points excluding the target time point from the time points within a threshold time interval based on the target time point.

[0049] A method in which the device 100 determines a second loss function is as follows. For example, the device 100 may obtain pose change information by applying an input image of a time point different from the target time point and an input image of the target time point to a pose estimation model. The device 100 may obtain a first point cloud of the target time point by applying the inverse of an intrinsic parameter related to the lidar to the depth estimation map. The device 100 may obtain a second point cloud of a time point different from the target time point by applying the pose change information to the first point cloud. Finally, the device 100 may determine a second loss function different from the first loss function based on the second point cloud.The detailed method by which the device 100 determines the second loss function is described later in . Fig. 6 described below.

[0050] The pose estimation model may generate pose change information corresponding to a pose change between the input image at the target time point and the input image at a time point different from the target time point. For example, the pose estimation model may generate pose change information based on a first pose at the target time point of an image sensor and a second pose at a time point different from the target time point.

[0051] The pose change information may be information indicating a relationship between a camera position of the target time point and a camera position of a time point different from the target time point, and may be information corresponding to a rotation and translation matrix of an object.

[0052] The processor 110 may execute software and control at least one other component (e.g., a hardware or software component) connected to the processor 110. The processor 110 may also perform various data processing or operations. For example, the processor 110 may store the depth distribution map, the depth estimation map, and the loss function set in the memory 120. For reference, the processor 110 may perform some or all of the operations performed by the device 100. Therefore, for ease of description in this specification, an operation performed by the device 100 will be described primarily as an operation performed by the processor 110.

[0053] Furthermore, for ease of description, in this specification, processor 110 is primarily described as, but not limited to, a single processor. For example, device 100 may include at least one processor. The at least one processor may perform some or all of the operations associated with training the monocular depth estimation model.

[0054] Memory 120 may temporarily and / or permanently store various data and / or information for training the monocular depth estimation model. For example, memory 120 may store the depth distribution map, the depth estimation map, and the loss function set.

[0055] Fig. 2 shows an example flowchart for describing a method for training a monocular depth estimation model according to an example of the present disclosure.

[0056] In S210, a device (e.g., the device 100 in Fig. 1) Obtain a depth distribution map based on a depth map obtained from a point cloud of a target time point.

[0057] The point cloud can be collected by a lidar or an RGB-D sensor. The point cloud can comprise a set of points obtained by calculating distance information per light / signal of an object measured by the lidar or RGB-D sensor. Unlike a two-dimensional (2D) image, the point cloud can contain information in a depth direction (e.g., z-axis), and thus the point cloud can be expressed in three dimensions.

[0058] The depth map may represent an image created by extracting only information in the depth direction from the point cloud. For example, the depth map may have pixels that omit information (e.g., x-axis) in the horizontal direction and information (e.g., y-axis) in the vertical direction from each of the pixels contained in the point cloud.

[0059] In S220, the device may obtain a depth estimation map by applying the input image of the target time point to a monocular depth estimation model.

[0060] The monocular depth estimation model may have a plurality of layers, and each layer may have a plurality of nodes. The node may have a node value determined based on an activation function. A node at any layer may be connected to a node at another layer (e.g., another node) via a link (e.g., a connecting edge) with a connection weight. The node value of a node may be propagated to other nodes via the link. In an inference operation of the monocular depth estimation model, node values ​​may be propagated from the previous layer to the next layer.

[0061] For example, the forward propagation operation in the monocular depth estimation model may indicate an operation of propagating node values ​​based on input data in one direction from an input layer of the monocular depth estimation model to an output layer of the monocular depth estimation model. In other words, the node value of the corresponding node may be propagated (e.g., forward propagated) to a node (e.g., the next node) of the next layer connected by the node and the connecting edge. For example, the node may receive a value weighted by a connection weight from the previous node (e.g., a plurality of nodes) connected by the connecting edge.

[0062] The node value of a node can be determined based on the application of an activation function (e.g., a sigmoid function) to the sum (e.g., weighted sum) of weighted values ​​received from previous nodes. For example, a parameter of a neural network can include the connection weight described above. The parameters of the neural network can be updated such that a value of a loss function (e.g., a first loss function and a second loss function), described later, changes in a target direction (e.g., a direction in which a loss is minimized).

[0063] The monocular depth estimation model may refer to a model that can be trained by machine learning and may be a machine learning model that outputs a training output (e.g., a depth estimation map) from a training input (e.g., an input image).

[0064] For example, the learning algorithm can use supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the above example. However, for ease of description in this specification, a training algorithm of the monocular depth estimation model is primarily described as semi-supervised learning.

[0065] The monocular depth estimation model may include multiple layers of an artificial neural network. The artificial neural network may be one of the following: a deep neural network (DNN), a convolutional neural network (CNN), a U-Net for image segmentation (U-Net), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-Net, or at least one combination among combinations thereof, but it is not limited to the example described above. However, for the convenience of description in this specification, the monocular depth estimation model is mainly described as a network with a U-Net structure and may be a model in which an encoder is a ResNet model and the decoder is a model that converts a sigmoid output into a depth estimation map.

[0066] The U-Net, a monocular depth estimation model used by the device, may consist of an encoder, a decoder, and a combination thereof. The encoder can generally extract a hierarchical structure of an image feature map from low to high complexity. On the other hand, the decoder can transform a feature and reconstruct an output from low resolution to high resolution.

[0067] The monocular depth estimation model may include a convolutional layer (conv) that performs a linear transformation operation, a batch normalization layer (BN) that performs a normalization operation, a rectified linear unit (ReLU) that performs a nonlinear function operation, and a channel concatenation layer or channel sum layer that combines the outputs of a plurality of layers.

[0068] The monocular depth estimation model can be trained to output a training output from a training input. The monocular depth estimation model can generate a temporary output in response to the training input during training and can be trained such that the loss between the temporary output and the training output (e.g., a training target) is minimized. During a training process, a parameter (e.g., a connection weight between nodes / layers in a neural network) of the machine learning model can be updated depending on the loss. This training can, for example, be performed by the device itself in which the training of the monocular depth estimation model takes place and can be done via a separate server. The monocular depth estimation model in which the training is completed can be stored in a memory (e.g., the memory 120 in Fig. 1) are stored.

[0069] In S230, the device may update a plurality of weights included in the monocular depth estimation model by applying a loss function group including the first loss function obtained based on the depth distribution map and the depth estimation map to the monocular depth estimation model. However, a method for training the monocular depth estimation model is not limited to this. As described above in Fig. 1, the apparatus may train the monocular depth estimation model based on the loss function group including the first loss function and the second loss function.

[0070] Fig. 3 shows an example of a method for training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device, according to an example of the present disclosure.

[0071] A device (e.g., the device 100 in Fig. 1) may obtain a depth distribution map 305 based on a depth map 303 obtained from a point cloud 301 of a target time point. The device may obtain a depth estimation map 311 by applying an input image 307 of the target time point to a monocular depth estimation model 309. The device may obtain a first loss function 313 based on the depth distribution map 305 and the depth estimation map 311.

[0072] The device may obtain pose change information 319 by applying an input image 315 of a time point different from the target time point and the input image 307 of the target time point to a pose estimation model 317. The device may obtain a reconstruction image 321 based on the pose change information 319. The device may obtain a second loss function 323 based on the obtained reconstruction image 321 and the input image 307 of the target time point. A detailed description of the obtainment of the reconstruction image 321 will be given later in Fig. 6 described below.

[0073] The input image 315 of a time point different from the target time point and the input image 307 of the target time point may be obtained from a single image taken while the vehicle is traveling. For example, the input image 315 of a time point different from the target time point may indicate an image obtained at a location where the vehicle is traveling at a time point t' different from the target time point. Similarly, the input image 307 of the target time point may indicate an image obtained at a location where the vehicle is traveling at a target time point t (i.e., at a time point subsequent to time point t'). Accordingly, the device may obtain the pose change information 319 in the form of a matrix comprising information about pose changes of objects included in temporally continuous images by applying the input image 307 and the input image 315 to the pose estimation model 317.

[0074] The apparatus may update a plurality of weights included in the monocular depth estimation model 309 (i.e., train the monocular depth estimation model 309) by applying the loss function group comprising the first loss function 313 and the second loss function 323 to the monocular depth estimation model 309.

[0075] The device may obtain the uncertainty 329 of each of a plurality of pixels included in the depth estimation map based on the depth estimation map 311 obtained from the trained monocular depth estimation model 309. For example, the device may obtain an estimated depth 325 of each of a plurality of pixels present in the depth estimation map 311.

[0076] Here, the depth estimation map 325 may include distance information of an object (e.g., it may indicate a portion of an object) contained in a target pixel. For example, pixel A of the depth estimation map 311 may have an estimated depth with a value of "a," and pixel B of the depth estimation map 311 may have an estimated depth with a value of "b."

[0077] In addition or alternatively to the estimation depth of the target pixel, the device may obtain the average of the channels of the target pixel and the standard deviation of the channels through the standard deviation estimation 327. The device may obtain the uncertainty 329 of the target pixel based on the mean of the channels of the target pixel and the standard deviation of the channels. A detailed description of this will be given later in Fig. 6 described below.

[0078] As described above, the device may use the first loss function 313 and the second loss function 323 to train the monocular depth estimation model 309. Furthermore, the device may obtain the uncertainty 329 of the target pixel corresponding to the estimated depth 325 based on the estimated depth 325 obtained from the monocular depth estimation model 309.

[0079] If the device does not perform a proposal process (hereinafter referred to as “proposal process”) to obtain the depth distribution map 305 from the point cloud 301, which will be described later in the Fig. 4 and Fig. 5, the performance of the monocular depth estimation model 309 that outputs the depth estimation map 311 and the uncertainty 329 may be degraded. For example, if only the one depth estimation map 311 is obtained from the monocular depth estimation model 309, the device may train the monocular depth estimation model 309 without performing the proposal process. In this case, the performance of the monocular depth estimation model 309 may not be degraded. On the other hand, if the device obtains the uncertainty 329 of the target pixel in addition to or alternatively to the depth estimation map 311 from the monocular depth estimation model 309, the performance of the monocular depth estimation model 309 may be degraded if the device does not perform the proposal process.Accordingly, a method for training the monocular depth estimation model 309 is described below by a detailed description of the proposal method with reference to FIG. Fig. 4 and Fig. 5 described.

[0080] Fig. 4 shows an example of a method for obtaining a depth distribution map based on a depth map in a training device according to an example of the present disclosure.

[0081] A device (e.g. the device 100 in Fig. 1) may obtain a point cloud (e.g., a point cloud 410) from a scene 400 by a lidar. The device may obtain a depth map by extracting portions of the depth information from the point cloud 410. As described above in Fig. 1, the depth map may display a 2D image created by extracting information in a depth direction from the point cloud 410.

[0082] The device may obtain a three-dimensional (3D) representation from a two-dimensional (2D) image (e.g., a depth tensor 420) by augmenting a learned feature of input data (e.g., a channel of the depth map, a single-layer 2D array within a data structure containing information about a depth at each pixel, etc.) by a predetermined first condition based on each of a plurality of pixels included in the depth map. In other words, the depth tensor 420 may display a tensor having as many probability values ​​as the number of channels, corresponding to an estimated depth of each pixel included in the depth map, which is a 2D image. Referring to Fig. 4 describes an example where the number of channels is “K”.

[0083] Thereafter, the device may obtain a depth distribution map to obtain a first loss function based on the depth tensor 420. A method for obtaining a depth distribution map from the depth tensor 420 will be described later in Fig. 5 described below.

[0084] Fig. 5 shows an example of generating a depth distribution map in a training device according to an example of the present disclosure.

[0085] With reference to Fig. 5, a sentence 510 having probabilities corresponding to an estimated depth contained in a target pixel 505 can be described primarily with a focus on the target pixel 505 in a depth tensor 500. Furthermore, the depth of an actual object of the target pixel 505 is described as 5.2 m.

[0086] A table 515 may include a discretization value (e.g., a threshold or interval used to divide continuous data into discrete ranges, a step size or incremental value, a scaling factor used to map floating-point numbers to integers, etc.), a representative discretization value, and probabilities for each of a plurality of channels included in the set 510. For example, the device (e.g., device 100 in Fig. 1) Expand the channel a number of times as required to satisfy a first condition (e.g., 22 channel conditions) by using the depth information of each pixel based on the plurality of pixels contained in the depth map. Set 510 may include probability values ​​corresponding to the channels, respectively, when the target pixel's depth estimate is expanded to 22 channels.

[0087] The device may determine a first discretization value (e.g., a minimum discretization value or a relatively small discretization value) and a second discretization value (e.g., a maximum discretization value or a discretization value greater than the first discretization value) based on a sensor (e.g., a lidar) (i.e., expand the state of 22 channels based on a lidar specification) to determine the discretization value of each of the channels included in each individual pixel or the target pixel of the depth tensor. For example, based on Fig. 5, the minimum discretization value is 0.01 and the maximum discretization value is 120.

[0088] The device may determine a discretization value of each of the channels present in each individual pixel or the target pixel 505 based on an index (e.g., channel 0, channel 1, channel 2, ..., channel 21), a minimum discretization value (e.g., 0.01), a maximum discretization value (e.g., 120), and the number of channels (e.g., a first condition of 22) of each of the channels present in the individual pixel or the target pixel 505.

[0089] A method in which the device determines the discretization value of each channel can be expressed by Equations 1 to 3 below. di=dmin+(dmax−dmin)×i / nbins

[0090] Here, equation 1 can mean a uniform discretization. d min can denote a minimal discretization value; d max can denote a maximum discretization value; i can denote a channel index; and n binscan denote the number of channels, i.e. the number of targets determined by discretization values. di=exp (log(dmin)+log(dmaxdmin)∗inbins)

[0091] Here, equation 2 can mean that the distance increases the discretization. d min , d max , i and n bins can be the same as described in equation 1. di=dmin+dmax−dminnbins(nbins+1)×i×(i+1)

[0092] Here, Equation 3 can represent a linearly increasing discretization. d_min, d_max, i, and n_bins can be the same as described in Equation 1 and Equation 2.

[0093] For ease of explanation in this specification, it is specifically described that the device determines the discretization values ​​of the channels based on the linearly increasing discretization of Equation 3. Accordingly, as for the discretization values ​​in a table 515, the discretization values ​​in the table 515 may include values ​​obtained by determining the discretization value of each channel included in the individual pixel by determining the index, the minimum discretization value, the maximum discretization value, and the number of channels according to the linearly increasing discretization in Equation 3.

[0094] Thereafter, the device may determine the ratio between the discretization value of the N-th channel and the discretization value of the (N+1)-th channel as the representative discretization value of the N-th channel based on the case where a discretization value of the N-th channel of the single pixel or the target pixel 505 and a discretization value of the (N+1)-th channel of the single pixel or the target pixel 505 following the N-th channel are determined.

[0095] A method in which the device determines the representative discretization value can be expressed by Equation 4 below. mid=BN+BN+12

[0096] Here, mid can denote a representative discretization value; B N can denote a discretization value of the N-th channel; and B N+1 can denote a discretization value of the (N+1)th channel.

[0097] Accordingly, with reference to the representative discretization values ​​in table 515, the device may determine that the first representative discretization value is 0.340476 based on a discretization value of 0.01 for the first channel and a discretization value of 0.6709524 for the second channel. Furthermore, the device may determine representative discretization values ​​of a plurality of channels according to Equation 4.

[0098] The device may set a probability value for each channel based on the case where the representative discretization values ​​of the channels are determined. For example, the device may set a probability value that matches the representative discretization value for each channel.

[0099] Specifically, in a situation where the target pixel 505 has an estimated depth corresponding to 5.2 m, which is the depth of the actual object, the device may set a probability value that matches the representative discretization value while increasing or decreasing the probability value by 0.05 from 0 to 1. For example, to obtain the probability distribution of an estimated depth corresponding to a depth of 5.2 m, the device may set the representative discretization value (e.g., 4.667619) of channel 4 and the representative discretization value (e.g., 7.236905) of channel 5, which are closest to 5.2 m, as the region of interest. The device may increase or decrease the probability value that matches the representative discretization value corresponding to two channels present in the region of interest by 0.05 from 0 to 1.

[0100] More specifically, the device may successively set probability values ​​of 1, 0.95, 0.9, 0.85, and 0.8 as a probability value corresponding to the representative discretization value (e.g., 4.667619) of channel 4. Similarly, the device may successively (i.e., the second condition that the sum can be 1) set probability values ​​of 0, 0.05, 0.1, 0.15, and 0.2 as a probability value corresponding to the representative discretization value (e.g., 7.236905) of channel 5. In summary, the representative discretization value (e.g., 4.667619) of channel 4 and the representative discretization value (e.g., 7.236905) of channel 5 can follow a first distribution (1, 0), a second distribution (0.95, 0.05), a third distribution (0.9, 0.1), a fourth distribution (0.85, 0.15), and a fifth distribution (0.8, 0.2).

[0101] However, the method for adjusting the probability value is not limited to this. For example, to obtain the probability distribution of an estimated depth corresponding to a depth of 5.2 m, the device may increase or decrease the probability values ​​of three channels, for example, the representative discretization value (e.g., 4.667619) of channel 4 closest to 5.2 m, the representative discretization value (e.g., 7.236905) of channel 5, and the representative discretization value (e.g., 10.37714) of channel 6, by 0.05 from 0 to 1.

[0102] Next, the device may determine the obtained expected value as the estimation depth of the target pixel 505 by obtaining the expected value of each of the first to fifth distributions.

[0103] The method by which the device obtains the expected value of the distribution can be expressed by the following equation 5. E(x)=∑i=1KPi×Bi+Bi+12

[0104] Here E(x) can denote an expected value; P i can denote a probability value corresponding to the i-th representative discretization value; and (B i +B i+1 ) / 2 can denote the i-th representative discretization value.

[0105] For example, according to Equation 5, the device can obtain "4.667619," which is the result of "4.667619×1 + 7.236905×0" as the expected value of the first distribution. According to Equation 5, the device can obtain "4.796083," which is the result of "4.667619×0.95 + 7.236905×0.05" as the expected value of the second distribution. According to Equation 5, the device can obtain "4.924548," which is the result of "4.667619×0.9 + 7.236905×0.1" as the expected value of the third distribution. According to Equation 5, the device can obtain "5.053012," which is the result of "4.667619×0.85 + 7.236905×0.15" as the expected value of the fourth distribution. According to Equation 5, the device can obtain "5.181476," which is the result of "4.667619×0.8 + 7.236905×0.2" as the expected value of the fifth distribution.

[0106] Here, with respect to the fifth distribution, the expected value 540 of the fifth distribution may be 5.181476, indicating the nearest value as the estimation depth, which corresponds to 5.2 m, which is the depth of the actual object at the target pixel 505. Accordingly, the device may set the fifth distribution to a target distribution 530 and determine the target distribution 530 as the probability distribution of the estimation depth present in the target pixel 505.

[0107] As described above, the device can not only determine the probability distribution of the target pixel 505, but also determine the probability distribution of each pixel other than the target pixel 505. When the probability distribution of pixels (e.g., some or all) included in the depth tensor 500 is determined, the device can determine the depth tensor 500 described above as a depth estimation map.

[0108] Fig. 6 shows an example of a method for training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device according to an example of the present disclosure.

[0109] A device (e.g. the device 100 in Fig. 1) may obtain pose conversion information 607 by applying an input image 601 at time t' and an input image 603 at time t to a pose estimation model 605. The device may obtain a depth estimation map 611 by applying the input image 603 of time t to a monocular depth estimation model 609.

[0110] The device may obtain a first point cloud (e.g., a first point cloud 613), which is a 3D point cloud at time t, by applying the inverse of an intrinsic parameter corresponding to a specific image sensor (e.g., a lidar or a camera) to a depth estimation map 611. Then, the device may obtain a second point cloud (e.g., a second point cloud 615), which is a 3D point cloud of time t', by applying the pose conversion information 607 to the first point cloud 613. Furthermore, the device may convert the second point cloud 615 of time t' into 2D image coordinates by applying the intrinsic parameter to the second point cloud 615. Furthermore, the device may create a reconstruction image 617 based on pixels of the input image 603 at time t according to the 2D image coordinates.

[0111] The device may convert a 3D point cloud of time t corresponding to the position (10, 20) of the depth estimation map 611 into a 3D point cloud of time t' based on the pose conversion information 607. Furthermore, the device may convert the 3D point cloud of time t' corresponding to the location (10, 20) of the depth estimation map 611 into 2D image coordinates (e.g., (15, 18)) based on the intrinsic parameter. Furthermore, the device may generate the reconstruction image 617 by mapping a pixel value (e.g., 218) of the input image 603 corresponding to the 2D image coordinates (15, 18) of time t' to (10, 20).

[0112] The device may determine a first loss function based on the depth estimation map 611 and a depth distribution map 619, and may determine a second loss function based on the reconstruction image 617 and the input image 603 of time t.

[0113] The device may obtain a first estimated depth of the first pixel from a plurality of pixels included in the depth distribution map 619, and may obtain a second estimated depth of the second pixel, related to a location corresponding to the first pixel, from a plurality of pixels included in the depth estimation map 611. For reference, the estimated depth may indicate the expected value of the probability distribution corresponding to the corresponding pixel. The device may determine the first loss function based on a difference between the first estimated depth and the second estimated depth. Subsequently, the device may update a plurality of weights included in the monocular depth estimation model 609 to obtain the second estimated depth based on the thus determined first loss function.

[0114] However, the method for determining the first loss function is not limited to this. For example, if the first estimation depth satisfies a predetermined third condition, the device may determine a difference between the first estimation depth and the second estimation depth as the first loss function. If the first estimation depth does not satisfy the third condition, the device may skip updating at least one weight included in the monocular depth estimation model for obtaining the second estimation depth. Specifically, the third condition may indicate a condition that the estimation depth is not 0, and the first loss function may be expressed by Equation 6 and Equation 7 below. depthmask=(estimateddept h>0?1:0)

[0115] Here depth maskdenote a mask for adapting the training of the monocular depth estimation model 609 depending on a value of a first estimated depth. depth can indicate the first estimation depth.

[0116] That is, depth mask may have a “1” if the first estimate depth is greater than “0” and may have a “0” in other cases. Loss=avg[−log{1−abs((Ed1×dept hmask)−(Ed2×dept hmask))}]

[0117] Here Loss can denote a first loss function; E d1 can indicate a first estimation depth; E d2 can denote a second estimation depth; and depth mask can be the same as that described in equation 6.

[0118] In summary, the first loss function may be determined based on the difference between the first estimation depth and the second estimation depth when the first estimation depth is greater than "0." If the first estimation depth is less than or equal to "0," the first loss function may be "0." In other words, if the first estimation depth is less than or equal to "0," the device determines the first loss function as "0." The device may skip updating the plurality of weights that the monocular depth estimation model has to obtain the second estimation depth.

[0119] Through the above example, the device can train the monocular depth estimation model 609 by applying a loss function group comprising the first loss function and the second loss function to the monocular depth estimation model 609. The device can obtain the RSD indicating the uncertainty of a target pixel by having a first factor indicating the mean value of the channels of the target pixel among the pixels included in the depth estimation map 611 and a second factor indicating the standard deviation of the channels of the target pixel. Detailed descriptions regarding the RSD will be provided later in Fig. 7 and Fig. 8 described below.

[0120] Fig. 7 shows an example of a target image, an estimation depth of each of the pixels included in an input image, and the uncertainty of each of the pixels included in the input image in an inference device according to an example of the present disclosure.

[0121] An inference device according to an example may include a processor and a memory containing instructions. For example, the inference device may indicate an inference device that uses a monocular depth estimation model trained by a training device (e.g., the training device 100 in Fig. 1).

[0122] The inference device may obtain a target image 700 for testing. The inference device may obtain a depth estimation map 710 for the target, including the estimation depth of each pixel among a plurality of pixels in the target image, by applying the target image 700 to a monocular depth estimation model (i.e., a monocular depth estimation model trained by a training device) having updated weights. The inference device may obtain a target uncertainty map 720, including RSD for each of a plurality of estimation depths included in the target depth estimation map 710.

[0123] For example, the inference device may obtain a first factor indicating the mean value of the channels of the target pixel among the pixels included in the depth estimation map 710 of the target, and a second factor indicating the standard deviation of the channels of the target pixel. The inference device may obtain the RSD indicating the uncertainty of the target pixel based on the first factor and the second factor.

[0124] The RSD according to an example can be expressed by Equation 8 below. RSDh,w=δh,wμh,w

[0125] Here (h,w) can denote coordinates that indicate a location of the target pixel; µ (h,w) can denote the first factor; δ (h,w) can denote the second factor; and RSD (h,w) can refer to the RSD.

[0126] Fig. 8 shows an example of a correlation between an estimation depth and the RSD in a training device according to an example of the present disclosure.

[0127] With reference to Fig. 8, a horizontal axis of a graph may represent an error ratio of the trained monocular depth estimation model, and a vertical axis of the graph may represent RSD. For example, a region 800 represented in a graph may include the error ratio and RSD of each of a plurality of pixels in the depth estimation map obtained from the trained monocular depth estimation model.

[0128] Next, a straight line 810 shown in the graph may be a straight line indicating the correlation between the error ratio and the RSD in the range 800. That is, the straight line 810 may have a relationship where the error ratio increases with increasing RSD, which indicates the uncertainty. The correlation corresponding to the straight line 810 can be expressed by the following Equation 9. errorratio=max(dept hpredrealdept h,realdept hdept hpred)∝Uncertaintyh,w

[0129] Here error ratio =max ((depth pred ) / (real depth ), (real depth ) / (depth pred )) denotes an error ratio, which represents the horizontal axis of the graph. Specifically, it may refer to the error ratio obtained based on the estimated depth and the actual depth of each of the plurality of pixels included in the depth estimation map. Uncertainty(h,w) can denote the RSD, which indicates the uncertainty of the target pixel (h, w).

[0130] Fig. 9 shows an example of a computing system associated with a training device or method according to an example of the present disclosure.

[0131] Referring to Fig. 9, a computing system 1000 associated with a training device or method may include at least a processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 interconnected via a bus 1200.

[0132] Processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory 1300 and / or storage 1600. Both memory 1300 and storage 1600 may include various types of volatile or non-volatile storage media. For example, memory 1300 may include read-only memory (ROM) and random access memory (RAM).

[0133] Accordingly, the operations of the method or algorithm described in connection with the examples disclosed in the specification may be implemented directly with a hardware module, a software module, or a combination of the hardware module and the software module executed by the processor 1100. The software module may be located on a storage medium (i.e., the memory 1300 and / or the memory 1600), such as random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrical EPROM (EEPROM), a register, a hard disk drive, a removable disk, or a compact disc-ROM (CD-ROM).

[0134] The storage medium may be connected to the processor 1100. The processor 1100 may read information from the storage medium and write information to the storage medium. Alternatively or additionally, the storage medium may be integrated with the processor 1100. The processor and the storage medium may be implemented with an application-specific integrated circuit (ASIC). The ASIC may be provided in a user terminal. Alternatively or additionally, the processor and the storage medium may be implemented with separate components in the user terminal.

[0135] The present disclosure has been made to solve the above-mentioned problems of the prior art while maintaining the advantages of the prior art.

[0136] An example of the present disclosure provides a training apparatus that can obtain pixel estimation depth by using only a monocular depth estimation model without an image generation model to obtain a complex model structure, a loss function, and a disparity for self-supervision-based depth estimation by obtaining a depth distribution map based on a depth map obtained from a point cloud, an inference apparatus, and a training method.

[0137] Furthermore, an example of the present disclosure provides a training apparatus capable of estimating uncertainties while maintaining the depth estimation performance of the monocular depth estimation model by obtaining a first loss function based on a depth distribution map and a depth estimation map and a second loss function based on images classified using a plurality of time points, an inference apparatus, and a training method.

[0138] Furthermore, an example of the present disclosure includes a training device that can obtain a valid correlation between an estimation depth and a relative standard deviation (RSD) of each of the plurality of pixels included in a target depth estimation map by obtaining the RSD indicating the estimation depth and the uncertainty of each of the plurality of pixels included in the target depth estimation map, as well as an inference device and a training method.

[0139] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned here will be clearly understood by those skilled in the art to which the present disclosure refers from the following description.

[0140] According to one example of the present disclosure, a training device may include a memory storing computer-executable instructions and at least one processor executing the instructions by accessing the memory. The at least one processor may obtain a depth distribution map based on a depth map obtained from a point cloud of a target time point, may obtain a depth estimation map by applying an input image of the target time point to a monocular depth estimation (MDE) model, and may update a plurality of weights included in the MDE model by applying a loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map to the MDE model.

[0141] In one example, the at least one processor may obtain the depth map by extracting portions of depth information from the point cloud, may obtain a depth tensor by extending a channel of the depth map by a predetermined first condition using the portions of depth information of each of a plurality of pixels based on the plurality of pixels included in the depth map, and may obtain the depth distribution map based on the depth tensor.

[0142] In one example, the at least one processor may determine a minimum discretization value and a maximum discretization value based on a sensor (e.g., a lidar sensor) to determine a discretization value of each of the channels present in a single pixel of the depth tensor, and may determine the discretization value of each of the channels present in the single pixel based on an index of each of the channels present in the single pixel, the minimum discretization value, the maximum discretization value, and the number of channels.

[0143] In one example, the at least one processor may determine a ratio of a discretization value of an Nth channel and a discretization value of an (N+1)th channel as a representative discretization value of the Nth channel based on a case where the discretization value of the Nth channel of the individual pixel and the discretization value of the (N+1)th channel of the individual pixel are determined after the Nth channel. The Nth channel may be a channel with a natural number "N" less than or equal to the number of channels of the depth tensor.

[0144] For example, the depth distribution map may comprise pixels for each of which a representative discretization value is determined corresponding to the channels contained in each of the pixels of the depth tensor. A sum of probabilities corresponding to channels contained in each of the pixels of the depth tensor may have probabilities that satisfy a predefined second condition.

[0145] In one example, the at least one processor may obtain a first estimation depth of a first pixel from a plurality of pixels included in the depth distribution map, may obtain a second estimation depth of a second pixel related to a location corresponding to the first pixel from a plurality of pixels included in the depth estimation map, may determine the first loss function based on the first estimation depth and the second estimation depth, and may update the plurality of weights included in the MDE model to obtain the second estimation depth based on the determined first loss function.

[0146] In one example, the at least one processor may determine a difference between the first estimation depth and the second estimation depth as the first loss function when the first estimation depth satisfies a predetermined third condition, and may skip updating the plurality of weights present in the MDE model to obtain the second estimation depth when the first estimation depth does not satisfy the third condition.

[0147] For example, the at least one processor may obtain pose change information by applying an input image of a time point different from the target time point and an input image of the target time point to a pose estimation model, may obtain a first point cloud of the target time point by applying an inverse of an intrinsic parameter related to a lidar to the depth estimation map, may obtain a second point cloud of a time point different from the target time point by applying the pose change information to the first point cloud, and may determine a second loss function different from the first loss function based on the second point cloud. The loss function set may further include the second loss function.

[0148] In one example, the at least one processor may obtain a reconstruction image by applying the intrinsic parameter to the second point cloud, and may determine the second loss function based on the input image and the reconstruction image.

[0149] In one example, the at least one processor may obtain a first factor comprising a mean of the channels of a target pixel among the pixels included in the depth estimation map and a second factor comprising a standard deviation of the channels of the target pixel, and may obtain a relative standard deviation (RSD) indicating the uncertainty of the target pixel based on the first factor and the second factor.

[0150] According to one example of the present disclosure, an inference device using an MDE model trained by the training device may include a memory configured to store instructions executable by a computer and at least one processor configured to execute the instructions by accessing the memory. The at least one processor may receive a target image for testing, may obtain a target depth estimation map including an estimation depth for each of a plurality of pixels included in the target image by applying the target image to an MDE model having updated weights, and may obtain a target uncertainty map including an RSD for each of a plurality of estimation depths included in the target depth estimation map.

[0151] According to an example of the present disclosure, a training method may include: obtaining a depth distribution map based on a depth map obtained from a point cloud of a target time point, obtaining a depth estimation map by applying an input image of the target time point to an MDE model, and updating a plurality of weights included in the MDE model by applying a loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map to the MDE model.

[0152] For example, obtaining the depth distribution map may include: obtaining the depth map by extracting depth information from the point cloud, obtaining a depth tensor by expanding a channel of the depth map by a predetermined first condition using the depth information of each of a plurality of pixels based on the plurality of pixels included in the depth map, and obtaining the depth distribution map based on the depth tensor.

[0153] In one example, obtaining the depth distribution map may include determining a minimum discretization value and a maximum discretization value based on a lidar to determine a discretization value of each of the channels included in a single pixel of the depth tensor, and determining the discretization value of each of the channels included in the single pixel based on an index of each of the channels included in the single pixel, the minimum discretization value, the maximum discretization value, and the number of channels.

[0154] In one example, obtaining the depth distribution map may include determining a ratio of a discretization value of an Nth channel and a discretization value of an (N+1)th channel as a representative discretization value of the Nth channel based on a case where the discretization value of the Nth channel of the individual pixel and the discretization value of the (N+1)th channel of the individual pixel following the Nth channel are determined. The Nth channel may be a channel with a natural number of "N" less than or equal to the number of channels of the depth tensor. The depth distribution map may include pixels for each of which a representative discretization value corresponding to the channels included in each of the pixels of the depth tensor is determined.A sum of probabilities corresponding to channels contained in each of the pixels of the depth tensor may have probabilities that satisfy a given second condition.

[0155] In one example, updating the plurality of weights included in the MDE model may include obtaining a first estimation depth of a first pixel from a plurality of pixels included in the depth distribution map, obtaining a second estimation depth of a second pixel related to a location corresponding to the first pixel from a plurality of pixels included in the depth estimation map, determining the first loss function based on the first estimation depth and the second estimation depth, and updating the plurality of weights included in the MDE model to obtain the second estimation depth based on the determined first loss function.

[0156] For example, updating the plurality of weights included in the MDE model may include determining a difference between the first estimation depth and the second estimation depth as the first loss function when the first estimation depth satisfies a predetermined third condition, and skipping updating the plurality of weights included in the MDE model to obtain the second estimation depth when the first estimation depth does not satisfy the third condition.

[0157] In one example, the training method may further include obtaining pose change information by applying an input image of a time point different from the target time point and an input image of the target time point to a pose estimation model; obtaining a first point cloud of the target time point by applying an inverse of an intrinsic parameter related to a lidar to the depth estimation map; obtaining a second point cloud of a time point different from the target time point by applying the pose change information to the first point cloud; and determining a second loss function different from the first loss function based on the second point cloud. The loss function set may further include the second loss function.

[0158] In one example, determining the second loss function may include obtaining a reconstruction image by applying the intrinsic parameter to the second point cloud, and determining the second loss function based on the input image and the reconstruction image.

[0159] In one example, the training method may further comprise a first factor indicating a mean value of the channels of a target pixel among the pixels included in the depth estimation map, and a second factor indicating a standard deviation of the channels of the target pixel, and obtaining an RSD indicating the uncertainty of the target pixel based on the first factor and the second factor.

[0160] The above description is merely an example of the technical idea of ​​the present disclosure, and various alterations and modifications can be made by a person skilled in the art without departing from the essential feature of the present disclosure.

[0161] The examples described above may be implemented using hardware elements, software elements, and / or a combination of hardware elements and software elements. For example, the devices, methods, and components described in the examples of the present disclosure may be implemented using general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any device capable of executing instructions and responding. A processing device may execute an operating system (OS) or a software application running on the OS.Furthermore, the processing device may access, store, manipulate, process, and generate data in response to software execution. It should be understood that although a single processing device is illustrated for ease of understanding, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the device may include a plurality of processors or a processor and a controller. The device may also include another processing configuration, such as a parallel processor.

[0162] Software may comprise computer programs, codes, instructions, or one or more combinations thereof, and may configure a device to operate in a desired manner or to control the device independently or jointly. Software and / or data may be embodied permanently or temporarily in any type of machine, component, physical equipment, virtual equipment, computer storage media or devices, or transmitted signal waves, to be interpreted by the processing device or to provide instructions or data to the processing device. Software may be dispersed among computer systems connected by networks and stored or executed in a distributed form. Software and data may be recorded in a computer-readable storage medium.

[0163] The methods according to the examples described above may be recorded on a computer-readable medium having program instructions executable by various computing devices. The computer-readable medium may also include program instructions, data files, data structures, or a combination thereof. The program instructions recorded on the medium may be specifically designed and configured for the examples of the present disclosure, or they may be known and available to those skilled in computer software. The computer-readable medium may include hardware devices specifically configured for storing and executing program instructions, such as magnetic media (e.g., a hard disk, a floppy disk, or magnetic tape), optical recording media (e.g., CD-ROM and DVD), magneto-optical media (e.g.,a floppy disk), read-only memory (ROM), random-access memory (RAM), and flash memory. Examples of computer programs include not only machine language codes created by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or similar device.

[0164] The hardware device described above may be configured to function as one or more software modules to perform the operations of the examples of the present disclosure described above, or vice versa.

[0165] Although the examples are described with reference to limited drawings, it may be apparent to one skilled in the art that the examples may be changed or modified in various ways based on the above description. For example, adequate effects may be achieved even if the aforementioned methods and processes are performed in a different order than described above, and / or the aforementioned elements, such as systems, structures, devices, or circuits, are combined or connected in forms and modes other than those described above, or are replaced or exchanged with other components or equivalents.

[0166] Therefore, other embodiments, other examples, and equivalents to the claims are intended to fall within the scope of the following claims.

[0167] Accordingly, the examples of the present disclosure are not intended to limit but to explain the technical idea of ​​the present disclosure, and the scope and spirit of the present disclosure are not limited by the above examples. The scope of the present disclosure should be interpreted by the appended claims, and any equivalents thereof should be construed to encompass the scope of the present disclosure.

[0168] The descriptions of a training apparatus, an inference apparatus, and a training method according to an example of the present disclosure, as well as a method thereof, are as follows.

[0169] According to at least one of the examples of the present disclosure, it is possible to obtain a pixel estimation depth by using only a monocular depth estimation model without an image formation model to obtain a complex model structure, a loss function, and a disparity for self-supervision-based depth estimation by obtaining a depth distribution map based on a depth map obtained from a point cloud.

[0170] Furthermore, according to at least one of the examples of the present disclosure, it is possible to estimate the uncertainty while maintaining the depth estimation performance of the monocular depth estimation model by obtaining a first loss function based on a depth distribution map and a depth estimation map and a second loss function based on images classified using a plurality of time points.

[0171] Furthermore, according to at least one of the examples of the present disclosure, it is possible to obtain a valid correlation between an estimation depth and a relative standard deviation (RSD) of each of the plurality of pixels included in a depth estimation map of the target by determining the RSD indicating the estimation depth and the uncertainty of each of the plurality of pixels included in the depth estimation map of the target.

[0172] In addition, a variety of effects can be provided, which are understood directly or indirectly through the specification.

[0173] Although the present disclosure has been described with reference to examples and the accompanying drawings, the present disclosure is not limited thereto, but may be variously modified and changed by those skilled in the art to which the present disclosure relates without departing from the spirit and scope of the present disclosure as claimed in the following claims.

Claims

[1] Device comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the device to obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point; obtain a depth estimation map based on an input image associated with the target time point and applied to a monocular depth estimation (MDE) model; update a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, the loss function group comprising a first loss function obtained based on the depth distribution map and the depth estimation map; and output a signal indicating the updated plurality of weights. [2] The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtaining the depth map by extracting pieces of depth information from the point cloud, wherein the pieces of depth information are associated with a plurality of pixels present in the depth map; to obtain a depth tensor by extending a channel of the depth map, wherein the channel of the depth map is extended by a first condition based on the pieces of depth information; and to obtain the depth distribution map based on the depth tensor. [3] The apparatus of claim 2, wherein the instructions, when executed by the at least one processor, further cause the apparatus to based on sensing information obtained from a sensor, determine a minimum discretization value and a maximum discretization value associated with channels present in a single pixel of the depth tensor; and to determine a discretization value of each of the channels, wherein the discretization value of each of the channels is determined based on an index of each of the channels present in the individual pixel, the minimum discretization value, the maximum discretization value and a number of the channels. [4] The apparatus of claim 3, wherein the instructions, when executed by the at least one processor, further cause the apparatus to based on a discretization value of an Nth channel of a single pixel of the depth tensor and a discretization value of an (N+1)th channel of the single pixel following the Nth channel, to determine a ratio of the discretization value of the Nth channel and the discretization value of the (N+1)th channel as a representative discretization value of the Nth channel, and where N is a natural number and less than or equal to the total number of channels of the depth tensor. [5] Device according to claim 4, wherein the depth distribution map comprises pixels, and wherein a representative discretization value of a pixel of the pixels is associated with channels contained in the pixel of the pixels of the depth tensor, and where a sum of probabilities has probabilities satisfying a second condition, the sum of probabilities being associated with channels contained in each of the pixels of the depth tensor. [6] The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain a first estimated depth of a first pixel among a plurality of pixels present in the depth distribution map; obtain a second depth estimate of a second pixel among a plurality of pixels present in the depth estimation map, the second pixel being associated with a location corresponding to the first pixel; to determine the first loss function based on the first estimation depth and the second estimation depth; and update the plurality of weights present in the MDE model based on the determined first loss function to obtain the second estimation depth. [7] The apparatus of claim 6, wherein the instructions, when executed by the at least one processor, further cause the apparatus to based on the first estimation depth satisfying a third condition, to determine a difference between the first estimation depth and the second estimation depth as the first loss function; or based on the first estimation depth that does not satisfy the third condition, skip updating the plurality of weights present in the MDE model to obtain the second estimation depth. [8] The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, further cause the apparatus to Obtaining pose change information by applying an input image at a time different from the target time and an input image at the target time to a pose estimation model; obtain a first point cloud at the target time by applying an inverse of an intrinsic parameter related to a sensor to the depth estimation map; obtain a second point cloud at a time different from the target time by applying the pose change information to the first point cloud; and to determine a second loss function, which is different from the first loss function, based on the second point cloud, and wherein the loss function group further comprises the second loss function. [9] The apparatus of claim 8, wherein the instructions, when executed by the at least one processor, further cause the apparatus to to obtain a reconstruction image by applying the intrinsic parameter to the second point cloud; and to determine the second loss function based on the input image and the reconstruction image. [10] The apparatus of claim 8, wherein the instructions, when executed by the at least one processor, further cause the apparatus to obtain a first factor indicating a mean value associated with channels of a target pixel among pixels included in the depth estimation map and obtain a second factor indicating a standard deviation value associated with the channels of the target pixel; and based on the first factor and the second factor to obtain a relative standard deviation value indicating the uncertainty of the target pixel. [11] Device comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the device to obtain a target image for testing; obtain a target depth estimation map by applying the target image to a monocular depth estimation model having updated weights, the target depth estimation map having an estimation depth of each of a plurality of pixels included in the target image; obtain a target uncertainty map, the target uncertainty map having a relative standard deviation value of each of a plurality of estimation depths included in the target depth estimation map; and to output a signal indicating the target uncertainty map. [12] A method performed by a processor, the method comprising: Obtaining a depth distribution map based on a depth map obtained from a point cloud at a target time point; Obtaining a depth estimation map by applying an input image associated with the target time point to a monocular depth estimation (MDE) model; Updating a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, the loss function group comprising a first loss function obtained based on the depth distribution map and the depth estimation map; and Output a signal indicating the updated plurality of weights. [13] The method of claim 12, wherein obtaining the depth distribution map comprises: Obtaining the depth map by extracting pieces of depth information from the point cloud, wherein the pieces of depth information are associated with a plurality of pixels present in the depth map; Obtaining a depth tensor by extending a channel of the depth map, wherein the channel of the depth map is extended by a first condition based on the pieces of depth information; and Obtained, based on the depth tensor, the depth distribution map. [14] The method of claim 13, wherein obtaining the depth distribution map comprises: Determining, based on sensing information obtained from a sensor, a minimum discretization value and a maximum discretization value associated with channels present in a single pixel of the depth tensor; and Determining a discretization value of each of the channels, wherein the discretization value of each of the channels is determined based on an index of each of the channels present in the individual pixel, the minimum discretization value, the maximum discretization value, and a number of the channels. [15] The method of claim 14, wherein obtaining the depth distribution map comprises: Determining, based on a discretization value of an N-th channel of a single pixel and a discretization value of an (N+1)-th channel of the single pixel following the N-th channel, a ratio of the discretization value of the N-th channel and the discretization value of the (N+1)-th channel as a representative discretization value of the N-th channel, and where N is a natural number and less than or equal to the total number of channels of the depth tensor, wherein the depth distribution map comprises pixels, and wherein a representative discretization value of a pixel of the pixels is associated with channels present in the pixel of the pixels of the depth tensor, and where a sum of probabilities has probabilities satisfying a second condition, the sum of probabilities being associated with channels contained in each of the pixels of the depth tensor. [16] The method of claim 12, wherein updating the plurality of weights present in the MDE model comprises: Obtaining a first estimated depth of a first pixel among a plurality of pixels present in the depth distribution map; Obtaining a second depth estimate of a second pixel among a plurality of pixels present in the depth estimation map, the second pixel being associated with a location corresponding to the first pixel; Determining the first loss function based on the first estimation depth and the second estimation depth; and Updating the plurality of weights present in the MDE model based on the determined first loss function to obtain the second estimation depth. [17] The method of claim 16, wherein updating the plurality of weights present in the MDE model comprises: Determining a difference between the first estimation depth and the second estimation depth as a first loss function, based on the first estimation depth satisfying a third condition; or Skipping updating the plurality of weights present in the MDE model based on the first estimation depth that does not satisfy the third condition to obtain the second estimation depth. [18] The method of claim 12, further comprising: Obtaining pose change information by applying an input image at a time different from the target time and an input image at the target time to a pose estimation model; Obtaining a first point cloud at the target time by applying an inverse of a sensor-related intrinsic parameter to the depth estimation map; Obtaining a second point cloud at a time different from the target time by applying the pose change information to the first point cloud; and Determining, based on the second point cloud, a second loss function that differs from the first loss function, and wherein the loss function group further comprises the second loss function. [19] The method of claim 18, wherein determining the second loss function comprises: Obtaining a reconstruction image by applying the intrinsic parameter to the second point cloud; and Determine the second loss function based on the input image and the reconstruction image. [20] The method of claim 18, further comprising: Obtaining a first factor indicative of a mean value associated with channels of a target pixel among pixels included in the depth estimation map and a second factor indicative of a standard deviation value associated with the channels of the target pixel; and Obtain, based on the first factor and the second factor, a relative standard deviation value indicating the uncertainty of the target pixel.

Images