Device for training and reasoning and method for training

Abstract

The invention relates to a device for training and reasoning and a method for training. The apparatus may include: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: obtain a depth profile based on a depth map obtained from a point group of target points in time; obtaining a depth estimation map based on an input image associated with the target point in time and applied to a monocular depth estimation (MDE) model; updating a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, the loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map; and outputting a signal indicating the updated plurality of weights.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2023-0145795 filed in the Korean Intellectual Property Office on October 27, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a training device, an inference device and a training method, and more specifically, to a technology for training a monocular depth estimation model. Background Art

[0004] With the development of computer vision technology based on deep neural networks in autonomous driving technology, various artificial intelligence models such as object detection, semantic segmentation, depth map estimation and lane detection are being studied.

[0005] In particular, the output of a monocular depth estimation model for autonomous driving can be used to identify the drivable range of the autonomous vehicle. However, it is difficult to quantitatively obtain the uncertainty about the output of the monocular depth estimation model.

[0006] For example, in regression tasks that estimate specific values, Bayesian and ensemble methods can be used, that is, using multiple outputs for one input to quantitatively capture uncertainty. However, using this method in actual driving vehicles is inefficient. In addition, in classification tasks that capture specific categories, uncertainty can be quantitatively captured by utilizing entropy. However, the number of each case that can express deep information may be reduced.

[0007] To address these issues, it is necessary to develop a technique for training a monocular depth estimation model to estimate depth in pixels and estimate the uncertainty of depth in actual moving vehicles. Summary of the invention

[0008] According to the present disclosure, a device may include: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the device to: obtain a depth distribution map based on a depth map obtained from a point group at a target time point; obtain a depth estimation map based on an input image associated with the target time point and applied to a monocular depth estimation (MDE) model; update multiple weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group may include a first loss function obtained based on the depth distribution map and the depth estimation map; and output a signal indicating the updated multiple weights.

[0009] The device, wherein, when executed by at least one processor, the instructions further cause the device to perform autonomous driving control of the vehicle based on the updated MDE model, wherein the updated MDE model is updated based on the updated multiple weights.

[0010] The device, wherein, when executed by at least one processor, the instructions further cause the device to: obtain a depth map by extracting multiple depth information from the point group, wherein the multiple depth information is associated with multiple pixels included in the depth map; obtain a depth tensor by expanding channels of the depth map, wherein the channels of the depth map are expanded based on the multiple depth information by a first condition; and obtain a depth distribution map based on the depth tensor.

[0011] The device, wherein, when executed by at least one processor, the instructions further cause the device to: determine a minimum discrete value and a maximum discrete value associated with channels included in a single pixel of a depth tensor based on sensing information obtained by a sensor; and determine a discrete value for each channel, wherein the discrete value for each channel is determined based on an index of each channel included in the single pixel, the minimum discrete value, the maximum discrete value, and the number of channels.

[0012] The device, wherein, when executed by at least one processor, the instructions further cause the device to: determine a ratio of the discrete value of the Nth channel to the discrete value of the (N+1)th channel based on the discrete value of the Nth channel of a single pixel of the depth tensor and the discrete value of the (N+1)th channel after the Nth channel of the single pixel, as a representative discrete value of the Nth channel, and wherein N is a natural number and is less than or equal to the total number of channels of the depth tensor.

[0013] The apparatus, wherein the depth profile may include a plurality of pixels, and wherein a representative discrete value of a pixel in the plurality of pixels is associated with a channel included in a pixel in the plurality of pixels of a depth tensor, and wherein a sum of probabilities may include a probability of satisfying a second condition, wherein the sum of probabilities is associated with a channel included in each pixel of the depth tensor.

[0014] The device, wherein, when executed by at least one processor, the instructions further cause the device to: obtain a first estimated depth of a first pixel among a plurality of pixels included in a depth distribution map; obtain a second estimated depth of a second pixel among a plurality of pixels included in the depth estimation map, wherein the second pixel is related to a position corresponding to the first pixel; determine a first loss function based on the first estimated depth and the second estimated depth; and update a plurality of weights included in the MDE model for obtaining a second estimated depth based on the determined first loss function.

[0015] The device, wherein, when executed by at least one processor, the instructions cause the device to: determine the difference between the first estimated depth and the second estimated depth as a first loss function based on the first estimated depth satisfying a third condition; or skip updating multiple weights included in the MDE model for obtaining the second estimated depth based on the first estimated depth not satisfying the third condition.

[0016] The device, wherein, when executed by multiple processors, the instructions further cause the device to: obtain posture change information by applying an input image at a time point different from a target time point and an input image at the target time point to a posture estimation model; obtain a first point group at the target time point by applying the inverse of the intrinsic parameters related to the sensor to the depth estimation map; obtain a second point group at a time point different from the target time point by applying the posture change information to the first point group; and determine a second loss function different from the first loss function based on the second point group, and wherein the loss function group may further include the second loss function.

[0017] The apparatus, wherein when executed by at least one processor, the instructions further cause the apparatus to: obtain a reconstructed image by applying the intrinsic parameters to the second point group; and determine a second loss function based on the input image and the reconstructed image.

[0018] The device, wherein, when executed by at least one processor, the instructions further cause the device to: obtain a first factor representing an average value associated with a channel of a target pixel among pixels included in a depth estimation map and a second factor representing a standard deviation value associated with the channel of the target pixel; and based on the first factor and the second factor, obtain a relative standard deviation value representing the uncertainty of the target pixel.

[0019] According to the present disclosure, a device may include: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the device to: obtain a target image for testing; obtain a target depth estimation map by applying the target image to a monocular depth estimation model including updated weights, wherein the target depth estimation map may include an estimated depth for each of a plurality of pixels included in the target image; obtain a target uncertainty map, wherein the target uncertainty map may include a relative standard deviation value for each of a plurality of estimated depths included in the target depth estimation map; and output a signal indicating the target uncertainty map.

[0020] According to the present disclosure, a method executed by a processor may include: obtaining a depth distribution map based on a depth map obtained from a point group at a target time point; obtaining a depth estimation map by applying an input image associated with the target time point to a monocular depth estimation (MDE) model; updating multiple weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group may include a first loss function obtained based on the depth distribution map and the depth estimation map; and outputting a signal indicating the updated multiple weights.

[0021] The method, wherein obtaining a depth distribution map may include: obtaining a depth map by extracting multiple depth information from a point group, wherein the multiple depth information is associated with multiple pixels included in the depth map; obtaining a depth tensor by expanding channels of the depth map, wherein the channels of the depth map are expanded based on the multiple depth information by a first condition; and obtaining a depth distribution map based on the depth tensor.

[0022] The method, wherein obtaining a depth distribution map may include: determining a minimum discrete value and a maximum discrete value associated with channels included in a single pixel of a depth tensor based on sensing information obtained by a sensor; and determining a discrete value for each channel, wherein the discrete value for each channel is determined based on an index, a minimum discrete value, a maximum discrete value, and the number of channels of each channel included in the single pixel.

[0023] The method, wherein obtaining a depth distribution map may include: determining a ratio of a discrete value of the Nth channel to a discrete value of the (N+1)th channel after the Nth channel of a single pixel based on a discrete value of the Nth channel and a discrete value of the (N+1)th channel after the Nth channel of the single pixel, as a representative discrete value of the Nth channel, and wherein N is a natural number and is less than or equal to the total number of channels of a depth tensor, wherein the depth distribution map may include multiple pixels, and wherein a representative discrete value of one pixel among the multiple pixels is associated with a channel included in one pixel among the multiple pixels of the depth tensor, and wherein a sum of probabilities may include a probability of satisfying a second condition, wherein the sum of probabilities is associated with a channel included in each pixel of the depth tensor.

[0024] The method, wherein updating multiple weights included in the MDE model may include: obtaining a first estimated depth of a first pixel among multiple pixels included in a depth distribution map; obtaining a second estimated depth of a second pixel among multiple pixels included in the depth estimation map, wherein the second pixel is related to a position corresponding to the first pixel; determining a first loss function based on the first estimated depth and the second estimated depth; and updating multiple weights included in the MDE model for obtaining the second estimated depth based on the determined first loss function.

[0025] The method, wherein updating multiple weights included in the MDE model may include: determining the difference between the first estimated depth and the second estimated depth as a first loss function based on the first estimated depth satisfying a third condition; or skipping updating the multiple weights included in the MDE model for obtaining the second estimated depth based on the first estimated depth not satisfying the third condition.

[0026] The method may further include: obtaining posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model; obtaining a first point group at the target time point by applying the inverse of the intrinsic parameters related to the sensor to the depth estimation map; obtaining a second point group at a time point different from the target time point by applying the posture change information to the first point group; and determining a second loss function different from the first loss function based on the second point group, and wherein the loss function group may further include the second loss function.

[0027] The method, wherein determining the second loss function may include: obtaining a reconstructed image by applying the intrinsic parameters to the second point group; and determining the second loss function based on the input image and the reconstructed image.

[0028] The method may further include: obtaining a first factor representing an average value associated with a channel of a target pixel among pixels included in the depth estimation map and a second factor representing a standard deviation value associated with the channel of the target pixel; and based on the first factor and the second factor, obtaining a relative standard deviation value representing the uncertainty of the target pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0030] Figure 1 An example of a training apparatus for a monocular depth estimation model according to an example of the present disclosure is shown;

[0031] Figure 2 An example of a flowchart for describing a method of training a monocular depth estimation model according to an example of the present disclosure is shown;

[0032] Figure 3 An example of a process of training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device according to an example of the present disclosure is shown;

[0033] Figure 4 An example of a method for obtaining a depth distribution map based on a depth map in a training device according to an example of the present disclosure is shown;

[0034] Figure 5An example of generating a depth profile in a training device according to an example of the present disclosure is shown;

[0035] Figure 6 An example of a process of training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device according to an example of the present disclosure is shown;

[0036] Figure 7 An example showing a target image, an estimated depth of each pixel included in an input image, and uncertainty of each pixel included in the input image in an inference device according to an example of the present disclosure;

[0037] Figure 8 An example showing the correlation between estimated depth and RSD in a training apparatus according to an example of the present disclosure; and

[0038] Fig. 9 An example of a computing system related to a training apparatus or a training method according to an example of the present disclosure is shown.

[0039] With regard to the description of the drawings, the same or similar components will be marked with the same or similar reference numerals.

[0040] Description of Reference Numerals

[0041] 400: Scene

[0042] 410: Point Cloud

[0043] 420: Deep Tensor

[0044] 500: Depth Tensor

[0045] 505: Target pixels

[0046] 510: Collection

[0047] 515: Table

[0048] 530: Target Distribution

[0049] 540: Expected value

[0050] 700: Target image

[0051] 710: Target depth estimation diagram

[0052] 720: Target Uncertainty Diagram

[0053] 800: Area

[0054] 810: Straight line DETAILED DESCRIPTION

[0055] Hereinafter, some examples of the present disclosure will be described in detail with reference to the accompanying drawings. When adding reference numerals to the components in each figure, it should be noted that although the same components are shown in another figure, they also include the same reference numerals. In addition, when describing the examples of the present disclosure, if the detailed descriptions related to the well-known functions or configurations may make the subject matter of the present disclosure unnecessarily vague, these detailed descriptions will be omitted. Hereinafter, various examples of the present disclosure may be described with reference to the accompanying drawings. Therefore, those of ordinary skill in the art will recognize that various modifications, equivalents and / or substitutions may be made to the various examples described herein without departing from the scope and spirit of the present disclosure. With regard to the description of the accompanying drawings, similar components may be indicated with similar reference numerals.

[0056] When describing the elements of the examples of the present disclosure, the terms first, second, A, B, (a), (b), etc. may be used herein. These terms are only used to distinguish one element from another element, but do not limit the corresponding elements, regardless of the nature, order or priority of the corresponding elements. In addition, unless otherwise defined, all terms (including technical terms and scientific terms) used herein should be interpreted according to the custom of the field to which the present disclosure belongs. It will be understood that the terms used herein should be interpreted as including the meaning consistent with the meaning in the context of the present disclosure and the related art, and unless clearly defined in this article, it will not be interpreted as an idealized or overly formalized meaning. For example, terms such as "first", "second", etc. used in this article can refer to various components in various examples of the present disclosure, but do not limit these elements. For example, "first user equipment" and "second user equipment" can indicate different user equipment, regardless of their order or priority. For example, without departing from the scope of the present disclosure, the first component can be referred to as the second component, and similarly, the second component can also be referred to as the first component.

[0057] In this specification, the expressions "having", "may have", "including" and "comprising" or "may include" and "may include" used herein indicate the existence of corresponding features (for example, elements such as values, functions, operations or components), but do not exclude the existence of other features.

[0058] It will be understood that if an element (e.g., a first element) is referred to as being “(operably or communicatively) coupled to / (operably or communicatively) coupled to” another element (e.g., a second element) or “connected to” another element (e.g., a second element), it may be directly coupled to / coupled to or connected to the other element, and there may also be an intervening element (e.g., a third element). Conversely, if an element (e.g., a first element) is referred to as being “directly coupled to / directly coupled to” another element (e.g., a second element) or “directly connected to” another element (e.g., a second element), it will be understood that there are no intervening elements (e.g., a third element).

[0059] As used herein, the expression “configured to” may be used as an expression such as “suitable for,” “capable of,” “designed to,” “adapted to,” “made to,” or “able to,” depending on the circumstances.

[0060] The term "configured to" does not necessarily refer only to hardware "specially designed to". On the contrary, the expression "a device configured to..." may indicate that the device "can" operate with other devices or other components. For example, "a processor configured (or set to) perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) that performs the corresponding operations, or may refer to a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that performs the corresponding operations by running one or more software programs stored in a memory device. The terms used in the specification are only used to describe specific examples and are not intended to limit the scope of the present disclosure. Unless otherwise indicated, terms in the singular may include plural forms. All terms used herein, including technical terms or scientific terms, may include the same meanings commonly understood by those skilled in the art. It will be further understood that for terms defined in dictionaries and commonly used, they should also be interpreted in accordance with the customs of the relevant technical field, and should not be interpreted as idealized or overly formal meanings, unless explicitly defined in various examples of the present disclosure. In some cases, even if the terms are defined in the specification, they should not be interpreted as excluding examples of the present disclosure.

[0061] In the present disclosure disclosed herein, the expressions "A or B", "at least one of A or / and B", or "one or more of A or / and B", etc. used herein may include any one and all combinations of one or more of the listed related items. For example, the term "A or B", "at least one of A and B", or "at least one of A or B" may refer to all of the following: the case (1) including at least one A, the case (2) including at least one B, or the case (3) including at least one A and at least one B. In addition, when describing the components of the examples of the present disclosure, the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", or "at least one of A, B or C", or any combination thereof may include any one and all combinations of one or more of the listed related items. In particular, the expression "at least one of A, B or C, or any combination thereof" may include A, B or C, or any combination thereof, such as AB, ABC, etc.

[0062] In the following, reference will be made to Figures 1 to 9 Various examples of the present disclosure are described in detail.

[0063] Figure 1 An example of a training apparatus for a monocular depth estimation model according to an example of the present disclosure is shown.

[0064] According to an example, the apparatus 100 may include a processor 110 and a memory 120 containing instructions 122 .

[0065] The apparatus 100 may represent a device for training a monocular depth estimation model. For example, the apparatus 100 may update a plurality of weights included in the monocular depth estimation model by applying a loss function group to the monocular depth estimation model. In other words, the apparatus 100 may train the monocular depth estimation model by updating a plurality of weights included in the monocular depth estimation model.

[0066] The monocular depth estimation model can be a network of a U-net structure, including an encoder and a decoder. Here, the encoder can be a ResNet model, and the decoder can be a model that converts a sigmoid output into a depth estimation map. Specifically, the device can obtain a depth estimation map from the decoder by applying the input image at the target time point to the encoder of the monocular depth estimation model. A detailed description of the monocular depth estimation model is provided later below. Figure 2 Described in.

[0067] The loss function group may include a first loss function and a second loss function. The apparatus 100 may update a plurality of weights included in the monocular depth estimation model by applying each of the first loss function and the second loss function to the monocular depth estimation model.

[0068] A first loss function may be obtained based on the depth distribution map and the depth estimation map.

[0069] A depth distribution map may represent a set of depth information (e.g., information about the distance between an object and a laser radar), which represents three-dimensional (3D) distance information of objects and backgrounds that exist in an image obtained when the laser radar acquires an actual scene including a predetermined area based on the position of the vehicle. Specifically, the depth distribution map may include a plurality of pixels that are obtained by dividing the objects and background present in the above-mentioned image using a grid having a predetermined size. The depth distribution map may include depth information for each of the plurality of pixels. In particular, the depth distribution map may include a probability distribution of depth information for each of the plurality of pixels. Therefore, the depth distribution map may represent a graph including a probability distribution of depth information for each of the plurality of pixels in an image that displays an actual scene using the plurality of pixels.

[0070] The depth estimation map may represent a set of depth information (e.g., information about the distance between an object and a camera including an image sensor) obtained by a monocular depth estimation model of 3D distance information of objects and backgrounds present in an image obtained when a monocular camera (or a monocular camera) captures an actual scene including a predetermined area based on a vehicle position. Specifically, the depth estimation map may include a plurality of pixels obtained by dividing the objects and background present in the above-mentioned image using a grid having a predetermined size. The depth estimation map may include depth information of each of the plurality of pixels obtained by the monocular depth estimation model.

[0071] Therefore, for the convenience of the description of this specification, it is mainly described that the depth distribution map may include the depth information of the image acquired by the lidar, and the depth estimation map may include the depth information obtained by applying the image acquired by the monocular camera to the monocular depth estimation model.

[0072] The method for the device 100 to determine the first loss function is as follows. For example, the device 100 may obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point. In addition, the device 100 may obtain a depth estimation map by applying an input image at a target time point to a monocular depth estimation model. The device 100 may determine the first loss function based on the depth distribution map and the depth estimation map. The detailed method for the device 100 to determine the first loss function is described later below. Figure 6 Described in.

[0073] The second loss function can be obtained based on the reconstructed image and the input image at the target time point.

[0074] The input image at the target time point may be generated by the image sensor at the target time point. The input image at a time point different from the target time point may be generated by the image sensor at a time point different from the target time point. In this case, the time point different from the target time point may be at least one of the other time points other than the target time point among the time points within the threshold time interval based on the target time point.

[0075] The method for the device 100 to determine the second loss function is as follows. For example, the device 100 may obtain posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model. The device 100 may obtain a first point cloud at the target time point by applying the inverse of the intrinsic parameters related to the lidar to the depth estimation map. The device 100 may obtain a second point cloud at a time point different from the target time point by applying the posture change information to the first point cloud. Finally, the device 100 may determine a second loss function different from the first loss function based on the second point cloud. The detailed method for the device 100 to determine the second loss function is described later below. Figure 6 Described in.

[0076] The pose estimation model may generate pose change information corresponding to a pose change between an input image at a target time point and an input image at a time point different from the target time point. For example, the pose estimation model may generate pose change information based on a first pose of the image sensor at the target time point and a second pose at a time point different from the target time point.

[0077] The posture change information may be information indicating a relationship between a camera posture at a target time point and a camera posture at a time point different from the target time point, and may be information corresponding to a rotation and translation matrix of an object.

[0078] The processor 110 may run software and may control at least one other component (e.g., hardware or software component) connected to the processor 110. The processor 110 may also perform various data processing or operations. For example, the processor 110 may store a depth distribution map, a depth estimation map, and a loss function group in the memory 120. For reference, the processor 110 may perform some or all of the operations performed by the device 100. Therefore, for ease of description in this specification, the operations performed by the device 100 are mainly described as operations performed by the processor 110.

[0079] In addition, for the convenience of description in this specification, the processor 110 is mainly described as a single processor, but is not limited thereto. For example, the device 100 may include at least one processor. The at least one processor may perform some or all operations related to the training of the monocular depth estimation model.

[0080] The memory 120 may temporarily and / or permanently store various data and / or information for performing training of a monocular depth estimation model. For example, the memory 120 may store a depth distribution map, a depth estimation map, and a loss function group.

[0081] Figure 2 An example of a flowchart for describing a method of training a monocular depth estimation model according to an example of the present disclosure is shown.

[0082] In S210, a device (eg, Figure 1 The device 100 in FIG. 1 can obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point.

[0083] Point clouds can be collected by lidar or RGB-D sensors. Point clouds can include a set of points obtained by calculating the distance information of each light / signal of an object measured by a lidar or RGB-D sensor. Unlike a two-dimensional (2D) image, a point cloud can include information in the depth direction (e.g., Z axis), so the point cloud can be represented in three dimensions.

[0084] The depth map may represent an image created by extracting only information in the depth direction from the point cloud. For example, the depth map may include pixels in which information in the horizontal direction (e.g., x-axis) and information in the vertical direction (e.g., y-axis) are omitted from each pixel included in the point cloud.

[0085] In S220 , the apparatus may obtain a depth estimation map by applying the input image at the target time point to a monocular depth estimation model.

[0086] The monocular depth estimation model may include multiple layers, each layer may include multiple nodes. The node may include a node value determined based on an activation function. A node on any layer may be connected to a node on another layer (e.g., another node) via a link (e.g., a connecting edge) with a connection weight. The node value of a node may be propagated to other nodes via a link. In the reasoning operation of the monocular depth estimation model, the node value may be propagated forward from the previous layer to the next layer.

[0087] For example, the forward propagation operation in the monocular depth estimation model may represent an operation of propagating node values ​​in a direction from the input layer of the monocular depth estimation model to the output layer of the monocular depth estimation model based on the input data. In other words, the node value of the corresponding node may be propagated (e.g., forward propagated) to the node of the next layer connected by the node and the connecting edge (e.g., the next node). For example, the node may receive a value weighted by a connection weight from a previous node (e.g., multiple nodes) connected by a connecting edge.

[0088] The node value of the node can be determined based on applying an activation function (e.g., a sigmoid function) to the sum of weighted values ​​received from previous nodes (e.g., a weighted sum). For example, the parameters of the neural network may include the above-mentioned connection weights. The parameters of the neural network may be updated so that the values ​​of the loss functions (e.g., the first loss function and the second loss function) described later change along the target direction (e.g., the direction in which the loss is minimized).

[0089] A monocular depth estimation model may refer to a model that can be trained by machine learning, and may be a machine learning model that outputs a training output (e.g., a depth estimation map) from a training input (e.g., an input image).

[0090] For example, the learning algorithm may include supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but is not limited to the above examples. However, for the convenience of description in this specification, the training algorithm of the monocular depth estimation model is mainly described as semi-supervised learning.

[0091] The monocular depth estimation model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a U-Net (U-net) for image segmentation, a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN) or a deep Q network, or at least one combination of their combinations, but may not be limited to the above examples. However, for the convenience of description of this specification, the monocular depth estimation model in this specification is mainly described as a network of a U-Net structure, and may be the following model, in which the encoder is a ResNet model, and the decoder is a model that converts the sigmoid output into a depth estimation map.

[0092] U-Net is a monocular depth estimation model used by the device, which can be composed of an encoder, a decoder and their connections. The encoder can usually extract the hierarchical structure of the image feature map from low complexity to high complexity. On the other hand, the decoder can transform the features and reconstruct the output from low resolution to high resolution.

[0093] The monocular depth estimation model may include a convolution layer (Conv) that performs a linear transformation operation, a batch normalization layer (BN) that performs a normalization operation, a rectified linear unit (ReLU) layer that performs a nonlinear function operation, and a channel concatenation layer or a channel sum layer that merges the outputs of multiple layers.

[0094] The monocular depth estimation model can be trained to output a training output based on a training input. During the training process, the monocular depth estimation model can generate a temporary output in response to the training input, and can be trained to minimize the loss between the temporary output and the training output (e.g., a training target). During the training process, the parameters of the machine learning model (e.g., the connection weights between nodes / layers in a neural network) can be updated based on the loss. For example, such training can be performed by the device itself that performs the monocular depth estimation model training, and can be performed by a separate server. The trained monocular depth estimation model can be stored in a memory (e.g., Figure 1 In the memory 120).

[0095] In S230, the apparatus may update a plurality of weights included in the monocular depth estimation model by applying a loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map to the monocular depth estimation model. However, the method for training the monocular depth estimation model is not limited thereto. Figure 1 The device can train a monocular depth estimation model based on a loss function group including a first loss function and a second loss function.

[0096] Figure 3 An example of a process of training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device according to an example of the present disclosure is shown.

[0097] Device (e.g. Figure 1 The device 100 in FIG. 1 can obtain a depth distribution map 305 based on a depth map 303 obtained from a point cloud 301 at a target time point. The device can obtain a depth estimation map 311 by applying an input image 307 at the target time point to a monocular depth estimation model 309. The device can obtain a first loss function 313 based on the depth distribution map 305 and the depth estimation map 311.

[0098] The device may obtain posture change information 319 by applying the input image 315 at a time point different from the target time point and the input image 307 at the target time point to the posture estimation model 317. The device may obtain a reconstructed image 321 based on the posture change information 319. The device may obtain a second loss function 323 based on the obtained reconstructed image 321 and the input image 307 at the target time point. A detailed description of obtaining the reconstructed image 321 is described later below. Figure 6 Described in.

[0099] The input image at a time point different from the target time point (I t' )315 and the input image at the target time point (I t) 307 can be obtained from image frames obtained when the vehicle is traveling. For example, the input image 315 at a time point different from the target time point can represent an image obtained at a time point t' different from the target time point, at the position where the vehicle is traveling. Similarly, the input image 307 at the target time point can represent an image obtained at the position where the vehicle is traveling at the target time point t (i.e., a time point after time point t'). Therefore, the device can obtain posture change information 319 in the form of a matrix including posture change information about objects included in the time-continuous images by applying the input image 307 and the input image 315 to the posture estimation model 317.

[0100] The apparatus may update a plurality of weights included in the monocular depth estimation model 309 (ie, train the monocular depth estimation model 309 ) by applying a loss function group including the first loss function 313 and the second loss function 323 to the monocular depth estimation model 309 .

[0101] The device may obtain uncertainty 329 for each of a plurality of pixels included in the depth estimation map based on the depth estimation map 311 obtained from the trained monocular depth estimation model 309. For example, the device may obtain an estimated depth 325 for each of a plurality of pixels included in the depth estimation map 311.

[0102] Here, the estimated depth 325 may represent distance information of an object included in the target pixel (e.g., it may represent a portion of an object). For example, pixel A of the depth estimation map 311 may have an estimated depth of "a", and pixel B of the depth estimation map 311 may have an estimated depth of "b".

[0103] In addition to or as an alternative to the estimated depth of the target pixel, the device may also obtain the channel mean and the standard deviation of the channel of the target pixel through standard deviation estimation 327. The device may obtain the uncertainty 329 of the target pixel based on the channel mean and the standard deviation of the channel of the target pixel. A detailed description is provided later below. Figure 6 Described in.

[0104] As described above, the device may train the monocular depth estimation model 309 using the first loss function 313 and the second loss function 323. In addition, the device may obtain uncertainty 329 of the target pixel corresponding to the estimated depth 325 based on the estimated depth 325 obtained from the monocular depth estimation model 309.

[0105] If the device does not perform the process of obtaining the depth distribution map 305 from the point cloud 301 (hereinafter referred to as the "suggested process"), the performance of the monocular depth estimation model 309 that outputs the depth estimation map 311 and the uncertainty 329 may be reduced. Figure 4and Figure 5 For example, when only one depth estimation map 311 is obtained from the monocular depth estimation model 309, the device may train the monocular depth estimation model 309 without performing the suggestion process. In this case, the performance of the monocular depth estimation model 309 may not be reduced. On the other hand, when the device obtains the uncertainty 329 of the target pixel in addition to or as an alternative to obtaining the depth estimation map 311 from the monocular depth estimation model 309, if the device does not perform the suggestion process, the performance of the monocular depth estimation model 309 may be reduced. Therefore, the following will be referred to later by referring to Figure 4 and Figure 5 The proposed process of the monocular depth estimation model 309 is described in detail, thereby describing the training method of the monocular depth estimation model 309.

[0106] Figure 4 An example of a method of obtaining a depth distribution map based on a depth map in a training device according to an example of the present disclosure is shown.

[0107] Device (e.g. Figure 1 The device 100 in FIG. 1 can obtain a point group (e.g., point cloud 410) from the scene 400 through a laser radar. The device can obtain a depth map by extracting multiple depth information from the point cloud 410. For reference, as described above Figure 1 As described above, the depth map may represent a 2D image created by extracting only information in a depth direction from the point cloud 410 .

[0108] The device can obtain a three-dimensional (3D) representation from a 2D image (e.g., a depth tensor 420) by expanding a learning feature of input data (e.g., a channel of a depth map, a single-layer two-dimensional (2D) array containing depth information of each pixel in a data structure, etc.) by a predetermined first condition based on each of a plurality of pixels included in the depth map. In other words, the depth tensor 420 can represent a tensor including as many probability values ​​as the number of channels, which corresponds to the estimated depth of each pixel included in the depth map as a 2D image. Referring to Figure 4 , where an example where the number of channels is “K” is described.

[0109] Afterwards, the apparatus may obtain a depth distribution map for obtaining the first loss function based on the depth tensor 420. The method of obtaining the depth distribution map from the depth tensor 420 is described below. Figure 5 Described in.

[0110] Figure 5 An example of generating a depth distribution map in a training device according to an example of the present disclosure is shown.

[0111] Reference Figure 5, a set 510 including probabilities corresponding to the estimated depths included in the target pixel 505 can be described mainly focusing on the target pixel 505 in the depth tensor 500. In addition, the actual object depth of the target pixel 505 is described as 5.2 m.

[0112] Table 515 may include discrete values ​​(e.g., thresholds or intervals for placing continuous data into discrete buckets, step sizes or increment values, scaling factors for mapping floating point numbers to integers, etc.), representative discrete values, and probabilities for each of the multiple channels included in set 510. For example, a device (e.g., Figure 1 The device 100 in the embodiment may expand the channels to the same number as that corresponding to the first condition (e.g., the 22 channel condition) by using the depth information of each of the multiple pixels based on the multiple pixels included in the depth map. When the estimated depth of the target pixel is expanded to 22 channels, the set 510 may include probability values ​​corresponding to the channels respectively.

[0113] The device may determine a first discrete value (e.g., a minimum discrete value or a relatively small discrete value) and a second discrete value (e.g., a maximum discrete value or a discrete value greater than the first discrete value) based on a sensor (e.g., a lidar) (i.e., a condition of extending 22 channels based on the lidar specification) to determine a discrete value of each channel included in each individual pixel or target pixel of the depth tensor. For example, referring to Figure 5 The minimum discrete value is 0.01 and the maximum discrete value is 120.

[0114] The device can determine the discrete value of each channel included in each individual pixel or target pixel 505 based on the index of each channel included in the single pixel or target pixel 505 (e.g., channel 0, channel 1, channel 2, ..., channel 21), the minimum discrete value (e.g., 0.01), the maximum discrete value (e.g., 120), and the number of channels (e.g., the first condition 22).

[0115] The method by which the device determines the discrete value of each channel can be expressed by the following Equations 1 to 3.

[0116] d i =d min +(d max -d min )×i / n bins d i =d min +(d max -d min )×i / n bins Equation 1

[0117] Here, Equation 1 may refer to uniform discretization. min Can represent the minimum discrete value; dmax Can represent the maximum discrete value; i can represent the channel index; n bins The number of channels can be expressed, i.e. the target number determined by a discrete value.

[0118]

[0119] Here, Equation 2 can represent the discretization of the spacing increase. min ,d max , i and n bins can be the same as described in Equation 1.

[0120]

[0121] Here, Equation 3 can represent a linearly increasing discretization. min ,d max , i and n bins It can be the same as described in Equation 1 and Equation 2.

[0122] For reference, in order to facilitate the description of this specification, it is mainly described that the device determines the discrete value of the channel based on the linear increase discretization of Equation 3. Therefore, referring to the discrete values ​​in Table 515, the discrete values ​​in Table 515 may include values ​​obtained by determining the discrete value of each channel included in a single pixel by determining the index, the minimum discrete value, the maximum discrete value, and the number of channels according to the linear increase discretization in Equation 3.

[0123] Thereafter, the device may determine the ratio of the discrete value of the Nth channel to the discrete value of the (N+1)th channel as the representative discrete value of the Nth channel based on determining the discrete value of the Nth channel of the single pixel or target pixel 505 and the discrete value of the (N+1)th channel after the Nth channel of the single pixel or target pixel 505.

[0124] The method by which the device determines the representative discrete value can be expressed by the following Equation 4.

[0125]

[0126] Here, mid can represent a representative discrete value; B N Can represent the discrete value of the Nth channel; B N+1 Can represent the discrete value of the (N+1)th channel.

[0127] Therefore, referring to the representative discrete values ​​in Table 515, the device can determine the first representative discrete value to be 0.340476 based on the discrete value of the first channel 0.01 and the discrete value of the second channel 0.6709524. In addition, the device can determine the representative discrete values ​​of the plurality of channels according to Equation 4.

[0128] The device may set the probability value of each channel based on the situation where the representative discrete value of the channel is determined. For example, the device may set the probability value that matches the representative discrete value corresponding to each channel.

[0129] Specifically, in the case where the target pixel 505 includes an estimated depth corresponding to 5.2 m (i.e., the depth of the actual object), the device may set a probability value matching the representative discrete value while increasing or decreasing the probability value by 0.05 from 0 to 1. For example, to obtain a probability distribution of an estimated depth corresponding to a depth of 5.2 m, the device may set the representative discrete value of channel 4 (e.g., 4.667619) and the representative discrete value of channel 5 (e.g., 7.236905) closest to 5.2 m as the target region. The device may increase or decrease the probability value matching the representative discrete values ​​corresponding to the two channels included in the target region by 0.05 from 0 to 1.

[0130] In more detail, the device may sequentially set probability values ​​1, 0.95, 0.9, 0.85, and 0.8 as probability values ​​that match the representative discrete value (e.g., 4.667619) of channel 4. Accordingly, the device may sequentially set probability values ​​0, 0.05, 0.1, 0.15, and 0.2 as probability values ​​that match the representative discrete value (e.g., 7.236905) of channel 5 (i.e., the second condition that the sum may be 1). In summary, the representative discrete value (e.g., 4.667619) of channel 4 and the representative discrete value (e.g., 7.236905) of channel 5 may follow a first distribution (1, 0), a second distribution (0.95, 0.05), a third distribution (0.9, 0.1), a fourth distribution (0.85, 0.15), and a fifth distribution (0.8, 0.2).

[0131] However, the method of setting the probability value is not limited to this. For example, in order to obtain the probability distribution of the estimated depth corresponding to the depth of 5.2 m, the device can increase or decrease the probability values ​​of the three channels, such as the representative discrete value of channel 4 closest to 5.2 m (e.g., 4.667619), the representative discrete value of channel 5 (e.g., 7.236905), and the representative discrete value of channel 6 (e.g., 10.37714), from 0 to 1 by 0.05.

[0132] Next, the apparatus may determine the obtained expected value as the estimated depth of the target pixel 505 by obtaining the expected value of each of the first to fifth distributions.

[0133] The method by which the device obtains the distribution expectation value can be expressed by the following equation 5.

[0134]

[0135] Here, E(x) may represent an expected value; Pi may represent a probability value matching the i-th representative discrete value; can represent the i-th representative discrete value.

[0136] For example, according to Equation 5, the device can obtain "4.667619", which is the result of the expected value "4.667619×1+7.236905×0" of the first distribution. According to Equation 5, the device can obtain "4.796083", which is the result of the expected value "4.667619×0.95+7.236905×0.05" of the second distribution. According to Equation 5, the device can obtain "4.924548", which is the result of the expected value "4.667619×0.9+7.236905×0.1" of the third distribution. According to Equation 5, the device can obtain "5.053012", which is the result of the expected value "4.667619×0.85+7.236905×0.15" of the fourth distribution. According to Equation 5, the device can obtain "5.181476" which is a result of "4.667619×0.8+7.236905×0.2" as the expected value of the fifth distribution.

[0137] Here, referring to the fifth distribution, the expected value 540 of the fifth distribution may be 5.181476 and may represent a value closest to the estimated depth corresponding to 5.2 m, which is the depth of the actual object at the target pixel 505. Therefore, the device may set the fifth distribution as the target distribution 530, and may determine the target distribution 530 as the probability distribution of the estimated depth included in the target pixel 505.

[0138] As described above, the device may determine not only the probability distribution of the target pixel 505 but also the probability distribution of each pixel different from the target pixel 505. If the probability distribution of the pixels (e.g., part or all) included in the depth tensor 500 is determined, the device may determine the above-mentioned depth tensor 500 as a depth estimation map.

[0139] Figure 6 An example of a process of training a monocular depth estimation model by obtaining a first loss function and a second loss function in a training device according to an example of the present disclosure is shown.

[0140] Device (e.g. Figure 1 The device 100 in FIG. 1 can obtain the posture conversion information 607 by applying the input image 601 at time point t' and the input image 603 at time point t to the posture estimation model 605. The device can obtain the depth estimation map 611 by applying the input image 603 at time point t to the monocular depth estimation model 609.

[0141] The device may obtain a first point group (e.g., a first point cloud 613) as a 3D point cloud at time point t by applying the inverse of the intrinsic parameters corresponding to a specific image sensor (e.g., a lidar or a camera) to the depth estimation map 611. Thereafter, the device may obtain a second point group (e.g., a second point cloud 615) as a 3D point cloud at time point t' by applying the pose conversion information 607 to the first point cloud 613. In addition, the device may convert the second point cloud 615 at time point t' into 2D image coordinates by applying the intrinsic parameters to the second point cloud 615. In addition, the device may create a reconstructed image 617 based on pixels corresponding to the 2D image coordinates in the input image 603 at time point t.

[0142] The device may convert the 3D point cloud at time point t corresponding to the position (10, 20) of the depth estimation map 611 into a 3D point cloud at time point t' based on the posture conversion information 607. In addition, the device may convert the 3D point cloud at time point t' corresponding to the position (10, 20) of the depth estimation map 611 into 2D image coordinates (e.g., (15, 18)) based on the intrinsic parameters. In addition, the device may generate a reconstructed image 617 by assigning the pixel value (e.g., 218) corresponding to the 2D image coordinates (15, 18) at time point t' of the input image 603 to (10, 20).

[0143] The device may determine a first loss function based on the depth estimation map 611 and the depth distribution map 619 , and may determine a second loss function based on the reconstructed image 617 and the input image 603 at the time point t.

[0144] The device can obtain a first estimated depth of a first pixel among a plurality of pixels included in the depth distribution map 619, and can obtain a second estimated depth of a second pixel among a plurality of pixels included in the depth estimation map 611, the second pixel being associated with a position corresponding to the first pixel. For reference, the estimated depth can represent an expected value of a probability distribution corresponding to the corresponding pixel. The device can determine a first loss function based on the difference between the first estimated depth and the second estimated depth. Thereafter, the device can update a plurality of weights included in the monocular depth estimation model 609 based on the first loss function thus determined, so as to obtain a second estimated depth.

[0145] However, the method for determining the first loss function is not limited to this. For example, if the first estimated depth satisfies a predetermined third condition, the device may determine the difference between the first estimated depth and the second estimated depth as the first loss function. If the first estimated depth does not satisfy the third condition, the device may skip updating at least one weight included in the monocular depth estimation model for obtaining the second estimated depth. Specifically, the third condition may represent a condition that the estimated depth is not 0, and the first loss function may be represented by the following equations 6 and 7.

[0146] depth mask =(estimated depth >0?1:0) Equation 6

[0147] Here, depth mask It may represent a mask used to adjust the training of the monocular depth estimation model 609 according to the value of the first estimated depth. depth A first estimated depth may be represented.

[0148] That is, if the first estimated depth is greater than "0", then depth mask can include "1", and in other cases, depth mask "0" may be included.

[0149] Loss = avg[-log{1-abs((E d1 ×depth mask )-(E d2 ×depth mask ))}] Equation 7

[0150] Here, Loss can represent the first loss function; E d1 It can represent the first estimated depth; E d2 Can represent the second estimated depth; depth mask can be the same as described in Equation 6.

[0151] In summary, if the first estimated depth is greater than "0", the first loss function may be determined based on the difference between the first estimated depth and the second estimated depth. If the first estimated depth is less than or equal to "0", the first loss function may be "0". In other words, if the first estimated depth is less than or equal to "0", the device determines that the first loss function is "0". The device may skip updating a plurality of weights included in the monocular depth estimation model for obtaining the second estimated depth.

[0152] Through the above example, the device can train the monocular depth estimation model 609 by applying the loss function group including the first loss function and the second loss function to the monocular depth estimation model 609. The device can obtain the RSD representing the uncertainty of the target pixel by obtaining the first factor representing the channel average of the target pixel among the pixels included in the depth estimation map 611 and the second factor representing the channel standard deviation of the target pixel. The detailed description of RSD is as follows Figure 7 and Figure 8 Described in.

[0153] Figure 7An example of a target image, an estimated depth of each pixel included in an input image, and uncertainty of each pixel included in the input image in an inference device according to an example of the present disclosure is shown.

[0154] According to an example, the reasoning device may include a processor and a memory containing instructions. For example, the reasoning device may represent a method of using a training device (e.g., Figure 1 An inference device for a monocular depth estimation model trained by the training device 100).

[0155] The reasoning device may obtain a target image 700 for testing. The reasoning device may obtain a target depth estimation map 710 including an estimated depth of each of a plurality of pixels included in the target image by applying the target image 700 to a monocular depth estimation model including an updated weight (i.e., a monocular depth estimation model trained by a training device). The reasoning device may obtain a target uncertainty map 720 including an RSD of each of a plurality of estimated depths included in the target depth estimation map 710.

[0156] For example, the inference device may obtain a first factor representing a channel average of target pixels and a second factor representing a channel standard deviation of target pixels among pixels included in the target depth estimation map 710. The inference device may obtain RSD representing uncertainty of the target pixel based on the first factor and the second factor.

[0157] The RSD according to the example can be expressed by the following Equation 8.

[0158]

[0159] Here, (h, w) may represent coordinates indicating the position of a target pixel; μ h,w can represent the first factor; δ h,w Can represent the second factor; RSD h,w Can represent RSD.

[0160] Figure 8 An example of correlation between estimated depth and RSD in a training apparatus according to an example of the present disclosure is shown.

[0161] Reference Figure 8 , the horizontal axis in the figure may represent the error ratio of the trained monocular depth estimation model, and the vertical axis in the figure may represent the RSD. For example, the area 800 shown in the figure may include the error ratio and RSD of each of the multiple pixels in the depth estimation map obtained from the trained monocular depth estimation model.

[0162] Next, the straight line 810 shown in the figure may be a straight line representing the correlation between the error ratio and the RSD in the region 800. That is, the straight line 810 may include a relationship in which the error ratio increases as the RSD representing uncertainty increases. The correlation corresponding to the straight line 810 may be represented by the following equation 9.

[0163]

[0164] Here, It can represent the error ratio, that is, the horizontal axis in the figure. Specifically, it can refer to the error ratio obtained based on the estimated depth and the actual depth of each pixel in the depth estimation map. h,w The RSD indicating the uncertainty of the target pixel (h,w) may be represented.

[0165] Fig. 9 An example of a computing system related to a training apparatus or a training method according to an example of the present disclosure is shown.

[0166] Reference Fig. 9 , a computing system 1000 related to a training device or a training method may include at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage device 1600 , and a network interface 1700 connected to each other via a bus 1200 .

[0167] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. Each of the memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read-only memory (ROM) and a random access memory (RAM).

[0168] Therefore, the operation of the method or algorithm described in conjunction with the examples disclosed in this specification may be directly implemented as a hardware module or a software module, or a combination of a hardware module and a software module, executed by the processor 1100. The software module may reside on a storage medium (i.e., the memory 1300 and / or the storage device 1600) such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk drive, a removable disk, or a compact disk-ROM (CD-ROM).

[0169] The storage medium may be coupled to the processor 1100. The processor 1100 may read information from the storage medium and may write information into the storage medium. Alternatively or additionally, the storage medium may be integrated with the processor 1100. The processor and the storage medium may be implemented as an application specific integrated circuit (ASIC). The ASIC may be provided in a user terminal. Alternatively or additionally, the processor and the storage medium may be implemented as separate components in a user terminal.

[0170] The present disclosure aims to solve the above-mentioned problems arising in the prior art while maintaining the advantages achieved by the prior art unchanged.

[0171] The examples of the present disclosure provide a training device, an inference device, and a training method, which can obtain pixel estimated depth by obtaining a depth distribution map based on a depth map obtained from a point cloud by utilizing only a monocular depth estimation model without using a complex model structure, loss function, and disparity image generation model for obtaining self-supervised depth estimation.

[0172] In addition, the examples of the present disclosure provide a training device, an inference device and a training method, which can estimate uncertainty while maintaining the depth estimation performance of a monocular depth estimation model by obtaining a first loss function based on a depth distribution map and a depth estimation map, and obtaining a second loss function based on images classified using multiple time points.

[0173] In addition, the examples of the present disclosure provide a training device, an inference device, and a training method, which can obtain an effective correlation between the estimated depth of each of the multiple pixels included in the target depth estimation map and the relative standard deviation (RSD) representing the uncertainty by obtaining the estimated depth of each of the multiple pixels included in the target depth estimation map.

[0174] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems. For those skilled in the art to which the present disclosure belongs, any other technical problems not mentioned in this document can be clearly understood through the following description.

[0175] According to an example of the present disclosure, a training device may include a memory storing computer executable instructions and at least one processor that executes the instructions by accessing the memory. The at least one processor may obtain a depth distribution map based on a depth map obtained from a point cloud at a target time point, may obtain a depth estimation map by applying an input image at the target time point to a monocular depth estimation (MDE) model, and may update a plurality of weights included in the MDE model by applying a loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map to the MDE model.

[0176] In an example, at least one processor may obtain a depth map by extracting multiple depth information from a point cloud, may obtain a depth tensor based on multiple pixels included in the depth map by extending a channel of the depth map by a predetermined first condition using multiple depth information of each pixel in the multiple pixels, and may obtain a depth distribution map based on the depth tensor.

[0177] In an example, at least one processor may determine a minimum discrete value and a maximum discrete value based on a sensor (e.g., a light detection and ranging (lidar) sensor) to determine a discrete value for each channel included in a single pixel of a depth tensor, and may determine the discrete value for each channel included in the single pixel based on an index of each channel included in the single pixel, a minimum discrete value, a maximum discrete value, and the number of channels.

[0178] In an example, at least one processor may determine a ratio of a discrete value of the Nth channel to a discrete value of the (N+1)th channel as a representative discrete value of the Nth channel based on a case where a discrete value of the Nth channel of a single pixel and a discrete value of the (N+1)th channel after the Nth channel of the single pixel are determined. The Nth channel may be a channel with respect to a natural number "N" that is less than or equal to the number of channels of the depth tensor.

[0179] In an example, the depth profile may include pixels, for each pixel, a representative discrete value corresponding to a channel included in each pixel of the depth tensor is determined. The sum of probabilities corresponding to the channel included in each pixel of the depth tensor may include a probability of satisfying a predetermined second condition.

[0180] In an example, at least one processor can obtain a first estimated depth of a first pixel among a plurality of pixels included in a depth distribution map, can obtain a second estimated depth of a second pixel among a plurality of pixels included in the depth estimation map and related to a position corresponding to the first pixel, can determine a first loss function based on the first estimated depth and the second estimated depth, and can update a plurality of weights included in the MDE model for obtaining a second estimated depth based on the determined first loss function.

[0181] In an example, if the first estimated depth satisfies a predetermined third condition, at least one processor may determine the difference between the first estimated depth and the second estimated depth as a first loss function, and if the first estimated depth does not satisfy the third condition, updating multiple weights included in the MDE model for obtaining the second estimated depth may be skipped.

[0182] In an example, at least one processor may obtain posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model, may obtain a first point cloud at the target time point by applying an inverse of an intrinsic parameter associated with a laser radar to a depth estimation map, may obtain a second point cloud at a time point different from the target time point by applying the posture change information to the first point cloud, and may determine a second loss function different from the first loss function based on the second point cloud. The loss function group may further include a second loss function.

[0183] In an example, the at least one processor may obtain a reconstructed image by applying the intrinsic parameters to the second point cloud, and may determine a second loss function based on the input image and the reconstructed image.

[0184] In an example, at least one processor may obtain a first factor representing an average value of a channel of a target pixel among pixels included in a depth estimation map, and a second factor representing a standard deviation of the channel of the target pixel, and may obtain a relative standard deviation (RSD) representing uncertainty of the target pixel based on the first factor and the second factor.

[0185] According to an example of the present disclosure, an inference device using an MDE model trained by a training device may include a memory configured to store instructions that can be executed by a computer and at least one processor configured to execute the instructions by accessing the memory. The at least one processor may obtain a target image for testing, may obtain a target depth estimation map including an estimated depth of each of a plurality of pixels included in the target image by applying the target image to an MDE model including updated weights, and may obtain a target uncertainty map including the RSD of each of a plurality of estimated depths included in the target depth estimation map.

[0186] According to an example of the present disclosure, a training method may include: obtaining a depth distribution map based on a depth map obtained from a point cloud at a target time point; obtaining a depth estimation map by applying an input image at the target time point to an MDE model; and updating multiple weights included in the MDE model by applying a loss function group including a first loss function obtained based on the depth distribution map and the depth estimation map to the MDE model.

[0187] In an example, obtaining a depth distribution map may include: obtaining a depth map by extracting multiple depth information from a point cloud; based on multiple pixels included in the depth map, obtaining a depth tensor by extending a channel of the depth map by a predetermined first condition using multiple depth information of each pixel in the multiple pixels; and obtaining a depth distribution map based on the depth tensor.

[0188] In an example, obtaining a depth distribution map may include: determining a minimum discrete value and a maximum discrete value based on a lidar to determine a discrete value of each channel included in a single pixel of a depth tensor; and determining a discrete value of each channel included in a single pixel based on an index, a minimum discrete value, a maximum discrete value, and a number of channels of each channel included in the single pixel.

[0189] In an example, obtaining a depth distribution map may include: determining a ratio of a discrete value of the Nth channel to a discrete value of the (N+1)th channel after the Nth channel of a single pixel based on determining a discrete value of the Nth channel and a discrete value of the (N+1)th channel of the single pixel as a representative discrete value of the Nth channel. The Nth channel may be a channel with respect to a natural number "N", which is less than or equal to the number of channels of the depth tensor. The depth distribution map may include pixels, and for each pixel, a representative discrete value corresponding to a channel included in each pixel of the depth tensor is determined. The sum of probabilities corresponding to the channels included in each pixel of the depth tensor may include a probability of satisfying a predetermined second condition.

[0190] In an example, updating multiple weights included in the MDE model may include: obtaining a first estimated depth of a first pixel among multiple pixels included in the depth distribution map, obtaining a second estimated depth of a second pixel among multiple pixels included in the depth estimation map related to a position corresponding to the first pixel, determining a first loss function based on the first estimated depth and the second estimated depth, and updating multiple weights included in the MDE model for obtaining the second estimated depth based on the determined first loss function.

[0191] In an example, updating the multiple weights included in the MDE model may include: if the first estimated depth satisfies a predetermined third condition, determining the difference between the first estimated depth and the second estimated depth as a first loss function; and if the first estimated depth does not satisfy the third condition, skipping updating the multiple weights included in the MDE model for obtaining the second estimated depth.

[0192] In an example, the training method may further include: obtaining posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model; obtaining a first point cloud at the target time point by applying an inverse of an intrinsic parameter associated with a laser radar to a depth estimation map; obtaining a second point cloud at a time point different from the target time point by applying the posture change information to the first point cloud; and determining a second loss function different from the first loss function based on the second point cloud. The loss function group may further include a second loss function.

[0193] In an example, determining the second loss function may include: obtaining a reconstructed image by applying the intrinsic parameters to the second point cloud; and determining the second loss function based on the input image and the reconstructed image.

[0194] In an example, the training method may further include: obtaining a first factor representing the average value of a channel of a target pixel among pixels included in the depth estimation map and a second factor representing the standard deviation of the channel of the target pixel; and obtaining an RSD representing the uncertainty of the target pixel based on the first factor and the second factor.

[0195] The above description is merely an example of the technical idea of ​​the present disclosure, and those skilled in the art may make various modifications and changes without departing from the basic features of the present disclosure.

[0196] The above examples can be implemented by hardware elements, software elements and / or a combination of hardware elements and software elements. For example, the devices, methods and components described in the examples of the present disclosure can be implemented by utilizing a general-purpose computer or a special-purpose computer, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor or any device that can run instructions and respond. The processing device can execute an operating system (OS) or a software application running on the OS. In addition, the processing device can access, store, operate, process and generate data in response to the operation of the software. It will be understood by those skilled in the art that, although a single processing device is shown for ease of understanding, the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. Moreover, the processing device may include different processing configurations, such as parallel processors.

[0197] Software may include computer programs, codes, instructions, or one or more combinations thereof, and configure a processing device to operate in a desired manner or to independently or jointly control a processing device. Software and / or data may be embodied permanently or temporarily in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal waves for interpretation by or provision of instructions or data to a processing device. Software may be distributed throughout a computer system connected by a network and stored or run in a decentralized manner. Software and data may be recorded in a computer-readable storage medium.

[0198] The method according to the above example can be recorded in a computer-readable medium, which includes program instructions that can be run by various computer devices. The computer-readable medium can also include program instructions, data files, data structures, or combinations thereof. The program instructions recorded in the medium can be specially designed and configured for the examples of the present disclosure, or they can be known and available to computer software technicians. The computer-readable medium can include hardware devices specially configured to store and run program instructions, such as magnetic media (e.g., hard disks, floppy disks, or tapes), optical recording media (e.g., CD-ROMs and DVDs), magneto-optical media (e.g., optical magnetic disks), read-only memories (ROMs), random access memories (RAMs), and flash memories. Examples of computer programs include not only machine language codes created by compilers, but also high-level language codes that can be run by computers by utilizing interpreters, etc.

[0199] The aforementioned hardware devices may be configured to function as one or more software modules to perform the operations of the aforementioned examples of the present disclosure, and vice versa.

[0200] Even though the examples are described with reference to the limiting figures, it is obvious that those skilled in the art can make various changes or modifications to the examples based on the above description. For example, even if the above processes and methods are performed in a different order from that described above, and / or the above elements (e.g., systems, structures, devices, or circuits) are combined or connected in a different form and mode from that described above, or replaced or exchanged with other elements or equivalent elements, sufficient effects can be achieved.

[0201] Accordingly, other embodiments, other examples, and equivalents to the claims are within the scope of the appended claims.

[0202] Therefore, the examples of the present disclosure are not intended to limit but to illustrate the technical ideas of the present disclosure, and the scope and spirit of the present disclosure are not limited by the above examples. The protection scope of the present disclosure should be interpreted by the attached claims, and all equivalent schemes should be interpreted as included in the scope of the present disclosure.

[0203] A description of a training device, an inference device, and a training method and methods thereof according to examples of the present disclosure is as follows.

[0204] According to at least one of the examples of the present disclosure, pixel estimated depth can be obtained by obtaining a depth distribution map based on a depth map obtained from a point cloud by utilizing only a monocular depth estimation model without utilizing an image generation model of a complex model structure, loss function, and gap for obtaining self-supervised depth estimation.

[0205] In addition, according to at least one example among the examples of the present disclosure, the uncertainty can be estimated while maintaining the depth estimation performance of the monocular depth estimation model by obtaining a first loss function based on a depth distribution map and a depth estimation map, and obtaining a second loss function based on images classified using multiple time points.

[0206] In addition, according to at least one example among the examples of the present disclosure, an effective correlation between the estimated depth of each of the multiple pixels included in the target depth estimation map and the relative standard deviation (RSD) representing the uncertainty can be obtained.

[0207] Furthermore, various effects directly or indirectly understood through the present specification can also be provided.

[0208] Although the present disclosure is described above with reference to examples and drawings, the present disclosure is not limited thereto, and those skilled in the art may make various modifications and changes to the present disclosure without departing from the spirit and scope of the present disclosure as claimed in the claims.

Claims

1. A device for training, comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Obtaining a depth distribution map based on the depth map obtained from the point group at the target time point; Obtaining a depth estimation map based on an input image associated with the target time point and applied to a monocular depth estimation model, i.e., an MDE model; updating a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group includes a first loss function obtained based on the depth distribution map and the depth estimation map; and A signal indicative of the updated plurality of weights is output.

2. The device according to claim 1, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: Obtaining the depth map by extracting a plurality of pieces of depth information from the point group, wherein the plurality of pieces of depth information are associated with a plurality of pixels included in the depth map; Obtaining a depth tensor by expanding channels of the depth map, wherein the channels of the depth map are expanded by a first condition based on the plurality of depth information; as well as Based on the depth tensor, the depth distribution map is obtained.

3. The device according to claim 2, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: Determining, based on sensing information obtained by a sensor, a minimum discrete value and a maximum discrete value associated with a channel included in a single pixel of the depth tensor; as well as A discrete value for each of the channels is determined, wherein the discrete value for each of the channels is determined based on an index of each of the channels included in the single pixel, the minimum discrete value, the maximum discrete value, and the number of the channels.

4. The device according to claim 3, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: Based on the discrete value of the Nth channel of a single pixel of the depth tensor and the discrete value of the (N+1)th channel after the Nth channel of the single pixel, determine a ratio of the discrete value of the Nth channel to the discrete value of the (N+1)th channel as a representative discrete value of the Nth channel, and N is a natural number and is less than or equal to the total number of channels of the depth tensor.

5. The device according to claim 4, wherein: The depth profile includes a plurality of pixels, and a representative discrete value of a pixel in the plurality of pixels is associated with a channel included in a pixel in the plurality of pixels of the depth tensor, and The sum of probabilities includes a probability of satisfying a second condition, the sum of probabilities being associated with a channel included in each of a plurality of pixels of the depth tensor.

6. The device according to claim 1, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: Obtaining a first estimated depth of a first pixel among a plurality of pixels included in the depth distribution map; Obtaining a second estimated depth of a second pixel among a plurality of pixels included in the depth estimation map, wherein the second pixel is related to a position corresponding to the first pixel; determining the first loss function based on the first estimated depth and the second estimated depth; and Based on the determined first loss function, a plurality of weights included in the MDE model for obtaining the second estimated depth are updated.

7. The device according to claim 6, wherein: When executed by the at least one processor, the instructions cause the apparatus to: Based on the first estimated depth satisfying a third condition, determining a difference between the first estimated depth and the second estimated depth as the first loss function; or Based on the first estimated depth not satisfying the third condition, updating of a plurality of weights included in the MDE model for obtaining the second estimated depth is skipped.

8. The device according to claim 1, wherein: When executed by the plurality of processors, the instructions further cause the apparatus to: Obtaining posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model; Obtaining a first point group at the target time point by applying an inverse of intrinsic parameters associated with the sensor to the depth estimation map; Obtaining a second point group at a time point different from the target time point by applying the posture change information to the first point group; as well as Based on the second point group, determining a second loss function different from the first loss function, and The loss function group further includes the second loss function.

9. The device according to claim 8, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: Obtaining a reconstructed image by applying the intrinsic parameters to the second point group; and Based on the input image and the reconstructed image, the second loss function is determined.

10. The device according to claim 8, wherein: When executed by the at least one processor, the instructions further cause the apparatus to: obtaining a first factor representing an average value associated with a channel of a target pixel among a plurality of pixels included in the depth estimation map and a second factor representing a standard deviation value associated with the channel of the target pixel; as well as Based on the first factor and the second factor, a relative standard deviation value representing the uncertainty of the target pixel is obtained.

11. A device for reasoning, comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Obtain a target image for testing; Obtaining a target depth estimation map by applying the target image to a monocular depth estimation model including updating weights, wherein the target depth estimation map includes an estimated depth of each pixel of a plurality of pixels included in the target image; obtaining a target uncertainty map, wherein the target uncertainty map includes a relative standard deviation value for each of a plurality of estimated depths included in the target depth estimation map; as well as A signal indicative of the target uncertainty map is output.

12. A training method, the method being executed by a processor, the method comprising: Obtaining a depth distribution map based on the depth map obtained from the point group at the target time point; Obtaining a depth estimation map by applying an input image associated with the target time point to a monocular depth estimation model, i.e., an MDE model; updating a plurality of weights included in the MDE model based on a loss function group applied to the MDE model, wherein the loss function group includes a first loss function obtained based on the depth distribution map and the depth estimation map; and A signal indicative of the updated plurality of weights is output.

13. The method according to claim 12, wherein: Obtaining the depth distribution map includes: Obtaining the depth map by extracting a plurality of pieces of depth information from the point group, wherein the plurality of pieces of depth information are associated with a plurality of pixels included in the depth map; Obtaining a depth tensor by expanding channels of the depth map, wherein the channels of the depth map are expanded by a first condition based on the plurality of depth information; and Based on the depth tensor, the depth distribution map is obtained.

14. The method according to claim 13, wherein: Obtaining the depth distribution map includes: Determining a minimum discrete value and a maximum discrete value associated with a channel included in a single pixel of the depth tensor based on sensing information obtained by a sensor; and A discrete value for each of the channels is determined, wherein the discrete value for each of the channels is determined based on an index of each of the channels included in the single pixel, the minimum discrete value, the maximum discrete value, and the number of the channels.

15. The method according to claim 14, wherein: Obtaining the depth distribution map includes: Based on the discrete value of the Nth channel of a single pixel and the discrete value of the (N+1)th channel following the Nth channel of the single pixel, determine a ratio of the discrete value of the Nth channel to the discrete value of the (N+1)th channel as a representative discrete value of the Nth channel, and N is a natural number and is less than or equal to the total number of channels of the depth tensor, The depth profile includes a plurality of pixels, and a representative discrete value of a pixel in the plurality of pixels is associated with a channel included in a pixel in the plurality of pixels of the depth tensor, and The sum of probabilities includes a probability of satisfying a second condition, the sum of probabilities being associated with a channel included in each of a plurality of pixels of the depth tensor.

16. The method according to claim 12, wherein: Updating the plurality of weights included in the MDE model includes: Obtaining a first estimated depth of a first pixel among a plurality of pixels included in the depth distribution map; Obtaining a second estimated depth of a second pixel among a plurality of pixels included in the depth estimation map, the second pixel being related to a position corresponding to the first pixel; determining the first loss function based on the first estimated depth and the second estimated depth; and Based on the determined first loss function, a plurality of weights included in the MDE model for obtaining the second estimated depth are updated.

17. The method according to claim 16, wherein: Updating the plurality of weights included in the MDE model includes: Based on the first estimated depth satisfying a third condition, determining a difference between the first estimated depth and the second estimated depth as the first loss function; or Based on the first estimated depth not satisfying the third condition, updating of a plurality of weights included in the MDE model for obtaining the second estimated depth is skipped.

18. The method according to claim 12, further comprising: Obtaining posture change information by applying an input image at a time point different from the target time point and an input image at the target time point to a posture estimation model; Obtaining a first point group at the target time point by applying an inverse of intrinsic parameters associated with the sensor to the depth estimation map; Obtaining a second point group at a time point different from the target time point by applying the posture change information to the first point group; as well as Based on the second point group, determining a second loss function different from the first loss function, and Wherein the loss function group further includes the second loss function.

19. The method according to claim 18, wherein: Determining the second loss function includes: Obtaining a reconstructed image by applying the intrinsic parameters to the second point group; and Based on the input image and the reconstructed image, the second loss function is determined.

20. The method of claim 18, further comprising: obtaining a first factor representing an average value associated with a channel of a target pixel among a plurality of pixels included in the depth estimation map and a second factor representing a standard deviation value associated with the channel of the target pixel; as well as Based on the first factor and the second factor, a relative standard deviation value representing the uncertainty of the target pixel is obtained.

Images