Learning device, learning method thereof, test equipment and test method

Abstract

The invention discloses a learning device, a learning method thereof, a test device and a test method. A learning device is presented. The apparatus may include a processor and a memory storing instructions that, when executed by the processor, may cause the apparatus to obtain at least one first depth map based on at least one piece of cloud data associated with ambient information, and at least one first image associated with the at least one first depth map, determining variance estimation information indicating a variance between the at least one first depth map and the at least one first image based on the at least one first depth map and the at least one first image, back-propagating a variance loss based on the first variance estimation information and variance reference true value (GT) information associated with the first variance estimation information, and updating a parameter associated with determining the first variance estimation information based on the back-propagating variance loss.

Description

[0001] Related Applications

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0114011 filed on August 29, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a learning device, a learning method thereof, a test apparatus using the same, and a test method using the same, and more particularly, to a learning device for constructing training data for training another network, a learning method thereof, a test apparatus using the same, and a test method using the same. Background Art

[0004] For deep neural network-based computer vision technology in autonomous driving technology, various artificial intelligence models can be considered, such as object detection, semantic segmentation, depth estimation, and lane detection.

[0005] For example, depth estimation can be used variously for identifying surroundings and environmental objects using cameras in autonomous driving situations, free space, etc.

[0006] High-quality depth images for training, which are generated based on point cloud data obtained by light detection and ranging (LiDAR) sensors, are needed for training depth map estimation networks that perform such depth estimation operations.

[0007] A high-quality depth map for training may refer to a depth map in which the consistency between the camera-based image and the LiDAR sensor-based depth map is high. High consistency may mean that the depth information at a specific location (e.g., pixel coordinates (50, 90)) on the depth map accurately represents the depth information about a specific object present at the specific location (e.g., pixel coordinates (50, 90)) on the image corresponding to the depth map.

[0008] However, the consistency between the depth map and the image may be low due to various reasons, for example, due to errors in calibration for the difference between the location where the LiDAR is installed and the location where the camera is installed, errors in calibration for correcting camera distortion, or differences in relative speed between a moving object where the LiDAR is installed and surrounding objects.

[0009] For example, refer to Figure 1A , which shows that the consistency between the depth map and the image is low, and it can be identified that multiple point cloud data corresponding to trees and buildings are projected to incorrect locations on the image. Figure 1B, which shows that there is a high consistency between the depth map and the image, and it can be identified that multiple point cloud data corresponding to trees and buildings are accurately projected to the positions of the corresponding objects on the image.

[0010] If a depth map with low consistency is used to train a depth map estimation network, the accuracy of the depth map estimation network may be low. An inspector may check the consistency of each of the depth maps individually and select a high-quality depth map to prevent it. Therefore, a lot of time and cost may be consumed when constructing a large number of high-quality depth maps for training. Summary of the invention

[0011] According to the present disclosure, a device may include: a processor; and a memory storing instructions, which, when executed by the processor, may enable the device to: obtain at least one first depth map based on at least one cloud data associated with surrounding environment information; and at least one first image associated with the at least one first depth map; determine first variance estimation information indicating the variance between the at least one first depth map and the at least one first image based on the at least one first depth map and the at least one first image; back-propagate variance loss based on the first variance estimation information and variance reference true value (ground truth, GT) information related to the first variance estimation information; and update parameters associated with determining the first variance estimation information based on the back-propagated variance loss.

[0012] The device, wherein the instructions, when executed by the processor, may cause the device to: apply an encoding operation to the at least one first depth map; obtain a first feature map corresponding to the at least one first depth map based on the encoding operation; apply a second encoding operation to the at least one first image; obtain a second feature map corresponding to the at least one first image based on the second encoding operation; generate an integrated feature map based on the first feature map and the second feature map; apply a decoding operation to the integrated feature map; and determine the first variance estimation information based on the decoding operation.

[0013] The device, wherein the first variance estimation information may include: width variance estimation information indicating width variance; and height variance estimation information indicating height variance, wherein the width variance and the height variance are variances between each pixel of the at least one first depth map and each pixel of the at least one first image, and each pixel of the at least one first image corresponds to each pixel of the at least one first depth map.

[0014] The device, wherein the instructions, when executed by the processor, may cause the device to determine suitability determination information indicating whether a second depth map is acceptable as training data for training a depth map estimation network based on second variance estimation information, wherein the second variance estimation information indicates a variance between the second depth map and a second image.

[0015] The device, wherein the instructions, when executed by the processor, cause the device to: obtain the second variance estimation information; back-propagate fitness loss based on the fitness determination information and fitness GT information associated with the fitness determination information; and update parameters associated with determining the fitness determination information based on the back-propagation of the fitness loss.

[0016] The device, wherein the instructions, when executed by the processor, may cause the device to update parameters associated with determining the suitability determination information based on a state in which the parameters associated with determining the first variance estimation information are fixed.

[0017] The device, wherein the instructions, when executed by the processor, may cause the device to: apply a suitability determination operation to the second depth map; based on the suitability determination operation, generate first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map, respectively; and determine suitability determination information based on the first to n-th pieces of region suitability determination information.

[0018] The device, wherein the instructions, when executed by the processor, may cause the device to: apply a noise adding operation to at least one piece of cloud data; and generate the at least one first depth map based on the noise adding operation.

[0019] The device, wherein the instructions, when executed by the processor, can cause the device to: perform a filtering operation to remove data from the at least one cloud data, the removed data corresponding to the location of the surrounding environment information; and generate the at least one first depth map based on the filtering operation.

[0020] A device (eg, a learning device), wherein the instructions, when executed by the processor, may cause the device to generate another depth map having a higher accuracy than the at least one first depth map based on updated parameters.

[0021] According to the present disclosure, a device may include: a processor; and a memory storing instructions, which, when executed by the processor, may enable the device to: obtain a depth map for testing based on point cloud data associated with surrounding environment information for testing; and an image for testing associated with the depth map for testing; determine variance estimation information for testing, which indicates the variance between the depth map for testing and the image for testing; and determine suitability determination information for testing based on the variance estimation information for testing, the suitability determination information indicating whether the depth map for testing can be accepted for training a depth map estimation network.

[0022] The method may also include generating another depth map having higher accuracy than the depth map based on the trained depth map estimation network.

[0023] According to the present disclosure, a method may include: obtaining at least one first depth map based on at least one cloud data associated with surrounding environment information; and at least one first image associated with the at least one first depth map; determining first variance estimation information indicating the variance between the at least one first depth map and the at least one first image based on the at least one first depth map and the at least one first image; and back-propagating variance loss based on the first variance estimation information and variance reference true value (GT) information related to the first variance estimation information; updating parameters associated with determining the first variance estimation information based on back-propagating the variance loss; and generating another depth map having higher accuracy than the at least one first depth map based on the updated parameters.

[0024] The method, wherein determining the first variance estimation information may include: applying an encoding operation to at least one first depth map; obtaining a first feature map corresponding to the at least one first depth map based on the encoding operation; applying a second encoding operation to the at least one first image; obtaining a second feature map corresponding to the at least one first image based on the second encoding operation; generating an integrated feature map based on the first feature map and the second feature map; applying a decoding operation to the integrated feature map; and determining the first variance estimation information based on the decoding operation.

[0025] The method, wherein the first variance estimation information may include: width variance estimation information indicating width variance; and height variance estimation information indicating height variance, wherein the width variance and the height variance are variances between respective pixels of the at least one first depth map and respective pixels of the at least one first image, and respective pixels of the at least one first image correspond to respective pixels of the at least one first depth map.

[0026] The method may also include, after updating the parameters associated with determining the first variance estimation information: obtaining second variance estimation information indicating the variance between the second depth map and the second image; determining suitability determination information based on the second variance estimation information, which indicates whether the second depth map can be accepted as training data for training a depth map estimation network; back-propagating a fitness loss based on the suitability determination information and fitness GT information associated with the suitability determination information; and updating the parameters associated with the determination of the suitability determination information based on back-propagating the fitness loss.

[0027] The method, wherein parameters associated with determining the first variance estimate information are fixed.

[0028] The method, wherein determining the suitability determination information may include: applying a suitability determination operation to the second depth map; based on the suitability determination operation, generating first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map, respectively; and determining the suitability determination information based on the first to n-th pieces of region suitability determination information.

[0029] Before acquiring the at least one first depth map and the at least one first image, the method may further include: performing a noise adding operation on the at least one piece of cloud data; and generating the at least one first depth map based on the noise adding operation.

[0030] The method, wherein generating at least one first depth map may include: performing a filtering operation to remove data from at least one piece of cloud data, the removed data corresponding to a location of surrounding environment information; and generating the at least one first depth map based on the filtering operation.

[0031] The method may further include generating another depth map having higher accuracy than the at least one first depth map based on the updated parameters.

[0032] According to the present disclosure, a method may include: obtaining a depth map for testing based on point cloud data associated with surrounding environment information for testing; and an image for testing associated with the depth map for testing; determining variance estimation information for testing based on the depth map for testing and the image for testing, which indicates the variance between the depth map for testing and the image for testing; and determining suitability determination information for testing based on the variance estimation information for testing, the suitability determination information indicating whether the depth map for testing can be accepted for training a depth map estimation network.

[0033] The method may also include generating another depth map having higher accuracy than the depth map based on the trained depth map estimation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0035] Figure 1A and Figure 1B An embodiment showing consistency between a depth map and an image;

[0036] Figure 2 An embodiment of a configuration of a learning device according to an embodiment disclosed in the present disclosure is shown;

[0037] Figure 3 , Figure 4A , Figure 4B , Figure 4C , Figure 5 , Figure 6 and Figure 7 An embodiment of the operation of a learning device according to an embodiment disclosed in the present disclosure is shown;

[0038] Figure 8 An embodiment of a learning method according to an embodiment disclosed in the present disclosure is shown;

[0039] Fig. 9 An embodiment of a configuration of a test device according to an embodiment disclosed in the present disclosure is shown;

[0040] Fig.10 An embodiment illustrating the operation of a test device according to an embodiment disclosed in the present disclosure; and

[0041] Fig.11 An embodiment of a testing method according to an embodiment disclosed in the present disclosure is shown.

[0042] Regarding the description of the drawings, the same or similar reference numerals may be used for the same or similar components. DETAILED DESCRIPTION

[0043] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, when reference numerals are attached to the constituent elements of each of the accompanying drawings, when the same constituent elements are shown in other drawings, the constituent elements are also represented by the same reference numerals. In addition, when describing the embodiments of the present disclosure, detailed descriptions of well-known features or functions will be excluded to avoid unnecessarily obscuring the main purpose of the present disclosure.

[0044] When describing the components according to the embodiments of the present disclosure, terms such as first, second, "A", "B", (a), (b), etc. may be used. These terms are intended only to distinguish one component from another, and these terms do not limit the properties, order, or sequence of the corresponding components. In addition, unless otherwise defined, all terms including technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs. Those terms as defined in commonly used dictionaries should be interpreted as having meanings equal to the contextual meanings in the relevant art, and should not be interpreted as having ideal or overly formal meanings, unless explicitly defined in this application as having ideal or overly formal meanings.

[0045] In the following, reference will be made to Figures 2 to 11 Embodiments of the present disclosure are described in detail.

[0046] Figure 2 An example of a configuration of the learning device 1000 according to an embodiment disclosed in the present disclosure is shown. Figures 3 to 7 An example of the operation of the learning device according to the embodiment disclosed in the present disclosure is shown.

[0047] refer to Figure 2 , the learning device 1000 may include an acquisition device 1100 , a variance estimation network 1200 , a parameter updating device 1300 , and a suitability determination network 1400 .

[0048] In addition, the learning device 1000 may additionally include a training depth map generator (not shown) for generating a first depth map for training and / or a second depth map for training. For reference, the first depth map for training may be training data for training the variance estimation network 1200 to be described below, and the second depth map for training may be training data for training the fitness determination network 1400 to be described below.

[0049] To facilitate understanding, a process of generating a first depth map for training will be described first, and then other components of the learning device 1000 will be described.

[0050] refer to Figure 3 , the training depth map generator may apply a noise adding operation to at least some of the first point cloud data for training obtained in multiple environments, at multiple time points, in multiple spaces, etc., to generate a first depth map for training. At this time, the first point cloud data for training may be point cloud data having excellent consistency with the corresponding image.

[0051] For example, a transformation matrix (e.g., an extrinsic matrix) may be used to convert the first point cloud data used for training in a coordinate system corresponding to a light detection and ranging (LiDAR) sensor to a coordinate system corresponding to a camera. 1 、n 2 、n 3 Etc. are added to the transformation matrix to generate the first depth map for training, which is used to train the variance estimation network.

[0052] For reference, when Figure 3 When comparing a depth map generated by applying a transformation matrix without adding noise (i.e., a depth map without adding noise) with a first depth map for training, although the depth map and the first depth map for training are depth maps generated from the same point cloud data, it can be recognized that the positions on the depth maps for the same object are different from each other. In other words, the first depth map for training may be a depth map in which noise is intentionally added to move the position of the object for training the variance estimation network.

[0053] At this time, the first depth map for training may be generated based on specific first point cloud data for training included in the field of view (FOV) of the camera among the plurality of first point cloud data for training whose coordinates are converted into the camera coordinate system.

[0054] First variance ground truth (GT) information may be generated based on a depth map without noise added and a first depth map for training.

[0055] In other words, the first variance GT information may include information about the variance (or displacement) of each pixel on the first depth map used for training to move to the position of each corresponding pixel on the depth map without adding noise.

[0056] For example, the first variance GT information may be a graph having two channels. In this case, one of the two channels (e.g., the first channel) may include variance information corresponding to a width direction (or a horizontal direction; width), and the other channel (e.g., the second channel) may include variance information corresponding to a height direction (or a vertical direction; height).

[0057] For example, for Figure 3To move a vehicle located just to the left of the center of the first depth map used for training to the position of the same vehicle in the depth map without added noise, the vehicle should move some pixels (e.g., 10 pixels) in the left direction (or -width direction) and some pixels (e.g., 30 pixels) in the downward direction (or -height direction). Therefore, in the first channel map of the first variance GT information (e.g., the channel map corresponding to the width variance), the value corresponding to the pixel where the vehicle is located may be -10, and in the second channel map (e.g., the channel map corresponding to the height variance), the value corresponding to the pixel where the vehicle is located may be -30.

[0058] For reference, in Figure 3 , the first depth map for training with added noise is shown to help understanding, but noise does not have to be added to the first depth map for training generated according to the embodiment disclosed in the present disclosure. For example, noise may be added to 80 first depth maps out of 100 first depth maps for training, and noise may not be added to 20 first depth maps out of 100 first depth maps for training.

[0059] In addition, the training depth map generator may additionally apply a filtering operation for removing specific first point cloud data for training corresponding to a specific position of the first surrounding environment information from among the plurality of first point cloud data for training to generate a first depth map for training. For reference, the specific position of the first surrounding environment information may be a position that is not sensed by the camera due to a difference between a position where the camera is installed and a position where the LiDAR sensor is installed, while the specific position of the first surrounding environment information may be a position sensed by the LiDAR sensor.

[0060] Reference Figure 4A , an image generated by sensing first surrounding environment information about a specific place at a specific time by means of a camera can be recognized. In addition, referring to Figure 4B , it is possible to identify a depth map generated by sensing the first surrounding environment information by means of the LiDAR sensor. It is possible to identify that a second object located behind the first object and not shown at the position of the camera is also sensed by the LiDAR sensor. In this case, Figure 4C As shown, point cloud data corresponding to locations not shown at the camera position (ie, specific first point cloud data used for training) may be removed to allow the depth map to include information that is more similar to the image.

[0061] For example, a convex hull algorithm or statistical noise filtering may be used as a filtering technique for removing specific first point cloud data for training.

[0062] A description is given above of an embodiment in which a training depth map generator of a learning device generates a first depth map for training and first variance GT information corresponding to the first depth map for training. Hereinafter, a description will be given of components included in the learning device.

[0063] The acquisition device can obtain at least one first depth map for training and at least one first image for training corresponding to the at least one first depth map for training based on at least one first point cloud data for training for at least one piece of surrounding environment information.

[0064] For reference, each of the at least one first surrounding environment information may be different from each other in time and / or space information. In addition, the first depth map for training and the first image for training corresponding to the first depth map for training may be multiple pieces of data for the same surrounding environment information (i.e., at the same point in time or at the same place in the same time interval).

[0065] The variance estimation network may output first variance estimation information for the first depth map based on the first depth map for training and the first image for training to match the first image for training.

[0066] At this time, if Figure 5 As shown, the variance estimation network may apply a first encoding operation to a first depth map for training by means of a first encoder to obtain a feature map corresponding to the first depth map for training, may apply a second encoding operation to a first image for training by means of a second encoder to obtain a feature map corresponding to the first image for training, may use a feature map corresponding to the first depth map for training and a feature map corresponding to the first image for training to generate an integrated feature map, and may apply a decoding operation to the integrated feature map by means of a decoder to output first variance estimation information. At this time, the feature map corresponding to the first depth map for training and the feature map corresponding to the first image for training may be cascaded to generate an integrated feature map.

[0067] For reference, a network mainly used in scene estimation tasks, such as Resnet 50, can be used for the first encoder and the second encoder, but this is only helpful for understanding. The present disclosure is not limited to this. In addition, a decoder that performs an upsampling operation using a feature map can be used as the decoder.

[0068] At this time, the variance estimation network may output first width variance estimation information and first height variance estimation information of each pixel of the first depth map for training to match each pixel of the first image for training corresponding to each pixel of the first depth map for training as first variance estimation information. In other words, the first variance estimation information may be a graph with two channels. For reference, since the description of the first variance estimation information is the same as that of the reference above, Figure 3 The description of the first variance GT information described is similar, so its repeated description will be omitted.

[0069] The parameter updating device may back-propagate the variance loss based on the first variance estimation information and the first variance GT information corresponding to the first variance estimation information to update the parameters of the variance estimation network. For reference, the loss (eg, L1 loss) mainly used in scene estimation tasks may be used as the variance loss.

[0070] As a result, the variance estimation network can be trained to accurately output displacement information (ie, first variance estimation information) necessary for the input depth map to match the image corresponding to the depth map.

[0071] Meanwhile, the learning device may additionally include a suitability determination network 1400 .

[0072] For example, the suitability determination network may output suitability determination information corresponding to whether the second depth map for training is suitable as training data for training another network (e.g., a depth map estimation network) based on second variance estimation information of the second depth map for training that matches the second image for training. For reference, the second depth map for training may be the same as a portion of the first depth map for training, but is not limited thereto.

[0073] At this time, the suitability determination network may output suitability determination information for each region of the second depth map for training.

[0074] For example, the suitability determination network may apply a suitability determination operation to the second depth map used for training to generate first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map used for training, respectively, and may output suitability determination information based on the first to n-th pieces of region suitability determination information.

[0075] For example, the suitability determination network may apply a convolution operation and a classification operation to the second variance estimation information corresponding to the second depth map used for training.

[0076] Reference Figure 6, when the second variance estimation information having a size of 192×192×2 (which is an exemplary numerical value to help understanding) is obtained, the fitness determination network may apply a convolution operation and a classification operation to the second variance estimation information to output fitness determination information having a size of 3×3×1. At this time, the receptive field (or small block) of each pixel of the fitness determination information may be 64×64×2. This may be the result of performing fitness determination on a 64×64×2 region (or small block) of the second depth map for training, which corresponds to the value of a specific value of the fitness determination information in the fitness determination network 1400.

[0077] For reference, and to facilitate description, Figure 6 Shows that with Figure 5 However, the second variance estimation information disclosed in the present disclosure is not necessarily the same as the first variance estimation information.

[0078] For example, a value of 0.851 for a pixel located at the upper left end of the suitability determination information may mean that an area (or small block) with a size of 64×64×2 located at the upper left end of the second depth map used for training corresponding to the value 0.851 has a low variance (i.e., high consistency with the corresponding image).

[0079] On the other hand, a value of 0.220 for a pixel located on the left side of the middle row of the suitability determination information may mean that a region (or patch) having a size of 64×64×2 located on the right side of the second depth map used for training corresponding to a value of 0.851 has a high variance (i.e., low consistency with the corresponding image).

[0080] At the same time, reference Figure 7 Descriptively given Figure 6 The description of the scheme of suitability GT information is shown at the bottom. For reference, for ease of description, Figure 7 Shown with Figure 3 The first variance GT information has the same value as the second variance GT information. However, the second variance GT information disclosed in the present disclosure is not necessarily the same as the first variance GT information.

[0081] Reference Figure 7 , when the amplitude of the second variance GT information is 192×192×2, as shown in reference Figure 6 As described above, suitability GT information can be generated for each region (or patch).

[0082] For example, the result of the second variance addition in the upper left region (64×64×2) of the second variance GT information may be 30. In addition, when the threshold is set to 50, since the sum of the variance additions of the region 30 is less than the threshold 50, it can be determined that the corresponding region (or small block) of the second depth map used for training has high consistency with the image. A value of 1 (i.e., a GT value meaning high consistency) may be assigned to the region (or small block).

[0083] On the other hand, the result of adding the second variance in the right region (64×64×2) of the center of the second variance GT information may be 201. When the threshold is set to 50, since the sum of the variances in the region 201 is greater than the threshold 50, it may be determined that the region (or patch) of the second depth map used for training has low consistency with the image. A value of 0 (i.e., a GT value meaning low consistency) may be assigned to the region (or patch).

[0084] The parameter updating device may back-propagate the fitness loss based on the fitness determination information output from the fitness determination network and the fitness GT information corresponding to the fitness determination information to further update the parameters of the fitness determination network. For reference, a loss mainly used in a classification operation (e.g., binary cross entropy, etc.) may be used as the fitness loss.

[0085] For example, the parameter updating device can back-propagate the suitability loss based on the first to nth regional suitability determination information output from the suitability determination network and the first to nth regional suitability GT information corresponding to the first to nth regional suitability determination information, respectively, to further update the parameters of the suitability determination network.

[0086] At this time, the parameter updating device may additionally update the parameters of the suitability determination network while the parameters of the variance estimation network are frozen.

[0087] Figure 8 is a flowchart showing the operation of the learning device.

[0088] refer to Figure 8 In operation 810, at least one first depth map for training based on at least one first point cloud data for training for at least one first surrounding environment information and at least one first image for training corresponding to the at least one first depth map for training may be obtained.

[0089] In operation 820, first variance estimation information for the first depth map for training may be output based on the first depth map for training and the first image for training to match the first image for training.

[0090] In operation 830, a variance loss based on the first variance estimation information and first variance GT information corresponding to the first variance estimation information may be back-propagated to update a parameter of a variance estimation network that outputs the first variance estimation information.

[0091] In operation 840, second variance estimation information of the second depth map for training may be obtained to match the second image for training. At this time, the second variance estimation information may be estimation information generated based on the same depth map and image as the first depth map for training and the first image for training in operation 810. However, since the second variance estimation information is estimation information generated by the variance estimation network that has completed training, although the second variance estimation information is generated based on the same depth map and image, the second variance estimation information may be higher in accuracy than the first variance estimation information generated during training of the variance estimation network.

[0092] In operation 850, suitability determination information corresponding to whether the second depth map for training is suitable as training data for training the depth map estimation network may be output based on the second variance estimation information.

[0093] In operation 860 , a fitness loss based on the fitness determination information and fitness GT information corresponding to the fitness determination information may be back-propagated to update a parameter of a fitness determination network that outputs the fitness determination information.

[0094] Meanwhile, the configuration of the test apparatus 2000 will be described below. The test apparatus 2000 includes a Fig. 9 The network whose parameters are updated by the learning device as described above. For reference, descriptions that are the same as or similar to the descriptions of the learning device will be omitted.

[0095] Fig. 9 An example of a configuration of a test device 2000 according to an embodiment disclosed in the present disclosure is shown.

[0096] refer to Fig. 9 , the test device 2000 according to the embodiment disclosed in the present disclosure may include an acquisition device 2100 , a variance estimation network 2200 , and a fitness determination network 2300 .

[0097] For example, the acquisition device 2100 may obtain a depth map for testing based on the point cloud data for testing of the surrounding environment information for testing, and an image for testing corresponding to the depth map for testing. Figure 2 The acquisition device 1100 of the learning device 1000 shown in FIG.

[0098] The variance estimation network 2200 may output variance estimation information for the depth map for testing based on the depth map for testing and the image for testing to match the image for testing.

[0099] The suitability determination network 2300 may output suitability determination information for testing, which indicates whether the depth map for testing is suitable for training the depth map estimation network, based on the variance estimation information for testing.

[0100] Fig.10 is a diagram illustrating the operation of the test device.

[0101] Reference Fig.10 , when a depth map for testing and an image for testing are obtained, the variance estimation network 2200 may apply a first encoding operation to the depth map for testing by means of a first encoder to obtain a feature map corresponding to the depth map for testing, may apply a second encoding operation to the image for testing by means of a second encoder to obtain a feature map corresponding to the image for testing, may use the feature map corresponding to the depth map for testing and the feature map corresponding to the image for testing to generate an integrated feature map for testing, and may apply a decoding operation to the integrated feature map for testing by means of a decoder to output variance estimation information for testing. At this time, the feature map corresponding to the depth map for testing and the feature map corresponding to the image for testing may be cascaded to generate an integrated feature map.

[0102] The suitability determination network 2300 may output suitability determination information for testing based on the variance estimation information for testing, which indicates whether the depth map for testing is suitable for training another network (e.g., a depth map estimation network). Fig.10 As shown, suitability determination information for testing can be output for seven areas.

[0103] The test device may allow a filter device (not shown) to compare the suitability determination information for testing with a threshold value (e.g., 0.5) to classify the suitability (true or false) of each region. For example, since the suitability determination information 0.851 for testing in the first region is greater than the threshold value 0.5, the suitability corresponding to the first region may be classified as true. On the other hand, since the suitability determination information 0.321 for testing in the third region is less than the threshold value 0.5, the suitability corresponding to the third region may be classified as false.

[0104] The testing device may determine whether specific point cloud data for testing or a specific depth map for testing is suitable as training data for training another network based on the suitability of the classification for each region.

[0105] For example, when there are one or more regions classified as false among all regions, the test apparatus may determine that the point cloud data for testing or the depth map for testing is not suitable as training data.

[0106] For another example, if the ratio of regions classified as false to all regions is less than a threshold, the testing device may determine that the point cloud data for testing or the depth map for testing is suitable as training data.

[0107] Fig.11 is a flow chart illustrating the operation of the testing device.

[0108] Reference Fig.11 In operation 1110, a depth map for testing based on point cloud data of surrounding environment information for testing and an image for testing corresponding to the depth map for testing may be obtained.

[0109] In operation 1120 , variance estimation information for testing with respect to the depth map for testing may be output based on the depth map for testing and the image for testing to match the image for testing.

[0110] In operation 1130 , suitability determination information for testing may be output based on the variance estimation information for testing, the suitability determination information indicating whether the depth map for testing is suitable for training a depth map estimation network.

[0111] As a result, it is possible to accurately determine whether any of the depth maps used as training data for training another network (e.g., a depth map estimation network) has a high consistency with the image. Thus, the inspector can efficiently select data for constructing a training data set without even having to go through a time-consuming and costly process of inspecting the depth maps.

[0112] Additionally or alternatively, it should be understood that one or more of the following methods or embodiments thereof may be performed by one or more processors. The learning device 1000 may be implemented as a hardware device comprising a memory and a processor. The memory is configured to store program instructions, and the processor is specifically programmed to execute program instructions to perform one or more of the processes described in more detail below. The learning device 1000 may control the operation of a unit, module, component, device, etc., as described herein. It should be understood that, as recognized by those skilled in the art, the following methods may be performed by a device comprising the learning device 1000 in combination with one or more other components.

[0113] In addition or alternatively, the learning device 1000 of the present disclosure may be implemented as a non-volatile computer-readable recording medium including executable program instructions executed by a processor. Examples of computer-readable recording media include, but are not limited to, ROM, RAM, compact disk (CD) ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device.

[0114] The present disclosure has been made to solve the above-mentioned problems occurring in the prior art while keeping the advantages achieved by the prior art intact.

[0115] One embodiment of the present disclosure provides a learning device and a learning method for constructing training data for training a depth map estimation network, a testing device using the learning device, and a testing method using the testing device.

[0116] Another embodiment of the present disclosure provides a learning device for outputting variance estimation information about a variance necessary for matching a depth map to an image, a learning method thereof, a test apparatus using the same, and a test method using the same.

[0117] Another embodiment of the present disclosure provides a learning device for outputting suitability determination information corresponding to whether a depth map is suitable as training data for training a depth map estimation network, and a learning method thereof, a test device using the same, and a test method using the same.

[0118] Another embodiment of the present disclosure provides a learning device and a learning method thereof for reducing the time and cost consumed in constructing training data for training a depth map estimation network, a testing device using the learning device, and a testing method using the testing device.

[0119] Another embodiment of the present disclosure provides a learning device and a learning method thereof for efficiently selecting data to construct training data for training a depth map estimation network, a testing device using the learning device, and a testing method using the testing device.

[0120] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0121] According to an embodiment of the present disclosure, a learning device may include: an acquisition device, which obtains at least one first depth map for training based on at least one first point cloud data for training for at least one first surrounding environment information, and at least one first image for training, and the at least one first image corresponds to the at least one first depth map for training; a variance estimation network, which outputs first variance estimation information for the first depth map for training to match the first image for training based on the first depth map for training and the first image for training; and a parameter updating device, which back-propagates the variance loss based on the first variance estimation information and the first variance reference true value (GT) information corresponding to the first variance estimation information to update the parameters of the variance estimation network.

[0122] In one embodiment, the variance estimation network may apply a first encoding operation to a first depth map for training to obtain a feature map corresponding to the first depth map for training, may apply a second encoding operation to a first image for training to obtain a feature map corresponding to the first image for training, may use the feature map corresponding to the first depth map for training and the feature map corresponding to the first image for training to generate an integrated feature map, and may apply a decoding operation to the integrated feature map to output first variance estimation information.

[0123] In an embodiment, the variance estimation network may output first width variance estimation information and first height variance estimation information for each pixel of the first depth map used for training as first variance estimation information to match each pixel of the first image used for training, each pixel corresponding to each pixel of the first depth map used for training.

[0124] In an embodiment, the learning device may further include a suitability determination network that outputs suitability determination information corresponding to whether the second depth map for training is suitable as training data for training the depth map estimation network based on the second variance estimation information of the second depth map for training to match the second image for training.

[0125] In an embodiment, the acquisition device may also obtain second variance estimation information. The parameter updating device may back-propagate the fitness loss based on the fitness determination information and the fitness GT information corresponding to the fitness determination information to further update the parameters of the fitness determination network.

[0126] In an embodiment, the parameter updating device may additionally update the parameters of the suitability determination network while the parameters of the variance estimation network are fixed.

[0127] In an embodiment, the suitability determination network may apply a suitability determination operation to the second depth map for training to generate first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map for training, respectively, and may output suitability determination information based on the first to n-th pieces of region suitability determination information.

[0128] In an embodiment, the learning apparatus may further include a training depth map generator that applies a noise adding operation to some of the at least one first point cloud data for training to generate a first depth map for training.

[0129] In an embodiment, the training depth map generator may additionally apply a filtering operation to generate a first depth map for training, wherein the filtering operation is used to remove specific first point cloud data corresponding to a specific position of the first surrounding environment information in at least one first point cloud data for training.

[0130] According to another embodiment of the present disclosure, a testing device may include: an acquisition device, which obtains a depth map for testing based on point cloud data for testing with respect to surrounding environment information for testing, and an image for testing corresponding to the depth map for testing; a variance estimation network, which outputs variance estimation information for the depth map for testing based on the depth map for testing and the image for testing to match the image for testing; and a suitability determination network, which outputs suitability determination information for testing based on the variance estimation information for testing, wherein the suitability determination information indicates whether the depth map for testing is suitable for training a depth map estimation network.

[0131] According to another embodiment of the present disclosure, a learning method may include: obtaining at least one first depth map for training and at least one first image for training based on at least one first point cloud data for training for at least one first surrounding environment information, the at least one first image corresponding to the at least one first depth map for training; based on the first depth map for training and the first image for training, outputting first variance estimation information for the first depth map for training to match the first image for training; back-propagating variance loss based on the first variance estimation information and first variance reference true value (GT) information corresponding to the first variance estimation information to update parameters of a variance estimation network for outputting the first variance estimation information.

[0132] In an embodiment, the output of the first variance estimation information may include applying a first encoding operation to a first depth map for training to obtain a feature map corresponding to the first depth map for training, applying a second encoding operation to the first image for training to obtain a feature map corresponding to the first image for training, generating an integrated feature map using the feature map corresponding to the first depth map for training and the feature map corresponding to the first image for training, and applying a decoding operation to the integrated feature map to output the first variance estimation information.

[0133] In an embodiment, outputting the first variance estimation information may include outputting first width variance estimation information and first height variance estimation information for each pixel of the first depth map used for training as the first variance estimation information to match each pixel of the first image used for training, which corresponds to each pixel of the first depth map used for training.

[0134] In an embodiment, the learning method may further include: after updating the parameters of the variance estimation network, obtaining second variance estimation information for the second depth map used for training to match the second image used for training, outputting suitability determination information corresponding to whether the second depth map used for training is suitable as training data for training the depth map estimation network based on the second variance estimation information, and back-propagating suitability loss based on the suitability determination information and suitability GT information corresponding to the suitability determination information to update the parameters of the suitability determination network used to output the suitability determination information.

[0135] In an embodiment, the parameters of the variance estimation network may be frozen.

[0136] In one embodiment, outputting suitability determination information may include: applying a suitability determination operation to a second depth map used for training to generate first to n-th pieces of region suitability determination information corresponding to first to n-th regions of the second depth map used for training, respectively; and outputting suitability determination information based on the first to n-th pieces of region suitability determination information.

[0137] In an embodiment, the learning method may further include: before obtaining the first depth map for training and the first image for training, applying a noise adding operation to at least some of the at least one first point cloud data for training to generate the first depth map for training.

[0138] In an embodiment, generating a first depth map for training may include: additionally applying a filtering operation for removing specific first point cloud data for training to generate a first depth map for training, wherein the specific first point cloud data corresponds to a specific position of the first surrounding environment information in at least one first point cloud data for training.

[0139] According to another embodiment of the present disclosure, a testing method may include: obtaining a depth map for testing and an image for testing based on point cloud data for testing with respect to surrounding environment information for testing, the image for testing corresponding to the depth map for testing; based on the depth map for testing and the image for testing, outputting variance estimation information for the depth map for testing to match the image for testing; and outputting suitability determination information for testing based on the variance estimation information for testing, the suitability determination information indicating whether the depth map for testing is suitable for training a depth map estimation network.

[0140] The present technology may provide a learning device for constructing training data for training a depth map estimation network and a learning method thereof, a testing device using the learning device, and a testing method using the testing device.

[0141] The present technology can provide a learning device for outputting variance estimation information on a variance necessary for a depth map to match an image, a learning method thereof, a test apparatus using the learning device, and a test method using the test apparatus.

[0142] The present technology can provide a learning device and a learning method for outputting suitability determination information corresponding to whether a depth map is suitable as training data for training a depth map estimation network, a test device using the information, and a test method using the information.

[0143] The present technology can provide a learning device and a learning method thereof for reducing the time and cost consumed in constructing training data for training a depth map estimation network, a testing device using the learning device, and a testing method using the testing device.

[0144] The present technology can provide a learning device and a learning method for efficiently selecting data to construct training data for training a depth map estimation network, a test device using the learning device, and a test method using the test device.

[0145] Furthermore, various effects directly or indirectly determined by the present disclosure can be provided.

[0146] Although the present disclosure has been described above with reference to the embodiments and the accompanying drawings, the present disclosure is not limited thereto, but various modifications and changes may be made by those skilled in the art without departing from the spirit and scope of the present disclosure as claimed in the appended claims.

[0147] Therefore, the embodiments of the present disclosure are provided to explain the spirit and scope of the present disclosure, but not to limit them, so that the spirit and scope of the present disclosure are not limited by the embodiments. The scope of the present disclosure should be interpreted based on the attached claims, and all technical concepts within the scope equivalent to the claims should be included in the scope of the present disclosure.

Claims

1. A learning device, comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the learning device to: get: at least one first depth map based on at least one piece of cloud data associated with surrounding environment information; as well as at least one first image associated with the at least one first depth map; determining, based on the at least one first depth map and the at least one first image, first variance estimation information indicating a variance between the at least one first depth map and the at least one first image; Backpropagating the variance loss based on: the first variance estimation information; as well as first variance reference true value information associated with the first variance estimation information; as well as Parameters associated with determining the first variance estimate information are updated based on the back-propagated variance loss.

2. The learning device according to claim 1, wherein: The instructions, when executed by the processor, cause the learning device to: applying a first encoding operation to the at least one first depth map; Obtaining a first feature map corresponding to the at least one first depth map based on the first encoding operation; applying a second encoding operation to the at least one first image; obtaining a second feature map corresponding to the at least one first image based on the second encoding operation; Generates an integrated feature map based on: the first feature map; as well as the second characteristic map; Applying a decoding operation to the integrated feature map; as well as The first variance estimate information is determined based on the decoding operation.

3. The learning device according to claim 1, wherein: The first variance estimation information includes: width variance estimation information indicating a width variance; and Height variance estimation information indicating height variance, The width variance and the height variance are variances between each pixel of the at least one first depth map and each pixel of the at least one first image, and each pixel of the at least one first image corresponds to each pixel of the at least one first depth map.

4. The learning device according to claim 1, wherein: When executed by the processor, the instructions cause the learning device to: determine suitability determination information based on second variance estimation information, the suitability determination information indicating whether the second depth map can be accepted as training data for training a depth map estimation network, wherein the second variance estimation information indicates a variance between the second depth map and a second image.

5. The learning device according to claim 4, wherein: The instructions, when executed by the processor, cause the learning device to: obtaining the second variance estimation information; Backpropagate the fitness loss based on: the suitability determination information; as well as suitability benchmark truth value information associated with the suitability determination information; as well as Based on the back propagation of the fitness loss, parameters associated with determining the fitness determination information are updated.

6. The learning device according to claim 4, wherein: The instructions, when executed by the processor, cause the learning device to update a parameter associated with determining the suitability determination information based on a state in which the parameter associated with determining the first variance estimation information is fixed.

7. The learning device according to claim 4, wherein: The instructions, when executed by the processor, cause the learning device to: applying a suitability determination operation to the second depth map; Based on the suitability determination operation, generating first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map, respectively; as well as The suitability determination information is determined based on the first to n-th pieces of area suitability determination information.

8. The learning device according to claim 1, wherein: The instructions, when executed by the processor, cause the learning device to: applying a noise adding operation to the at least one piece of cloud data; as well as The at least one first depth map is generated based on the noise adding operation.

9. The learning device according to claim 8, wherein: The instructions, when executed by the processor, cause the learning device to: performing a filtering operation to remove data corresponding to the location of the surrounding environment information from the at least one piece of cloud data; as well as The at least one first depth map is generated based on the filtering operation.

10. A learning device comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the learning device to: get: a depth map for testing based on point cloud data associated with surrounding environment information for testing; and an image for testing associated with the depth map for testing; Determining variance estimation information for testing, the variance estimation information for testing indicating a variance between the depth map for testing and the image for testing; as well as Suitability determination information for testing is determined based on the variance estimation information for testing, the suitability determination information for testing indicating whether the depth map for testing can be accepted for training of a depth map estimation network.

11. A learning method comprising: get: at least one first depth map based on at least one piece of cloud data associated with surrounding environment information; as well as at least one first image associated with the at least one first depth map; determining, based on the at least one first depth map and the at least one first image, first variance estimation information indicating a variance between the at least one first depth map and the at least one first image; Backpropagating the variance loss based on: the first variance estimation information; as well as variance reference true value information associated with the first variance estimation information; as well as Based on back-propagating the variance loss, parameters associated with determining the first variance estimate information are updated.

12. The learning method according to claim 11, wherein: The determining of the first variance estimation information comprises: applying a first encoding operation to the at least one first depth map; Obtaining a first feature map corresponding to the at least one first depth map based on the first encoding operation; applying a second encoding operation to the at least one first image; obtaining a second feature map corresponding to the at least one first image based on the second encoding operation; Generates an integrated feature map based on: the first feature map; and the second characteristic map; applying a decoding operation to the integrated feature map; and The first variance estimate information is determined based on the decoding operation.

13. The learning method according to claim 11, wherein: The first variance estimation information includes: width variance estimation information indicating a width variance; and Height variance estimation information indicating height variance, The width variance and the height variance are variances between each pixel of the at least one first depth map and each pixel of the at least one first image, and each pixel of the at least one first image corresponds to each pixel of the at least one first depth map.

14. The learning method according to claim 11, after updating the parameter associated with determining the first variance estimation information, further comprising: obtaining second variance estimation information indicating a variance between the second depth map and the second image; determining suitability determination information based on the second variance estimation information, the suitability determination information indicating whether the second depth map can be accepted as training data for training a depth map estimation network; Backpropagate the fitness loss based on: the suitability determination information; as well as suitability benchmark truth value information associated with the suitability determination information; as well as Based on back-propagating the fitness loss, parameters associated with determining the fitness determination information are updated.

15. The learning method according to claim 14, wherein: Parameters associated with determining the first variance estimate information are fixed.

16. The learning method according to claim 14, wherein: Determining the suitability determination information includes: applying a suitability determination operation to the second depth map; generating, based on the suitability determination operation, first to n-th pieces of region suitability determination information corresponding to the first to n-th regions of the second depth map, respectively; and The suitability determination information is determined based on the first to n-th pieces of area suitability determination information.

17. The learning method according to claim 11, before acquiring the at least one first depth map and the at least one first image, further comprising: performing a noise adding operation on the at least one piece of cloud data; as well as The at least one first depth map is generated based on the noise adding operation.

18. The learning method according to claim 17, wherein: Generating the at least one first depth map comprises: performing a filtering operation to remove data corresponding to the location of the surrounding environment information from the at least one piece of cloud data; and The at least one first depth map is generated based on the filtering operation.

19. A testing method comprising: get: a depth map for testing based on point cloud data associated with surrounding environment information for testing; as well as an image for testing associated with the depth map for testing; Determine variance estimation information for testing based on the depth map for testing and the image for testing, wherein the variance estimation information for testing indicates a variance between the depth map for testing and the image for testing; as well as Based on the variance estimation information for testing, suitability determination information for testing is determined, the suitability determination information for testing indicating whether the depth map for testing can be accepted for training of a depth map estimation network.

Images