Learning apparatus, learning method, test apparatus, and test method

Abstract

The invention provides a learning apparatus, a learning method of the learning apparatus, a test apparatus using the learning method, and a test method using the test apparatus. The learning device may acquire a target image and a source image, generate an estimated depth map based on the target image via a first network, generate pose change information corresponding to a pose change between the target image and the source image, generate a composite image corresponding to the target image, determine a first loss based on the composite image and the target image, and generate a second loss based on the first loss. A second loss is determined, and the first loss and the second loss are back propagated and parameters of the first network and parameters of the second network are updated.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0125735 filed in the Korean Intellectual Property Office on September 20, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a learning device, a learning method for a learning device, a testing device using the learning method, and a testing method using the testing device. More specifically, it relates to a learning device based on self-supervised learning, a learning method for a learning device, a testing device using the learning method, and a testing method using the testing device. Background Art

[0004] Recently, with the development of computer vision technology based on deep neural networks in autonomous driving technology, various artificial intelligence models such as object detection, semantic segmentation, depth map estimation, and lane detection have been studied.

[0005] For example, depth map estimation is used in various ways to use a camera to recognize the surrounding environment and the space of surrounding objects, free space, etc. under autonomous driving conditions. Generally, a large amount of labeled training data (e.g., depth maps for training) is required to train a network for generating depth maps.

[0006] However, it is basically necessary to perform a process of assigning labels to each of a large number of images and checking them in order to construct data for training. The stage of securing training data for training the deep image generation network consumes a lot of time and resources. Summary of the invention

[0007] The present disclosure aims to solve the above-mentioned problems occurring in some embodiments while maintaining the advantages achieved by these embodiments unchanged.

[0008] One aspect of the present disclosure provides a learning device based on self-supervised learning, a learning method of the learning device, a testing device using the learning method, and a testing method using the testing device.

[0009] Another aspect of the present disclosure provides a learning device for reducing the time and cost consumed to ensure training data for training a depth map generation network, a learning method of the learning device, a testing device using the learning method, and a testing method using the testing device.

[0010] Another aspect of the present disclosure provides a learning device for allowing a depth map generation network to accurately estimate depth information based on an image sequence, a learning method of the learning device, a testing device using the learning method, and a testing method using the testing device.

[0011] Another aspect of the present disclosure provides a learning device for accurately estimating depth information of a dynamic object or an occluded object in an image, a learning method of the learning device, a testing device using the learning method, and a testing method using the testing device.

[0012] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and those skilled in the art to which the present disclosure belongs will clearly understand any other technical problems not mentioned herein through the following description.

[0013] According to one or more example embodiments, a learning device may include: one or more processors and a memory. The memory may store instructions, and when executed by the one or more processors, the instructions may configure the learning device to: acquire a target image and a source image; generate an estimated depth map based on the target image via a first network; generate pose change information corresponding to a pose change between the target image and the source image via a second network; generate a synthetic image corresponding to the target image by using the estimated depth map, the pose change information, and the source image; determine a first loss based on the synthetic image and the target image; determine a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map; and back-propagate the first loss and the second loss, and update the parameters of the first network and the parameters of the second network.

[0014] When executed by one or more processors, the instructions may configure the learning device to determine the second loss by: determining the second loss via a loss calculation device, further based on at least one of brightness information, contrast information, or structure information of each of the pseudo depth map and the estimated depth map.

[0015] The first weight corresponding to the brightness information may be smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

[0016] The target image may be generated by the image sensor at a first time. The source image may be generated by the image sensor at a second time within a threshold range around the first time.

[0017] The second network may include a posture change information generation network. When executed by one or more processors, the instructions may configure the learning device to generate posture change information by: generating posture change information based on a first posture at a first time and a second posture at a second time via the posture change information generation network.

[0018] When executed by one or more processors, the instructions may configure the learning device to generate a synthetic image by: acquiring first three-dimensional (3D) point cloud information at a first time based on an estimated depth map; converting the first 3D point cloud information at the first time into second 3D point cloud information at a second time based on posture change information; converting the second 3D point cloud information into two-dimensional (2D) image coordinates based on image sensor parameters corresponding to the image sensor; and generating a synthetic image based on pixel values ​​of the source image corresponding to the 2D image coordinates.

[0019] The first network may include an estimated depth map generation network. When executed by one or more processors, the instructions may further configure the learning device to: generate a pseudo depth map based on the target image via the pseudo depth map generation network.

[0020] The pseudo depth map generation network may include parameters in a frozen state.

[0021] According to one or more example embodiments, a system may include a testing device and a learning device. The testing device may include: an acquisition device configured to acquire a target image for testing; and a first network configured to generate an estimated depth map for testing based on the target image. The learning device may be configured to: acquire a target image and a source image; generate posture change information corresponding to the posture change between the target image and the source image via a second network; generate a synthetic image corresponding to the target image by using the estimated depth map, the posture change information and the source image; determine a first loss based on the synthetic image and the target image; determine a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map; and back-propagate the first loss and the second loss, and update the parameters of the first network and the parameters of the second network. The testing device may be configured to perform testing based on the updated parameters.

[0022] The test device may be configured to determine the second loss by: determining the second loss further based on at least one of brightness information, contrast information, or structure information of each of the pseudo depth map and the estimated depth map.

[0023] The first weight corresponding to the brightness information may be smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

[0024] The target image may be generated by the image sensor at a first time. The source image may be generated by the image sensor at a second time within a threshold range around the first time.

[0025] The second network may include a posture change information generation network. The test device may be configured to generate the posture change information by: generating the posture change information based on the first posture at the first time and the second posture at the second time via the posture change information generation network.

[0026] According to one or more example embodiments, a learning method may include: acquiring a target image and a source image through one or more processors; generating an estimated depth map based on the target image through one or more processors; generating posture change information corresponding to the posture change between the target image and the source image through one or more processors; generating a synthetic image corresponding to the target image through one or more processors by using the estimated depth map, the posture change information and the source image; determining a first loss based on the synthetic image and the target image through one or more processors; determining a second loss based on a pseudo depth map corresponding to the target image and the estimated depth map through one or more processors; and back-propagating the first loss and the second loss through one or more processors, and updating parameters of a first network for generating an estimated depth map and parameters of a second network for generating posture change information.

[0027] Determining the second loss may include determining the second loss further based on at least one of brightness information, contrast information, or structure information of each of the pseudo depth map and the estimated depth map.

[0028] The first weight corresponding to the brightness information may be smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

[0029] The target image may be generated by the image sensor at a first time. The source image may be generated by the image sensor at a second time within a threshold range around the first time.

[0030] Generating the posture change information may include generating the posture change information based on a first posture at a first time and a second posture at a second time.

[0031] Generating a synthetic image may include: acquiring first three-dimensional (3D) point cloud information at a first time; converting the first 3D point cloud information at the first time into second 3D point cloud information at a second time based on posture change information; converting the second 3D point cloud information into two-dimensional (2D) image coordinates based on image sensor parameters corresponding to the image sensor; and generating a synthetic image based on pixel values ​​of a source image corresponding to the 2D image coordinates.

[0032] The learning method may further include: before determining the second loss, generating a pseudo depth map based on the target image by a pre-trained pseudo depth map generation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings:

[0034] Figure 1 is a block diagram showing a configuration of a learning device according to an embodiment disclosed in the present disclosure;

[0035] Figure 2 , Figure 3 , Figure 4A and Figure 4B is a diagram for describing the operation of a learning device according to an embodiment disclosed in the present disclosure;

[0036] Figure 5 is a flowchart for describing a learning method according to an embodiment disclosed in the present disclosure;

[0037] Figure 6 is a block diagram showing a configuration of a test device according to an embodiment disclosed in the present disclosure;

[0038] Figure 7 is a diagram for describing the operation of the test device according to the embodiment disclosed in the present disclosure; and

[0039] Figure 8 is a flowchart for describing a testing method according to an embodiment disclosed in the present disclosure.

[0040] Regarding the description of the drawings, the same or similar reference numerals may be used for the same or similar components. DETAILED DESCRIPTION

[0041] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. When adding reference numerals to the components of each figure, it should be noted that the same reference numerals should be used even if the same components are marked on different figures. In addition, when describing the embodiments of the present disclosure, detailed descriptions of well-known features or functions will be omitted to avoid unnecessary confusion of the main points of the present disclosure.

[0042] When describing components according to embodiments of the present disclosure, terms such as first, second, "A", "B", (a), (b), etc. may be used. These terms are intended only to distinguish one component from another, and these terms do not limit the properties, order, or sequence of the corresponding components. In addition, all terms used herein, including technical and scientific terms, have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs, unless they are defined differently. These terms defined in general dictionaries should be interpreted as having meanings that match the background meanings in the relevant technical field, and should not be interpreted as having idealized or overly formalized meanings unless they are clearly defined in this application.

[0043] In the following, reference will be made to Figures 1 to 8 Embodiments of the present disclosure are described in detail.

[0044] Figure 1 is a block diagram showing the configuration of a learning device 1000 according to an embodiment disclosed in the present disclosure. Figures 2 to 4B is a diagram for describing the operation of the learning device according to the embodiment disclosed in the present disclosure.

[0045] Reference Figure 1 , the learning device 1000 may include a learning device 1100 , an estimated depth map generation network 1200 , a posture change information generation network 1300 , a synthetic image generator 1400 , a loss calculation device 1500 and a parameter updating device 1600 .

[0046] In addition, the learning device 1000 may further include a pseudo depth map generation network 1700. However, although described below, the pseudo depth map generation network 1700 is not an essential component of the learning device 1000 according to the embodiment disclosed in the present disclosure.

[0047] The learning device 1000 may include a memory for storing program instructions and a processor configured to execute the program instructions. The above-mentioned learning device 1100, estimated depth map generation network 1200, posture change information generation network 1300, synthetic image generator 1400, loss calculation device 1500, parameter updating device 1600 and pseudo depth map generation network 1700 can perform related functions through the processor included in the learning device 1000.

[0048] In the following, reference will be made to Figure 2 The operation of the learning device 1000 according to the embodiment disclosed in the present disclosure is described in detail.

[0049] First, the learning device 1100 may acquire a target image and at least one source image. The learning device may be a computing device.

[0050] For example, the target image may be generated by a specific image sensor at a specific time point, and the source image may be generated by a specific image sensor at a nearby time point corresponding to the specific time point. In this case, the nearby time point may be at least one time point among the time points within the threshold time interval for the specific time point, except for the specific time point.

[0051] For example, the target image and the source image may be a sequence of images acquired within a time interval including a specific time point.

[0052] In addition, the estimated depth map generation network 1200 can generate an estimated depth map based on the target image.

[0053] For example, the estimated depth map generation network 1200 may be a network of a U-net (Palm Broadband) structure including an encoder and a decoder. In this case, the encoder may be a ResNet (Residual Network) model, and the decoder may be a model for performing an operation of converting a sigmoid (Σ) output into a depth map.

[0054] Furthermore, the pose change information generation network 1300 may generate pose change information corresponding to a pose change between a target image and a source image.

[0055] For example, the posture change information generation network 1300 may generate posture change information based on a first posture of a specific image sensor at a specific time point and a second posture at a nearby time point.

[0056] For example, the posture change information may be information indicating a relationship between a camera posture at a specific time point and a camera posture at a nearby time point, which may be information corresponding to a rotation and translation matrix.

[0057] In addition, the synthetic image generator 1400 may generate a synthetic image corresponding to a target image using the estimated depth map, the pose change information, and the source image.

[0058] For example, the synthetic image generator 1400 may obtain three-dimensional (3D) point cloud information at a specific time point based on the estimated depth map.

[0059] The synthetic image generator 1400 may convert 3D point cloud information at a specific time point into 3D point cloud information at a nearby time point based on the posture change information.

[0060] The synthetic image generator 1400 may convert 3D point cloud information of nearby time points into two-dimensional (2D) image coordinates based on image sensor parameters (eg, intrinsic parameters) corresponding to a specific image sensor.

[0061] The synthetic image generator 1400 may generate a synthetic image based on pixel values ​​of a source image corresponding to the 2D image coordinates.

[0062] For example, the synthetic image generator 1400 may convert the 3D point cloud information of a specific time point (e.g., the first time) corresponding to (10, 20) of the estimated depth map into 3D point cloud information of a nearby time point based on the posture change information. The synthetic image generator 1400 may convert the 3D point cloud information of the nearby time point corresponding to (10, 20) of the estimated depth map into 2D image coordinates (e.g., (15, 18)) based on the image sensor parameters.

[0063] The composite image generator 1400 may assign pixel values ​​(e.g., 218) of a source image corresponding to a 2D image coordinate (15, 18) at a nearby time point (e.g., a second time within a threshold range near the first time) to (10, 20) to generate a composite image.

[0064] When the pose change information is accurate, the pixel value (e.g., 250) of the source image assigned to each pixel (e.g., (30, 50)) of the synthesized image may be the same or similar to the pixel value (e.g., 251) assigned to the pixel at the same position as the target image (e.g., (30, 50)). When the pose change information is inaccurate, the pixel value (e.g., 250) of the source image assigned to each pixel (e.g., (30, 50)) of the synthesized image may be significantly different from the pixel value (e.g., 89) assigned to the pixel at the same position as the target image (e.g., (30, 50)).

[0065] In addition, the loss calculation device 1500 may calculate a first loss based on the synthesized image and the target image, and may calculate a second loss based on the pseudo depth map and the estimated depth map corresponding to the target image. The loss calculation device may be a calculation device.

[0066] For reference, the estimated depth map generation network 1200 trained using only the first loss based on the synthetic image and the target image can generate an estimated depth map in which the object boundary is represented in a blurred manner for dynamic objects or partially occluded objects.

[0067] Therefore, in order to generate an estimated depth map in which the object boundaries of dynamic objects or partially occluded objects are clearly represented, in addition to the first loss, a second loss based on the error between the pseudo depth map and the estimated depth map can be additionally used to train the estimated depth map generation network 1200.

[0068] At this time, the pseudo depth map may be acquired from the pseudo depth map generation network 1700. The learning device 1000 according to the embodiment disclosed in the present disclosure may acquire, but is not limited to, the pseudo depth map generated by the pseudo depth map generation network 1700. For example, the learning device 1000 may further include the pseudo depth map generation network 1700.

[0069] At this time, the pseudo depth map generation network 1700 may generate a pseudo depth map based on the target image. For reference, the pseudo depth map generation network 1700 may have parameters in a frozen state.

[0070] For example, the pseudo depth map generation network 1700 may be a high-capacity network that is pre-trained using a large amount of training data sets, which may include a greater number of layers than the estimated depth map generation network 1200 according to the embodiment disclosed in the present disclosure, and may include a greater number of parameters than the estimated depth map generation network 1200 according to the embodiment disclosed in the present disclosure.

[0071] For reference, since the pseudo depth map generation network 1700 is a high-capacity network, it is difficult to load such a high-capacity network into a resource-limited device (e.g., a vehicle). On the other hand, the accuracy of the depth map output from the pseudo depth map generation network 1700 can be higher than the accuracy of the depth map output from the estimated depth map generation network 1200 according to the embodiment disclosed in the present disclosure.

[0072] Therefore, only the advantages of this high-capacity network (i.e., high accuracy) can be used to improve the accuracy of the network actually loaded (i.e., the estimated depth map generation network 1200) and reduce the cost required to construct training data. In other words, the output of the pseudo depth map generation network 1700 can be used as the correct answer data for training the estimated depth map generation network 1200.

[0073] For reference, the accuracy of the pseudo depth map of the pseudo depth map generation network 1700 may be relatively higher than the accuracy of the estimated depth map of the estimated depth map generation network 1200, but may be lower than the accuracy of the ground truth (GT) depth map generated by a process in which an inspector manually performs inspection alone.

[0074] Therefore, when the estimated depth map generation network 1200 is trained by simply using only a loss generated based on a regression-based loss function corresponding to a pseudo depth map and an estimated depth map, the accuracy of the estimated depth map generation network 1200 may be low.

[0075] However, if you can Figure 3 The pseudo depth map (i.e., located at Figure 3 As can be seen in the second row of images in , the pseudo depth map generated from a pre-trained high-capacity network can include information that is very similar to the original image in terms of shape or structure.

[0076] Therefore, such advantages can be used to train the estimated depth map generation network 1200. In other words, the estimated depth map generation network 1200 can be trained using the second loss based on the pseudo depth map, which is relatively lower than the GT depth map in terms of accuracy, but is as accurate as the GT depth map in terms of information about the shape or structure of the original image.

[0077] In other words, unsupervised learning (or self-supervised learning) can be performed using a first loss based on the image sequence, and supervised learning can be performed using a second loss based on the pseudo GT map (or pseudo depth map) from the pseudo depth map generation network 1700 at the same time, thereby significantly improving the performance of the estimated depth map generation network 1200.

[0078] For example, the loss calculation device 1500 may calculate a first loss based on the synthesized image and the target image.

[0079] For example, the loss calculation device 1500 may calculate the first loss with reference to at least some of the brightness information, contrast information, and structure information of each of the synthesized image and the target image.

[0080] For reference, the equation for the first loss can be as follows:

[0081] [Equation 1]

[0082]

[0083] At this time, pe(I a , I b ) may be a first loss (e.g., photometric loss), I a It can be the target image, I b It can be a synthetic image, SSIM (I a , I b ) can be I a and I b The similarity between (the similarity between at least some of the brightness information, contrast information and structure information), ||I a -I b || 1 It can be for I a and I b The regression loss (loss L1) of the pixel value of each pixel between , and a can be a weighted and ratio adjustment parameter of the structural similarity index map (SSIM) and the loss L1. For reference, the first loss of the above equation 1 is only an example to help understanding, and the first loss according to the embodiment disclosed in the present disclosure is not limited to the above equation 1. For example, the first loss may not include ||I a -I b || 1 and a.

[0084] For reference, the equation for SSIM is as follows:

[0085] [Equation 2]

[0086] SSIM(I a , I b )=[l(I a , Ib )] α ·[c(I a , I b )] β ·[s(I a , I b )] γ

[0087] At this time, α, β, and γ may have predetermined values. For example, the values ​​of α, β, and γ may be "1".

[0088] The equations for each of l (brightness), c (contrast), and s (structure) are as follows:

[0089] [Equation 3]

[0090]

[0091] At this time, c 1 、c 2 and c 3 can be a predetermined constant, μ can be the mean value, and σ can be the standard deviation.

[0092] In addition, the loss calculation device 1500 can calculate the second loss based on the pseudo depth map and the estimated depth map corresponding to the target image.

[0093] For example, the loss calculation device 1500 may calculate the second loss with reference to at least some of the brightness information, contrast information, and structure information of each of the pseudo depth map and the estimated depth map.

[0094] For reference, the equation for the second loss can be as follows.

[0095] [Equation 4]se(I a , I b )=(1-SSIM(I a , I b ))

[0096] At this time, se(I a , I b ) may be a second loss (e.g., similarity error loss), I a It can be a pseudo depth map, I b It can be an estimated depth map, and SSIM(I a , I b ) can be I a and I b For reference, since the equation of SSIM has been described with reference to Equation 2 above, a repeated description thereof will be omitted.

[0097] At this time, the first weight α corresponding to the brightness information may be smaller than the second weight β corresponding to the contrast information and the third weight γ corresponding to the structure information.

[0098] For example, the value of α may be “0”, and the values ​​of β and γ may be “1.” However, this is an example for helping understanding, and the present disclosure is not limited to this example.

[0099] For example, the first weight α and the second weight β may be smaller than the third weight γ.

[0100] In addition, the parameter updating device 1600 may back-propagate the first loss and the second loss, and may update the parameters of the estimated depth map generation network 1200 and the parameters of the pose change information generation network 1300 .

[0101] Figure 5 is a flowchart showing the operation of the learning device.

[0102] Reference Figure 5 , in operation 510, the learning apparatus 1000 may acquire a target image and at least one source image.

[0103] In operation 520-1, the learning device 1000 may generate an estimated depth map based on the target image. In operation 520-2, the learning device 1000 may generate pose change information corresponding to a pose change between the target image and the source image.

[0104] In operation 530 , the learning device 1000 may generate a synthesized image corresponding to the target image using the estimated depth map, the pose change information, and the source image.

[0105] In operation 540-1, the learning apparatus 1000 may calculate a first loss based on the synthesized image and the target image. In operation 540-2, the learning apparatus 1000 may calculate a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map.

[0106] Furthermore, in operation 550 , the learning apparatus 1000 may back-propagate the first loss and the second loss, and may update the parameters of the estimated depth map generation network 1200 and the parameters of the pose change information generation network 1300 .

[0107] On the other hand, reference will be made to Figures 6 to 8 A test apparatus 2000 including an estimated depth map generation network 2200 whose parameters are updated using the above-described learning apparatus 1000 is described. For reference, repeated descriptions identical or similar to those of the learning apparatus 1000 will be omitted.

[0108] Figure 6is a block diagram showing a configuration of a test apparatus 2000 according to an embodiment disclosed in the present disclosure. Figure 7 is a diagram for describing the operation of the test device according to the embodiment disclosed in the present disclosure.

[0109] Reference Figure 6 , a test device 2000 according to an embodiment disclosed in the present disclosure may include a learning device 2100 and an estimated depth map generation network 2200 .

[0110] For reference, since the pose change information generation network 1300, the synthetic image generator 1400, the loss calculation device 1500, the parameter updating device 1600, and the pseudo depth map generation network 1700 correspond to components required in the training operation for improving the accuracy of the estimated depth map generation network 2200, when compared with Figure 1 When compared with the components of the learning device 1000 in , it can be determined that the test device 2000 does not include the posture change information generation network 1300, the synthetic image generator 1400, the loss calculation device 1500, the parameter updating device 1600 and the pseudo depth map generation network 1700.

[0111] In the following, reference will be made to Figure 7 The operation of the testing device 2000 according to the embodiment disclosed in the present disclosure is described in detail.

[0112] First, the learning device 2100 may obtain a target image for testing. As described above, since the estimated depth map generation network 2200 is in a training completed state, the inference of the estimated depth map generation network 2200 may not require additional source images required for training the estimated depth map generation network 2200.

[0113] The estimated depth map generation network 2200 may generate an estimated depth map for testing based on a target image for testing.

[0114] Figure 8 is a diagram illustrating the operation of the test apparatus.

[0115] Reference Figure 8 , in operation 810 , the testing apparatus 2000 may acquire a target image for testing.

[0116] In operation 820 , the test apparatus 2000 may generate an estimated depth map for testing based on a target image for testing.

[0117] On the other hand, (i) the performance of the depth map estimation network trained using the first loss and the second loss according to the embodiment disclosed in the present disclosure is compared with (ii) the performance of the network trained using only the first loss. At this time, the image acquired by the forward-looking camera and the laser radar (LiDAR) data corresponding to the image are used for the performance comparison between the two networks.

[0118] It can be determined that as a result of identifying the performance according to the score a1 using the same data, the accuracy of the network trained using only the first loss is shown as 72.34%, while the accuracy of the network trained using the first loss and the second loss is shown as 87.93%. It can be determined that the accuracy of the estimated depth map generation network trained using the first loss and the second loss according to the embodiment disclosed in the present disclosure is much better.

[0119] You can refer to Figure 4A and Figure 4B For reference, the left column is an input image, the middle column is an output depth map generated for the input image by a network trained using only the first loss, and the right column is an output depth map generated for the input image by an estimated depth map generation network trained using the first loss and the second loss according to an embodiment disclosed in the present disclosure.

[0120] Reference Figure 4A and Figure 4B It can be determined that the depth map generated by the estimated depth map generation network trained using the first loss and the second loss according to the embodiment disclosed in the present disclosure is much better in accuracy than the depth map generated by the network trained using only the first loss.

[0121] In detail, it can be determined that the estimated depth map generation network according to the embodiment disclosed in the present disclosure is much greater in information about the shape and / or structure of the object as well as depth information than a depth map generated by a network trained using only the first loss.

[0122] In particular, it can be determined that even in Figure 4A When such an image includes a dynamic object, the shape of the object will not be crushed on the estimated depth map.

[0123] Therefore, a network for generating estimated depth maps with high accuracy can be trained using only image sequences acquired from a monocular camera.

[0124] Furthermore, the estimated depth map generation network can be trained by a second loss to generate an estimated depth map in which object boundaries are clearly represented for dynamic objects or partially occluded objects.

[0125] The present technology can provide a learning device based on self-supervised learning, a learning method for the learning device, a testing device using the learning method, and a testing method using the testing device.

[0126] The present technology can provide a learning device for reducing the time and cost consumed for securing training data for training a depth map generation network, a learning method for the learning device, a testing device using the learning method, and a testing method using the testing device.

[0127] The present technology can provide a learning device for allowing a depth map generation network to accurately estimate depth information based on an image sequence, a learning method for the learning device, a testing device using the learning method, and a testing method using the testing device.

[0128] The present technology can provide a learning device for accurately estimating depth information of a dynamic object or an occluded object in an image, a learning method for the learning device, a testing device using the learning method, and a testing method using the testing device.

[0129] Furthermore, various effects directly or indirectly determined by the present disclosure can be provided.

[0130] Hereinabove, although the present disclosure has been described with reference to the exemplary embodiments and the accompanying drawings, the present disclosure is not limited thereto but various modifications and changes may be made by those skilled in the art to which the present disclosure belongs without departing from the spirit and scope of the present disclosure claimed in the claims.

[0131] Therefore, the exemplary embodiments of the present disclosure are provided to explain the spirit and scope of the present disclosure, rather than to limit it, so that the spirit and scope of the present disclosure are not limited by the embodiments. The scope of the present disclosure should be interpreted on the basis of the appended claims, and all technical ideas within the scope equivalent to the claims should be included within the scope of the present disclosure.

Claims

1. A learning device, comprising: one or more processors; as well as A memory storing instructions, which, when executed by the one or more processors, configure the learning device to: Get the target image and the source image; generating an estimated depth map based on the target image via a first network; generating, via a second network, pose change information corresponding to a pose change between the target image and the source image; generating a synthesized image corresponding to the target image by using the estimated depth map, the pose change information, and the source image; determining a first loss based on the composite image and the target image; determining a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map; and The first loss and the second loss are back-propagated, and parameters of the first network and parameters of the second network are updated.

2. The learning device according to claim 1, wherein: When executed by the one or more processors, the instructions configure the learning device to determine the second loss by: The second loss is further determined based on at least one of brightness information, contrast information or structure information of each of the pseudo depth map and the estimated depth map via the loss calculation device.

3. The learning device according to claim 2, wherein: The first weight corresponding to the brightness information is smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

4. The learning device according to claim 1, wherein: The target image is generated by an image sensor at a first time, and The source image is generated by the image sensor at a second time within a threshold range around the first time.

5. The learning device according to claim 4, wherein: The second network includes a posture change information generation network, and When executed by the one or more processors, the instructions configure the learning device to generate the posture change information by: generating the posture change information based on a first posture at the first time and a second posture at the second time via the posture change information generation network.

6. The learning device according to claim 4, wherein: When executed by the one or more processors, the instructions configure the learning device to generate the composite image by: Based on the estimated depth map, obtaining first three-dimensional point cloud information at the first time, that is, first 3D point cloud information; Based on the posture change information, converting the first 3D point cloud information at the first time into the second 3D point cloud information at the second time; Based on image sensor parameters corresponding to the image sensor, converting the second 3D point cloud information into two-dimensional image coordinates, i.e., 2D image coordinates; and The synthesized image is generated based on pixel values ​​of the source image corresponding to the 2D image coordinates.

7. The learning device according to claim 1, wherein: The first network includes an estimated depth map generation network, and When executed by the one or more processors, the instructions further configure the learning device to: The pseudo depth map is generated based on the target image via a pseudo depth map generation network.

8. The learning device according to claim 7, wherein: The pseudo depth map generation network includes parameters in a frozen state.

9. A system comprising a testing device and a learning device, wherein The testing device comprises: a learning device configured to acquire a target image for testing; as well as A first network is configured to generate an estimated depth map for testing based on the target image; and The learning device is configured as follows: Acquire the target image and the source image; generating, via a second network, pose change information corresponding to a pose change between the target image and the source image; generating a synthesized image corresponding to the target image by using the estimated depth map, the pose change information, and the source image; determining a first loss based on the composite image and the target image; determining a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map; and Back-propagating the first loss and the second loss, and updating the parameters of the first network and the parameters of the second network, The testing device is configured to perform a test based on the updated parameters.

10. The system according to claim 9, wherein: The test apparatus is configured to determine the second loss by: The second loss is further determined based on at least one of brightness information, contrast information, or structure information of each of the pseudo depth map and the estimated depth map.

11. The system according to claim 10, wherein: The first weight corresponding to the brightness information is smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

12. The system according to claim 9, wherein: The target image is generated by an image sensor at a first time, and The source image is generated by the image sensor at a second time within a threshold range around the first time.

13. The system according to claim 12, wherein: The second network includes a posture change information generation network, and the test device is configured to generate the posture change information by: generating the posture change information based on a first posture at the first time and a second posture at the second time via the posture change information generation network.

14. A learning method comprising: Acquire a target image and a source image via one or more processors; generating, by the one or more processors, an estimated depth map based on the target image; generating, by the one or more processors, posture change information corresponding to a posture change between the target image and the source image; generating, by the one or more processors, a composite image corresponding to the target image by using the estimated depth map, the pose change information, and the source image; determining, by the one or more processors, a first loss based on the composite image and the target image; determining, by the one or more processors, a second loss based on the pseudo depth map corresponding to the target image and the estimated depth map; and The first loss and the second loss are back-propagated through the one or more processors, and parameters of a first network for generating the estimated depth map and parameters of a second network for generating the pose change information are updated.

15. The learning method according to claim 14, wherein: Determining the second loss includes: The second loss is further determined based on at least one of brightness information, contrast information, or structure information of each of the pseudo depth map and the estimated depth map.

16. The learning method according to claim 15, wherein: The first weight corresponding to the brightness information is smaller than the second weight corresponding to the contrast information, and smaller than the third weight corresponding to the structure information.

17. The learning method according to claim 14, wherein: The target image is generated by an image sensor at a first time, and The source image is generated by the image sensor at a second time within a threshold range around the first time.

18. The learning method according to claim 17, wherein: Generating the posture change information includes: The posture change information is generated based on a first posture at the first time and a second posture at the second time.

19. The learning method according to claim 17, wherein: Generating the composite image comprises: Acquire first three-dimensional point cloud information at the first time, that is, first 3D point cloud information; Based on the posture change information, converting the first 3D point cloud information at the first time into the second 3D point cloud information at the second time; Based on image sensor parameters corresponding to the image sensor, converting the second 3D point cloud information into two-dimensional image coordinates, i.e., 2D image coordinates; and The synthesized image is generated based on pixel values ​​of the source image corresponding to the 2D image coordinates.

20. The learning method according to claim 14, further comprising: Before determining the second loss, the pseudo depth map is generated based on the target image by a pre-trained pseudo depth map generation network.

Images