Novel knowledge distillation method from binocular parallax to monocular depth

Abstract

A novel knowledge distillation method from binocular parallax to monocular depth comprises the following steps: S1, training a parallax depth conversion module: inputting first training data into a pre-trained binocular stereo parallax teacher model to obtain a parallax map; the first training data comprises a left eye color image and a right eye color image; inputting the disparity map and the original disparity reference map into a disparity depth conversion module to obtain a pseudo label depth map; calculating a loss function according to the pseudo-label depth map, and updating parameters of a parallax depth conversion module; s2, mixing the pseudo-label depth map output by the parallax depth conversion module with the original true label depth map to obtain a mixed label depth map; s3, training a monocular depth estimation student model: training the monocular depth estimation student model through the second training data; the second training data comprises a mixed label depth map and a corresponding original reference map, and the method can achieve the training on a data set with a lower data size, and improves the global precision.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a novel knowledge distillation method from binocular parallax to monocular depth. Background Art

[0002] Visual geolocation is a process used to determine the geographic location of an entity. Visual geolocation technology for objects has important applications in both life and industry. It requires at least one real-world image as input to locate the location where the image was taken or the object in the image. A real-world image refers to a color image captured in the real world. It can be a photo taken with a camera in daily life, or an aerial image taken by a satellite or electro-optical targeting pod.

[0003] In visual geolocation, depth estimation is a key step in positioning. It can confirm the distance between objects in the image and the camera, helping the system understand the spatial position of objects in multiple locations, thereby achieving more accurate positioning.

[0004] At present, the technical routes for depth estimation mainly include multi-view and single-view.

[0005] Among them, multi-view methods will reach performance bottlenecks in some areas that violate the above assumptions, such as parts blocked by objects or areas with discontinuous geometric clues due to blurred textures and strong reflective surfaces. In addition, due to the need for a computationally intensive inference process for each candidate depth and the high calibration cost for sensor hardware devices such as cameras, the paradigm based on calculating pixel similarity adopted by modern multi-view stereo disparity estimation still has great limitations in deployment on low-end performance devices. Single-view depth estimation methods can only obtain relevant texture and information from one input image. Based on this condition, single-view methods first evaluate the change in the size of objects in the image by learning the gradient features of the texture, and finally estimate the distance from the object in the image to the sensor plane. In order to extract these clues from the image, existing methods usually use deep feature extractors to encode this information, and then regress pixel-level depth values ​​through supervised learning methods. The low number of input images and faster inference speed make single-view methods a strong competitor in the industry for depth estimation. However, the monocular depth estimation problem has an inherent ambiguity in nature, that is, a single image is only one of the projection results of countless three-dimensional objects. A real-life image may correspond to the layout of multiple objects, which means that object size changes beyond the distribution of the training data set will lead to unpredictable depth prediction results. This ambiguity further limits the global accuracy and generalization ability of the current model. In addition, the existing supervised learning methods' understanding of objects and texture gradients depends largely on the density of depth labels. Object features with fewer labels often cannot be correctly integrated and learned by monocular cues.

[0006] Therefore, how to improve the global accuracy and generalization ability of the model and reduce the demand for data volume is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] In view of this, the present invention provides a novel knowledge distillation method from binocular disparity to monocular depth, which can be trained on data sets with lower data volume and improves global accuracy. By enhancing visual semantics, the model has stronger generalization ability.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A novel knowledge distillation method from binocular disparity to monocular depth includes the following steps:

[0010] S1: Training the parallax depth conversion module:

[0011] First training data is constructed, and the first training data is input into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes a left-eye color image and a right-eye color image.

[0012] The disparity map and the original disparity reference map are input into the disparity depth conversion module to obtain a pseudo-label depth map.

[0013] A loss function is calculated according to the pseudo-label depth map, and parameters of the disparity depth conversion module are updated.

[0014] S2: Mixing the pseudo label output by the disparity depth conversion module with the original true label to obtain a mixed label depth map.

[0015] S3: Training the monocular depth estimation student model:

[0016] Constructing second training data, and training the monocular depth estimation student model through the second training data; the second training data includes the mixed label depth map and the corresponding original reference map.

[0017] Preferably, the disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator;

[0018] The backbone model feature extractor is used to perform multi-scale feature extraction on the original color image to obtain a multi-scale color feature map

[0019] The multi-scale fusion generator is used to mix the disparity map and the multi-scale color feature map layer by layer and output a depth map.

[0020] Preferably, the step of layer-by-layer mixing comprises:

[0021] The disparity map is continuously downsampled multiple times by a downsampling module to obtain downsampled feature maps of multiple scales;

[0022] The downsampled feature map and the color feature map of the corresponding scale are dimensionally spliced ​​and then upsampled to obtain a fused feature map;

[0023] The fused feature map is further spliced ​​with the color feature map and the downsampled feature map of the corresponding scale and then upsampled; and so on, the final fused feature map is obtained.

[0024] Preferably, the S1 further includes: mixing the pseudo-label depth map and the original true label through an adaptive filling strategy, and calculating a size loss function.

[0025] Preferably, the mixing in S2 is specifically: using a morphological mask module to perform density adaptive filling.

[0026] Preferably, the step of density adaptive filling includes:

[0027] Binarize the original true label depth map and perform a dilation based on the true label, and extract the dense area of ​​the original true label in the original true label depth map as the unfilled area through the dilation;

[0028] Perform secondary expansion based on the true label in the original true label depth map to obtain a maximum filling area, wherein the maximum filling area is used to calibrate the farthest filling distance of the original true label geometric distribution;

[0029] Determining a first pseudo-label filling area according to the unfilled area and the maximum filled area;

[0030] Marking a second pseudo-label filling area corresponding to the first pseudo-label filling area in the pseudo-label depth map;

[0031] In the second pseudo-label filling area, generating a random mask;

[0032] The first pseudo-label filling area and the second pseudo-label filling area are mixed under the random mask to obtain a mixed label depth map.

[0033] Preferably, generating a random mask comprises:

[0034] Setting a first threshold, and dividing the second pseudo-label area into different depth partitions according to the first threshold;

[0035] A corresponding second threshold is set for each of the depth partitions; the second threshold is a pseudo-label density hyperparameter in the corresponding depth partition;

[0036] A mask is generated according to the second threshold to perform element occlusion or non-occlusion on each of the depth partitions.

[0037] Preferably, generating a mask according to the second threshold to perform element occlusion on each of the depth partitions includes:

[0038] A random value is derived through a normal distribution, and each element in the depth partition is judged in turn; when the corresponding random value meets the second threshold, occlusion or non-occlusion is performed.

[0039] Preferably, the morphological mask module is specifically:

[0040] T=G+M d *M*K

[0041] Wherein, T is the mixed label depth map output by the morphological mask module; G is the original discrete depth map corresponding to the original true label; M d is the pseudo-label filled area, M is the random mask, and K is the pseudo-label depth map.

[0042] Preferably, the pseudo-label filling area is confirmed in the following manner:

[0043]

[0044] Among them, M G is the effective true label mask, S 1 and S 2 They represent the expansion structures used for primary expansion and secondary expansion respectively.

[0045] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a novel knowledge distillation method from binocular disparity to monocular depth, which realizes the conversion of the teacher's binocular disparity map to the monocular depth estimation map of the student model through the disparity depth conversion module Dis2Dep.

[0046] The present invention uses morphological masking under the guidance of an adaptive density filling strategy to realize the training of the student model with a mixed label map of the original discrete true labels and the converted dense pseudo labels, which can globally reduce the indicator errors of the scale invariant error SILog and the mean square error RMSE.

[0047] Student models with different structures are trained on a dataset enhanced with a mixture of discrete true labels and dense pseudo labels using this method. This can effectively improve the accuracy of the main indicators of the KITTI dataset without changing the structure and parameter quantity of the student monocular depth estimation model.

[0048] After filling with pseudo labels, the depth label map can enhance the features of object edges and object interfaces. This enhanced feature can make the student monocular depth estimation model have stronger generalization and the ability to regress object shapes, especially for small objects. This enhanced semantic information can benefit downstream tasks that use depth maps, including robotics, autonomous driving, augmented and mixed reality.

[0049] In order to optimize the label distillation effect between heterogeneous models, this paper proposes a density adaptive filling strategy. It can fill the blanks of the original discrete true labels with pseudo labels generated by the teacher model in a way that takes into account factors such as the density, geometric relationship and depth range of the true labels and pseudo labels. This ensures the robustness of the model, while reducing the interference of the errors of the teacher model on the training results of the student model, and optimizing the distillation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 Schematic diagram of a novel knowledge distillation method from binocular parallax to monocular depth provided by the present invention.

[0052] Figure 2 Schematic diagram of a knowledge distillation framework in an embodiment of the present invention;

[0053] Figure 3 A flowchart generated by multi-scale fusion in an embodiment of the present invention;

[0054] Figure 4 Schematic diagram of label mixing method of density adaptive filling strategy in an example of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 and Figure 2The embodiment of the present invention discloses a novel knowledge distillation method for binocular disparity to monocular depth, comprising the following steps:

[0058] S1: Training the parallax depth conversion module. The specific process includes:

[0059] S11: construct first training data, and input the first training data into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes a left-eye color image and a right-eye color image.

[0060] S12: Input the disparity map and the original disparity reference map into the disparity depth conversion module to obtain a pseudo-label depth map. The disparity reference map refers to the left eye color image or the right eye color image used as a reference when calculating the disparity. The disparity map result will be different depending on the selection of the reference, and thus have a corresponding relationship.

[0061] S13: Calculate the loss function according to the pseudo-label depth map and update the parameters of the disparity depth conversion module.

[0062] S2: Mix the pseudo labels output by the disparity depth conversion module and the original true labels to obtain a mixed label depth map.

[0063] S3: Train the monocular depth estimation student model. The specific process includes:

[0064] Constructing second training data, and training a monocular depth estimation student model through the second training data; the second training data includes a mixed label depth map and a corresponding original reference map.

[0065] In this embodiment, the present invention establishes a disparity depth conversion module (Dis2Dep) based on the probability distribution mapping between the learned disparity map and the original discrete depth label map. Compared with the deterministic formula conversion, Dis2Dep has been shown in experiments to further improve the performance of accuracy and stability indicators such as SILog and RMSE for teacher models of any accuracy. Dis2Dep can also be trained on datasets with lower data volumes, which ensures the low data requirements of Dis2Dep, which is crucial to further reducing both computational costs and data collection costs.

[0066] In order to further implement the above technical solution, the disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator; the backbone model feature extractor is used to extract multi-scale features from the original color image; the multi-scale fusion generator is used to mix the disparity map and the results of multi-scale feature extraction layer by layer and output the depth map. Among them, the multi-scale fusion generator includes a downsampling feature extractor and an upsampling fusion generator.

[0067] like Figure 3In the training phase of the disparity depth conversion module, the disparity map with dimensions H×W×1 and the corresponding original color image with dimensions H×W×3 generated by the upstream teacher model are first used as input, and the discrete original depth map with dimensions H×W×1 is used as the training label. The pre-trained feature extraction backbone model first sequentially extracts the original color images of four different size tensors, namely and Among them C i Represents the number of layers of the feature map. The disparity map with a dimension of H×W×1 is input to the multi-scale feature fusion module. The disparity map will first be downsampled successively by a convolution kernel of size 3×3 and stride 1 and a batch normalization layer to and is finally downsampled to At this point, the generation process begins, corresponding to The disparity feature map and the previously extracted color feature map are concatenated according to the feature channel dimension to generate a shape of The mixed feature map is input to an upsampling layer with a convolution kernel of 3×3, a step size of 2, and padding and output padding of 1. After the concatenation and upsampling convolution, the first size is The feature fusion map is generated by . The overall size of this map is small, and it contains and fuses the information of the disparity map and the original color image at a coarse granularity under a large receptive field. This layer of output can help reduce noise in low-texture areas. The first layer of feature fusion map will be spliced ​​with the color feature map and disparity feature map of corresponding size according to the feature map dimension, and will be gradually expanded to H×W×uC through up convolution. 5 After being expanded to the original width and height, this set of fused feature maps will be input into two layers of sharpening convolution layers with a convolution kernel of 3×3 and a stride of 1, and a Sigmoid activation function, and finally restored to H×W×1, while the depth of all pixels is compressed to (0,1). This output depth map will be calculated with the original discrete depth map using the SILog Loss function, and the gradient will be back-propagated to the multi-scale feature fusion generation module for iterative training.

[0068] In this embodiment, the present invention designs a lightweight module based on learning, namely a disparity depth conversion module, to help convert the disparity map into a depth map; the disparity depth conversion module can extract the corresponding multi-scale information from the color image and the original output disparity map, and generate the original size depth map by upsampling. This module can effectively reduce the error in the conversion of the disparity map to the medium and long distance and low texture areas of the depth map. As a learning-based module, Dis2Dep needs to be trained before the student model is trained. If the memory space and computing power of the computing system allow, it can also be trained together with the student model using an end-to-end framework.

[0069] In the training process of Dis2Dep. The training process of Dis2Dep first requires a pair of color images of the left and right eyes to be input into the pre-trained binocular disparity teacher model to generate a disparity map corresponding to the left eye. This disparity map and the color image corresponding to the left eye will be input into the Dis2Dep module, which can generate a dense pseudo-label depth map. This pseudo-label depth map will be used to calculate the SILog loss function with the original discrete depth map and cross-validated on the training data set, forming a gradient back-propagation training mode. When the training loss function of Dis2Dep converges, the weight matrix of Dis2Dep will be frozen.

[0070] In order to further implement the above technical solution, the present invention mixes the original true labels and pseudo labels, and uses the mixed results to supervise the student model for training.

[0071] It should be noted that for the part with true labels, it can be regarded as additional supervision provided by the teacher model, while for the part without labels, it actually belongs to the category of semi-supervised learning. Existing benchmarks can only calculate the prediction accuracy on the existing discrete original depth labels. Obviously, it is impossible to know the prediction performance in the unlabeled area, especially in the area with very sparse surrounding labels. The reliability of the prediction results is low, and the semi-supervised information provided by the low-reliability area will be slightly less than the supervision information of the original true label. Therefore, in order to improve its reliability, it is necessary to determine the geometric position, density distribution and distillation loss function of the pseudo labels filled by the teacher model. The original discrete depth map provides a natural geometric distribution. However, since the original depth map is actively captured by the lidar through a deterministic physical method, it can maintain good accuracy within a certain distance. However, it cannot obtain effective information on strongly reflective surfaces such as paint on ground traffic markings, car paint and metal surfaces. At the same time, due to the geometric layout of the lidar and the optical sensor group, when the point cloud obtained by the lidar is projected to the left or right camera plane, the object in the color image will be blocked due to the position offset of the sensor, and the surface of the object visible in the optical sensor cannot be observed on the lidar. This occlusion is more likely to occur when the object is closer to the image sensor plane, which means that the point cloud has no labels as hints before it actually reaches the edge of the object. The dense depth map converted by Dis2Dep still retains the high-precision characteristics of the original binocular stereo disparity method for depth estimation of close-range areas of the scene, especially for objects that stand out from the surrounding background in color visible light images. However, as the disparity decreases, the inherent myopia defect of binocular stereo vision gradually emerges. At the same time, the blur and lack of information caused by the sky, road surface, lens motion, lens depth of field, etc. also make the binocular stereo vision method perform poorly at longer distances.

[0072] In general, the original depth map often lacks information about the true edge of the object, but the color image can be used to infer the overall appearance of the object and aggregate features from the texture information of visible light. The information of the original depth map and the converted dense depth map can complement each other in their respective disadvantageous and advantageous areas, which requires reasonable filling of soft labels for these occluded areas.

[0073] The purpose of using dense pseudo labels to fill the original discrete true labels is to supervise the unlabeled parts of the original discrete depth map through credible labels. However, since these filled pseudo labels still have inherent errors, this strategy hopes that the pseudo labels filled into the original discrete depth map should interfere with the pixel accuracy of the original labels as little as possible. The existence of these pseudo labels is more to enable the student model to strengthen the edges of objects and their front-to-back relationships as much as possible through the difference in depth and combined with the features extracted from the input color image.

[0074] These pseudo-labels provide continuous information about the actual shape of the object on the blocked surface and the strongly reflective surface. Therefore, the pseudo-label will first be farther away from the pixel position of the existing original label, but still on the same surface or object, and preferably can contain the depth on both sides of the edge of the object at the same time. In order to further reduce the incidental errors, according to the currently obtained conversion rules, low-precision areas that may diverge at long distances should also be eliminated as much as possible. In order to fully utilize the advantages of the two depth maps, the present invention proposes a density adaptive label filling strategy, which uses morphological masks to mix the original depth map and the pseudo-label depth map generated by the teacher model. This strategy mainly considers three main factors.

[0075] 1. Density ratio of original labels and pseudo labels. Although the converted pseudo-label depth map retains high pixel accuracy, the accuracy of these pseudo labels still varies with the scene as a whole. Taking the KITTI dataset as an example, due to the relatively complex lighting conditions of outdoor scenes, methods using color real-world images may fluctuate in performance in these scenes. In order to ensure the robustness of the training process, this strategy can maintain the semi-supervised additional knowledge and potential interference brought by pseudo-label filling by adjusting the density ratio of pseudo-labels and true labels.

[0076] 2. Distribution of depth range of pseudo-label filling. The scenes contained in the KITTI autonomous driving dataset are varied, ranging from urban roads, main roads, boulevards to national highways. The depth in some scenes is only concentrated within 80 meters, while some are as far as 255 meters. Generally speaking, the farther the distance, the fewer valid original real labels. If too many pseudo-labels with low reliability are introduced in long-distance areas, it will inevitably interfere with the correct labels. In order to overcome the inherent myopia problem of binocular stereo models, this strategy needs to further aggregate the information of pseudo-labels and original labels by controlling the filling depth range of pseudo-labels.

[0077] 3. The geometric relationship between pseudo labels and original labels. The distribution of original labels on the imaging plane of color images is not uniform. Therefore, the geometric distribution relationship between pseudo labels and original labels after filling becomes an important factor in providing prompt information to the student model. When too dense pseudo labels are used during filling, the information focus of the student model will be placed on the pseudo labels. Other experiments have shown that when the density of labels is too large, overfitting will occur, reducing the overall generalization ability of the model. The sampling and mixing process of the original true labels and pseudo labels can be regarded as a logical or operation of two mask images.

[0078] In this embodiment, the present invention uses a morphological mask module to fully consider the above three factors to achieve a density-adaptive label filling strategy. After the hyperparameter controls the filling probability of the original real label and the pseudo label on this morphological mask module, the morphological mask module can realize adaptive random filling using dilation operation. This method is based on the position of each original true label to dilate, so as to confirm the pseudo label to determine the pseudo label filling range, so that the pseudo label can be consistent with the geometric distribution of the original true label; using a controllable random mask, fill in pseudo labels of arbitrary density within the pseudo label filling range, combined with the depth information within the label filling range, so that the filled pseudo label can be consistent with the original true label The corresponding density relationship and depth relationship. For example: if the density of the filled pseudo label is to be the same as the density of the true label and have the same depth ratio, it is necessary to first confirm the density of the true label according to the non-filled area and use it as the filling density of the pseudo label; then the number of pseudo labels can be confirmed according to the pseudo label filling range, combined with the depth ratio of the real label, and the number of pseudo labels at different depths can be confirmed; finally, the number of pseudo label fillings is adjusted by setting different hyperparameters to make it consistent with the real label.

[0079] Specifically, Figure 4 , the input color image is recorded as x, its corresponding original true label depth map G, the pseudo label dense depth map K is the pseudo label depth map, and the overall random mask M = M 1 +M 2 . Random mask at distance threshold r maxUnder the control of 1 and medium and long distance random mask M 2 , effective true label mask M G , and SILog threshold t 0 All depth maps have a shape of H×W×1. The dilation operation is denoted as The matrix product is recorded as * according to the corresponding position, and the exclusive OR operation is recorded as XOR. During training, the SIlog error relative to the original true label is first calculated based on the dense pseudo-label generated by the stereo model. When the error is lower than the SILog threshold t 0 When , the true label can be used directly for training, otherwise it needs to be mixed with the pseudo label. The steps of density adaptive filling include:

[0080] First, the original discrete depth map, i.e., the original true label depth map, is binarized and expanded once, and the dense area M of the original true label in the original discrete depth map is extracted through the expansion min , as the unfilled area; this will avoid the interference of pseudo labels on the accuracy of the original true labels, because in the dense area of ​​the calibrated original true labels, at least one out of every 9 pixels has been calibrated with depth, which can reflect the surface shape of the object in the image after the common downsampling operation in the student model. In this embodiment, a 3×3 square structure element S is used. 1 Perform a dilation on the discretized depth map.

[0081] Secondly, the original true label depth map is dilated twice to obtain the maximum filling area, which is used to calibrate the farthest filling distance of the original true label geometric distribution. In this embodiment, a 115×115 square structure element S is used for the original true label depth map. 2 Expand to get the maximum filling area M max When there is no depth value within the surrounding 115 pixels, it means that the pixel is located in an area that cannot be marked even if it is offset by one-third of the screen height. These pixels are usually too far away from the camera plane, exceeding the 255 meters that the lidar can capture, and are beyond the prediction range of the model, so they do not need to be considered.

[0082] After two expansion operations, the maximum filling area and the original true label dense area are XORed to obtain the pseudo label filling area M d , calculated as follows:

[0083] M d =M min XORM max

[0084] After confirming the pseudo-label filling area, it is necessary to perform filling and mixing in the pseudo-label filling area. To this end, the present invention uses a random mask to control the density ratio and depth ratio during filling and mixing. The overall random mask M is composed of a close-range random mask M1 and a long-range random mask M2. The close-range random mask M1 and the long-range random mask M2 randomly select elements for masking within the corresponding depth range. In order to control the number of masking and the label proportion of different depth areas, a probability hyperparameter can be set, and then a random value is derived using a normal distribution. Each element is judged in turn, so that the density of the selected elements corresponds to the probability of the probability hyperparameter; the density ratio control of the true and false label ratio and the label ratio control at different depths are achieved.

[0085] Specifically, the steps of generating a random mask include:

[0086] A first threshold is set, and the second pseudo-label area is divided into different depth partitions according to the first threshold.

[0087] A corresponding second threshold is set for each of the depth partitions; the second threshold is a pseudo-label density hyperparameter in the corresponding depth partition.

[0088] Generate a mask according to the second threshold to block or unblock elements in each depth partition; for example: derive a random value through a normal distribution, and judge each element in the depth partition in turn; when the corresponding random value meets the second threshold, block or unblock.

[0089] In the current common benchmark, only depth prediction within 80 meters is considered. Based on the divergence of the conversion, this paper sets 40 meters as the medium and long distance threshold r max That is, the first threshold value, the depth partitions exceeding the threshold value are medium- and long-distance depth partitions, and the depth partitions not exceeding the threshold value are short- or close-distance depth partitions.

[0090]

[0091]

[0092] in, is the occlusion element value in the close-range mask. When the value is 1, it means no occlusion, and when it is 0, it means occlusion; σ 1 is a random value derived from a normal distribution, t 1 is the short-distance cutoff threshold, which is used to control the probability of each close-distance point being taken.

[0093]

[0094] in, is the occlusion element value in the close-range mask. When the value is 1, it means no occlusion, and when it is 0, it means occlusion; σ 2 is a random value derived from a normal distribution, t 2 It is the medium and long distance cutoff threshold, which is used to control the probability of each medium and long distance point being taken.

[0095] In order to simultaneously calculate the area outside the pseudo-label filling area, the labels outside the pseudo-label filling area, that is, the valid true labels, are blocked to avoid filling.

[0096]

[0097] in, is the occlusion element value in the effective true label mask. When the value is 1, it means no occlusion, and when it is 0, it means occlusion; G i,j is the pixel value in the original true label depth image.

[0098] In this embodiment, the pseudo-label filling area is confirmed by dilation so that the filling of pseudo-labels can fit the distribution of real labels. In addition, the density ratio of real and fake labels and the label ratio at different depths can be further controlled by the hyperparameter t 1 , t 2 and r max During training, the distillation framework will first calculate the SILog loss of the pseudo-label map on the original true label. When this loss is greater than the initial threshold t 0 When , the pseudo label will not be mixed with the original true label. As the training progresses, the training SILog of the student model will continue to decrease, and the threshold t 0 It will also gradually decrease to reduce the additional error interference of the supplementary information.

[0099] Furthermore, when training the disparity depth module, this method also designs a size-invariant loss function weighted by the true label and the original label to enable the SILog loss function to balance the gradient loss of pseudo labels and true labels from a denser mixed depth map.

[0100] The following is the expression of the weighted SILog loss function. This method adds ν as a weight to the original loss function to weight the training errors of the pseudo-labels and true labels.

[0101]

[0102] in, is the true label depth value at pixel i, is the predicted depth value at pixel i. It is the difference between the predicted depth value and the true label at pixel i. is the pseudo-label depth value at pixel j, is the predicted depth value at pixel j. is the difference between the predicted depth value and the pseudo label at pixel j. w represents the calculation weight of the pseudo label in the loss function, n is the total number of pixels, and ν represents the recall parameter.

[0103] Before the converted depth map is input into the monocular student model, the pseudo labels generated by the teacher model are mixed with the original true labels. This adaptive filling strategy can reduce the inherent error attached to the additional pseudo labels while considering the geometric distribution of the original true labels.

[0104] In order to give full play to the advantages of single-view estimation and multi-view estimation, this method adopts the idea of ​​label distillation commonly used in heterogeneous models from the perspective of knowledge distillation, and uses the teacher model to perform semi-supervised training on the student model. In actual operation, a multi-view stereo disparity model with higher accuracy and large computing resource usage is used as the teacher model in the framework. It uses multiple color real-scene pictures as input and finally returns the disparity map corresponding to one of the perspectives. In order to further convert the disparity map into a depth map, this method found in the experiment that the use of the classic deterministic algorithm as the disparity-depth conversion method has the disadvantages of high noise, low accuracy, and small depth. Therefore, a disparity-depth conversion module based on the probability distribution mapping between the learned disparity map and the original discrete depth label map is established. Compared with the deterministic formula conversion, Dis2Dep has been proved in experiments that it can further improve the performance of accuracy and stability indicators such as SILog and RMSE for any precision teacher model. Dis2Dep can also be trained on datasets with lower data volume, which ensures the low data requirement of Dis2Dep, which is crucial to further reduce the computing cost and data collection cost.

[0105] Subsequently, in order to better mix the original true labels and the converted dense pseudo-label depth map generated by the teacher model, this method proposes a density adaptive filling strategy. The density adaptive filling strategy takes into account the density ratio, geometric distribution and depth filling interval between the true label and the pseudo-label, and uses morphological masks to put the density adaptive filling strategy into practice. Experimentally, the adaptive filling strategy can effectively reduce the errors attached to the teacher model and improve the efficiency of the teacher model's additional supervision of the student model. Overall, through the teacher-student framework of binocular stereo disparity to monocular depth estimation based on knowledge distillation designed by this method, the student model can obtain an average SILog error reduction of 1.4% in full field of view training. In low field of view tasks, BtS can achieve a 29.9% SILog error reduction after distillation compared to simply using the original label. Therefore, this distillation framework can provide effective supervision outside the sensor field of view.

[0106] The following table shows the comparison data of indicators of various monocular models before and after using the knowledge distillation in this embodiment.

[0107] Table 1 Comparison of knowledge distillation effects under various models

[0108]

[0109] It can be found from the above table that the present invention uses morphological masking under the guidance of an adaptive density filling strategy to realize the training of the student model with a mixed label map of the original discrete true label and the converted dense pseudo label, which can globally reduce the indicator errors of the size invariant error SILog and the mean square error RMSE.

[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0111] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A novel knowledge distillation method from binocular disparity to monocular depth, characterized in that: The following steps are involved: S1: Training the parallax depth conversion module: Constructing first training data, and inputting the first training data into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes a left-eye color image and a right-eye color image; Inputting the disparity map and the original disparity reference map into the disparity depth conversion module to obtain a pseudo-label depth map; Calculating a loss function according to the pseudo-label depth map, and updating parameters of the disparity depth conversion module; S2: Mixing the pseudo-label depth map output by the disparity depth conversion module and the original true label depth map to obtain a mixed label depth map; S3: Training the monocular depth estimation student model: Constructing second training data, and training the monocular depth estimation student model through the second training data; the second training data includes the mixed label depth map and the corresponding original reference map.

2. According to claim 1, a novel binocular disparity to monocular depth knowledge distillation method is characterized in that: The disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator; The backbone model feature extractor is used to perform multi-scale feature extraction on the original color image to obtain a multi-scale color feature map The multi-scale fusion generator is used to mix the disparity map and the multi-scale color feature map layer by layer and output a depth map.

3. The novel binocular disparity to monocular depth knowledge distillation method according to claim 2, characterized in that: The step of layer-by-layer mixing comprises: The disparity map is continuously downsampled multiple times by a downsampling module to obtain downsampled feature maps of multiple scales; The downsampled feature map and the color feature map of the corresponding scale are dimensionally spliced ​​and then upsampled to obtain a fused feature map; The fused feature map is further spliced ​​with the color feature map and the downsampled feature map of the corresponding scale and then upsampled; and so on, the final fused feature map is obtained.

4. The novel binocular disparity to monocular depth knowledge distillation method according to claim 1, characterized in that: The S1 also includes: mixing the pseudo-label depth map and the original true label through an adaptive filling strategy, and calculating a size loss function.

5. The novel binocular disparity to monocular depth knowledge distillation method according to claim 1 or 4, characterized in that: The mixing in S2 is specifically: using a morphological mask module to perform density adaptive filling.

6. The novel binocular disparity to monocular depth knowledge distillation method according to claim 5, characterized in that: The step of density adaptive filling includes: Binarize the original true label depth map and perform a dilation based on the true label, and extract the dense area of ​​the original true label in the original true label depth map as the unfilled area through the dilation; Perform secondary expansion based on the true label in the original true label depth map to obtain a maximum filling area, wherein the maximum filling area is used to calibrate the farthest filling distance of the original true label geometric distribution; Determining a first pseudo-label filling area according to the unfilled area and the maximum filled area; Marking a second pseudo-label filling area corresponding to the first pseudo-label filling area in the pseudo-label depth map; In the second pseudo-label filling area, generating a random mask; The first pseudo-label filling area and the second pseudo-label filling area are mixed under the random mask to obtain a mixed label depth map.

7. The novel binocular disparity to monocular depth knowledge distillation method according to claim 6, characterized in that: The generating of the random mask comprises: Setting a first threshold, and dividing the second pseudo-label area into different depth partitions according to the first threshold; A corresponding second threshold is set for each of the depth partitions; the second threshold is a pseudo-label density hyperparameter in the corresponding depth partition; A mask is generated according to the second threshold to perform element occlusion or non-occlusion on each of the depth partitions.

8. The novel binocular disparity to monocular depth knowledge distillation method according to claim 7, characterized in that: Generating a mask according to the second threshold to perform element occlusion on each of the depth partitions includes: A random value is derived through a normal distribution, and each element in the depth partition is judged in turn; when the corresponding random value meets the second threshold, occlusion or non-occlusion is performed.

9. The novel binocular disparity to monocular depth knowledge distillation method according to claim 5, characterized in that: The morphological mask module is specifically: T=G+M d *M*K Wherein, T is the mixed label depth map output by the morphological mask module; G is the original discrete depth map corresponding to the original true label; M d is the pseudo-label filled area, M is the random mask, and K is the pseudo-label depth map.

10. A novel binocular disparity to monocular depth knowledge distillation method according to claim 6 or 9, characterized in that: The method for confirming the pseudo-label filling area is as follows: Among them, M G is the effective true label mask, S1 and S2 represent the expansion structures adopted by the first expansion and the second expansion, respectively.

Images