A novel knowledge distillation method for binocular parallax to monocular depth

Abstract

A novel knowledge distillation method for binocular disparity to monocular depth is disclosed. S1: Training the disparity-depth conversion module: First training data is input into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes left-eye and right-eye color images; the disparity map and the original disparity reference map are input into the disparity-depth conversion module to obtain a pseudo-labeled depth map; a loss function is calculated based on the pseudo-labeled depth map, and the parameters of the disparity-depth conversion module are updated; S2: The pseudo-labeled depth map output by the disparity-depth conversion module and the original true-labeled depth map are mixed to obtain a mixed-labeled depth map; S3: Training the monocular depth estimation student model: The monocular depth estimation student model is trained using second training data; the second training data includes the mixed-labeled depth map and the corresponding original reference map. This invention enables training on datasets with relatively low data volume and improves global accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a novel knowledge distillation method for binocular parallax to monocular depth. Background Technology

[0002] Visual geolocation is a process used to determine the geographical location of an entity. Visual geolocation technology has important applications in both daily life and industry. It requires at least one real-world image as input to locate the image's shooting location or the objects within it. A real-world image refers to a color image captured in the real world; this can be a photograph taken with a camera in everyday life, or an aerial image taken by a satellite or electro-optical targeting pod.

[0003] In visual geolocation, depth estimation is a key step in localization. It can determine the distance between objects in an image and the camera, helping the system understand the spatial location of objects in multiple places, thereby achieving more accurate localization.

[0004] Currently, the main technical approaches to depth estimation include multi-view and single-view methods.

[0005] Multi-view methods often reach performance bottlenecks in areas that violate the above assumptions, such as occluded parts of objects or regions with discontinuous geometric cues due to texture blurring and highly reflective surfaces. Furthermore, the computationally intensive inference process required for each candidate depth, coupled with the high calibration costs for sensor hardware like cameras, limits the deployment of the pixel-similarity-based paradigm used in modern multi-view stereo disparity estimation on low-performance devices. Single-view depth estimation methods can only extract relevant texture and information from a single input image. Given this limitation, single-view methods first assess changes in object size in the image by learning texture gradient features, ultimately estimating the distance from the object to the sensor plane. To extract these cues from the image, existing methods typically use depth feature extractors to encode this information and then regress pixel-level depth values ​​through supervised learning. The lower number of input images and faster inference speed make single-view methods a strong contender for practical application in depth estimation within industry. However, monocular depth estimation inherently suffers from ambiguity. A single image represents only one projection of countless possible arrangements of objects in 3D space, and a single real-world image may correspond to multiple object layouts. This means that variations in object size beyond the training dataset's distribution can lead to unpredictable depth predictions, further limiting the global accuracy and generalization ability of current models. Moreover, existing supervised learning methods' understanding of objects and texture gradations largely depends on the density of depth labels; features of objects with fewer labels often cannot be correctly integrated and learned by monocular cues.

[0006] Therefore, how to improve the global accuracy and generalization ability of the model while reducing the need for large amounts of data is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a novel knowledge distillation method from binocular parallax to monocular depth, which can be trained on datasets with relatively low data volume and improves global accuracy. By enhancing visual semantics, the model has stronger generalization ability.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A novel knowledge distillation method for binocular parallax to monocular depth includes the following steps:

[0010] S1: Training the disparity-depth transformation module:

[0011] First training data is constructed and input into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes a left-eye color image and a right-eye color image.

[0012] The disparity map and the original disparity reference map are input into the disparity depth conversion module to obtain a pseudo-label depth map.

[0013] The loss function is calculated based on the pseudo-label depth map, and the parameters of the disparity depth conversion module are updated.

[0014] S2: Mix the pseudo-labels output by the parallax depth conversion module with the original true labels to obtain a hybrid label depth map.

[0015] S3: Training a monocular depth estimation student model:

[0016] Construct a second training data set and train the monocular depth estimation student model using the second training data set; the second training data set includes the hybrid labeled depth map and the corresponding original reference map.

[0017] Preferably, the disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator;

[0018] The backbone model feature extractor is used to perform multi-scale feature extraction on the original color image to obtain multi-scale color feature maps.

[0019] The multi-scale fusion generator is used to blend the disparity map and the multi-scale color feature map layer by layer to output a depth map.

[0020] Preferably, the layer-by-layer mixing step includes:

[0021] The disparity map is downsampled multiple times by the downsampling module to obtain downsampled feature maps at multiple scales;

[0022] The downsampled feature map and the color feature map at the corresponding scale are concatenated dimensionally and then upsampled to obtain the fused feature map;

[0023] The fused feature map is further concatenated with color feature maps and downsampled feature maps of the corresponding scale, and then upsampled; this process is repeated to obtain the final fused feature map.

[0024] Preferably, step S1 further includes: mixing the pseudo-label depth map and the original true label using an adaptive filling strategy, and calculating a size loss function.

[0025] Preferably, the mixing in S2 specifically involves using a morphological masking module for density adaptive filling.

[0026] Preferably, the density adaptive filling step includes:

[0027] The original true label depth map is binarized and dilated once based on the true labels. The dense region of the original true labels in the original true label depth map is extracted through the first dilation and used as the unfilled region.

[0028] A secondary dilation is performed on the true labels in the original true label depth map to obtain the maximum filling region. The maximum filling region is used to mark the farthest filling distance of the geometric distribution of the original true labels.

[0029] The first pseudo-label filling area is determined based on the unfilled area and the maximum filled area;

[0030] In the pseudo-label depth map, a second pseudo-label filling region corresponding to the first pseudo-label filling region is marked;

[0031] In the second pseudo-label filling area, a random mask is generated;

[0032] Under the random mask, the first pseudo-label filling area and the second pseudo-label filling area are mixed to obtain a mixed label depth map.

[0033] Preferably, the generation of the random mask includes:

[0034] Set a first threshold, and divide the second pseudo-label region into different depth partitions according to the first threshold;

[0035] A corresponding second threshold is set for each of the depth partitions; the second threshold is the pseudo-label density hyperparameter in the corresponding depth partition.

[0036] Based on the second threshold, a mask is generated to either occlude or not occlude elements in each of the depth partitions.

[0037] Preferably, the step of generating a mask based on the second threshold to occlude elements in each of the depth partitions includes:

[0038] Random values ​​are derived from a normal distribution, and each element in the depth partition is judged sequentially; when the corresponding random value meets the second threshold, occlusion or non-occlusion is performed.

[0039] Preferably, the morphological masking module specifically comprises:

[0040] T = G + M d *M*K

[0041] Where T is the hybrid label depth map output by the morphological masking module; G is the original discrete depth map corresponding to the original true label; M d M represents the pseudo-label filling area, M is the random mask, and K is the pseudo-label depth map.

[0042] Preferably, the method for confirming the pseudo-label filling area is as follows:

[0043]

[0044] Among them, M G For effective true label masking, S1 and S2 represent the expansion structures used for primary and secondary expansion, respectively.

[0045] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a novel knowledge distillation method for binocular disparity to monocular depth, which realizes the conversion of the teacher's binocular disparity map to the monocular depth estimation map of the student model through the disparity-depth conversion module Dis2Dep.

[0046] This invention uses morphological masking guided by an adaptive density filling strategy to train a student model with a hybrid label map consisting of the original discrete true labels and the transformed dense pseudo labels. This can reduce the index errors of size invariant error (SILog) and mean square error (RMSE) globally.

[0047] Training student models with different structures on datasets augmented with discrete true labels and dense pseudo labels using this method can effectively improve their accuracy on key metrics of the KITTI dataset without changing the structure and number of parameters of the student monocular depth estimation model.

[0048] By using pseudo-labels for padding, the depth label map can enhance features at object edges and boundaries. This enhanced feature allows student monocular depth estimation models to have stronger generalization capabilities and the ability to regress object shapes, especially for small objects. This enhanced semantic information can benefit downstream tasks that use depth maps, including robotics, autonomous driving, augmented reality, and mixed reality.

[0049] To optimize label distillation between heterogeneous models, this invention proposes a density-adaptive filling strategy. This strategy fills the blank spaces in the original discrete true labels with pseudo-labels generated by the teacher model, taking into account factors such as the density, geometric relationship, and depth range of true and pseudo labels. This ensures model robustness while reducing the interference of errors from the teacher model on the training results of the student model, thus optimizing the distillation effect. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of a novel knowledge distillation method for converting binocular parallax to monocular depth, provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the knowledge distillation framework in an embodiment of the present invention;

[0053] Figure 3 This is a flowchart illustrating the multi-scale fusion generation process in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the label mixing method of the density adaptive filling strategy in an example of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 and Figure 2This invention discloses a novel knowledge distillation method for converting binocular parallax to monocular depth, comprising the following steps:

[0058] S1: Training the disparity-depth transformation module, the specific process includes:

[0059] S11: Construct the first training data and input the first training data into the pre-trained binocular stereo disparity teacher model to obtain the disparity map; the first training data includes the left eye color image and the right eye color image.

[0060] S12: Input the disparity map and the original disparity reference map into the disparity-depth conversion module to obtain a pseudo-labeled depth map. The disparity reference map refers to the left-eye color image or right-eye chromatic image used as a reference during disparity calculation. The disparity map result will vary depending on the selected reference, thus exhibiting a corresponding relationship.

[0061] S13: Calculate the loss function based on the pseudo-label depth map and update the parameters of the disparity depth conversion module.

[0062] S2: Mix the pseudo-labels output by the parallax depth conversion module with the original true labels to obtain a hybrid label depth map.

[0063] S3: Training the monocular depth estimation student model, the specific process includes:

[0064] Construct a second training dataset and train a monocular depth estimation student model using the second training dataset; the second training dataset includes a mixed-label depth map and the corresponding original reference map.

[0065] In this embodiment, the present invention establishes a disparity depth transformation module (Dis2Dep) based on the probability distribution mapping between the learned disparity map and the original discrete depth label map. Compared with deterministic formula transformation, Dis2Dep has been experimentally demonstrated to further improve the performance of teacher models of any accuracy on accuracy and stability metrics such as SILog and RMSE. Dis2Dep can also be trained on datasets with relatively small amounts of data, which ensures its low data requirements and is crucial for further reducing computational and data collection costs.

[0066] To further implement the above technical solution, the disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator. The backbone model feature extractor is used to extract multi-scale features from the original color image. The multi-scale fusion generator is used to blend the disparity map and the results of the multi-scale feature extraction layer by layer to output a depth map. The multi-scale fusion generator includes a downsampling feature extractor and an upsampling fusion generator.

[0067] like Figure 3In the training phase of the disparity-depth conversion module, the disparity map of dimension H×W×1 generated by the upstream teacher model and its corresponding original color image of dimension H×W×3 are first used as input, and the discrete original depth map of dimension H×W×1 is used as the training label. The pre-trained feature extraction backbone model first sequentially extracts original color images of four different tensor sizes, namely... and Where C i This represents the number of layers in the feature map. A disparity map of dimension H×W×1 is input to the multi-scale feature fusion module. The disparity map is first sequentially downsampled by a 3×3 convolutional kernel with a stride of 1 and a batch regularization layer. And was eventually downsampled to At this point, the generation process begins, corresponding to The disparity feature map and the previously extracted color feature map are concatenated along the feature channel dimension to generate a shape of... The mixed feature map is input into an upsampling layer with a 3×3 kernel, a stride of 2, and padding of 1 for both the output and convolutions. After this concatenation and upsampling convolution, the first size is obtained. The feature fusion generation map is relatively small in overall size, containing and fusing information from the disparity map under a large receptive field and the original color image at a coarse-grained level. This output layer helps reduce noise in low-texture regions. The first feature fusion map is concatenated with the corresponding size color feature map and disparity feature map along the feature map dimensions, and is progressively enlarged to H×W×uC5 through convolution. After being enlarged to the original width and height dimensions, this fused feature map is fed into two sharpening convolutional layers with 3×3 kernels and a stride of 1, and a sigmoid activation function, and is finally restored to H×W×1, while the depth of all pixels is compressed to (0,1). This output depth map is then subjected to size-invariant loss (SILog Loss) with the original discrete depth map, and the resulting gradient is backpropagated to the multi-scale feature fusion generation module for iterative training.

[0068] In this embodiment, the present invention designs a learning-based lightweight module, namely a disparity-depth conversion module, to help convert disparity maps into depth maps. The disparity-depth conversion module can extract corresponding multi-scale information from the color image and the original output disparity map, and generate a depth map of the original size through upsampling. This module can effectively reduce the error in mid-to-long-range and low-texture regions during the conversion from disparity map to depth map. As a learning-based module, Dis2Dep needs to be trained before the student model is trained. If the memory space and computing power of the computing system allow, an end-to-end framework can also be used to train it together with the student model.

[0069] The training process of Dis2Dep begins by inputting a pair of color images for both the left and right eyes into a pre-trained binocular disparity teacher model to generate a disparity map corresponding to the left eye. This disparity map and the corresponding color image for the left eye are then input into the Dis2Dep module, which generates a dense pseudo-labeled depth map. This pseudo-labeled depth map is then compared with the original discrete depth map using an SILog loss function and cross-validated on the training dataset, forming a gradient backpropagation training pattern. Once the Dis2Dep training loss function converges, the Dis2Dep weight matrix is ​​frozen.

[0070] To further implement the above technical solution, this invention mixes the original true labels and pseudo labels, and uses the mixed result to supervise the training of the student model.

[0071] It should be noted that the portions with existing true labels can be considered as additional supervision provided by the teacher model, while the portions without labels fall under the category of semi-supervised learning. Existing benchmarks can only calculate prediction accuracy on existing discrete original depth labels, and obviously cannot know the prediction performance in unlabeled regions, especially in areas with very sparse surrounding labels, where the reliability of the prediction results is low. The semi-supervised information provided in low-reliability regions will be slightly less than the supervision information from the original true labels. Therefore, to improve its reliability, it is necessary to determine the geometric position, density distribution, and distillation loss function of the pseudo-labels filled by the teacher model. The original discrete depth map provides a natural geometric distribution. However, since the original depth map is actively captured by LiDAR using deterministic physical methods, it can maintain good accuracy within a certain distance. But it cannot obtain effective information on highly reflective surfaces, including paint on ground traffic markings, car paint, and metal surfaces. At the same time, due to the geometric layout of the LiDAR and optical sensor group, when the point cloud acquired by the LiDAR is projected onto the left or right camera plane, objects in the color image will be occluded due to the sensor position offset. Object surfaces visible in the optical sensor cannot be observed on the LiDAR. This occlusion is more likely to occur when the object is closer to the image sensor plane, meaning that the point cloud loses its labeling as a cue before it actually reaches the object's edge. The dense depth map transformed by Dis2Dep retains the high accuracy of the original binocular stereo parallax method for estimating depth in near-field areas of the scene, especially for objects that stand out significantly from the surrounding background in color visible light images. However, as parallax decreases, the inherent nearsightedness of binocular stereo vision gradually becomes apparent. Simultaneously, blurring and lack of information caused by the sky, road surface, camera movement, and depth of field also make the binocular stereo vision method perform poorly at longer distances.

[0072] In general, the original depth map often lacks information about the true edges of objects, but the overall shape of the object can be inferred and features aggregated from the visible light texture information of the color image. The information in the original depth map and the transformed dense depth map can complement each other in their respective disadvantage and advantage areas, which requires appropriate soft labeling to fill in these occluded areas.

[0073] The purpose of using dense pseudo-labels to fill the original discrete true labels is to supervise the unlabeled parts of the original discrete depth map with reliable labels. However, since these pseudo-labels still have inherent errors, this strategy aims to minimize the interference of the pseudo-labels filling the original discrete depth map with the pixel accuracy of the original labels. The significance of these pseudo-labels is more to enable the student model to strengthen the edges of objects and the spatial relationships between them by using depth differences and combining features extracted from the input color image.

[0074] These pseudo-labels provide continuous information about the actual shape of the object on occluded and highly reflective surfaces. Therefore, the pseudo-labels will initially be located at pixel positions far from the original labels, but still on the same surface or object, and ideally, they should simultaneously contain the depth of both sides of the object's edge. To further reduce incidental errors, based on the current transformation rules, low-precision regions that may diverge at greater distances should be eliminated as much as possible. To fully utilize the advantages of the two depth maps, this invention proposes a density-adaptive label filling strategy that uses morphological masking to blend the original depth map and the pseudo-label depth map generated by the teacher model. This strategy primarily considers three main factors.

[0075] 1. Density ratio of original labels to pseudo-labels. Although the transformed pseudo-label depth maps retain high pixel accuracy, the overall accuracy of these pseudo-labels still varies with the scene. Taking the KITTI dataset as an example, due to the relatively complex lighting conditions in outdoor scenes, the method using color real-world images may experience performance fluctuations in these scenarios. To ensure the robustness of the training process, this strategy can maintain the semi-supervised additional knowledge and potential interference brought about by pseudo-label padding by adjusting the density ratio of pseudo-labels to real labels.

[0076] 2. Distribution of Pseudo-Label Filling Depth. The KITTI autonomous driving dataset includes diverse scenes, ranging from urban roads, main roads, and tree-lined streets to national highways. In some scenes, the filling depth is concentrated within 80 meters, while in others it exceeds 255 meters. Generally, the greater the distance, the fewer effective original true labels there are. Introducing too many pseudo-labels, which are inherently less reliable, into distant regions inevitably interferes with the correct labels. To overcome the inherent nearsightedness of stereo models, this strategy requires controlling the filling depth range of pseudo-labels to further aggregate information from both the pseudo-labels and the original labels.

[0077] 3. Geometric Relationship Between Pseudo-Labels and Original Labels. The original labels are not uniformly distributed on the imaging plane of a color image. Therefore, the geometric distribution relationship between the pseudo-labels and original labels after padding becomes an important factor in providing hints to the student model. When overly dense pseudo-labels are used during padding, the student model will focus its information on the pseudo-labels. Furthermore, experiments have shown that excessively high label density can lead to overfitting and reduce the overall generalization ability of the model. The sampling and mixing process of the original and pseudo-labels can be viewed as a logical OR operation between two mask images.

[0078] In this embodiment, the present invention utilizes a morphological masking module to fully consider the above three factors to achieve a density-adaptive label filling strategy. After controlling the filling probabilities of the original real labels and pseudo labels on this morphological masking module using hyperparameters, the morphological masking module can achieve adaptive random filling using dilation operations. This method performs dilation based on the position of each original real label to determine the pseudo label filling range, so that the pseudo labels can match the geometric distribution of the original real labels. Using a controllable random mask, pseudo labels of arbitrary density are filled within the pseudo label filling range. Combined with the depth information within the label filling range, the filled pseudo labels can conform to the corresponding density and depth relationships with the original real labels. For example, if the density of the filled pseudo labels is the same as the density of the real labels and has the same depth ratio, it is necessary to first determine the density of the real labels based on the non-filled areas and use it as the filling density of the pseudo labels; then, the number of pseudo labels can be determined based on the pseudo label filling range, and combined with the depth ratio of the real labels, the number of pseudo labels at different depths can be determined; finally, the number of pseudo labels is adjusted by setting different hyperparameters to match the real labels.

[0079] Specifically, such as Figure 4 Let the input color image be denoted as x, its corresponding original true label depth map G, and the pseudo-label dense depth map K be the pseudo-label depth map. The overall random masking M = M1 + M2. The random masking is at a distance threshold r. maxUnder the control of [the system], the random mask is divided into a near-range random mask M1 and a mid-to-long-range random mask M2, with an effective true label mask M1. G And the SILog threshold t0. All depth maps have a shape of H×W×1. The dilation operation is denoted as The matrix product at corresponding positions is denoted as *, and the XOR operation is denoted as XOR. During training, the SIlog error relative to the original true labels is first calculated based on the dense pseudo-labels generated by the stereo model. When the error is below the SILog threshold t0, the true labels can be used directly for training; otherwise, they need to be mixed with the pseudo-labels. The density adaptive padding steps include:

[0080] First, the original discrete depth map, i.e., the original true label depth map, is binarized and dilated once. The dense region M of the original true label in the original discrete depth map is extracted through the first dilation. min This is used as an unfilled region; this avoids the interference of false labels on the accuracy of the original true labels, because in the dense region of the original true labels, at least one in every nine pixels has been labeled with depth. After performing common downsampling operations in the student model, this is sufficient to reflect the surface shape of objects in the image. In this embodiment, a 3×3 square structuring element S1 is used to dilate the discrete depth map once.

[0081] Secondly, the original true label depth map is dilated a second time to obtain the maximum filling region. The maximum filling region is used to determine the farthest filling distance of the original true label geometric distribution. In this embodiment, the original true label depth map is dilated using a 115×115 square structuring element S2 to obtain the maximum filling region M. max When there is no depth value within a 115-pixel radius, it indicates that the pixel is located in an area that cannot be marked even at an offset of one-third of the screen height. These pixels are typically located too far from the camera plane, exceeding the 255-meter range that the LiDAR can capture, and are beyond the model's prediction range, thus needing to be disregarded.

[0082] After two dilation operations, the pseudo-label filling region M can be obtained by XORing the maximum filling region and the original dense region of true labels. d The calculation method is as follows:

[0083] M d =M min XORM max

[0084] After confirming the pseudo-label filling area, it is necessary to perform filling and mixing within the pseudo-label filling area. To this end, this invention utilizes random masking to control the density ratio and depth ratio during filling and mixing. The overall random mask M consists of a near-range random mask M1 and a far-range random mask M2. The near-range random mask M1 and the far-range random mask M2 randomly select elements for masking within their respective depth ranges. To control the number of masked elements and the proportion of labels in different depth regions, a probability hyperparameter can be set. Then, a random value is derived using a normal distribution, and each element is judged sequentially. This ensures that the density of the selected elements corresponds to the probability of the probability hyperparameter, thereby achieving control over the density ratio of genuine and pseudo labels and the label proportion at different depths.

[0085] Specifically, the steps for generating a random mask include:

[0086] Set a first threshold, and divide the second pseudo-label region into different depth partitions according to the first threshold.

[0087] A corresponding second threshold is set for each of the depth partitions; the second threshold is the pseudo-label density hyperparameter in the corresponding depth partition.

[0088] Based on the second threshold, a mask is generated to occlude or not occlude elements in each of the depth partitions; for example, random values ​​are derived through a normal distribution, and each element in the depth partition is judged sequentially; when the corresponding random value meets the second threshold, occlusion or non-occlusion is performed.

[0089] Current benchmark tests only consider depth prediction within 80 meters. Based on the divergence of the transformation, this paper sets 40 meters as the mid-to-long-distance threshold r. max That is, the first threshold. Depth partitions exceeding the threshold are considered medium-to-long-distance depth partitions, while those not exceeding the threshold are considered short-distance or near-distance depth partitions.

[0090]

[0091]

[0092] in, σ1 represents the occlusion element value in the near-field masking. A value of 1 indicates no occlusion, and a value of 0 indicates occlusion. σ1 is a random value derived from the normal distribution, and t1 is the short-range cutoff threshold used to control the probability of each near-field point being selected.

[0093]

[0094] in, σ2 represents the occlusion element value in the near-field masking, where a value of 1 indicates no occlusion and a value of 0 indicates occlusion; σ2 is a random value derived from the normal distribution, and t2 is the mid-to-long-range cutoff threshold, used to control the probability of each mid-to-long-range point being selected.

[0095] In order to perform calculations simultaneously with the areas outside the pseudo-tag filling area, tags outside the pseudo-tag filling area, i.e., valid real tags, are masked to prevent filling from occurring.

[0096]

[0097] in, The value of the occluded element in the valid true tag mask is 1, which means no occlusion, and 0 means occlusion; G i,j These are the pixel values ​​in the original true-label depth image.

[0098] In this embodiment, the filling area of ​​the pseudo-label is confirmed by dilation, so that the filling of the pseudo-label can conform to the distribution of the real label. In addition, the density ratio of the real and pseudo labels and the label ratio at different depths can be further determined by hyperparameters t1, t2 and r. max The process is controlled. During training, the distillation framework first calculates the SILog loss of the pseudo-label map on the original real labels. When this loss exceeds an initially set threshold t0, the pseudo-labels will not be mixed with the original real labels. As training progresses, the training SILog of the student model will continuously decrease, and the threshold t0 will also gradually decrease to reduce the additional error interference from supplementary information.

[0099] Furthermore, when training the disparity depth module, this method also designs a size-invariant loss function weighted by the real and original labels to enable the SILog loss function to balance the gradient loss of the pseudo and real labels from the denser mixed depth map.

[0100] The following is the expression for the weighted SILog loss function. This method adds ν as a weight to the original loss function to weight and integrate the training errors of pseudo-labels and real labels.

[0101]

[0102] in, It is the true label depth value on pixel i. It is the predicted depth value on pixel i. It is the difference between the predicted depth value on pixel i and the true label. It is the pseudo-label depth value at pixel j. It is the predicted depth value at pixel j. ν is the difference between the predicted depth value at pixel j and the pseudo-label. w represents the weight of the pseudo-label in the loss function, n is the total number of pixels, and ν represents the recall parameter.

[0103] Before the transformed depth map is input into the monocular student model, the pseudo-labels generated by the teacher model and the original true labels are mixed. This adaptive filling strategy can reduce the inherent error of the additional pseudo-labels while taking into account the geometric distribution of the original true labels.

[0104] To fully leverage the advantages of single-view and multi-view estimation, this method adopts the label distillation approach commonly used in heterogeneous models, employing a teacher model to semi-supervised train the student model. In practice, a more accurate but computationally resource-intensive multi-view stereo disparity model is used as the teacher model within the framework. This model takes multiple color real-world images as input and ultimately returns a disparity map corresponding to one of the viewpoints. To further transform the disparity map into a depth map, experiments revealed that using classic deterministic algorithms for disparity-depth conversion suffers from high noise, low accuracy, and limited depth. Therefore, a disparity-depth conversion module based on the probability distribution mapping between the learned disparity map and the original discrete depth label map is established. Compared to deterministic formula conversion, Dis2Dep demonstrates in experiments that it further improves performance on accuracy and stability metrics such as SILog and RMSE for teacher models of any accuracy. Dis2Dep can also be trained on datasets with relatively small amounts of data, ensuring its low data requirements, which is crucial for further reducing computational and data collection costs.

[0105] Subsequently, to better blend the original true labels and the transformed dense pseudo-label depth maps generated by the teacher model, this method proposes a density-adaptive filling strategy. This strategy considers the density ratio, geometric distribution, and depth filling interval between the true and pseudo labels, and applies it in practice using morphological masking. Experimentally, the adaptive filling strategy effectively reduces the errors carried by the teacher model and improves the additional supervision efficiency of the teacher model for the student model. Overall, using the knowledge distillation-based binocular stereo disparity to monocular depth estimation teacher-student framework designed in this method, the student model achieves an average SILog error reduction of 1.4% during training across the entire field of view. In low field-of-view tasks, BtS achieves a 29.9% reduction in SILog error after distillation compared to simply using the original labels. Therefore, this distillation framework can provide effective supervision outside the sensor's field of view.

[0106] The table below shows the comparison data of various monocular models before and after knowledge distillation in this embodiment.

[0107] Table 1. Comparison of Knowledge Distillation Effects under Multiple Models (Reference Table)

[0108]

[0109] As can be seen from the table above, this invention uses morphological masking guided by an adaptive density filling strategy to train a student model with a mixed label map of the original discrete true labels and the transformed dense pseudo labels, which can reduce the index errors of size invariant error SILog and mean square error RMSE globally.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A novel knowledge distillation method for binocular parallax to monocular depth, characterized in that, Includes the following steps: S1: Training the disparity-depth transformation module: Construct first training data and input the first training data into a pre-trained binocular stereo disparity teacher model to obtain a disparity map; the first training data includes a left-eye color image and a right-eye color image; The disparity map and the original disparity reference map are input into the disparity depth conversion module to obtain a pseudo-label depth map; wherein, the disparity reference map refers to the left-eye color image or the right-eye color image used as a reference benchmark during disparity calculation; The disparity depth conversion module includes a backbone model feature extractor and a multi-scale feature fusion generator. The backbone model feature extractor is used to extract multi-scale features from the original color image to obtain a multi-scale color feature map. The multi-scale feature fusion generator is used to blend the disparity map and the multi-scale color feature map layer by layer to output a depth map. The step of layer-by-layer mixing includes: performing multiple consecutive downsampling operations on the disparity map using a downsampling module to obtain downsampled feature maps at multiple scales; concatenating the downsampled feature maps with the corresponding scale color feature maps dimensionally and then upsampling to obtain a fused feature map; further concatenating the fused feature map with the corresponding scale color feature map and the downsampled feature map and then upsampling; and so on, to obtain the final fused feature map. The loss function is calculated based on the pseudo-label depth map, and the parameters of the disparity depth conversion module are updated. S2: The pseudo-label depth map output by the disparity depth conversion module and the original true label depth map are mixed to obtain a mixed label depth map; the mixing process uses a morphological masking module for density adaptive filling, and the specific steps include: The original true label depth map is binarized and dilated once based on the true labels. The dense region of the original true labels in the original true label depth map is extracted through the first dilation and used as the unfilled region. A secondary dilation is performed on the true labels in the original true label depth map to obtain the maximum filling region, which is used to determine the farthest filling distance of the geometric distribution of the original true labels. The first pseudo-label filling area is determined based on the unfilled area and the maximum filled area; In the pseudo-label depth map, a second pseudo-label filling region corresponding to the first pseudo-label filling region is marked; In the second pseudo-label filling area, a random mask is generated; Under the random mask, the first pseudo-label filling region and the second pseudo-label filling region are mixed to obtain a mixed label depth map; S3: Training a monocular depth estimation student model: Construct a second training data set and train the monocular depth estimation student model using the second training data set; the second training data set includes the hybrid labeled depth map and the corresponding original reference map.

2. The novel knowledge distillation method for binocular parallax to monocular depth according to claim 1, characterized in that, S1 further includes: after mixing the pseudo-label depth map and the original true label using a density adaptive filling strategy, calculating the size loss function.

3. The novel knowledge distillation method for binocular parallax to monocular depth according to claim 1, characterized in that, The generation of the random mask includes: Set a first threshold, and divide the second pseudo-label filling area into different depth partitions according to the first threshold; A corresponding second threshold is set for each of the depth partitions; the second threshold is the pseudo-label density hyperparameter in the corresponding depth partition. Based on the second threshold, a mask is generated to either occlude or not occlude elements in each of the depth partitions.

4. The novel knowledge distillation method for binocular parallax to monocular depth according to claim 3, characterized in that, The step of generating a mask based on the second threshold and occluding elements in each of the depth partitions includes: Random values ​​are derived from a normal distribution, and each element in the depth partition is judged sequentially; when the corresponding random value meets the second threshold, occlusion or non-occlusion is performed.

5. A novel knowledge distillation method for converting binocular parallax to monocular depth according to claim 1, characterized in that, The morphological masking module is specifically as follows: ； Where T is the hybrid label depth map output by the morphological masking module; G is the original discrete depth map corresponding to the original true label; M represents the pseudo-label filling area, M is the random mask, and K is the pseudo-label depth map.

6. A novel knowledge distillation method for converting binocular parallax to monocular depth according to claim 1 or 5, characterized in that, The method for confirming the pseudo-label filling area is as follows: ； in, For effective true label masking, and These represent the expansion structures used for primary and secondary expansion, respectively.

Images