An RGB monocular depth estimation method based on meta-learning dual-flow loss balancing

Abstract

This invention proposes an RGB monocular depth estimation method based on meta-learning dual-stream loss equalization. The steps include: acquiring monocular images and obtaining standardized RGB images through multi-stage image preprocessing; using a differentiated network structure of a dual-stream feature extraction module to extract shallow geometric features and deep semantic features in parallel from the standardized RGB images; constructing a cross-modal heterogeneous interaction module based on multi-head cross-attention to perform deep fusion and dynamic adaptation of shallow geometric features and deep semantic features, obtaining the final fused feature map; selecting the original dataset corresponding to the training set and obtaining the equalized total loss through meta-training; obtaining the trained depth estimation model through end-to-end joint training using the equalized total loss; inputting the RGB monocular image to be processed into the trained depth estimation model and outputting the final depth map. This invention can achieve accurate mapping from monocular images to 3D depth maps, adapt to complex interference scenes, and eliminate dependence on real depth labels.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision and deep learning, and in particular to an RGB monocular depth estimation method based on meta-learning two-stream loss equalization. Background Technology

[0002] Depth estimation is a core foundational task in computer vision. Its core objective is to deduce the 3D depth distribution of a scene from a 2D RGB image, providing crucial spatial dimension support for downstream vision tasks. Its performance directly determines the performance ceiling of a 3D perception system. In real-world scenarios, acquiring high-quality depth labels relies on specialized equipment such as LiDAR and depth cameras. This not only incurs high hardware costs but also significantly restricts data acquisition due to environmental and scene limitations, severely hindering the large-scale deployment and generalized application of depth estimation technology.

[0003] Existing depth estimation techniques can be mainly divided into three categories: Supervised learning methods rely on large-scale datasets with real depth labels. Although they can achieve high accuracy on specific datasets, label dependence leads to poor generalization ability, and the high cost of labeling limits the scale and diversity of training data; Self-supervised learning methods construct supervision signals by using stereo matching of binocular images or inter-frame motion estimation of video, which does not require manual labeling, but essentially can only handle multi-image input scenarios and cannot meet the single-image depth estimation needs of static image analysis, monocular camera deployment, etc.; Unsupervised learning methods are trained through indirect supervision signals such as image reconstruction, which get rid of the dependence on real labels, but generally use manually designed fixed loss functions, lack scene adaptation ability, and are prone to problems such as blurred edges of depth maps, loss of details, abrupt changes in depth values, or structural misalignment, making it difficult to balance accuracy and robustness.

[0004] In recent years, convolutional neural networks and visual Transformers have made breakthrough progress in RGB monocular depth estimation, propelling it into the end-to-end modeling era with their powerful multi-scale feature extraction and nonlinear mapping capabilities. However, existing end-to-end methods still suffer from three major bottlenecks: First, most models fail to achieve parallel extraction and depth interaction of depth geometric features and RGB semantic features, resulting in poor adaptability to heterogeneous features and insufficient feature representation capabilities for complex scenes. Second, the design of a single loss function does not consider the differences in supervision requirements of different feature streams, leading to an imbalance in supervision signals and a tendency for optimization bias. Third, the meta-learning mechanism is not fully integrated, making it difficult to explore the dynamic mapping relationship between scene features, loss weights, and model parameters, resulting in weak adaptability of the model to new scenes and limited generalization performance. Therefore, there is an urgent need to construct a novel monocular depth estimation method that integrates "meta-learning driven, loss balancing design, and depth cross-modal interaction" to break free from dependence on real depth labels and achieve more accurate, robust, and generalized depth information recovery in complex scenes.

[0005] Patent application number 202110570260.2 discloses a single-target tracking method and a computer-readable medium. The single-target tracking method includes: Step 1: acquiring a video frame sequence to be detected and performing data augmentation preprocessing on all video frames; Step 2: inputting the video frame sequence into a trained single-target detection model; Step 3: the single-target detection model outputs the target classification result and the corresponding bounding box estimate; Step 4: filtering the single target to be tracked from the target classification result, outputting its bounding box estimate, and completing the single-target tracking. Compared with the prior art, the above invention has advantages such as good target tracking performance. However, the core function of this invention is to locate and track the bounding box of the target in the video frame sequence, which only realizes the target position tracking in a two-dimensional plane and does not involve the estimation of three-dimensional depth information, thus failing to provide spatial dimension support for downstream vision tasks. At the same time, this invention relies on the input of the video frame sequence and cannot handle the depth perception requirements of static RGB monocular images. It also has weak adaptability to complex scenes such as texture interference, specular reflection, and low lighting, and its generalization performance is limited. Therefore, this invention has core shortcomings such as lack of depth estimation capability, poor scene adaptability, and inability to handle single image input. Summary of the Invention

[0006] To address the technical problems of insufficient feature representation ability and poor generalization of RGB monocular depth estimation methods in complex scenes, this invention proposes an RGB monocular depth estimation method based on meta-learning two-stream loss equalization. This method aims to eliminate reliance on real depth labels. Through a closed-loop logic of "image detection and enhancement - parallel feature extraction via dual-stream path - cross-modal heterogeneous feature interaction - meta-learning two-stream loss equalization - feedback adaptive parameter update - depth map generation and execution," a temporally differentiated dual-stream path and dynamic loss optimization mechanism are constructed to achieve accurate mapping from RGB monocular images to high-quality depth maps. This application focuses on RGB monocular depth estimation, enabling accurate mapping from 2D monocular images to 3D depth maps. The meta-learning two-stream loss equalization mechanism adapts to complex interference scenes, eliminating reliance on real depth labels.

[0007] To achieve the above objectives, the technical solution of this invention is implemented as follows: An RGB monocular depth estimation method based on meta-learning dual-stream loss equalization, comprising the following steps:

[0008] Step 1: Acquire RGB monocular images of the target scene, and obtain standardized RGB images through multi-stage image preprocessing to form a training set;

[0009] Step 2: Employ the differentiated network structure of the dual-stream feature extraction module to extract shallow geometric features and deep semantic features in parallel from the input standardized RGB image;

[0010] Step 3: Construct a cross-modal heterogeneous interaction module based on multi-head cross-attention to perform deep fusion and dynamic adaptation of shallow geometric features and deep semantic features to obtain the final fused feature map;

[0011] Step 4: Select the original dataset corresponding to the training set, and mine the mapping relationship between scene features and loss weights through meta-training to obtain the balanced total loss;

[0012] Step 5: Through end-to-end joint training and training set, optimize module parameters using backpropagation with equalized total loss to obtain the trained depth estimation model;

[0013] Step 6: Input the RGB monocular image to be processed into the trained depth estimation model. After passing through the dual-stream feature extraction module, the cross-modal heterogeneous interaction module and the decoder, the final depth map is output.

[0014] Preferably, the multi-stage image preprocessing method is as follows:

[0015] Based on image sharpness evaluation metrics, valid images without motion blur and severe noise are selected, while invalid data that are blurry, overexposed, or underexposed are removed.

[0016] Distortion correction is performed on the effective image, which is scaled to a uniform size according to the input requirements of dual-channel feature extraction, and bilinear interpolation is used to preserve the edge details and texture features of the image.

[0017] Noise reduction was performed by Gaussian filtering and bilateral filtering in sequence;

[0018] Random brightness adjustments and contrast enhancements are performed to improve the model's robustness to lighting changes. Finally, pixel normalization is applied to obtain a normalized RGB image. .

[0019] Preferably, the differentiated network structure includes a fast geometric feature path and a precise semantic feature path. The fast geometric feature path adopts a lightweight backbone network to quickly extract shallow geometric features of the scene with low computational complexity. The precise semantic feature path adopts a deep ResNet-50 architecture to accurately extract deep semantic features with higher computational complexity.

[0020] Preferably, the implementation method for generating a final fused feature map with both geometric integrity and semantic richness using the multi-head cross-attention cross-modal heterogeneous interaction module is as follows:

[0021] ① Deep semantic features are scaled using interpolation Upsampling is then performed, followed by 1×1 convolution to reduce the number of channels and obtain the aligned semantic feature map. ;

[0022] ② Shallow geometric features As query vector, semantic feature map As key and value vectors, a linear projection layer is used to generate query, key, and value matrices, respectively. Attention weights are obtained by performing a dot product operation on the query and key matrices and normalizing the result. These attention weights are then weighted and summed with the value matrices to obtain the interactive attention features for each attention head. The cross-attention features of all attention heads are concatenated along the channel dimension, and a linear projection layer is used for feature fusion and dimensional adjustment to generate the interactive geometric features. ;

[0023] ③ Similarly, semantic feature maps As a query vector, shallow geometric features As key vectors and value vectors, they generate semantic features after interaction. .

[0024] ④ An adaptive weighted fusion strategy is used to generate fusion weights. Then the fusion features Features are fused using 3×3 convolutional layers and batch normalization layers. Perform feature smoothing and output the final fused feature map.

[0025] Preferably, the lightweight backbone network includes multiple sequentially connected depthwise separable convolutional blocks, batch normalization, and activation layers, and shallow geometric features are obtained by adjusting the channel dimension through 1×1 convolution operations. ;

[0026] The deep ResNet-50 architecture includes multiple sequentially connected residual block groups, channel attention modules, and spatial attention modules. Deep semantic features are obtained by adjusting the channel dimensions through 1×1 convolution operations. ;

[0027] The adaptive weighted fusion strategy generates fusion weights. The method is as follows: calculate the geometric features after interaction. With semantic features Feature similarity is calculated using the cosine similarity method to determine the geometric features after interaction. With semantic features The cosine value of the corresponding pixel is used as the global average as the feature similarity. This feature similarity is then input into the Sigmoid activation function to generate the fusion weights. .

[0028] Preferably, the total equalization loss is:

[0029] ;

[0030] in, , This is the balance coefficient for the loss components; For pixel-level L1 reconstruction loss of shallow geometric features, For shallow geometric features, depth smoothing loss, For multi-scale structural similarity loss of deep semantic features, For edge consistency loss of deep semantic features, and These are the dynamic weights for shallow geometric features and deep semantic features, respectively. .

[0031] Preferably, the pixel-level L1 reconstruction loss is: ;

[0032] in, This is the pixel matrix of the original image after denoising. Based on shallow geometric features The reconstructed RGB image has H, W, and C as its height, width, and number of channels, respectively; i represents the height index, j represents the width index, and k represents the channel index.

[0033] The depth smoothing loss is: ;

[0034] in, Based on shallow geometric features The generated initial depth map, , These are the first-order gradient operators in the horizontal and vertical directions, respectively;

[0035] The multi-scale structural similarity loss formula is as follows: ;

[0036] in, Based on deep semantic features The generated initial depth map, Initial depth map Compared with the original image pixel matrix The structural similarity index, The multi-scale similarity index is obtained by weighting the structural similarity indices at different scales:

[0037] The edge consistency loss is ;

[0038] in, To extract the edge map of the original image pixel matrix I based on the Canny edge detection operator, Initial depth map Edge map;

[0039] The dynamic weight and The method for obtaining the dynamic weights is as follows: the final fused features output by the cross-modal heterogeneous interaction module are input into the meta-learning backbone network, and two initial weight values ​​are output through the output layer of the meta-learning network; the initial weight values ​​are then subjected to Softmax normalization to obtain the dynamic weights. and .

[0040] Preferably, the method for mining the mapping relationship between scene features and loss weights through meta-training is as follows:

[0041] ① Meta-dataset partitioning: Select the original dataset corresponding to the training set in step 1, and divide it into a meta-training set and a meta-test set according to the proportion. The meta-training set is used to learn the adjustment rules of the loss weight of the meta-learning network, and the meta-test set is used to verify the generalization performance of the model.

[0042] ② Meta-learning network: A two-layer Restormer block is used as the meta-learning backbone network. The Restormer block combines local enhanced window attention with global long-distance attention to efficiently capture the mapping relationship between scene features and loss weights.

[0043] ③ Meta-training objective: Through episode-based training of the meta-training set, minimize the loss function error of the query set contained in the episode, so that the meta-learning network can quickly output the optimal loss weights based on scene features.

[0044] The implementation method of step 5 includes: configuring the optimizer and setting effective parameters; adopting a warm-up + cosine annealing learning rate strategy; and balancing the total loss. The gradients of the parameters for the differentiated network structure and the cross-modal heterogeneous interaction module are calculated separately, and the parameters are updated synchronously at an update rate of 2:2:1. After each training round, the average loss of the meta-test set is calculated. If the decrease in the loss of the meta-test set for four consecutive rounds is less than a certain value, the loss is considered lost. If the model converges, the optimal parameters of each module are saved; otherwise, iterative training continues, with a total of no more than 100 training rounds, to obtain the trained depth estimation model.

[0045] Preferably, the method for outputting the final depth map is as follows: the RGB monocular image to be processed is processed through multi-stage image preprocessing to obtain a standardized RGB image. Standardized RGB images The fast geometric feature path and the precise semantic feature path of the parallel input differential network structure output shallow geometric features respectively. With deep semantic features The final fused feature map is generated through the cross-modal heterogeneous interaction module. Final fused feature map The input decoder and deconvolutional layers progressively upsample the feature maps to match the input RGB image. With consistent dimensions, the output convolutional layer reduces multi-channel features to a single channel, generating an initial depth map. The initial depth map is transformed using a linear transformation. By mapping the depth range to the actual scene, a normalized depth map is obtained. ; Using RGB images Guided filtering as a guide map for normalized depth maps The process involves smoothing the image, filling in tiny holes in the smoothed depth map using morphological closing operations, and then outputting the final depth map. .

[0046] Preferably, shallow geometric features The input consists of a decoder composed of deconvolutional layers and 1×1 convolutions, which process shallow geometric features through the deconvolutional layers. The image is upsampled to the same size as the original image pixel matrix I, then the number of feature channels is adjusted to 3 channels through a 1×1 convolution. Finally, a linear activation function is used to map the feature values ​​to the pixel value range of [0, 255], resulting in a shallow geometric feature-based image. Reconstructed RGB image ;

[0047] shallow geometric features The input to the decoder is progressively upsampled through deconvolutional layers to the same size as the pixel matrix I of the original image. Then, a 1×1 convolution is used to reduce the dimensionality of the multi-channel features to a single channel. Finally, the Sigmoid activation function is used to map the feature values ​​to a normalized depth range of [0,1] to obtain the initial depth map. ;

[0048] The structural similarity index is obtained by calculating the brightness similarity, contrast similarity, and structural similarity of corresponding regions in two images, and then weighting and summing them to obtain the final similarity score; the multi-scale similarity index is obtained by comparing the pixel matrix I of the original image with the initial depth map. Multi-scale downsampling is performed, the structural similarity index is calculated at each scale, and then the MS-SSIM value is obtained by weighted averaging of the structural similarity index values ​​at all scales.

[0049] The activation layer is implemented using the ReLU6 activation function;

[0050] The residual block group performs residual connections and convolutional feature extraction on the input standardized RGB image;

[0051] The fusion weight The closer the value is to 0.5, the more balanced the fusion of geometric and semantic features is achieved; fusion weight The weights are biased towards the side with better feature representation: if geometric features are better, the weights are fused. If the weights approach 1, then the semantic features are better, and the weights are fused. Approaching 0;

[0052] Parameters are configured using the AdamW optimizer;

[0053] The first 5 rounds of the learning rate are the warm-up phase, with the learning rate starting from... linear increase to After the warm-up phase, the learning rate decays using a cosine annealing strategy over a period of 20 rounds, with a minimum learning rate of [missing value]. ;

[0054] The formula for updating parameters is: ;in, The parameters before the update. For the updated parameters, The current learning rate, To balance the total loss on parameters The gradient;

[0055] The initial depth map ;in, , , These are the first, second, and third deconvolution layers, respectively. For output convolutional layers, Use the Sigmoid activation function;

[0056] The normalized depth map for: ;in, This is the depth scaling factor. This is the depth offset coefficient;

[0057] The fast geometric feature pathway uses a lightweight ShuffleNetV2×1.0 architecture as the backbone network, which includes one initial convolutional layer and four ShuffleNetV2 unit groups. The number of channels is unified to 128 dimensions through 1×1 convolution and a shallow geometric feature map is output. The Shuffle unit of the ShuffleNetV2 unit group randomly shuffles and recombines the feature channels after convolution of different groups through a channel shuffling mechanism. The traditional pointwise convolution is replaced by pointwise group convolution, which reduces the number of network parameters and computational cost while retaining the core feature extraction capability.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1) An innovative temporally differentiated dual-stream architecture is designed, where the two paths complement each other temporally (the fast path outputs basic features first, and the precise path outputs enhanced features later) to achieve efficient parallel extraction of heterogeneous features, solving the problem of insufficient adaptability of a single feature stream to complex scenarios; 2) A cross-modal heterogeneous interaction mechanism is constructed to achieve deep fusion of geometric and semantic features, forcibly constraining the structural consistency and semantic correlation between the depth map and the original image, significantly improving the adaptability and fusion accuracy of heterogeneous features; 3) A meta-learning-driven dual-stream loss balancing strategy is constructed, which learns the adjustment rules of loss weights under different scenarios through the meta-training set, and balances the supervision intensity of the two paths through dynamic weight allocation, avoiding the optimization bias dominated by a single loss, and improving the model's cross-scenario generalization ability; 4) Closed-loop training and inference without real label dependence are achieved, and the output depth map has both geometric accuracy and structural consistency through autonomous learning of the inherent rules of depth estimation, making it more suitable for the actual needs of various downstream visual tasks. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is an overall flowchart of the method of the present invention;

[0062] Figure 2 This is a diagram showing the processing results of the method of the present invention for texture interference scenarios.

[0063] Figure 3 This is a diagram showing the processing results of the method of the present invention for a specular reflection scenario.

[0064] Figure 4 This is a diagram showing the processing results of the method of the present invention in low-light scenarios. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, an RGB monocular depth estimation method based on meta-learning two-stream loss equalization includes the following steps:

[0067] Step 1: Acquire RGB monocular images of the target scene, and obtain standardized RGB images through multi-stage image preprocessing enhancement to form a training set.

[0068] The core of step 1 lies in acquiring a valid RGB monocular image and performing multi-stage preprocessing to eliminate noise, correct pixel distribution, and enhance feature representation, thereby providing a high-quality, interference-resistant input image for the dual-channel system. The specific implementation process is as follows:

[0069] ① Acquire RGB images of the target scene using a monocular camera, and select effective images without motion blur and severe noise based on image sharpness evaluation metrics, while eliminating invalid data that is blurry, overexposed, or underexposed; thus reducing interference with model training.

[0070] ② The effective image is distorted and then scaled to a uniform size according to the input requirements of the dual-channel system. Bilinear interpolation is used to calculate the gray value of the target pixel and weighted average the information of the adjacent pixels, effectively preserving the edge details and texture features of the image and avoiding structural distortion caused by size scaling.

[0071] ③ Adopt a combined denoising strategy of "Gaussian filtering + bilateral filtering": first suppress Gaussian noise through Gaussian filtering, and then remove salt-and-pepper noise and texture redundancy while preserving edge details through bilateral filtering;

[0072] ④ Through random brightness adjustment (brightness variation range ±15%) and contrast enhancement (gamma correction) To improve the model's robustness to changes in illumination, pixel normalization is performed to eliminate differences in pixel value distribution. The pixel normalization process follows the formula:

[0073] ;

[0074] in, A standardized RGB image pixel matrix; The pixel matrix of the original image after denoising; and These represent the pixel mean and standard deviation of the original image across the RGB channels, respectively, obtained through statistics from the training set. The specific statistical process is as follows: Iterate through all denoised valid images in the training set constructed in step 1, count the values ​​of all pixels in each of the three RGB channels, and calculate the arithmetic mean of the pixel values ​​for each channel, as shown below. , , That is, the pixel mean of the corresponding channel. Simultaneously, calculate the arithmetic mean of the squared differences between the pixel value of each channel and the mean of that channel, then take the square root to obtain the result. , , That is, the standard deviation of the corresponding channel. The statistical process needs to cover all images in the training set to ensure that the mean and standard deviation accurately reflect the pixel distribution characteristics of the training set. Finally, this pixel normalization operation maps pixel values ​​to a distribution range with a mean of 0 and a variance of 1, ensuring the stability and consistency of dual-path feature extraction.

[0075] Step 2: Using a dual-stream feature extraction module, shallow geometric features and deep semantic features are extracted in parallel from the input standardized RGB image through differentiated network structure configuration, resulting in temporally complementary heterogeneous feature pairs.

[0076] Specifically, a Fast Geometric Feature Path (FGFP) and a Precise Semantic Feature Path (PSFP) are constructed. The parallel extraction of heterogeneous features is achieved through differentiated network structure design. The temporal differences between the two paths (FGFP has a faster response speed than PSFP) and feature complementarity provide a foundation for subsequent interactive fusion.

[0077] The Fast Geometric Feature Path (FGFP) employs a lightweight backbone network, reducing computational cost while preserving feature representation capabilities. It rapidly extracts shallow geometric features of the scene with low computational complexity, and by combining it with the lightweight backbone network, the response time is kept below 30ms. The calculation process satisfies the following formula:

[0078] ;

[0079] in, It is the output shallow geometric feature; This represents a 1×1 convolution operation, used for adjusting the channel dimension. This represents the ReLU6 activation function, which restricts the output range to [0,6] to avoid gradient vanishing. This is a batch normalization operation used to stabilize feature distribution; Indicates the first Each depthwise separable convolutional block extracts features from the input image. The lightweight backbone network consists of multiple sequentially connected depthwise separable convolutional blocks, batch normalization, activation layers, and finally, channel dimension adjustment through a 1×1 convolution operation.

[0080] Deep geometric features refer to all depth-related geometric information in a scene, such as target contours, spatial layout, relative positions, and depth gradients. Shallow geometric features are an important component of deep geometric features, specifically referring to basic geometric features in the shallow layers of an image that are easily extracted and closely related to depth, such as the approximate outline of a target and a simple spatial layout. FGFP extracts shallow geometric features, providing a foundation for further mining and accurate estimation of subsequent depth information, avoiding the surge in computation and feature redundancy caused by directly extracting deep deep geometric features.

[0081] The Precise Semantic Feature Path (PSFP) employs a deep ResNet-50 architecture to accurately extract deep semantic features (such as object category, fine-grained edges, and texture details) with high computational complexity, enhancing the detail representation and structural consistency of the depth map, while keeping the response time within 80ms. The calculation process satisfies the following formula:

[0082] ;

[0083] in, It is the output deep semantic feature; This represents a 1×1 convolution operation, used for adjusting the channel dimension. and These are the channel attention module and the spatial attention module, respectively. Indicates the first Each residual block group performs residual connections and convolutional feature extraction on the input image. The deep ResNet-50 architecture includes multiple sequentially connected residual block groups, channel attention modules, and spatial attention modules, and finally adjusts the channel dimensions through a 1×1 convolution operation.

[0084] Through a differentiated structural design and parallel computation using dual-pathway architecture: FGFP employs a lightweight architecture and efficient convolutional operations, resulting in a fast response time (≤30ms) and prioritizing the output of shallow geometric features (basic geometric information); PSFP employs a deep ResNet-50 architecture, which has higher computational complexity and a slightly slower response time (≤80ms), but can output more accurate and richer deep semantic features (fine-grained detail information); the two pathways process the same input image in parallel, with FGFP outputting basic features first and PSFP outputting enhanced features later. The two are temporally complementary, and the feature types (shallow geometric features and deep semantic features) are heterogeneous, ultimately forming a temporally complementary heterogeneous feature pair.

[0085] Step 3: Construct a cross-modal heterogeneous interaction module to perform deep fusion and dynamic adaptation of heterogeneous feature pairs output by dual-channel flow, and enforce structural consistency and semantic relevance.

[0086] This step constructs a cross-modal heterogeneous interaction module based on Multi-Head Cross Attention (MHCA) to realize shallow geometric features. With deep semantic features Modal alignment, bidirectional guidance, and dynamic fusion are used to generate fused feature maps that possess both geometric integrity and semantic richness. The specific implementation process is as follows:

[0087] ① First, deep semantic features are scaled using interpolation. Upsampling is then performed, followed by 1×1 convolution to reduce the number of channels and obtain the aligned semantic feature map. To achieve shallow geometric features With deep semantic features Feature dimension alignment;

[0088] ② Shallow geometric features With semantic feature map Input MHCA module, shallow geometry features As a query vector, semantic feature map As key vectors and value vectors, through a linear projection layer Generate query matrix, key matrix, and value matrix respectively. The weight of each attention head is obtained by performing a dot product operation between the query matrix and the key matrix and then normalizing it, and the calculation satisfies the formula:

[0089] ;

[0090] in, The first The query matrix, key matrix, and value vector (matrix) of each attention head. For the attention head dimension, The function is used to normalize the weights; T represents the matrix transpose. This represents the cross-attention feature of the i1th attention head.

[0091] ③ The cross-attention output features of the attention heads are concatenated using a channel-dimensional concatenation method, through a linear projection layer ( Generate interactive geometric features The first step involves inputting shallow geometric features as the query vector Q and semantic feature maps as K and V into the attention heads of the MHCA module. Attention weights are obtained through the dot product of the query vector Q and the key vector K, followed by Softmax normalization. These attention weights are then weighted and summed with the value vector V to obtain the interactive attention features for each attention head. The second step involves concatenating the interactive attention features of all attention heads along the channel dimension and inputting them into a linear projection layer (WO) for feature fusion and dimension adjustment. This process eliminates feature redundancy during concatenation, ultimately outputting the interactive geometric features. Similarly, swap the inputs of the query vector Q and the value vector K. As the query vector Q, As key vector K and value vector V, generate semantic features after interaction. .

[0092] ④ An adaptive weighted fusion strategy is adopted, and the fusion weights are generated through the Sigmoid function. , Geometric features after interaction With semantic features The feature similarity is adaptively adjusted; the higher the similarity, the higher the fusion weight. The closer it is to 0.5, the better. Calculate the geometric features after interaction. With semantic features The feature similarity is calculated using the cosine similarity method, which calculates the cosine value of corresponding pixels in two feature maps and takes the global average as the final feature similarity. The feature similarity range is [0,1]. This feature similarity is then input into the Sigmoid activation function to generate fusion weights. Among them, the higher the feature similarity, the stronger the geometric features. With semantic features The stronger the feature consistency, the higher the fusion weight. The closer the similarity is to 0.5, the more balanced the fusion of the two types of features is achieved; the lower the feature similarity, the greater the difference between the two types of features, and the higher the fusion weight. The weights will be biased towards the feature representation that is better; for example, if the geometric features are better, the weights will be fused. If the weights approach 1, then the semantic features are better, and the weights are fused. Approaching 0 ensures the effectiveness of the fused features. The fusion process satisfies the formula: Fusion Features Finally, the fused features are processed using a 3×3 convolutional layer and a BN layer. Perform feature smoothing, remove fusion noise, and output the final fused feature map.

[0093] Fusion Feature Map The subsequent processing is divided into two scenarios, corresponding to the model training and inference stages respectively: (1) Training stage: fusing feature maps Input the dual-stream loss equalization module to generate the initial depth map. , This allows for the calculation of various loss components, providing supervisory signals for model parameter optimization; simultaneously, feature maps are fused. (2) Inference stage: fused feature maps The input decoder progressively upsamples the image to the same size as the input image through deconvolutional layers, and then reduces the dimensionality to a single channel through the output convolutional layer to generate the initial depth map. This provides a foundation for subsequent depth value scene adaptation and post-processing optimization, ultimately outputting a high-quality depth map.

[0094] Step 4: Construct a dual-stream loss equalization module. Through meta-training, mine the mapping relationship between scene features and loss weights, dynamically generate an adapted loss function, and obtain the equalized total loss.

[0095] The core objective of the meta-learning framework is to enable the model to learn how to learn. By learning the adjustment rules of loss weights under different scenarios through the meta-training set, the model can quickly and adaptively adjust the loss weights in new scenarios (meta-test set). The specific process is as follows:

[0096] ① Meta-dataset partitioning: Select indoor and outdoor scene datasets, and partition them proportionally into a meta-training set (containing various sub-scenes) and a meta-test set; Data reuse and scene coverage: The RGB monocular images of the target scene collected in step 1 are preprocessed to form the basic training set, covering various sub-scenes such as indoor and outdoor scenes (e.g., texture interference, specular reflection, low light, etc.); When partitioning the meta-dataset, select the original dataset corresponding to the training set in step 1 (e.g., NYU Depth V2, KITTI), and partition it proportionally (e.g., 4:1) into a meta-training set and a meta-test set. The meta-training set is used for the meta-learning network to learn the adjustment rules of the loss weights, and the meta-test set is used to verify the generalization performance of the model; The training set in step 1 and the meta-dataset are essentially related as "basic training data" and "meta-learning verification data". The scene coverage of the meta-dataset is consistent with that of the training set in step 1, ensuring that the loss weight adjustment rules learned by meta-learning can be adapted to the model trained on the training set in step 1, thereby improving the model's cross-scene generalization ability.

[0097] ② Meta-learning network: A two-layer Restormer Block is used as the meta-learning backbone network. The Restormer Block combines local enhanced window attention with global long-distance attention to efficiently capture the mapping relationship between scene features and loss weights.

[0098] ③ Meta-training objective: Through episodic training of the meta-training set (episode includes support set and query set), minimize the loss function error of the query set, so that the meta-learning network can quickly output the optimal loss weights based on scene features.

[0099] The dual-stream loss function design calculates the supervision loss of FGFP and PSFP respectively, and assigns dynamic weights generated by meta-learning to obtain the equalized total loss. . Specifically:

[0100] 1) Geometric feature path loss Focusing on the overall numerical accuracy and spatial smoothness of depth values, it comprises two components:

[0101] a. Pixel-level L1 reconstruction loss Based on image reconstruction supervision signals, measures the impact of shallow geometric features. The pixel difference between the generated reconstructed image and the original image is calculated using the following formula:

[0102] ;

[0103] in, Here is the original RGB image, and here is the pixel matrix of the original image after denoising in step 1. Based on shallow geometric features The reconstructed RGB image will contain shallow geometric features output by FGFP. The input is a simple decoder (consisting of 1×1 convolutional and deconvolutional layers), which uses the deconvolutional layers to extract shallow geometric features. The image is upsampled to the same size as the original RGB image I, then the number of feature channels is adjusted to 3 (RGB channels) through a 1×1 convolution. Finally, the feature values ​​are mapped to the pixel value range of [0, 255] through a linear activation function to obtain shallow geometric features. Reconstructed RGB image The core function of this reconstructed image is to perform pixel-level comparison with the original RGB image I, calculate the L1 reconstruction loss, and provide a supervisory signal for FGFP parameter optimization. H, W, and C represent the image height, width, and number of channels, respectively; here... The three-dimensional index of the image pixels is represented, where i represents the height index of the image (ranging from 1 to H, where H is the image height), j represents the width index of the image (ranging from 1 to W, where W is the image width), and k represents the channel index of the image (ranging from 1 to C, where C=3 corresponds to the RGB three channels). Used to locate a specific pixel in an image, such as i=1, j=1, k=1 representing the R channel of the first pixel in the top left corner of the image.

[0104] b. Depth smoothing loss To suppress noise and abrupt changes in depth values ​​in the depth map and ensure the spatial continuity of the depth distribution, the calculation formula is as follows:

[0105] ;

[0106] in, Based on shallow geometric features The generated initial depth map, initial depth map Acquisition and reconstruction of RGB images The reconstruction process is similar, using the shallow geometric features output by FGFP. The input to the decoder is progressively upsampled to the same size as the original image through deconvolutional layers, then multi-channel features are reduced to a single channel through 1×1 convolution, and finally the feature values ​​are mapped to a normalized depth range of [0,1] using the Sigmoid activation function to obtain the initial depth map. This depth map is generated based on shallow geometric features. Although it can reflect the basic depth layout of the scene, it lacks detail accuracy and needs to be further optimized by combining the semantic features and loss function of PSFP. , These are the first-order gradient operators in the horizontal and vertical directions, respectively. The two-dimensional pixel index represents the depth map, where i represents the height direction index of the depth map (ranging from 1 to H, where H is the depth map height, consistent with the original image height), and j represents the width direction index of the depth map (ranging from 1 to W, where W is the depth map width, consistent with the original image width). This index is used to locate a specific pixel in the depth map, corresponding to the pixel at the same location in the original image. It is used to calculate the depth gradient at that location and measure the spatial continuity of the depth distribution. This index is only used for calculating the depth smoothing loss and focuses on the spatial structural features of the depth map.

[0107] 2) Semantic feature path loss The focus is on the structural consistency and edge alignment between the depth map and the original image, comprising two components:

[0108] a. Multiscale structural similarity loss The MS-SSIM metric is used to comprehensively measure deep semantic features through multi-scale weighted calculations of brightness, contrast, and structural similarity. Initial depth map generated The structural similarity with the original image I is calculated using the following formula:

[0109]

[0110] Among them, the Structural Similarity Index (SSIM) is an indicator that measures the structural similarity between two images. It is calculated by weighting and summing the brightness similarity, contrast similarity, and structural similarity of corresponding regions in the two images to obtain the final similarity score (ranging from [0,1]). The closer the score is to 1, the more similar the structures of the two images are. The Multi-Scale Similarity Index (MS) refers to the SSIM calculated at different scales, that is, comparing the original image with the initial depth map. Multi-scale downsampling is performed, and the Structural Similarity Index (SSIM) is calculated at each scale. Then, a weighted average of the SSIM values ​​across all scales is taken to obtain the MS-SSIM value. Compared to single-scale SSIM, MS-SSIM can more comprehensively capture the multi-scale structural features of the image and more accurately measure the initial depth map. The structure is consistent with the original image I.

[0111] b. Edge consistency loss Edge map extraction of the original image based on the Canny edge detection operator Compared with the initial depth map edge map The formula for measuring the edge alignment between the two is:

[0112] ;

[0113] Meta-learning networks analyze the features of the current scene (such as texture density, illumination intensity, and reflection type) and output dynamic weights. and ( First, the fused feature map output by the cross-modal heterogeneous interaction module is... The input is a meta-learning backbone network (2-layer Restormer Block). The Restormer Block captures local scene features (such as texture density and edge details) through local enhancement window attention and captures global scene features (such as light intensity and reflection type) through global long-range attention, comprehensively analyzing the complexity and distribution of scene features. Then, the output layer (linear layer) of the meta-learning network outputs two initial weight values. '、 Finally, the initial weight values ​​are normalized using Softmax, i.e. =exp ( ') / [exp ( ')+exp ( ')], =exp ( ') / [exp ( ')+exp ( ')],make sure This allows for the reasonable allocation of dynamic weights, adapting to the supervision requirements of the current scenario. The total loss satisfies the formula:

[0114]

[0115] in, , The balance coefficient for the loss components was determined through experimental verification.

[0116] Step 5: Through end-to-end joint training and the training set in Step 1, optimize the parameters of all modules of the network by backpropagation using the equalized total loss. Without relying on the real depth label, iterate the training until the model converges to obtain the trained depth estimation model.

[0117] The specific implementation process of step 5 is as follows:

[0118] ① Configure the optimizer and set effective parameters to suppress overfitting; This invention uses the AdamW optimizer for parameter configuration. The specific implementation method is as follows: In the PyTorch framework, call the torch.optim.AdamW() interface, pass in the network parameters to be optimized (all parameters of FGFP, PSFP, and cross-modal heterogeneous interaction module), and set the core parameters including: 1. Learning rate: The initial learning rate is dynamically adjusted according to the "warm-up + cosine annealing" strategy (starting from 10 in the first 5 rounds). -6 Linear increase to 10 -3 Subsequent cosine annealing decays over 20 cycles, with a minimum of 10. -5 ); 2. Momentum parameters: β1=0.9, β2=0.999, used to smooth gradient updates and improve training stability; 3. Weight decay coefficient: set to 10 -5 By applying L2 regularization to the network parameters, overfitting is suppressed, preventing the model from overfitting on the training set and thus reducing its generalization ability. In addition, by combining data augmentation (random brightness adjustment and contrast enhancement) in step 1 and meta-test set validation (early stopping strategy) in step 5, overfitting is further suppressed, ensuring the model's generalization performance.

[0119] ② Adopt a learning rate strategy of "warm-up + cosine annealing". The first 5 rounds are the warm-up phase, and the learning rate starts from... linear increase to After the warm-up period, the learning rate decays using a cosine annealing strategy over a period of 20 rounds, with a minimum learning rate of [missing value]. ;

[0120] ③ Total loss The gradients of the parameters for FGFP, PSFP, and the cross-modal heterogeneous interaction module are calculated separately, and the parameters are updated synchronously at an update rate of 2:2:1 to ensure parameter compatibility among the three modules. The parameter update formula is:

[0121]

[0122] in, The parameters before the update (including the lightweight backbone network parameters of FGFP, the ResNet-50 network parameters of PSFP, and the MHCA module parameters). For the updated parameters, The current learning rate, For the total loss against the parameters The gradient.

[0123] ④ Calculate the average loss (Abs Rel metric) of the meta-test set after each training round. If the decrease in the meta-test set loss is less than 0.5% for four consecutive rounds, the result is considered negative. If the model converges, the optimal parameters of each module are saved; otherwise, iterative training continues, with a total training rounds not exceeding 100 rounds.

[0124] Step 6: Input the RGB monocular image to be processed into the trained depth estimation model, and after optimization by the dual-stream feature extraction module, cross-modal heterogeneous interaction module and decoder, output the final high-quality depth map.

[0125] This step is the model inference stage. Based on the optimal parameters obtained after training, it achieves rapid conversion and optimized output of RGB monocular images to high-quality depth maps. The specific implementation process is as follows:

[0126] ① Input preprocessing: Perform geometric correction (distortion correction), noise reduction, enhancement, and normalization on the RGB monocular image to be processed according to the process in step 1, and output a normalized RGB image. Geometric correction refers to the correction of radial and tangential distortion in images acquired by a monocular camera. Specifically, it involves correcting pixel shifts (such as barrel and pincushion distortion) caused by the camera lens's optical characteristics based on the camera's intrinsic parameter matrix (obtained in advance through camera calibration). This restores the image's true geometric structure, avoids feature extraction errors and depth estimation distortion caused by distortion, and ensures the geometric accuracy of the input image.

[0127] ② Dual-path feature extraction: Standardized RGB image Parallel inputs of FGFP and PSFP, outputting shallow geometric features respectively. With deep semantic features The fused feature map is generated through the cross-modal heterogeneous interaction module. ;

[0128] ③ Initial depth map generation: fusion of feature maps The input decoder and deconvolutional layers progressively upsample the feature maps to match the input image. With consistent dimensions, the output convolutional layer reduces multi-channel features to a single channel, generating an initial depth map. (Depth value range [0,1]), the calculation satisfies the formula:

[0129] ;

[0130] in, For the first Layer deconvolution, For output convolutional layers, This is the Sigmoid activation function.

[0131] ④ Depth value scene adaptation: Due to the initial depth map The depth value is a normalized result and needs to be mapped to the actual depth range of the scene (e.g., 0-10m for indoor scenes, 0-100m for outdoor scenes) through a linear transformation. The calculation formula is as follows:

[0132] ;

[0133] in, This is the depth scaling factor. This is the depth offset coefficient, determined using depth calibration data from the actual scene. For example, for indoor scenes, the depth scaling coefficient is a=10, and b=0; for outdoor scenes, the depth scaling coefficient is a=100, and b=0.

[0134] ⑤ Post-processing optimization: Guided filtering (guided image is the original RGB image) is used to optimize the normalized depth map. Smoothing is performed to remove isolated noise points while preserving edge details; finally, morphological closing operations (using 3×3 rectangular structuring elements) are used to fill in small holes in the depth map, outputting a high-quality final depth map. Final depth map The output is the result of a fully optimized process. Essentially, it's a precise reconstruction of the 3D depth distribution of an RGB monocular image. The core principles are as follows: Input preprocessing (distortion correction, denoising, and standardization) ensures the validity of the input data, providing a high-quality foundation for depth estimation; parallel extraction of geometric and semantic features via dual-stream channels, and cross-modal heterogeneous interaction to achieve deep feature fusion, ensures the integrity and richness of depth information; a meta-learning dual-stream loss equalization mechanism and end-to-end training allow the model to learn the mapping relationship between RGB images and depth information autonomously without relying on real depth labels, resulting in a high degree of geometric accuracy in the initial depth map output; scene adaptation maps normalized depth values ​​to the depth range of the actual scene (e.g., 0-10m indoors, 0-100m outdoors), aligning with practical application needs; post-processing optimization (guided filtering, morphological closing operations) removes noise and small holes, preserving edge details and ensuring the structural consistency and smoothness of the depth map. In summary, the final depth map... It fully restores the three-dimensional depth distribution of the target scene, and the value of each pixel corresponds to the depth distance in the actual scene, so it is the final depth estimation result.

[0135] Example 1

[0136] The following example illustrates an RGB monocular depth estimation method based on meta-learning dual-stream loss equalization according to the present invention. This embodiment focuses on RGB monocular depth estimation tasks in three typical interference scenarios: indoor scenes with texture interference, outdoor scenes with water surface specular reflection, and low-light outdoor scenes. The core solution addresses the problems of depth level confusion caused by texture interference, false depth caused by specular reflection, and difficulty in distinguishing near and far structures due to low contrast and similar hues under low light. Through the synergistic optimization of the three modules of the present invention, stable generation of high-quality depth maps under complex interference is achieved.

[0137] The system in this embodiment is deployed on a computing device equipped with an NVIDIA RTX 3090 GPU, using Python 3.9 and PyTorch 2.0, with CUDA version 11.8, to accelerate model training and inference. This invention constructs a temporally differentiated dual-flow path and dynamic loss optimization mechanism through a closed-loop logic of "image detection and enhancement - parallel feature extraction via dual-flow path - cross-modal heterogeneous feature interaction - meta-learning dual-flow loss equalization - feedback adaptive parameter update - depth map generation and execution," achieving accurate mapping from RGB monocular images to high-quality depth maps. The specific implementation process is as follows:

[0138] Step 1: Acquire RGB monocular images of the target scene, and enhance the output with standardized, interference-resistant image data through multi-stage image preprocessing. Specifically, firstly, RGB images are acquired and initially screened using a monocular camera. Based on image sharpness evaluation indicators such as variance and entropy, valid images without motion blur and severe noise are selected, while invalid data that is blurred, overexposed, or underexposed are removed. Next, radial and tangential distortion correction is performed on the valid images based on the camera intrinsic parameter matrix. Then, the images are scaled to a uniform size according to the dual-channel input requirements (512×512 pixels for indoor texture interference images, 640×480 pixels for outdoor specular reflection images, and 768×512 pixels for low-light images). Bilinear interpolation is used to preserve image details and avoid structural distortion caused by size scaling. Finally, a combined denoising strategy of "Gaussian filtering + bilateral filtering" is adopted. First, a 3×3 Gaussian filter (standard deviation) is applied. Suppress Gaussian noise, then use bilateral filtering (spatial domain standard deviation) grayscale standard deviation While preserving edge details, salt-and-pepper noise and redundant textures were removed; further improvements were made through random brightness adjustment (range ±15%) and contrast enhancement (gamma correction). To improve the model's robustness to changes in illumination, pixel normalization is performed. Based on the mean and standard deviation of the RGB three channels in the training set, pixel values ​​are mapped to a distribution range with a mean of 0 and a variance of 1, ensuring the stability and consistency of dual-path feature extraction.

[0139] Step 2: Using a dual-stream feature extraction module, deep geometric features and semantic features are extracted in parallel from the preprocessed input image. Temporally complementary heterogeneous feature pairs are obtained through differentiated network structure configuration. The Fast Geometric Feature Path (FGFP) uses a lightweight ShuffleNetV2×1.0 architecture as its backbone network. This architecture is optimized for low-computing scenarios and includes one initial convolutional layer (3×3 convolutional kernel, stride 2, output channels 24) and four ShuffleNetV2 unit groups (each group contains 3, 7, 3, and 1 Shuffle units respectively, with corresponding output sizes of 256×256, 128×128, 64×64, and 32×32). By using channel shuffling and point-by-point grouping convolution mechanisms, the number of parameters is controlled within 1.4M while retaining the core feature expression capabilities. It can quickly capture shallow geometric features of the scene (target contour, spatial layout, relative position) with low computational complexity and strictly control the response time within 30ms. Finally, the number of channels is unified to 128 dimensions through 1×1 convolution and shallow geometric feature maps are output. The Shuffle unit is a core component of the ShuffleNetV2 architecture. Its main functions are: first, to randomly shuffle and recombine the feature channels after different group convolutions through the ChannelShuffle mechanism, breaking the independence between channels, promoting feature interaction between channels, and improving feature expression capabilities; second, to replace traditional pointwise convolution with "pointwise group convolution", which reduces the number of network parameters and computation while retaining the core feature extraction capabilities, adapting to the "lightweight and fast feature extraction" requirements of FGFP, ensuring that the response time of FGFP is controlled within 30ms, and capturing shallow geometric features of the scene.

[0140] The Precise Semantic Feature Path (PSFP) adopts a deep ResNet-50 architecture, which includes one initial convolutional layer (7×7 convolutional kernel, stride 2, output channels 64) and four residual block groups (output channels 64, 128, 256, and 512 respectively). After each residual block group, a Channel Attention Module (CAM) and a Spatial Attention Module (SAM) are concatenated to mine deep semantic features (object category, fine-grained edges, texture details) with high computational complexity, enhance the detailed expression and structural consistency of the depth map, and control the response time to within 80ms. Finally, the number of channels is unified to 256 dimensions through 1×1 convolution and a deep semantic feature map is output. The two paths achieve feature complementarity through temporal differences. Hardware acceleration + network optimization: First, relying on the hardware performance of the deployed device (such as the NVIDIA RTX 3090 GPU in the example), CUDA parallel computing is used to accelerate the convolution operation of the ResNet-50 architecture, which greatly improves the feature extraction speed. Second, the ResNet-50 architecture is optimized for lightweighting. While retaining the ability to extract deep semantic features, some redundant convolutional layers are simplified. At the same time, key features are focused through CAM and SAM attention modules to reduce the amount of computation of invalid features. Finally, the PSFP feature extraction response time is controlled within 80ms. The temporal difference between the two pathways is reflected in the response speed of feature extraction (FGFP≤30ms, PSFP≤80ms), while the feature complementarity is reflected in the feature type and accuracy: FGFP quickly outputs shallow geometric features, which can quickly capture the basic spatial layout of the scene and the target outline, providing preliminary geometric constraints for depth estimation; although PSFP is slightly slower, it can output deep semantic features, supplementing the target category, fine-grained edge and other detailed information, and correcting the shortcomings of FGFP features; the two pathways are processed in parallel, with FGFP outputting basic features first and PSFP outputting enhanced features later. Subsequently, the two types of features are fused through a cross-modal heterogeneous interaction module to achieve the complementarity of "fast basic features + accurate detailed features", which not only ensures the speed of feature extraction, but also improves the accuracy of feature expression, providing support for high-quality depth estimation.

[0141] Step 3: Construct a cross-modal heterogeneous interaction module to perform deep fusion and dynamic adaptation of heterogeneous feature pairs output from dual-channel systems, and enforce structural consistency and semantic relevance. Specifically, the deep semantic feature map is first upsampled to 64×64 (consistent with the geometric feature map size) using bilinear interpolation, and then its channel count is reduced to 128 dimensions using 1×1 convolution, completing modal alignment of the two types of features. Next, the aligned features are input into an 8-head multi-head cross-attention (MHCA) module (each attention head is 16-dimensional). Geometric features are used as the query vector (Q), and semantic features as the key vector (K) and value vector (V). A 128×128-dimensional linear projection layer generates Q, K, and V matrices. Dot product operations and Softmax normalization capture the dependencies between the two types of features, generating enhanced geometric and semantic features. Subsequently, an adaptive weighted fusion strategy is used to dynamically generate fusion weights based on the similarity of the two types of interactive features. ( The higher the similarity The closer the feature size is to 0.5, the deeper the feature fusion is achieved through weighted summation; finally, a 3×3 convolutional layer and batch normalization (BN) are used to remove noise interference during the fusion process, outputting a fused feature map that has both geometric integrity and semantic richness. The fused features obtained from the training set... Figure 1 First, it is used for model parameter optimization. It is input into the dual-stream loss equalization module, generates an initial depth map and calculates the equalization total loss. Through loss backpropagation, it simultaneously optimizes the parameters of the FGFP, PSFP, and cross-modal heterogeneous interaction modules to ensure the parameter adaptability of the three modules. Second, it is used for training the meta-learning network. It contains the geometric and semantic features of the scene. After being input into the meta-learning network, it helps the meta-learning network to explore the mapping relationship between scene features and loss weights, so that the meta-learning network can learn to dynamically allocate loss weights according to the fusion features of different scenes, thereby improving the generalization ability of the model.

[0142] Step 4: Construct a dual-stream loss equalization module. Through meta-training, mine the mapping relationship between scene features and loss weights, dynamically generate an adapted loss function, and calculate the equalized total loss value. Regarding the construction of the meta-learning framework, the NYUDepth V2 (indoor) and KITTI (outdoor) datasets are selected and divided in a 4:1 ratio into a meta-training set (covering 10 sub-scenes such as dense textures, specular reflections, and low-light conditions) and a meta-test set. A two-layer Restormer Block is used as the meta-learning backbone network. By combining local enhanced window attention and global long-range attention, the mapping pattern between scene features and loss weights is efficiently captured. Meta-training is conducted in an episode-based training mode (each episode contains a support set and a query set), aiming to minimize the query set loss error, ensuring that the meta-learning network can quickly output the optimal loss weights based on scene features. Both the support set and the query set are randomly sampled from the meta-training set: First, a target sub-scene (such as a texture interference scene) can be randomly selected from the meta-training set; then, several images (such as 5 images) can be randomly sampled from this sub-scene as the support set, which is used by the meta-learning network to initially learn the mapping relationship between the features of this sub-scene and the loss weights; then, another set of images (such as 3 images) can be randomly sampled from the same sub-scene as the query set, which is used to verify the effectiveness of the mapping relationship learned by the meta-learning network. By minimizing the loss error of the query set, the parameters of the meta-learning network are optimized to ensure that the meta-learning network can quickly adapt to the loss weight adjustment requirements of new scenes. In the design of the dual-stream loss function, the geometric feature path loss focuses on the overall numerical accuracy and spatial smoothness of the depth values, including pixel-level L1 reconstruction loss (ensuring pixel-level consistency) and depth smoothing loss (suppressing abrupt changes in depth values); the semantic feature path loss focuses on structural consistency and edge alignment, including multi-scale structural similarity (MS-SSIM) loss (enhancing overall structural matching) and edge consistency loss (extracting edge maps based on the Canny operator to ensure local edge alignment); the meta-learning network dynamically allocates the loss weights of the two paths according to the current scene features and sets the loss component balance coefficient. Weighted fusion yields a balanced total loss, enabling targeted supervision.

[0143] Step 5: Through end-to-end joint training, optimize the parameters of all network modules using backpropagation with equalized total loss, without relying on the real depth labels, and iterate until the model converges. Specific operations are as follows: This embodiment uses the AdamW optimizer, and sets... The weight decay coefficient is set to The system effectively suppresses model overfitting by combining adaptive learning rate and weight decay mechanisms; it employs a "warm-up + cosine annealing" learning rate scheduling strategy, with the first 5 rounds serving as a warm-up phase, and the learning rate starting from... linear increase to After the warm-up period, the learning rate decays using a 20-round cosine annealing strategy, with a minimum learning rate set to [value missing]. To balance training speed and convergence accuracy; calculate the gradient of the total loss with respect to the parameters of FGFP, PSFP, and cross-modal heterogeneous interaction modules, and update the parameters synchronously at an update rate of 2:2:1 to ensure parameter fit between modules; calculate the mean absolute relative error (Abs Rel) of the meta-test set after each training round. If the error decreases by less than 1 / 4 consecutive rounds, the result is considered a negative result. If the model converges, the optimal parameters of each module are saved; otherwise, iterative training continues, with a total of no more than 100 rounds.

[0144] Step 6: Input the RGB monocular image to be processed into the trained model, and output the final high-quality depth map through dual-stream feature extraction, cross-modal heterogeneous interaction and decoding optimization. This step is the model inference stage, and the specific process is as follows: The image to be processed first undergoes distortion correction, denoising, enhancement, and standardization according to the process in step 1 to ensure the quality of the input data; then, it is input into a dual-channel parallel processing flow to quickly extract geometric and semantic features, and generates a fused feature map through a cross-modal heterogeneous interaction module; the fused feature map is input into a decoder (3 layers of deconvolution, 3×3 kernel, stride 2, gradually upsampling the feature map to the same size as the input image), and after mapping with a 1×1 convolution and a Sigmoid activation function, an initial depth map with a depth value range of [0,1] is generated; through linear transformation, the normalized depth value of the initial depth map is mapped to the actual scene depth range (indoor scene scaling factor a=10, offset factor b=0, corresponding to 0-10m; outdoor scene a=100, b=0, corresponding to 0-100m), adapting to the needs of different application scenarios; finally, guided filtering is used (the guide image is the original RGB image, radius r=5, regularization parameter...). Post-processing is performed on the morphological closing operation of the 3×3 rectangular structural element to remove isolated noise points and fill small holes while preserving edge details, outputting a high-quality final depth map.

[0145] Effect verification:

[0146] This embodiment completes the full-process test on RGB images of the three typical interference scenarios mentioned above. The system output fully demonstrates the comparison effect between the original image and the generated depth map, such as... Figure 2-4 As shown. (Through) Figure 2-4 The analysis of typical examples in three types of scenarios fully demonstrates the invention's ability to solve problems such as texture interference, specular reflection, and low lighting. This small-scale analysis verifies the model's generalization performance and robustness. The specific analysis is as follows:

[0147] 1. Indoor scenes with texture interference: such as Figure 2As shown, the original image contains densely textured pet fur and wood-grain wallpaper. The interweaving of these two textures can easily lead to confusion of depth levels using traditional methods. This invention strengthens the edge consistency loss weights through PSFP, and uses FGFP based on the lightweight architecture of ShuffleNetV2 to ensure rapid extraction and constraint of spatial layout. In the generated depth map, the depth boundaries between the pet and the wallpaper are clearly distinguishable. The pet's outline is complete and its depth value is significantly higher than the background. The fur texture does not interfere with the depth determination, and the depth value transition in the wallpaper area is smooth and natural, perfectly matching the real depth relationship of the interior space. This proves that this invention can effectively separate the target and background depth features in texture-interferenced scenes.

[0148] 2. Outdoor scenes involving mirror reflections from water surfaces: such as... Figure 3 As shown, the water reflection in the original image is similar to the structure of the real target. Traditional methods easily misjudge the reflection as a real spatial structure, generating false depth values. This invention dynamically improves the depth smoothing loss weights through meta-learning, efficiently captures the scene's geometric layout using the FGFP ShuffleNetV2 architecture, and suppresses the reflection edge response using the PSFP edge consistency loss. The generated depth map accurately restores the three-dimensional spatial position of real targets such as towers and trees, with sharp contour edges. There is no additional false depth information in the water reflection area, and the depth value remains stable, effectively avoiding depth distortion caused by specular reflection. This highlights the strong anti-interference capability of this invention against reflection interference.

[0149] 3. Outdoor scenes in low light: such as Figure 4 As shown, the original image was created in a low-light twilight environment, with low overall brightness, insufficient contrast, and similar tones, making it difficult for traditional methods to distinguish the distance levels of different bridges. This invention enhances spatial continuity through brightness / contrast enhancement in the preprocessing stage and uses FGFP based on ShuffleNetV2 depth smoothing loss to improve spatial continuity. The generated depth map shows significant differences in the distance depth of the three bridges, with depth values ​​gradually increasing from near to far, conforming to physical spatial laws. The bridge outlines are complete and unblurred, and the water reflection does not interfere with depth estimation. This successfully overcomes the limitations of feature extraction in low-light environments and achieves high-precision depth estimation.

[0150] In summary, the visualization results fully demonstrate that this invention achieves dynamic adaptation to complex interferences such as texture interference, specular reflection, and low illumination through a meta-learning dual-stream loss equalization mechanism. Differential feature extraction in the dual-stream path (combining targeted configurations of ShuffleNetV2 and ResNet-50) strengthens the representation of key structural features, cross-modal heterogeneous interaction ensures the effectiveness of feature fusion, and end-to-end joint training (AdamW optimizer + cosine annealing learning rate) improves model accuracy and generalization ability. Furthermore, this invention can stably generate high-quality depth maps with consistent structure and clear hierarchy without relying on additional labeled data, providing a reliable technical path for RGB monocular depth estimation in complex interference scenarios, demonstrating significant innovative value and practical application feasibility.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for RGB monocular depth estimation based on meta-learning two-stream loss equalization, characterized in that, The steps are as follows: Step 1: Acquire RGB monocular images of the target scene, and obtain standardized RGB images through multi-stage image preprocessing to form a training set; Step 2: Employ the differentiated network structure of the dual-stream feature extraction module to extract shallow geometric features and deep semantic features in parallel from the input standardized RGB image; Step 3: Construct a cross-modal heterogeneous interaction module based on multi-head cross-attention to perform deep fusion and dynamic adaptation of shallow geometric features and deep semantic features to obtain the final fused feature map; Step 4: Select the original dataset corresponding to the training set, and mine the mapping relationship between scene features and loss weights through meta-training to obtain the balanced total loss; Step 5: Through end-to-end joint training and training set, optimize module parameters using backpropagation with equalized total loss to obtain the trained depth estimation model; Step 6: Input the RGB monocular image to be processed into the trained depth estimation model. After passing through the dual-stream feature extraction module, the cross-modal heterogeneous interaction module and the decoder, the final depth map is output.

2. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 1, characterized in that, The implementation method of the multi-stage image preprocessing is as follows: Based on image sharpness evaluation metrics, valid images without motion blur and severe noise are selected, while invalid data that are blurry, overexposed, or underexposed are removed. Distortion correction is performed on the effective image, which is scaled to a uniform size according to the input requirements of dual-channel feature extraction, and bilinear interpolation is used to preserve the edge details and texture features of the image. Noise reduction was performed by Gaussian filtering and bilateral filtering in sequence; Random brightness adjustments and contrast enhancements are performed to improve the model's robustness to lighting changes. Finally, pixel normalization is applied to obtain a normalized RGB image. .

3. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 1 or 2, characterized in that, The differentiated network structure includes a fast geometric feature path and a precise semantic feature path. The fast geometric feature path uses a lightweight backbone network to quickly extract shallow geometric features of the scene with low computational complexity. The precise semantic feature path uses a deep ResNet-50 architecture to accurately extract deep semantic features with higher computational complexity.

4. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 3, characterized in that, The implementation method for generating a final fused feature map with both geometric integrity and semantic richness based on the multi-head cross-attention cross-modal heterogeneous interaction module is as follows: ① Deep semantic features are scaled using interpolation Upsampling is then performed, followed by 1×1 convolution to reduce the number of channels and obtain the aligned semantic feature map. ; ② Shallow geometric features As query vector, semantic feature map As key and value vectors, query matrix, key matrix, and value matrix are generated respectively through linear projection layer. Attention weights are obtained by performing dot product operation on query matrix and key matrix and normalizing. Attention weights are then weighted and summed with value matrix to obtain interactive attention features of each attention head. The cross-attention features of all attention heads are concatenated along the channel dimension, and feature fusion and dimensional adjustment are performed through a linear projection layer to generate the interactive geometric features. ; ③ Similarly, semantic feature maps As a query vector, shallow geometric features As key vectors and value vectors, they generate semantic features after interaction. . ④ An adaptive weighted fusion strategy is used to generate fusion weights. Then the fusion features ; Features are fused using 3×3 convolutional layers and batch normalization layers. Perform feature smoothing and output the final fused feature map.

5. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 4, characterized in that, The lightweight backbone network comprises multiple sequentially connected depthwise separable convolutional blocks, batch normalization, and activation layers. Shallow geometric features are obtained by adjusting the channel dimension through 1×1 convolution operations. ; The deep ResNet-50 architecture includes multiple sequentially connected residual block groups, channel attention modules, and spatial attention modules. Deep semantic features are obtained by adjusting the channel dimensions through 1×1 convolution operations. ; The adaptive weighted fusion strategy generates fusion weights. The method is as follows: calculate the geometric features after interaction. With semantic features Feature similarity is calculated using the cosine similarity method to determine the geometric features after interaction. With semantic features The cosine value of the corresponding pixel is used as the global average as the feature similarity. This feature similarity is then input into the Sigmoid activation function to generate the fusion weights. .

6. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 4 or 5, characterized in that, The total loss for equalization is: ； in, , This is the balance coefficient for the loss components; For pixel-level L1 reconstruction loss of shallow geometric features, For shallow geometric features, depth smoothing loss, For multi-scale structural similarity loss of deep semantic features, For edge consistency loss of deep semantic features, and These are the dynamic weights for shallow geometric features and deep semantic features, respectively. .

7. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 6, characterized in that, The pixel-level L1 reconstruction loss is: ; in, This is the pixel matrix of the original image after denoising. Based on shallow geometric features The reconstructed RGB image has H, W, and C as its height, width, and number of channels, respectively; i represents the height index, j represents the width index, and k represents the channel index. The depth smoothing loss is: ; in, Based on shallow geometric features The generated initial depth map, , These are the first-order gradient operators in the horizontal and vertical directions, respectively; The multi-scale structural similarity loss formula is as follows: ; in, Based on deep semantic features The generated initial depth map, Initial depth map Compared with the original image pixel matrix The structural similarity index, The multi-scale similarity index is obtained by weighting the structural similarity indices at different scales: The edge consistency loss is ; in, To extract the edge map of the original image pixel matrix I based on the Canny edge detection operator, Initial depth map Edge map; The dynamic weight and The method for obtaining the dynamic weights is as follows: the final fused features output by the cross-modal heterogeneous interaction module are input into the meta-learning backbone network, and two initial weight values ​​are output through the output layer of the meta-learning network; the initial weight values ​​are then subjected to Softmax normalization to obtain the dynamic weights. and .

8. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 7, characterized in that, The method for mining the mapping relationship between scene features and loss weights through meta-training is as follows: ① Meta-dataset partitioning: Select the original dataset corresponding to the training set in step 1, and divide it into a meta-training set and a meta-test set according to the proportion. The meta-training set is used to learn the adjustment rules of the loss weight of the meta-learning network, and the meta-test set is used to verify the generalization performance of the model. ② Meta-learning network: A two-layer Restormer block is used as the meta-learning backbone network. The Restormer block combines local enhanced window attention with global long-distance attention to efficiently capture the mapping relationship between scene features and loss weights. ③ Meta-training objective: Through episode-based training of the meta-training set, minimize the loss function error of the query set contained in the episode, so that the meta-learning network can quickly output the optimal loss weights based on scene features. The implementation method of step 5 includes: configuring the optimizer and setting effective parameters; adopting a warm-up + cosine annealing learning rate strategy; and balancing the total loss. The gradients of the parameters for the differentiated network structure and the cross-modal heterogeneous interaction module are calculated separately, and the parameters are updated synchronously at an update rate of 2:2:

1. After each training round, the average loss of the meta-test set is calculated. If the decrease in the loss of the meta-test set for four consecutive rounds is less than a certain value, the loss is considered lost. If the model converges, the optimal parameters of each module are saved; otherwise, iterative training continues, with a total of no more than 100 training rounds, to obtain the trained depth estimation model.

9. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 7 or 8, characterized in that, The method for outputting the final depth map is as follows: The RGB monocular image to be processed is subjected to multi-stage image preprocessing to obtain a standardized RGB image. Standardized RGB images The fast geometric feature path and the precise semantic feature path of the parallel input differential network structure output shallow geometric features respectively. With deep semantic features The final fused feature map is generated through the cross-modal heterogeneous interaction module. Final fused feature map The input decoder and deconvolutional layers progressively upsample the feature maps to match the input RGB image. With consistent dimensions, the output convolutional layer reduces multi-channel features to a single channel, generating an initial depth map. ; The initial depth map is transformed by linear transformation. By mapping the depth range to the actual scene, a normalized depth map is obtained. ; Using RGB images Guided filtering as a guide map for normalized depth maps The process involves smoothing the image, filling in tiny holes in the smoothed depth map using morphological closing operations, and then outputting the final depth map. .

10. The RGB monocular depth estimation method based on meta-learning two-stream loss equalization according to claim 9, characterized in that, shallow geometric features The input consists of a decoder composed of deconvolutional layers and 1×1 convolutions, which process shallow geometric features through the deconvolutional layers. The image is upsampled to the same size as the original image pixel matrix I, then the number of feature channels is adjusted to 3 channels through a 1×1 convolution. Finally, a linear activation function is used to map the feature values ​​to the pixel value range of [0, 255], resulting in a shallow geometric feature-based image. Reconstructed RGB image ; shallow geometric features The input to the decoder is progressively upsampled through deconvolutional layers to the same size as the pixel matrix I of the original image. Then, a 1×1 convolution is used to reduce the dimensionality of the multi-channel features to a single channel. Finally, the Sigmoid activation function is used to map the feature values ​​to a normalized depth range of [0,1] to obtain the initial depth map. ; The structural similarity index is obtained by calculating the brightness similarity, contrast similarity, and structural similarity of corresponding regions in two images, and then weighting and summing them to obtain the final similarity score; the multi-scale similarity index is obtained by comparing the pixel matrix I of the original image with the initial depth map. Multi-scale downsampling is performed, the structural similarity index is calculated at each scale, and then the MS-SSIM value is obtained by weighted averaging of the structural similarity index values ​​at all scales. The activation layer is implemented using the ReLU6 activation function; The residual block group performs residual connections and convolutional feature extraction on the input standardized RGB image; The fusion weight The closer the value is to 0.5, the more balanced the fusion of geometric and semantic features is achieved; fusion weight The weights are biased towards the side with better feature representation: if geometric features are better, the weights are fused. If the weights approach 1, then the semantic features are better, and the weights are fused. Approaching 0; Parameters are configured using the AdamW optimizer; The first 5 rounds of the learning rate are the warm-up phase, with the learning rate starting from... linear increase to After the warm-up phase, the learning rate decays using a cosine annealing strategy over a period of 20 rounds, with a minimum learning rate of [missing value]. ; The formula for updating parameters is: ;in, The parameters before the update. For the updated parameters, The current learning rate, To balance the total loss on parameters The gradient; The initial depth map ;in, , , These are the first, second, and third deconvolution layers, respectively. For output convolutional layers, Use the Sigmoid activation function; The normalized depth map for: ;in, This is the depth scaling factor. This is the depth offset coefficient; The fast geometric feature pathway uses a lightweight ShuffleNetV2×1.0 architecture as the backbone network, which includes one initial convolutional layer and four ShuffleNetV2 unit groups. The number of channels is unified to 128 dimensions through 1×1 convolution and a shallow geometric feature map is output. The Shuffle unit of the ShuffleNetV2 unit group randomly shuffles and recombines the feature channels after convolution of different groups through a channel shuffling mechanism. The traditional pointwise convolution is replaced by pointwise group convolution, which reduces the number of network parameters and computational cost while retaining the core feature extraction capability.

Images