A texture missing scene sharpening MDE method based on decoupled boundary structure prior

Abstract

This invention relates to the fields of computer vision and autonomous driving perception technology, and discloses a texture-deficient scene sharpening MDE method based on decoupled boundary structure priors. Instance masks of RGB images are extracted as structural priors; the feature flow is decoupled into a background depth branch and a foreground object depth completion branch. The background branch maintains the geometric continuity of texture-deficient regions through a global encoder, while the foreground branch independently processes each object region in blocks to restore sharpened edge details. After generating a preliminary depth map through Gaussian soft blending, an uncertainty-guided adaptive thinning module performs local linear correction using the original RGB image as the structural prior. This invention overcomes the boundary blurring problem of traditional global optimization paradigms in texture-deficient scenes, achieving a unity of foreground edge sharpening and background geometric continuity, and outputting a depth map with higher structural fidelity and boundary clarity.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and autonomous driving perception technology, specifically to a texture-deficient scene sharpening (MDE) method based on decoupled boundary structure priors. Background Technology

[0002] In practical applications (such as autonomous driving, augmented reality, 3D scene reconstruction, and robot manipulation), obtaining dense and accurate depth information from a single 2D image is a crucial foundation for perception. Traditional monocular depth estimation methods typically rely on global optimization paradigms (such as CNN-based encoders and decoders or Vision Transformer-based global attention networks), which treat all pixels in the scene equally.

[0003] However, the global optimization paradigm suffers from fundamental ambiguity when faced with textureless scenes (such as large areas of white walls, smooth roads, and cloudless skies). Due to the lack of photometric features, the network struggles to distinguish the sharpened boundaries of foreground objects from smooth background areas. This leads to severe boundary blur, foreground-background confusion, and patch-like artifacts in weakly textured regions in existing techniques (including cutting-edge methods like Depth Anything and ZoeDepth) in the depth map. Existing depth completion or uncertainty-guided thinning methods often only use the RGB image as a global guide, failing to explicitly decouple the foreground and background at the structural level. Consequently, once the initial depth is over-smoothed, it becomes difficult to recover the discontinuities in sharpening.

[0004] Therefore, there is an urgent need for a high-precision monocular depth estimation technology that can explicitly break through the limitations of global optimization and achieve extremely sharp foreground object boundaries while maintaining the geometric consistency of weak texture backgrounds. Summary of the Invention

[0005] The technical problem this invention aims to solve is to provide a Monocular Depth Estimation (MDE) method for sharpening texture-deficient scenes based on decoupled boundary structure priors. This method addresses the issues of excessively smoothed boundaries, foreground-background blurring, and geometric artifacts present in existing global depth estimation networks for texture-deficient scenes. It achieves a balance between foreground edge sharpening and background geometric continuity, resulting in an output depth map with higher structural fidelity and boundary clarity. The technical solution is as follows:

[0006] A texture-depleted scene sharpening method (MDE) based on decoupled boundary structure priors includes the following steps:

[0007] Step 1: Extract features from the input RGB image to obtain a high-dimensional latent space representation;

[0008] Step 2: Extract a set of binary instance masks as structural priors based on instance-aware cues;

[0009] Step 3: Decouple the feature flow of the high-dimensional latent space representation into a background depth branch and a foreground object depth completion branch; wherein, the background depth branch inherits the complete global features, and the foreground object depth completion branch, based on the binary instance mask set, constrains the high-dimensional latent space representation within each foreground object region, thus shielding background noise interference;

[0010] Step 4: The background depth branch uses a global visual Transformer encoder to extract scene-level features and calculate the base depth;

[0011] Step 5: The foreground object depth completion branch divides each foreground object region into blocks and calculates the specific depth of the object;

[0012] Step 6: Perform scale alignment and soft blending using the coupling module to obtain a preliminary depth map;

[0013] Step 7: Using an uncertainty-guided adaptive thinning module, the original high-resolution RGB image is used as a structural prior to correct artifacts and sharpen boundaries to obtain the final depth map.

[0014] Compared with the prior art, the beneficial effects of the present invention are:

[0015] This invention overcomes the limitations of single global optimization in existing monocular depth estimation by explicitly introducing decoupled boundary structure priors, achieving a balance between high-precision foreground completion and background geometric consistency. Combined with an uncertainty-guided adaptive thinning module (UG-ARM), this invention can accurately correct low-confidence regions using structural information from RGB images. Attached Figure Description

[0016] Figure 1 is a framework diagram of a sharpened monocular depth estimation based on decoupled boundary structure prior provided by the present invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] Step 1: Extract features from the input RGB image to obtain a high-dimensional latent space representation.

[0019] For the input RGB image Feature extraction is performed. This is to accommodate high-resolution images (such as those commonly used in autonomous driving). To reduce the receptive field and computational overhead, a weighted feature extractor is used. The input image is mapped to a high-dimensional latent feature space to obtain multi-scale basic features. :

[0020] ;

[0021] Where d is the downsampling factor of the feature extractor. The number of feature channels, and Input RGB images respectively The height and width. This step provides unified underlying feature support for subsequent background macro-geometric consistency inference and foreground micro-texture completion.

[0022] Step 2: Extract a set of binary instance masks as structural priors based on instance-aware cues.

[0023] Generate a set of binary instance masks using a pre-trained segmentation model (such as SAM, Segment Anything Model, or general segmentation model). Each mask By clearly isolating the foreground object region, the decoupling of subsequent depth prediction is guided, thereby completely eliminating the global smoothing interference of weak texture background on foreground edges and achieving explicit separation of structure prior and depth prediction.

[0024] Step 3: Decouple the feature flow into a background depth branch and a foreground object depth completion branch. The background depth branch inherits the complete global features, while the foreground object depth completion branch, based on a binary instance mask set, constrains the high-dimensional latent space representation within each foreground object region, thus masking background noise interference.

[0025] The feature flow is decoupled into a background depth branch and a foreground object depth completion branch. To overcome the depth ambiguity in texture-deficient regions during global optimization, this step utilizes the binary mask set extracted in step 2. Explicit spatial decoupling is performed between the input and feature ends:

[0026] 1) Background Depth Branch (Global Flow): Directly inherits the complete image input and global features to preserve the structural continuity of large-area backgrounds.

[0027] ;

[0028] 2) Foreground object depth completion branch (local flow): For the first object in the set... Instance mask The image is strictly constrained to the target region by using the Hadamard product, thus shielding it from background noise interference.

[0029] ;

[0030] ;

[0031] Step 4: The background depth branch uses the global ViT encoder (global visual Transformer encoder) to extract scene-level features and calculate the base depth.

[0032] The global encoding and decoding calculation formulas for the background depth branch are as follows:

[0033] ;

[0034] ;

[0035] in, For the Vision Transformer (ViT) global encoder, It is a depth decoder.

[0036] This branch ensures the geometric continuity of large areas of texture-deficient background regions (white walls, roads, sky, etc.) in the metric space and the consistency of the global scale, avoiding the excessive smoothing of local boundaries by traditional global models.

[0037] Step 5: Foreground object depth completion branch divides the object region into blocks and calculates the specific depth of the object.

[0038] Image of a foreground object region and the foreground object region image Divided into a set of overlapping image patches:

[0039] ;

[0040] Each patch is independently projected into the latent space:

[0041] ;

[0042] Reconstructing specific depths of objects after aggregation This process achieves high-frequency detail restoration that is completely independent of background noise statistics. In other words, the process focuses only on high-frequency texture cues inside the object and is completely independent of background noise statistics, restoring foreground sharpened edges and subtle geometric structures that have been smoothed out by the global model.

[0043] Step 6: Perform scale alignment and soft blending using the coupling module to obtain a preliminary depth map.

[0044] First, aggregate the masks:

[0045] ;

[0046] Generate a soft-weighted map using Gaussian convolution:

[0047] ;

[0048] Then the fusion formula is executed:

[0049] ;

[0050] The blending mechanism ensures a seamless transition between the foreground sharpening boundary and the background smooth geometry, while also aligning scale and offset to avoid blending artifacts.

[0051] Step 7: Using an uncertainty-guided adaptive thinning module, the original high-resolution RGB image is used as a structural prior to correct artifacts and sharpen boundaries to obtain the final depth map.

[0052] Adding an auxiliary head to the depth decoder to predict the log-variance, we obtain the uncertainty graph:

[0053] ;

[0054] in, pixel position The corresponding uncertainty value, To ensure that the uncertainty value is greater than zero, this embodiment uses a preset positive number. . Learnable weights for the uncertainty prediction auxiliary head. For the depth decoder at pixel position Output characteristics Learnable biases are used to assist the uncertain prediction head. Regions with high uncertainty (boundaries, transparent surfaces, etc.) receive more refined weights.

[0055] Subsequently, with high-resolution RGB images As a structural prior, combined with uncertainty diagrams Pixel-level filtering is performed: in high-uncertainty areas, RGB high-frequency details are used to correct oversmooth edges and fusion artifacts, while in low-uncertainty areas, the original depth prediction results are preserved, and the final sharpened depth map is output. Furthermore, the refinement process is modeled as a local linear transformation within a local window. Inside:

[0056] ;

[0057] in, pixel position The corresponding refinement depth value, For the first A vector of local linear coefficients acting on the RGB channels within a local window. For high-resolution RGB images at pixel locations The color vector at that location, For the first The bias coefficients corresponding to each local window For the first A local window.

[0058] By solving for linear coefficients using weighted least squares, edges in high-uncertainty regions are corrected using RGB high-frequency details, while the original depth is preserved in low-uncertainty regions.

[0059] The total loss function and output calculation are as follows:

[0060] The total loss combines deep regression, marginal consistency, and uncertainty regularization:

[0061] ;

[0062] The uncertainty-weighted consistency loss is defined as follows:

[0063] ;

[0064] The depth map refined by this loss function is physically similar to the initial depth map. Figure 1 The method focuses on penalizing biases in low-confidence regions. The final output is a refined high-resolution depth map, achieving a 10% lower RMSE and a 28% higher edge F1 score compared to existing methods, significantly improving the structural fidelity and boundary sharpness of scenes with missing textures.

[0065] like Figure 1 As shown, this invention proposes a monocular depth estimation method for texture-deficient scenes based on decoupled boundary structure priors (SepDepth framework). The overall process includes two core stages: decoupled encoding and coupled decoding (Stage 1) and structure-guided adaptive thinning (Stage 2).

[0066] In the first stage (decoupling coding and coupling decoding):

[0067] First, the input RGB image is fed into a shared feature extractor. To resolve the feature conflict between the weakly textured background and the sharpened foreground, this invention introduces a segmentation model to extract instance masks from the image. These masks serve as "prior" elements, driving the feature flow to decouple into two independent branches.

[0068] In the background depth branch, the network utilizes the Vision Transformer (ViT) to extract global contextual features. Because this branch has a global receptive field, its output... It can effectively constrain the measurement of depth continuity in large areas lacking texture (such as walls, sky, and unmarked roads), avoiding abrupt depth artifacts. However, this global attention can lead to blurred object edges.

[0069] Therefore, in parallel, the foreground object depth completion branch processes the foreground region under the aforementioned mask constraints. This utilizes the previously extracted mask. The network crops and masks the image into foreground local regions, then uses a patch-based local Transformer encoder. Formulaically, the input features are segmented into overlapping regions. And projected as By eliminating the interference of textureless background noise, this branch is able to capture the local high-frequency geometric details of objects with extreme sensitivity, outputting sharp depth-specific images of objects. .

[0070] In the coupling module, simple hard stitching leads to severe tomography artifacts in the depth map. This invention uses Gaussian convolution to smooth the mask, generating a soft-weighted map. The two are then softly blended in space and scale using the fusion formula in step 6 to obtain a preliminary depth map.

[0071] In the second stage (uncertainty-guided refinement, UG-ARM): although the initial depth map has separated the foreground and background, "oversmoothing" or "patch artifacts" may still exist in transition zones or areas with extremely low luminance features. To solve this problem, the present invention designs the UG-ARM module.

[0072] Unlike traditional global filtering, the refinement in this invention is spatially adaptive. We add a lightweight auxiliary prediction head at the end of the depth decoder, modeling it as a heteroscedastic Gaussian distribution to predict the log-variance of each pixel (i.e., the cognitive uncertainty map). Uncertainty value It will significantly increase at object edges, transparent surfaces, and areas with extremely weak textures.

[0073] Then, the high-resolution original RGB image As a structural guide diagram, a linear regression model is established within a local window, modeling the refinement process as a local linear transformation within the local window. .

[0074] Solving for the linear coefficient vector At that time, the inverse of the uncertainty graph This is used as a weight in the covariance calculation. This means that in areas with low network confidence (blurred boundary areas), the algorithm will forcibly rely on high-frequency RGB grayscale gradients for edge sharpening; while in areas with high network confidence (smooth walls), the algorithm will suppress excessive interference from RGB textures and retain the smooth depth predicted by the original network.

[0075] During the network training phase, to prevent the refinement process from destroying the physical meaning at the global scale, this invention designs an uncertainty-weighted consistency loss. This forces the final output to align with the initial depth on a macroscopic scale.

[0076] Through this decoupled design and adaptive refinement, the present invention achieves an excellent balance between computational efficiency and geometric reconstruction accuracy. Whether dealing with small, distant targets in autonomous driving or large, textureless workbenches faced by robotic arms, this system can output depth maps with extremely high fidelity, no artifacts, and sharp boundaries, demonstrating significant industrial application value.

[0077] Experiments show that, in scenarios with missing textures, the present invention achieves a 10% reduction in RMSE error and a 28% improvement in edge F1 score compared to existing methods. The output depth map has excellent structural fidelity and extremely clear object boundaries, which can perfectly adapt to the fine perception of obstacles in autonomous driving and the high-precision grasping tasks of robots.

Claims

1. A texture-deficient scene sharpening MDE method based on decoupled boundary structure prior, characterized in that, Includes the following steps: Step 1: Extract features from the input RGB image to obtain a high-dimensional latent space representation; Step 2: Extract a set of binary instance masks as structural priors based on instance-aware cues; Step 3: Decouple the feature flow of the high-dimensional latent space representation into a background depth branch and a foreground object depth completion branch; wherein, the background depth branch inherits the complete global features, and the foreground object depth completion branch, based on the binary instance mask set, constrains the high-dimensional latent space representation within each foreground object region, thus shielding background noise interference; Step 4: The background depth branch uses a global visual Transformer encoder to extract scene-level features and calculate the base depth; Step 5: The foreground object depth completion branch divides each foreground object region into blocks and calculates the specific depth of the object; Step 6: Perform scale alignment and soft blending using the coupling module to obtain a preliminary depth map; Step 7: Using an uncertainty-guided adaptive thinning module, the original high-resolution RGB image is used as a structural prior to correct artifacts and sharpen boundaries to obtain the final depth map.

2. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, In step 1, the input RGB image Feature extraction is performed, specifically by using a weighted feature extractor. , input RGB image Mapping to a high-dimensional latent feature space to obtain multi-scale basic features : ； Where d is the downsampling factor of the feature extractor. The number of feature channels, and Input RGB images respectively Height and width.

3. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, In step 2, extracting the binary instance mask set as a structural prior specifically involves generating the binary instance mask set using a pre-trained segmentation model. : ； in, For the first A binary instance mask, , The total number of instance masks; Each mask explicitly isolates the foreground object region, guiding the decoupling of subsequent depth prediction, thereby completely eliminating the global smoothing interference of weak texture background on foreground edges and achieving explicit separation of structure prior and depth prediction.

4. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Step 3 specifically involves: utilizing the extracted binary mask set Explicit spatial decoupling is performed between the input and feature ends: the background depth branch, i.e., the global flow, directly inherits the complete image input and global features to preserve the structural continuity of large-area backgrounds. ； in, The input background depth branch is a complete RGB image. The global base features are used as the input background depth branch. The multi-scale basic features output by the feature extractor; Foreground object depth completion branch, i.e., local flow: for the first object in the set... Instance mask The image is strictly constrained to the target region by using the Hadamard product, thus shielding it from background noise interference. ； ； in, For the first Image of the foreground object region corresponding to each instance mask. For the first Local features of a foreground object region To downsample the instance mask to the same level as Downsampling operations at the same spatial resolution This represents the Hadama product.

5. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Step 4 specifically involves: global encoding and decoding of the background depth branch. ； ； in, For global vision Transformer encoder, The scene-level encoded features output by the global visual Transformer encoder. For depth decoders, The base depth map output for the background depth branch.

6. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Step 5 specifically involves: For the first Instance mask For the corresponding foreground object, obtain the first one generated in step 3. Image of a foreground object region and the foreground object region image Divided into a set of overlapping image patches: ； in, For the first A set of overlapping image patches corresponding to each foreground object region For the position located at the line, number Overlapping image patches in columns, and These represent the number of image blocks in the height and width directions, respectively. Project each overlapping image patch into the latent feature space: ； in, Overlapping image patches The corresponding latent feature representation, To map overlapping image patches to a projection operation of feature embeddings, This is a Transformer processing module that performs local context encoding on the feature embedding; it performs overlapping region weighted aggregation on the latent feature representations of each overlapping image patch, and obtains the first value through the object depth reconstruction module. Object-specific depth map of a foreground object .

7. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Step 6 specifically involves: Step 6.1: Mask aggregation: ； in, For aggregation mask; Step 6.2: Perform Gaussian convolution to generate a soft weight map. : ； in, The standard deviation is Two-dimensional Gaussian convolution kernel, and These represent the horizontal and vertical offsets of the pixel position relative to the center of the convolution kernel, respectively. To control the Gaussian standard deviation of the width of the transition region between the foreground and background, the symbol " " indicates a convolution operation; Step 6.3: Execute the fusion formula: ； in, This is a preliminary depth map obtained after soft blending. To create specific depth maps for each object According to the corresponding instance mask The depth map of the foreground object is backfilled into the input image coordinate system and aggregated. The base depth map output for the background depth branch. To and A matrix of all ones of the same size.

8. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Step 7 specifically involves: Adding an auxiliary head to the depth decoder to predict the log-variance, we obtain the uncertainty graph: ； in, pixel position The corresponding uncertainty value, This is a preset positive number used to ensure that the uncertainty value is greater than zero. Learnable weights for the uncertainty prediction auxiliary head. For the depth decoder at pixel position Output characteristics Learnable bias for the uncertainty prediction auxiliary head; High-resolution RGB images As a structural prior, pixel-level filtering is performed in conjunction with the uncertainty map: in high uncertainty regions, RGB high-frequency details are preferentially used to correct oversmooth edges and fusion artifacts, while in low uncertainty regions, the original depth prediction results are preserved, and the final sharpened depth map is output. The refinement process is modeled as a local linear transformation within a local window. Inside: ； in, pixel position The corresponding refinement depth value, For the first A vector of local linear coefficients acting on the RGB channels within a local window. For high-resolution RGB images at pixel locations The color vector at that location, For the first The bias coefficients corresponding to each local window For the first A local window.

9. The texture-deficient scene sharpening MDE method based on decoupled boundary structure prior as described in claim 1, characterized in that, Total Loss Function and Output Calculation: Total Loss Function Combining deep regression, marginal consistency, and uncertainty regularization: ； in, For deep regression loss, For edge consistency loss, Consistency loss weighted by uncertainty; , and These are the corresponding weight coefficients; The uncertainty-weighted consistency loss is defined as follows: ； in, The number of effective pixels used in the loss calculation. pixel position The corresponding uncertainty value, pixel position The corresponding final sharpening depth value, pixel position The corresponding initial fusion depth value.

Images