Multi-scale visual converter

Abstract

The invention discloses a multi-scale visual converter, which is characterized in that a multi-scale feature coding and position embedding module performs multi-scale division after filling an RGB (Red, Green, Blue) image and obtains feature vectors of each scale; the encoder module is realized by using a hierarchical Transform and a cross-scale attention mechanism, feature vectors after fusion of all scales are obtained and extracted by using local branch features, feature vectors of the lowest resolution scale are obtained and extracted by using global branch features, and a feature information set is obtained by merging; the input decoder module performs feature fusion through cross-scale feature interaction based on an attention mechanism to obtain fused feature information, performs progressive up-sampling processing by using a multi-layer perceptron (MLP), and reconstructs an inverse depth map by using a DPT decoder; an input depth information generation module generates a metric depth map and a refined edge depth map. According to the method, multi-scale feature aggregation and edge perception geometric refinement can be enhanced, and high-quality monocular depth estimation is realized.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a multi-scale visual converter. Background Art

[0002] Visual Simultaneous Localization and Mapping (VSLAM) is an advanced visual perception technology that has attracted considerable attention in robotics research. VSLAM not only provides autonomous vehicles with environmental awareness and autonomous navigation capabilities, but also enables 3D reconstruction and remote sensing mapping of large surfaces. VSLAM relies on visual information to estimate the robot's position in unknown environments while simultaneously reconstructing the surroundings, resulting in significant theoretical and practical value.

[0003] Despite significant advances in visual simultaneous localization and mapping (VSLAM) technology in recent decades, particularly through algorithmic innovations driven by multi-sensor fusion and deep learning, monocular VSLAM systems still face fundamental challenges in depth estimation accuracy and high-fidelity scene reconstruction. Unlike stereo or RGB-D SLAM systems (SLAM technology that combines RGB-D sensors, which contain both color images (RGB) and depth information; SLAM stands for Simultaneous Localization and Mapping), monocular configurations inherently lack the ability to directly measure depth from a single viewpoint, necessitating complex depth inference mechanisms. Current monocular depth estimation methods can be broadly categorized into traditional geometric approaches and data-driven learning paradigms. Traditional geometric techniques employ direct methods, feature matching, and optical flow estimation to infer depth within geometric constraints, including triangulation principles and disparity analysis in stereo vision systems. These methods fundamentally rely on accurate camera calibration parameters and robust feature correspondence in textured environments. However, their reliance on rigid physical assumptions and hand-crafted constraints inherently limits their adaptability to complex real-world scenes with variable lighting, weak textures, or dynamic elements.

[0004] To address these limitations, machine learning-enhanced VSLAM frameworks have emerged as promising solutions. Early breakthroughs by Saxena et al. (2005-2007) demonstrated a Markov random field (MRF) model for multi-scale feature analysis, combining local and global image features through probabilistic depth modeling. While innovative, these parametric approaches required a priori knowledge of the relationship between RGB intensity and depth distributions. Subsequent nonparametric learning methods, such as Depth Transfer using motion-constrained depth sampling and a discrete-continuous optimization framework using particle belief propagation, eliminated model assumptions through data-driven inference. Advances in computing architectures and silicon processing platforms have further driven a paradigm shift in deep learning solutions. Modern convolutional neural networks surpass traditional machine learning through hierarchical feature abstraction and end-to-end nonlinear mapping between monocular images and depth fields. These architectures demonstrate superior performance in challenging environments by automatically learning complex scene representations, significantly improving depth estimation accuracy while maintaining computational efficiency.

[0005] Deep learning-based depth estimation methods are primarily categorized based on their learning paradigm: supervised, semi-supervised, and self-supervised. Supervised methods utilize per-pixel depth annotations to establish an explicit mapping function between the input image and the ground-truth depth values. Representative architectures include deep convolutional neural networks (DCNNs) and frameworks based on VGG (the classic convolutional neural network model developed by the Visual Geometry Group, a computer vision research team at the University of Oxford). While these methods achieve state-of-the-art accuracy when trained on fully annotated datasets, they face significant scalability limitations due to the prohibitive cost and labor-intensive nature of acquiring pixel-level depth annotations on a large scale. Semi-supervised methods reduce the annotation requirements by leveraging both labeled and unlabeled data through techniques such as consistency regularization and pseudo-labeling. However, these methods are still susceptible to error propagation from noisy pseudo-labels and exhibit sensitivity to domain shifts between the distributions of labeled and unlabeled data. The self-supervised paradigm is a promising alternative that formulates depth estimation as an image reconstruction problem using geometric constraints from stereo pairs or monocular sequences with known camera poses. These methods construct a supervisory signal via a view synthesis loss, where the estimated depth is warped between adjacent frames to minimize photometric differences. Recent advances include Dyna-MSDepth, which incorporates multi-scale high-order spatial-semantic interactions to generate stable depth maps in dynamic environments, and a geometric-neural hybrid framework that combines multi-frame temporal constraints with epipolar geometry principles. Despite these advances, self-supervised depth estimation still faces significant challenges: 1) dynamic object motion violates the static scene assumption in photometric consistency constraints, requiring explicit motion modeling; 2) scale ambiguity in monocular settings requires additional sensor fusion or prior knowledge integration; and 3) training complexity increases exponentially when processing multimodal remote sensing data with different spatial resolutions. Future research should prioritize efficient computational architectures and unified frameworks that can handle both static and dynamic features in Earth observation scenes.

[0006] For monocular depth estimation, self-supervised depth estimation has attracted considerable research attention due to its unlabeled training paradigm. However, two major challenges remain. First, self-supervised frameworks inherently lack metric scale recovery, requiring auxiliary sensor data (such as inertial measurements or camera intrinsics) to establish a physical scale reference. However, the inconsistency of camera configurations across training datasets hinders the development of unified scale-aware models. While mainstream datasets emphasize structured scenes, they remain limited in representing dynamic objects and extreme illumination variations. Therefore, addressing scale invariance and offset robustness across heterogeneous datasets has become a key research direction within the self-supervised paradigm. Second, the domain gap between datasets can significantly reduce model generalization. Recent progress in zero-shot cross-dataset generalization aims to bridge this gap, achieving metric-compatible predictions without dataset-specific fine-tuning. Pioneering work such as MegaDepth and MiDaS demonstrated the feasibility of large-scale multi-dataset training, while OmniData introduced a systematic strategy for cross-domain adaptation. Emerging architectures leveraging transformers, visual transformers (ViTs), and diffusion models have demonstrated enhanced cross-dataset feature decomposition capabilities, achieving significant progress in scale-consistent depth transfer. Despite these advances, three limitations remain: 1) suboptimal accuracy under occlusion and textureless regions; 2) sensitivity to camera parameter variations; and 3) inference efficiency in a multi-scale framework. Summary of the Invention

[0007] To address the above-mentioned problems in the prior art, the present invention provides a multi-scale visual converter. The technical problem to be solved by the present invention is achieved through the following technical solutions: A multi-scale visual converter, comprising: The multi-scale feature encoding and position embedding module is used to perform symmetrical padding on the input RGB image, then perform multi-scale division with different resolutions, and obtain the feature vector of each scale through linear projection and position encoding; The encoder module is implemented using a hierarchical Transformer and a cross-scale attention mechanism to obtain the fused feature vectors of all scales, extract features using local branches to obtain local branch feature information, obtain the feature vectors of the lowest resolution scale, extract features using global branches to obtain global branch feature information, and merge the local branch feature information with the global branch feature information to obtain a set of feature information containing different resolutions. The decoder module is used to fuse the feature information set through cross-scale feature interaction based on the attention mechanism to obtain fused feature information; the fused feature information is progressively upsampled using the introduced multi-layer perceptron (MLP) to restore the corresponding image information, and then the inverse depth map is reconstructed using the DPT decoder; A depth information generation module is configured to generate, based on the inverse depth map, a metric depth map representing absolute depth information and a refined edge depth map having a resolution consistent with that of the RGB image.

[0008] Beneficial effects of the present invention: The embodiment of the present invention provides a multi-scale visual converter, including: a multi-scale feature encoding and position embedding module, an encoder module, a decoder module and a depth information generation module; the multi-scale feature encoding and position embedding module is used to perform multi-scale division of different resolutions after symmetrical filling processing on the input RGB image, and obtain the feature vector of each scale through linear projection and position encoding; the encoder module is implemented by using a hierarchical Transformer and a cross-scale attention mechanism to obtain the feature vector after fusion of all scales, use local branches to perform feature extraction, and obtain local branch feature information; and obtain the feature vector of the lowest resolution scale, use the global branch to extract the feature vector of the lowest resolution scale, and use the global branch to extract the feature vector of the lowest resolution scale. Feature extraction to obtain global branch feature information; merging the local branch feature information and the global branch feature information to obtain a feature information set with different resolutions; a decoder module for fusing the feature information set through cross-scale feature interaction based on the attention mechanism to obtain fused feature information; using the introduced multi-layer perceptron MLP, the fused feature information is progressively upsampled to restore the corresponding image information, and then the DPT decoder is used to reconstruct the inverse depth map; a depth information generation module for generating a metric depth map representing absolute depth information and a refined edge depth map consistent with the resolution of the RGB image based on the inverse depth map.

[0009] The multi-scale visual converter of the present invention is implemented based on the open source ViT architecture, which can enhance multi-scale feature aggregation and edge-aware geometric refinement. For monocular depth estimation tasks, it accurately preserves object topology, terrain structure and absolute metric scale in depth prediction, and achieves high depth estimation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A structural block diagram of a multi-scale visual converter provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structural principle of a multi-scale visual converter provided by an embodiment of the present invention; Figure 3(a) and Figure 3(b) show the comparative analysis results of trajectory estimation accuracy of ViMGS and MonoGS on the TUM-RGBD fr3 dataset in the experiments of the embodiment of the present invention; Figures 4(a) and 4(b) show the comparative analysis results of trajectory estimation performance of ViMGS and MonoGS on the TUM-RGBD fr3 dataset in the experiments of the embodiment of the present invention; Figure 5 This figure shows the 3D Gaussian reconstruction process of ViMGS and MonoGS on the TUM RGB-D fr3 dataset in the experiments of the embodiment of the present invention. DETAILED DESCRIPTION

[0011] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0012] To address the critical need for geometrically accurate 3D reconstruction, embodiments of the present invention propose a Multiscale Visual Transformer (MViT) that integrates a pre-trained open-source ViT, enhances multiscale feature aggregation and edge-aware geometry refinement, and is specifically tailored for the monocular depth estimation task faced by SLAM. This MViT architecture accurately preserves object topology, terrain structure, and absolute metric scale in depth prediction.

[0013] The multi-scale visual converter of the present invention is implemented based on the open source ViT architecture. Figure 1 As shown in Figure 2, it specifically includes: multi-scale feature encoding and position embedding module, encoder module, decoder module and depth information generation module. Figure 2 The structural principle diagram shown in the figure, these four modules correspond to Figure 2 The four boxes in the upper left, upper right, lower right, and lower left of the middle image are shown. The MViT framework of our invention integrates: 1) a hierarchical cross-attention mechanism for joint local-global context modeling, 2) an adaptive fusion module to resolve scale-dependent feature conflicts, and 3) differentiable boundary constraints to ensure sub-pixel geometric fidelity. The multi-scale feature encoding and position embedding module is used to perform symmetrical padding on the input RGB image, then perform multi-scale division with different resolutions, and obtain the feature vector of each scale through linear projection and position encoding; The encoder module is implemented using a hierarchical Transformer and a cross-scale attention mechanism to obtain the fused feature vectors of all scales, extract features using local branches to obtain local branch feature information, obtain the feature vectors of the lowest resolution scale, extract features using global branches to obtain global branch feature information, and merge the local branch feature information with the global branch feature information to obtain a set of feature information containing different resolutions. The decoder module is used to fuse the feature information set through cross-scale feature interaction based on the attention mechanism to obtain fused feature information; the fused feature information is progressively upsampled using the introduced multi-layer perceptron (MLP) to restore the corresponding image information, and then the inverse depth map is reconstructed using the DPT decoder; A depth information generation module is configured to generate, based on the inverse depth map, a metric depth map representing absolute depth information and a refined edge depth map having a resolution consistent with that of the RGB image.

[0014] The following describes each module separately.

[0015] (1) Multi-scale feature encoding and position embedding module The multi-scale feature encoding and position embedding module performs symmetrical padding on the input RGB image, then performs multi-scale division with different resolutions, and obtains the feature vector of each scale through linear projection and position encoding. The process can include the following steps: Step A1, performing symmetrical filling processing on the input RGB image along the vertical dimension to obtain a symmetrically filled RGB image; The RGB image in the embodiment of the present invention may come from the TUM dataset.

[0016] See Figure 2 The module in the upper left corner symmetrically pads the input RGB image, originally at a resolution of 640×480 pixels, along its vertical dimension. This produces a symmetrically padded RGB image, resized to 640×640 pixels. Padding is performed to ensure an integer multiple of the resolution in subsequent segmentation operations. Symmetric padding is described in related technical documentation and will not be explained here.

[0017] The multi-scale processing framework of the present invention divides the RGB image after the symmetrical filling process into three levels, which are described in steps A2, A3 and A4 respectively. These three steps can be processed in parallel. The purpose of dividing the input image into multiple scales is to capture multi-level features, such as features at high resolution, which can retain local details (such as edges and textures). Features at low resolution can capture a wider range of context (such as object shape and overall layout).

[0018] Step A2: dividing the symmetrically filled RGB image into a plurality of image blocks according to an overlapping division method as first-scale image blocks; See Figure 2 The branch with s=1 in the graph, s=1 means the first scale.

[0019] In this branch, the symmetrically padded RGB image with a resolution of 640×640 pixels is divided into 16 image blocks of equal size using overlapping image blocks as the first scale image blocks. Each image block is called a patch.

[0020] Step A3: down-sampling the RGB image after the symmetrical filling process by a factor of 2, and dividing it into a plurality of image blocks in a non-overlapping manner as second-scale image blocks; See Figure 2 The branch with s=2 in the figure, s=2 means the second scale.

[0021] In this branch, the symmetrically padded RGB image with a resolution of 640×640 pixels is first downsampled by a factor of 2 to a resolution of 320×320 pixels; then, it is divided into 16 image blocks of the same size as the second-scale image blocks using a non-overlapping partitioning method.

[0022] Step A4: down-sampling the RGB image after the symmetrical filling process by a factor of 4, and using the obtained image as a third-scale image block; See Figure 2 The branch with s=3 in the figure represents the third scale. The image resolutions of the first scale, second scale, and third scale in the first scale image block, second scale image block, and third scale image block decrease in sequence.

[0023] In this branch, the symmetrically padded RGB image with a resolution of 640×640 pixels is downsampled by a factor of 4 to a resolution of 160×160 pixels as the third-scale image block.

[0024] In step A5, the image blocks at each scale are linearly projected into feature vectors with trainable position encoding to obtain feature vectors at each scale.

[0025] Linear projection means that each patch is mapped into a vector of fixed dimension through a fully connected layer, which is called an embedding vector.

[0026] Position encoding refers to adding position information (such as coordinates and scale) to each embedded vector to give it spatial semantics.

[0027] After linear projection and position encoding, the feature vector of each scale is expressed as: (1); Among them, the input RGB image is represented as , , and Represents the height and width of the input RGB image respectively; Indicates the The eigenvector of the scale, ; , represents a learnable embedding matrix; Indicates the Image patches at scale; , and Both indicate The size of the image patch at scale; It is The spatial dimension of scale, Represents the dimension of the feature vector after embedding, that is, through linear projection, each image block is mapped to a dimensional latent space; It is a flattening function for multidimensional arrays, used to convert image blocks into one-dimensional feature vectors. This step integrates spatial locality and global position perception, which is consistent with the paradigm of Visual Transformer (ViT) and can handle geometric changes at specific scales.

[0028] This part can be understood in conjunction with the relevant content of the ViT architecture and will not be explained in detail here.

[0029] (2) Encoder module See Figure 2 Understanding the module in the upper right corner. The encoder module is implemented using a hierarchical Transformer and a cross-scale attention mechanism.

[0030] After merging the feature vectors of the three scales, 33 patches are obtained, which then enter the local branch for feature extraction to obtain local branch feature information. At the same time, the feature vector of the third scale with the lowest resolution enters the global branch for feature extraction to obtain global branch feature information. Then, the local branch feature information and the global branch feature information are merged to obtain a feature information set with different resolutions. Please refer to the box on the right side of the global branch for the feature information set, which contains feature information of different resolutions. From left to right, the resolutions are: 10×10, 10×10, 20×20, 20×20, 40×40, 40×40; the red square on the far left is the global branch feature information output by the global branch, and the remaining squares are the merged local branch feature information.

[0031] The encoder module follows the standard ViT principle, modeling local-global feature interactions via a two-branch Transformer. Both the local and global branches utilize a multi-head self-attention (MSA) mechanism combined with a multi-layer perceptron (MLP) architecture, employing layer normalization (LN) to stabilize activations and a multi-head attention mechanism to model contextual dependencies. Both branches consist of L stacked transformer layers. The local branch uses 6 attention heads, while the global branch uses 12. The multi-layer perceptrons for both branches consist of two fully connected layers, using Gaussian Error Linear Units (GELUs) as intermediate activation functions.

[0032] In the L layers of transformers stacked sequentially, the The processing of the layer is expressed as: (2); (3); in, Represents a multi-head attention mechanism; Representation layer normalization is used to standardize features, accelerate model convergence and stabilize the training process; represents a multilayer perceptron; Indicates the The basic features of the layer input are transmitted through residual connections; Indicates the Layer Processed intermediate features; Represents the final output features after L layer processing.

[0033] (3) Decoder module See Figure 2 Module understanding in the lower right corner.

[0034] The decoder module fuses the feature information set through cross-scale feature interaction based on the attention mechanism to obtain the fused feature information. The process mainly includes feature alignment, dynamic weight adjustment and nonlinear interaction. Specifically, it includes the following steps: Through cross-scale feature alignment based on the attention mechanism, the dynamic weights generated by the interaction between the global query vector and the local key-value pairs are applied to the features of each scale, realizing context-aware multi-scale feature fusion and obtaining fused feature information. Figure 2 Adaptive feature fusion in . Through the global query vector With local key-value pairs 、 The cross-attention mechanism between the two, the framework proposed in this invention can dynamically adjust the weight coefficients of multi-scale features in a content-dependent manner, realize context-aware adaptive fusion, and thus optimize the perceptual granularity of hierarchical feature representation at different spatial resolutions. Among them, the local branch feature information in the feature information set is used as the multi-resolution feature. , the global branch feature information is the global context feature , fused via attention-based alignment.

[0035] The calculation formula of the fusion feature information is: (4); in, Indicates fusion feature information, corresponding to Figure 2 In the figure, the box below the upsampling represents the restored image information; the resolutions corresponding to the small boxes of multiple colors are: 20×20, 20×20, 40×40, 80×80, 160×160, and 320×320; represents the total number of multiple scales; represents the global query vector; Indicates the Local keys of scale; represents transpose; Represents the dimension of the feature vector after embedding; Indicates the Scale feature information. The framework of the present invention imitates the biological visual system, and features from coarse to fine are dynamically weighted to achieve contextual coherence.

[0036] The decoder module uses the introduced multi-layer perceptron MLP to perform progressive upsampling on the fused feature information to restore the corresponding image information, and then uses the DPT decoder to reconstruct the inverse depth map, including: The MLP is used to perform nonlinear refinement on the skip connection features, and the edge alignment capability is enhanced by combining the edge mask. The network is then gradually upsampled through transposed convolution and feature alignment operations, and finally the DPT decoder is used to reconstruct the inverse depth map. The formula used is: (5); During the feature fusion and upsampling process of the decoder, the present invention introduces an MLP to perform nonlinear refinement on the features of the skip connection to enhance the edge alignment capability.

[0037] In the above formula, the DPT decoder is an existing open source module. Transposed convolution and skip connection are the core operations in the upsampling process; Indicates the Image information restored by scale; represents a multilayer perceptron; , Indicates the The skip connection features corresponding to the scale. It can be understood by those skilled in the art that the skip connection refers to the connection part that directly connects the feature map of the encoder (feature extraction) part to the decoder (image reconstruction) part; It is The spatial dimension of scale, Represents the dimension of the feature vector after embedding; , Indicates the The edge mask corresponding to the scale; In the decoder module of the embodiment of the present invention, the nonlinear interaction between the edge mask and the multi-scale feature can improve the edge accuracy.

[0038] The decoder module uses transposed convolution and skip connection to reconstruct the inverse depth map, which is calculated as follows: (6); (7); in, represents transposed convolution; represents the feature alignment operation; Indicates the The image features after transposed convolution and feature alignment operations on the scale-restored image information; It means element-by-element addition; represents the inverse depth map; is the Sigmoid function; is a 1 The convolution kernel of 1 is used to perform channel feature transformation; Represents the image features obtained at all scales; Represents the bias term. Skip connections can reduce information loss and preserve high-frequency geometric details. The specific integration method of skip connections is channel splicing. The kernel size of the transposed convolution is 3×3 and the stride is 2×2.

[0039] (4) Depth Information Generation Module See Figure 2 The module understanding on the lower left is to obtain absolute depth based on geometric constraints.

[0040] In order to obtain a dense metric depth map The present invention performs zooming by horizontal field of view, which is determined by the focal length and width express.

[0041] Specifically, the depth information generation module generates a metric depth map representing absolute depth information according to the inverse depth map, including: According to the inverse depth map, based on geometric constraints, a metric depth map representing absolute depth information is generated; it can be expressed as: (8); in, represents the metric depth map; Indicates the focal length of the camera; Indicates the width of the object's projection on the image plane.

[0042] In the depth information generation module, an initial refined edge depth map is generated according to the inverse depth map ( Figure 2 The medium resolution is 640×640 pixels), and then cropped to obtain a refined edge depth map consistent with the resolution of the RGB image ( Figure 2 Medium resolution is 640×480 pixels).

[0043] In summary, the present invention has the following innovations: 1. A hierarchical cross-attention mechanism for unified modeling of local-global contextual dependencies.

[0044] 2. A multi-scale adaptive feature alignment module that addresses the discrepancy between features at different scales through learnable fusion weights.

[0045] 3. A geometrically constrained boundary refinement method with sub-pixel accuracy to improve spatial fidelity.

[0046] Comprehensive experimental verification shows that the multi-scale visual converter of the present invention can achieve accurate depth estimation and modeling with monocular data input.

[0047] To address the challenges of existing technologies and meet the critical requirement of geometrically accurate 3D reconstruction in monocular visual simultaneous localization and mapping (VSLAM), this paper proposes a multi-scale visual transformer, a hierarchical multi-scale ViT architecture, which can synergistically integrate local texture details and global contextual cues to construct depth information through three different resolution paths of the input image.

[0048] In order to verify the effectiveness of the solution of the present invention, relevant experimental descriptions are given below.

[0049] Our proposed approach was evaluated on two widely recognized benchmarks in the SLAM community: the Replica dataset and the TUM dataset, which include both monocular and depth-enhanced RGB-D sequences. The TUM RGB dataset presents inherent sensor acquisition challenges, characterized by significant motion blur in color images and sparse depth measurements due to limitations of the Kinect v1 sensor. These characteristics rigorously test the algorithm's ability to preserve detail amidst noisy real-world conditions. In contrast, the Replica dataset provides synthetic environments with millimeter-accurate ground truth poses and dense depth maps, making it an ideal benchmark for evaluating upper bounds in controlled scenarios. Our experimental approach adheres to established evaluation metrics and utilizes eight Replica sequences (office 0-4 and room 0-2) as well as three representative TUM RGB-D sequences (fr1 desk, fr2 xyz, fr3 office), ensuring comprehensive validation under both synthetic and real-world conditions.

[0050] For RGB rendering quality evaluation, three widely adopted perceptual metrics were used: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). For camera trajectory evaluation, the Root Mean Square Error (RMSE) metric of the Trajectory Absolute Error (ATE) was used to quantify localization accuracy. To evaluate RGB-only performance, ORB-SLAM2, MonoGS, and MotionGS were benchmarked on the TUM RGB dataset. For a comprehensive analysis of RGB-D scenes, the present invention was compared with NeRF-based SLAM implementations (Vox-Fusion, NICE-SLAM, Point-SLAM) and systems based on 3D Gaussian stitching (SplaTAM, GS-SLAM, MonoGS). All comparisons used benchmark results published in the original papers and reproduction experiments conducted by the present invention using open source implementations on the same hardware configuration.

[0051] As shown in Figures 3(a) and 3(b), we compared the trajectory estimation accuracy of ViMGS and MonoGS on the TUM-RGBD fr3 dataset. ViMGS is a SLAM algorithm based on MViT, which embeds MViT into SLAM to provide it with an inverse depth map and a refined edge depth map as the input depth map. MonoGS is an existing open-source algorithm. Quantitative evaluation results show that ViMGS achieves better global alignment with the ground-truth trajectory than MonoGS, with a trajectory absolute error (ATE) of 0.0236 m compared to 0.0437 m for MonoGS, resulting in a 46.0% improvement in estimation accuracy. This improvement stems from ViMGS's novel MViT architecture, which employs a hierarchical multi-scale pyramid structure to progressively extract high-frequency local features and low-frequency semantic information at different resolutions. This multi-scale paradigm significantly reduces the feature matching errors inherent in single-scale methods, especially in complex scenes with occlusions and texture variations, effectively suppressing the accumulated drift in long-term trajectory estimation. The Transformer-based self-attention mechanism establishes a global contextual relationship between visual features, thereby further enhancing the robustness of the system. Unlike MonoGS, which has a limited local receptive field that leads to gradual trajectory drift, the architecture of the present invention maintains spatiotemporal consistency by dynamically weighting the feature correlations in the entire sequence. It turns out that this function is particularly important for solving the problem of error propagation in loop closure detection and relocalization. In Figure 3(a), the horizontal axis Frame idx represents the frame number corresponding to the trajectory point of the system (or architecture); the vertical axis ATE represents the absolute error of the trajectory; the value 600 after Frame represents the frame number. In Figure 3(b), Input image represents the input image.

[0052] To quantitatively evaluate depth estimation performance, we analyzed keyframes at key trackpoints (frames 600, 1150, and 2400). At frame 600, the performance difference between the two systems becomes noticeable. ViMGS outperforms MonoGS with an RMSEATE of 0.68 cm compared to 2.29 cm for MonoGS. At frame 1150, MonoGS exhibits noticeable artifacts in depth estimation, including structural discontinuities in distant planar surfaces (e.g., distant ground regions), blurred object boundaries at small-scale structures (e.g., doll outlines), and inconsistent depth predictions in homogeneous regions (e.g., seat backs). ViMGS mitigates these limitations through multi-scale feature fusion, combining convolutional edge details with Transformer-derived semantic contours. This synergistic approach improves performance in weakly textured regions. In complex scenes (frame 2400), ViMGS widens the performance gap by maintaining temporal consistency through inter-frame feature propagation. ViMGS leverages inter-frame feature propagation to dynamically adjust geometric and semantic information between consecutive frames, achieving robust temporal consistency. This mechanism effectively alleviates the challenges of dynamic objects. MonoGS cannot track fast-moving objects (such as pedestrians), resulting in depth drift caused by motion blur. ViMGS refines object trajectories by propagating spatiotemporal features.

[0053] Figures 4(a) and 4(b) compare the trajectory estimation performance of ViMGS and MonoGS on the TUM-RGBD fr3 dataset. The horizontal axis represents x-values, the vertical axis represents y-values, both in meters, and gt represents the ground-truth trajectory. Figure 4(a) corresponds to MonoGS-SLAM, with an ATE RMSE of 0.04366508360528711, while Figure 4(b) corresponds to ViMGS-SLAM (the present invention), with an ATE RMSE of 0.023590304457389574. These results demonstrate that ViMGS maintains a consistently lower RMSE ATE than MonoGS throughout the entire run sequence. However, both methods exhibit transient error accumulation during the initialization and termination phases of the system run cycle, likely due to sensor calibration convergence dynamics and boundary condition uncertainties. Furthermore, both models exhibit significant error fluctuations during sharp turns, which are inherently challenging for dynamic object tracking due to rapid changes in viewpoint (especially in the presence of moving occluders). However, ViMGS demonstrates higher robustness with an average tracking error 50% lower than MonoGS. This performance gap highlights the ability of ViMGS to reduce error propagation through temporal feature aggregation and occlusion-aware attention.

[0054] like Figure 5As shown in the figure, the present invention analyzes the 3D Gaussian reconstruction process of ViMGS and MonoGS on the TUM RGB-D fr3 dataset. Figure 5 The left image in the middle corresponds to MonoGS-SLAM, and the right image corresponds to ViMGS-SLAM (i.e., the present invention). "splats" represents primitives, "time" represents time, and "ATE" represents absolute trajectory error. Experimental results show that ViMGS achieves significantly higher RMSE (Residual Value Sequence) and "ATE" (Absolute Error in Trajectory) accuracy than MonoGS. Notably, ViMGS generates 95,156 Gaussian primitives in the neighborhood, a 170.64% increase compared to MonoGS's 35,159 primitives. This significant difference in primitive density demonstrates ViMGS's ability to generate a more geometrically complete point cloud while maintaining millimeter-level localization accuracy.

[0055] Table 1 compares ViMGS-SLAM with several baseline models on the TUM-RGB and TUM-RGBD datasets. The TUM dataset, collected by the Machine Vision Group at the Technical University of Munich, provides ground-truth RGB-D sequences for evaluating visual SLAM systems. When using monocular input, ViMGS-SLAM outperforms GO-SLAM, MGSO, and MonoGS across all validation metrics. ViMGS-SLAM achieves 23.27 dB (PSNR), 0.78 (SSIM), and 0.27 (LPIPS), respectively, outperforming the next-best methods by Δ+2.01 dB, Δ+0.04, and Δ+0.06. This highlights ViMGS-SLAM's superior depth estimation and feature extraction capabilities, particularly in dynamic scenes and weakly textured areas. When using depth-assisted input, ViMGS-SLAM maintains a lead of 24.60 dB (PSNR), 0.81 (SSIM), and 0.21 (LPIPS), outperforming the next-best methods by +2.08 dB, +0.08, and +0.04, respectively. It is worth noting that the TUM dataset contains challenging dynamic indoor scenes (e.g., moving objects, people) and lighting changes, which exacerbate the complexity of depth estimation. The performance improvement comes from ViMGS-SLAM's MViT architecture, which captures global motion patterns through coarse-grained hierarchical feature fusion and enhances local edge and texture perception through a fine-grained attention mechanism. It is worth noting that the monocular depth estimation accuracy of ViMGS-SLAM on the TUM-RGB dataset is very close to the depth-assisted depth estimation accuracy on TUM-RGBD. This shows that the MViT architecture achieves near-sensor-level accuracy in depth inference and significantly improves the fidelity of scene reconstruction based on 3DGS images.

[0056] Table 1

[0057] Table 1. Comparison of rendering performance of different SLAM methods on the TUM dataset (ATE RMSE↓ [cm]). The proposed method consistently outperforms all baseline models on TUM RGB and TUM RGBD. Results are ranked first (bold numbers), second (bold numbers in brackets), and third (italic numbers) by data performance. The baseline model data for TUM was taken from public data from Go-SLAM and MGSO, and local data from a MonoGS run.

[0058] The multi-scale visual converter provided by the embodiment of the present invention can enhance multi-scale feature aggregation and edge-aware geometric refinement. For monocular depth estimation tasks, it accurately preserves object topology, terrain structure and absolute metric scale in depth prediction, and has high depth estimation quality.

[0059] It should be noted that, in the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A multi-scale visual converter, characterized in that The multi-scale visual converter comprises: The multi-scale feature encoding and position embedding module is used to perform symmetrical padding on the input RGB image, then perform multi-scale division with different resolutions, and obtain the feature vector of each scale through linear projection and position encoding; The encoder module is implemented using a hierarchical Transformer and a cross-scale attention mechanism to obtain the fused feature vectors of all scales, extract features using local branches to obtain local branch feature information, obtain the feature vectors of the lowest resolution scale, extract features using global branches to obtain global branch feature information, and merge the local branch feature information with the global branch feature information to obtain a set of feature information containing different resolutions. The decoder module is used to fuse the feature information set through cross-scale feature interaction based on the attention mechanism to obtain fused feature information; the fused feature information is progressively upsampled using the introduced multi-layer perceptron (MLP) to restore the corresponding image information, and then the inverse depth map is reconstructed using the DPT decoder; A depth information generation module is configured to generate, based on the inverse depth map, a metric depth map representing absolute depth information and a refined edge depth map having a resolution consistent with that of the RGB image.

2. The multi-scale visual converter according to claim 1, wherein The multi-scale feature encoding and position embedding module performs symmetrical filling processing on the input RGB image, then performs multi-scale division with different resolutions, and obtains the feature vector of each scale through linear projection and position encoding, including the following process: Perform symmetrical filling processing on the input RGB image along the vertical dimension to obtain a symmetrically filled RGB image; Dividing the symmetrically filled RGB image into a plurality of image blocks according to an overlapping division method as first-scale image blocks; The RGB image after the symmetrical filling process is downsampled by 2 times, and then divided into multiple image blocks according to a non-overlapping partitioning method as second-scale image blocks; Downsampling the RGB image after the symmetrical filling process by a factor of 4, and using the obtained image as a third-scale image block; wherein the image resolution decreases in the first scale, the second scale, and the third scale in sequence; The image patches at each scale are linearly projected into feature vectors with trainable positional encodings to obtain feature vectors at each scale.

3. The multi-scale visual converter according to claim 2, wherein The feature vector of each scale is expressed as: ； Among them, the input RGB image is represented as , , and Represents the height and width of the input RGB image respectively; Indicates the The eigenvector of the scale, ; , represents a learnable embedding matrix; Indicates the Image patches at scale; , and Both indicate The size of the image patch at scale; It is The spatial dimension of scale, Represents the dimension of the feature vector after embedding; It is a flattening function for multidimensional arrays, used to convert image blocks into one-dimensional feature vectors.

4. The multi-scale visual converter according to claim 1, wherein In the encoder module, both the local branch and the global branch are implemented using a multi-head attention mechanism combined with a multi-layer perceptron architecture, and both have L layers of transformer layers stacked in sequence; the local branch uses 6 attention heads, and the global branch uses 12 attention heads; the multi-layer perceptrons of the local branch and the global branch include two fully connected layers, and use Gaussian error linear units as intermediate activation functions.

5. The multi-scale visual converter according to claim 4, wherein: In the L layers of transformers stacked sequentially, the The processing of the layer is expressed as: ； in, Represents a multi-head attention mechanism; Representation layer normalization; represents a multilayer perceptron; Indicates the The basic features of the layer input are transmitted through residual connections; Indicates the Layer Processed intermediate features; Represents the final output features after L layer processing.

6. The multi-scale visual converter according to claim 1, wherein The decoder module performs feature fusion on the feature information set through cross-scale feature interaction based on the attention mechanism to obtain fused feature information, including: Through cross-scale feature alignment based on the attention mechanism, the dynamic weights generated by the interaction between the global query vector and the local key-value pairs are applied to the features of each scale, realizing context-aware multi-scale feature fusion and obtaining fused feature information.

7. The multi-scale visual converter according to claim 6, wherein: The calculation formula of the fusion feature information is: ； in, Indicates fusion feature information; Indicates the total number of multiple scales; represents the global query vector; Indicates the Local keys of scale; represents transpose; Represents the dimension of the feature vector after embedding; Indicates the Scale feature information.

8. The multi-scale visual converter according to claim 7, wherein: The decoder module uses the introduced multi-layer perceptron MLP to perform progressive upsampling on the fused feature information to restore the corresponding image information, and then uses the DPT decoder to reconstruct the inverse depth map, including: The MLP is used to perform nonlinear refinement on the skip connection features, and the edge alignment capability is enhanced by combining the edge mask. The network is then gradually upsampled through transposed convolution and feature alignment operations, and finally the DPT decoder is used to reconstruct the inverse depth map. The formula used is: ； in, Indicates the Image information restored by scale; represents a multilayer perceptron; , Indicates the Scale-corresponding skip connection features; It is The spatial dimension of scale, Represents the dimension of the feature vector after embedding; , Indicates the The edge mask corresponding to the scale; Represents element-wise multiplication.

9. The multi-scale visual converter according to claim 8, wherein In the decoder module, the calculation formula of the inverse depth map is expressed as: ； in, represents transposed convolution; represents the feature alignment operation; Indicates the The image features after transposed convolution and feature alignment operations on the scale-restored image information; It means element-by-element addition; represents the inverse depth map; is the Sigmoid function; is a 1 The convolution kernel of 1; Represents the image features obtained at all scales; represents the bias term.

10. The multi-scale visual converter according to claim 9, wherein: The process of the depth information generation module generating a metric depth map representing absolute depth information according to the inverse depth map includes: According to the inverse depth map, based on geometric constraints, a metric depth map representing absolute depth information is generated; it can be expressed as: ； in, represents the metric depth map; Indicates the focal length of the camera; Indicates the width of the object's projection on the image plane.

Images