Lightweight monocular depth estimation method based on selective scan state space model

Abstract

This invention provides a lightweight monocular depth estimation method based on a selective scanning state-space model, comprising the following steps: S1: Preprocessing the input monocular RGB image and then inputting it into a hierarchical encoder without position coding to extract multi-scale feature maps; S2: Performing spatial downsampling of the feature maps using separable convolution between different cascaded stages of the encoder; S3: Inputting the high-order semantic features output by the encoder into the decoder for progressive upsampling to gradually restore the spatial resolution to be consistent with the input image; S4: Using a regression head to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of that position relative to the camera; S5: Performing engineering optimization and deployment of the trained model, and systematically evaluating its inference speed, memory usage, and depth estimation accuracy on the target hardware platform.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a lightweight monocular depth estimation method based on a selective scanning state space model. Background Technology

[0002] Monocular depth estimation (MDE) aims to predict the distance of each pixel in a scene relative to a camera from a single RGB image. As a fundamental task in computer vision, it is crucial for achieving 3D scene understanding, autonomous navigation, and augmented reality (AR). In autonomous driving, depth information is a prerequisite for obstacle detection and path planning; in consumer electronics, it is widely used for mobile phone portrait blurring and interior design simulation. Because monocular vision systems offer advantages over binocular or LiDAR systems such as lower cost, simpler structure, and no baseline limitations, how to recover the 3D spatial structure from a 2D projection using purely algorithmic methods has always been a hot topic of interest in academia and industry.

[0003] To address the global modeling problem, the Visual Transformer (ViT) was introduced into the field of depth estimation. The Transformer achieves pixel-level global association through a self-attention mechanism. However, the performance improvement it brings comes at the cost of extremely high computational complexity. (1) Quadratic complexity challenge: The computational complexity of the self-attention mechanism is proportional to the square of the number of input pixels $N$. For high-resolution (such as 1080P or 4K) industrial applications, computational load and video memory usage will increase exponentially.

[0004] (2) Hardware deployment challenges: Transformer models typically contain a large number of parameters (often hundreds of megabytes or even several gigabytes), and their irregular memory access patterns are extremely unfriendly to embedded hardware such as mobile SoCs (System-on-a-Chip) and FPGAs. This means that high-precision Transformer models often can only run on expensive server-side graphics cards, which cannot meet the real-time processing requirements of terminal devices such as drones and smartphones.

[0005] While some lightweight solutions exist (such as MobileNet or feature distillation), these methods often sacrifice global visibility for speed, resulting in poor performance in complex outdoor long-range scenarios. How to achieve global geometric modeling capabilities with extremely low computational power consumption and a very small parameter scale has become a core technical challenge that must be addressed when migrating monocular depth estimation to edge computing. Summary of the Invention

[0006] This invention provides a lightweight monocular depth estimation method based on a selective scanning state-space model. Addressing the "accuracy-power consumption" trade-off faced by existing monocular depth estimation technologies when deployed on mobile devices, this method maintains Transformer-level global geometry modeling capabilities while reducing computational complexity from the quadratic order of the number of pixels. The model's performance is reduced to linear level (O(N)) and the number of model parameters is significantly reduced, making it suitable for real-time inference on embedded chips. The technical solution is as follows: A lightweight monocular depth estimation method based on a selective scanning state-space model includes the following steps: S1: Preprocess the input monocular RGB image, then input the preprocessed RGB image into a hierarchical encoder without position coding, and extract multi-scale feature maps using cascaded visual Mamba blocks. S2: Spatial downsampling of feature maps is performed using separable convolution between different cascaded stages of the encoder; S3: Input the high-order semantic features output by the encoder into the decoder for upsampling step by step, gradually restoring the spatial resolution to be consistent with the input image; S4: Use the regression head to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of that location relative to the camera; S5: Optimize and deploy the trained model in an engineering manner, and systematically evaluate its inference speed, memory usage, and depth estimation accuracy on the target hardware platform.

[0007] Furthermore, in step S1, the input monocular RGB image is preprocessed by scaling the monocular RGB image to the standard input resolution and normalizing it to the [0, 1] interval; then, the image is converted into an initial feature map using a convolutional layer with a kernel size of 4×4 and a stride of 4, achieving 4x downsampling to obtain the preprocessed feature map.

[0008] In step S1, the visual Mamba block adopts a dual-path parallel structure: the first path is the backbone path, which includes a depthwise separable convolutional layer and a two-dimensional selective scanning module, abbreviated as SS2D module, used for coordinating local spatial alignment and global context modeling with linear complexity; the second path is a gated path, which includes a linear mapping layer and a gated linear unit, used for nonlinear transformation and adaptive information filtering of the feature map; the outputs of the first path and the second path are fused by element-wise multiplication and aggregated with the input of the visual Mamba block via residual connections.

[0009] The SS2D module processes the input two-dimensional feature map X through three steps: unfolding, scanning, and merging. ; Where X is the output feature map from the layer before the VMB block, and its dimensions are B×C×H×W, which represent the batch size, channel number, feature map height and width, respectively. Unfolding involves flattening the two-dimensional features into a sequence along four preset paths: Sequence 1 from the top left to the bottom right; Sequence 2 from the bottom right to the top left; Sequence 3 from the bottom left to the top right; and Sequence 4 from the top right to the bottom left. Scanning involves inputting the sequences of the four scanning directions into four independent S6 state-space equations, thereby achieving parallel capture of the spatial dependencies of the image in the four directions. Merging involves rearranging the four sequence vectors processed by the S6 state-space equations back to the original spatial topology by performing the inverse mapping operation that is the opposite of the unfolding operation. The feature maps in the four directions are then weighted and fused using learnable weight parameters to obtain the final SS2D output.

[0010] Furthermore, in step S2, the hierarchical encoder is divided into four stages, and each stage is downsampled through a convolutional layer with a stride of 2. In the shallow stage Stage 1 / 2, fine-grained spatial details are preserved. In the deep stage Stage 3 / 4, high-order semantic information is extracted. To prevent memory overflow under high channel count, the state dimension d_state is limited to between 8 and 32.

[0011] Furthermore, in step S3, the high-order semantic features output by the encoder are input to the decoder for progressive upsampling to gradually restore the spatial resolution to match the input image, including the following steps: S31: In the decoding stage, the fine-grained spatial features extracted in the first stage of the encoder are passed to the front of the output layer through skip connections; S32: Lightweight channel attention units are used to recalibrate the weights of the fused features channel by channel. First, global average pooling is used to compress the two-dimensional spatial information of each channel into a scalar statistical value. Then, two fully connected layers with ReLU activation and Sigmoid gating are used to learn the non-linear dependencies between channels and generate importance weight coefficients for each channel. Finally, these weights are multiplied with the original fused features channel by channel to adaptively enhance the response strength of key channels in the depth estimation task.

[0012] Furthermore, in step S4, an end-to-end supervised training paradigm is adopted, using pairs of RGB images and their corresponding ground truth depth maps as training data; the training objective is to minimize the mixed loss function between the predicted depth map and the ground truth depth map, including the following steps: S41. Set the initial learning rate. End-to-end training was performed using the AdamW optimizer, with a total of 100 training epochs. The batch size was set to 8 or 16 based on the GPU memory capacity, and a cosine annealing strategy was used to gradually decay the learning rate from the initial value during training. ; S42. The loss function adopts a weighted combination of scale-invariant loss SILog and local gradient loss, with the weight ratio set to 1:0.5; The hybrid loss function The expression is: ; in, For scale-invariant logarithmic loss, This is a local gradient loss, and the ratio of their weight coefficients is... Set to 1:0.5.

[0013] Furthermore, in step S5, the trained PyTorch model is exported as an ONNX intermediate format, and FP16 half-precision quantization and computation graph optimization are performed through the NVIDIA TensorRT inference engine. Finally, it is deployed on an embedded computing platform to achieve high-performance real-time inference.

[0014] Furthermore, the S6 state-space equation utilizes the discretized state transition matrix. and the discretized input mapping matrix The state transition matrix after discretization is obtained by processing four sequences in parallel. and the discretized input mapping matrix They are defined as follows: ; ; Where A is the original state transition matrix of the continuous system, B is the original input mapping matrix of the continuous system, I is the identity matrix, and Δ is the discretization time step.

[0015] A lightweight monocular depth estimation system based on a selective scanning state space model includes a feature extraction module, a feature fusion module, and a depth prediction module. The feature extraction module is used to preprocess the input monocular RGB image, and then input the preprocessed RGB image into a hierarchical encoder without position coding, and extract multi-scale feature maps using cascaded visual Mamba blocks. The feature fusion module is used to perform spatial downsampling of feature maps between different cascaded stages of the encoder using separable convolution; the high-order semantic features output by the encoder are input into the decoder for progressive upsampling to gradually restore the spatial resolution to be consistent with the input image; then the regression head is used to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of the position relative to the camera. The depth prediction module is used to perform engineering optimization and deployment of the trained model, and to systematically evaluate its inference speed, memory usage and depth estimation accuracy on the target hardware platform.

[0016] The lightweight monocular depth estimation method based on a selective scanning state-space model introduces a State Space Model (SSM), specifically the Mamba algorithm (a structured state-space sequence model proposed by Albert Gu and Tri Dao in 2023, which overcomes the linear time-invariant (LTI) limitation of traditional SSMs by introducing an input-dependent selective mechanism, achieving Transformer-level performance in sequence tasks such as language modeling). Mamba, through its selective scanning mechanism, reduces computational complexity to linear levels (O(N)) while maintaining the global receptive field. This invention designs a novel lightweight vision architecture by reconstructing the Mamba algorithm in two dimensions and compressing the model, aiming to provide a high-precision, low-power depth perception solution for embedded devices. Attached Figure Description

[0017] Figure 1 This is the overall architecture diagram of the lightweight monocular depth estimation method based on the selective scanning state space model; Figure 2 This is the SS2D scanning path diagram; Figure 3 This is a diagram of the internal structure of the VMB block; Figure 4 It is a performance comparison curve. Detailed Implementation

[0018] The lightweight monocular depth estimation method based on a selective scan state-space model abandons the traditional self-attention mechanism and instead adopts a selective scan-based state-space model as the core of feature extraction. For continuous state space, the state-space model treats the image pixel sequence as a one-dimensional continuous signal. Through hidden state Establish a mapping relationship between input and output $y(t)$: While maintaining the global geometric modeling capabilities at the Transformer level, reduce the computational complexity from the quadratic order of the number of pixels (O(N)). 2 The computational resource consumption in high-resolution scenarios is significantly reduced to linear level (O(N)), enabling the model to perform real-time inference on embedded devices. ; ; In this model, A is the state transition matrix, describing the evolution of the hidden state; B is the input projection matrix, controlling the influence of the input signal on the hidden state; and C is the output projection matrix, mapping the hidden state to the output signal. h(t) represents the hidden state of the system at time t, x(t) represents the input signal at time t, and y(t) represents the output signal at time t. These two formulas together describe the core idea of ​​the state-space model: by using the hidden state h(t) as an intermediate representation, the one-dimensional input signal x(t) is dynamically mapped to the output signal y(t), thereby achieving global modeling of the sequence data.

[0019] Represents an L-dimensional real vector space. The input image features corresponding to time t are represented by an L-dimensional real vector, where L is the number of input feature channels; Represents an N-dimensional real vector space. The system hidden state at time t is represented by an N-dimensional real vector, where N is the dimension of the hidden state defined by the state-space model. The model continuously accumulates global context and image geometric information from the N-dimensional hidden state, achieving long-range dependency modeling without using self-attention. To implement the aforementioned continuous-time equations in a digital signal processing system, the continuous system is discretized by introducing a time step parameter Δ (Delta, i.e., sampling interval / discretization step size). The discretized system parameters are as follows: Discretized state transition matrix) and The discretized input mapping matrix is ​​defined as follows: ; ; In practical calculations, this invention uses a first-order Taylor expansion for simplification. Calculation: .

[0020] Where A is the original state transition matrix of the continuous system, B is the original input mapping matrix of the continuous system, I is the identity matrix, and Δ is the discretization time step. The meaning of the above two formulas is: by using the Zero-Order Hold (ZOH) rule, the continuous-time state-space model is converted into a discrete-time form—the first formula defines the discretized state transition matrix. (Obtained by adding the matrix exponent to the identity matrix), the second formula defines the discretized input mapping matrix. (Obtained by the product of Δ and B). After this transformation, the model can be run efficiently on a digital computer in a recursive manner.

[0021] The Instead of static, hard-coded parameters, the input function is used: ; The three formulas above express the core idea of ​​the Mamba Selective Mechanism: Unlike the traditional linear time-invariant SSM, the parameters B (input mapping matrix), C (output mapping matrix), and Δ (time step) in Mamba are no longer fixed constants, but are dynamically generated by the input x through learnable linear projection functions s_B(x), s_C(x), and s_Δ(x). This design allows the model to adaptively adjust the state update rate and output mapping method according to the input content—for important information (such as object edges in the depth map), Δ takes a smaller value to increase the state update frequency and capture detailed changes; for redundant information (such as sparsely textured sky regions), Δ takes a larger value to quickly skip over them. This is the key innovation that distinguishes Mamba from traditional SSM.

[0022] Unlike the fixed parameterization of traditional SSM, Mamba achieves "selective filtering" of spatial information by dynamically varying three input-dependent parameters—B(x), C(x), and Δ(x)—as they change with the input feature x. In depth estimation tasks, the model can automatically identify and ignore sparsely textured regions (such as flat surfaces like the sky and white walls), assigning larger Δ values ​​to these regions for quick skipping; while for critical regions with abrupt depth changes (such as object contours and occlusion boundaries), smaller Δ values ​​are assigned to finely capture geometric abrupt changes. This selectivity allows the lightweight model to achieve detail capture capabilities comparable to heavyweight Transformer models while maintaining low computational overhead.

[0023] The lightweight monocular depth estimation method based on a selective scanning state-space model includes the following steps: S1: As Figure 1As shown, the input monocular RGB image is preprocessed, and then the preprocessed RGB image is input into a hierarchical encoder without position coding. Multi-scale feature maps are extracted using cascaded visual Mamba blocks (VMBs).

[0024] The hierarchical encoder is a macroscopic architecture consisting of four cascaded stages. The visual Mamba blocks (VMBs) are the basic building blocks within each stage of the encoder. Each stage is composed of several stacked VMBs and is responsible for feature extraction and global context modeling at the corresponding resolution. In short, the relationship between the hierarchical encoder and VMBs is that of a whole to a part: the encoder defines a hierarchical feature extraction framework from high resolution to low resolution, while the VMBs are the core computational modules at each level of this framework.

[0025] Specifically The input monocular RGB image is preprocessed as follows: First, the image is uniformly scaled to the standard input resolution (e.g., 224×224 or 384×384 pixels) and normalized to the [0, 1] interval; then, a convolutional layer with a kernel size of 4×4 and a stride of 4 (i.e., Patch Embedding operation) is used to convert the image into an initial feature map, achieving a 4x downsampling (e.g., a 224×224 input image is converted into a 56×56 feature map), resulting in the preprocessed feature map.

[0026] The preprocessed feature map is then input into a hierarchical encoder without positional encoding. The hierarchical encoder internally uses cascaded visual Mamba blocks (VMBs) to extract multi-scale feature maps. For example... Figure 3 As shown, the visual Mamba block adopts a dual-path parallel structure: the first path is the main branch, which includes depthwise separable convolutional layers (DWConv) and a two-dimensional selective scan module (SS2D) for coordinating local spatial alignment and global context modeling with linear complexity; the second path is the gating branch, which includes linear mapping layers and gated linear units (GLUs) for performing nonlinear transformations and adaptive information filtering on the feature map. The outputs of the two paths are fused by element-wise multiplication.

[0027] The first path (the main path) comprises a 3×3 depthwise separable convolutional layer (DWConv), an SS2D module, and a SiLU activation function layer connected in sequence, used to coordinate local spatial alignment and global modeling with linear complexity. DWConv is responsible for capturing local spatial details (such as edge orientation and texture gradients) with extremely low computational overhead, while the SS2D module models global context dependencies (such as cross-region structural continuity) with O(N) linear complexity. The two complement each other, balancing local accuracy with global consistency.

[0028] The second path (gated path) includes a linear projection layer (used to project the input features along the channel dimension to increase expressive power) and a gated linear unit (GLU, which generates gating coefficients between 0 and 1 through the sigmoid activation function to achieve adaptive filtering of information flow—selectively retaining features useful for depth estimation and suppressing redundant information), so that the model has information selectivity similar to the attention mechanism, but the computational cost is much lower than that of traditional self-attention.

[0029] The outputs of the first path and the second path are fused by element-wise multiplication and then aggregated with the input of the visual Mamba block via residual connections.

[0030] Since native Mamba can only process one-dimensional sequences, this invention designs a two-dimensional selective scan module (SS2D), which achieves omnidirectional fusion of image spatial information through a four-way scan section strategy. The SS2D module processes the input two-dimensional feature map X through three steps: "unfolding-scanning-merging". ; Where X is the output feature map from the layer before the VMB block, with dimensions B×C×H×W, representing the batch size, number of channels, feature map height, and width, respectively.

[0031] Represents a three-dimensional real tensor space. Let X represent a real tensor with height H, width W, and number of channels C, where H and W are the spatial height and width of the feature map, respectively, and C is the feature channel dimension. The network actually uses a batch-first computation format, and the complete input tensor dimension is... B represents the training / inference batch size.

[0032] like Figure 2 As shown, the unfolding process involves flattening the two-dimensional features into a sequence along four preset paths. These four paths are: • Sequence 1: From top left to bottom right (forward horizontal scan); • Sequence 2: From bottom right to top left (reverse horizontal scan); • Sequence 3: From bottom left to top right (forward vertical scan); • Sequence 4: From the top right corner to the bottom left corner (reverse vertical scan).

[0033] Scanning and Merging Stage (S6): The sequences from the four scanning directions are respectively entered into four independent S6 state-space equations (i.e., the discretized SSM core computational units), realizing parallel capture of the spatial dependencies of the image in four directions. Merging Stage: The four sequence vectors processed by S6 are rearranged back to the original spatial topology (i.e., restored to the B×C×H×W feature map shape) through the inverse mapping operation of the unfolding. The feature maps in the four directions are then weighted and fused using learnable weight parameters to obtain the final SS2D output.

[0034] Within the SS2D module of the main path (first path), four independent S6 state-space equations are used to process the sequence vectors of the four scanning directions in parallel. Each S6 state-space equation selectively filters the input sequence through a dynamically learned time step parameter Δ, enabling the model to adaptively ignore redundant background information (such as sparsely textured regions like the sky and flat roads) while retaining key depth variation information (such as geometrically abrupt regions like object boundaries and occlusion edges). The four processed sequences are then remapped and fused into a unified two-dimensional feature map by a feature aggregation unit (composed of a linear mapping layer and element-wise addition operations), which serves as the final output of the SS2D module.

[0035] In this embodiment, a detailed description of the internal processing flow of the Visual Mamba Block (VMB) is provided, illustrating the complete processing of an input feature map as it passes through a single VMB block. The VMB block is the basic computational unit constituting the hierarchical encoder; each VMB block employs a dual-path parallel design to simultaneously perform local feature refinement and global context aggregation.

[0036] S11. After the feature map enters the VMB block, it first undergoes layer normalization (LayerNorm) to stabilize the numerical distribution during training. The internal structure of the Visual Mamba Block (VMB) is as follows: Each VMB block contains a parallel dual-path structure—the backbone path (first path) is responsible for spatial-global joint feature extraction, and the gated path (second path) is responsible for adaptive information filtering; the outputs of the two paths are fused through element-wise multiplication, and then added to the original input of the VMB block via residual connection to obtain the final output of the VMB block. The specific structure of each path is as follows.

[0037] The main path (i.e., the first path mentioned above) contains a 3×3 depthwise separable convolutional layer (DWConv) for local spatial feature extraction, followed by the SS2D module, which captures global contextual dependencies with linear complexity. This design allows DWConv to compensate for the shortcomings of SS2D in local fine-grained modeling, while SS2D provides a global receptive field that DWConv lacks.

[0038] Gated path (i.e., the second path mentioned above): Adaptive gating enhancement of the output features of the main path is achieved through gated linear units (GLUs). The GLU generates a gating coefficient matrix between 0 and 1 based on the input features, and performs element-wise multiplication with the output features of the main path, thereby enabling selective information passage—assigning higher gating values ​​to features important for depth estimation (such as edges and contours) to enhance the response, and assigning lower gating values ​​to redundant features (such as uniform texture regions) to suppress noise.

[0039] In this dual-path design, SS2D is responsible for global relevance, while DWConv is responsible for local spatial alignment.

[0040] S12. Feature extraction based on SS2D, using four-directional scanning processing: flatten the feature map into a sequence along the four directions of "top left to bottom right", "bottom right to top left", "top right to bottom left" and "bottom left to top right" respectively; S13, Selective Filtering: Utilizing the Discretized State Transition Matrix and input mapping matrix Four sequences are processed in parallel. The time step is specified. It dynamically changes with the input, enabling the automatic ignoring of redundant background information.

[0041] S2. Spatial Dimension Downsampling: Between different stages of the encoder, spatial downsampling of feature maps is performed using 3×3 depthwise separable convolutions with a stride of 2. This significantly reduces the number of computational parameters while halving the spatial resolution (e.g., from H×W to H / 2×W / 2) and doubling the channel dimension (e.g., from C to 2C). This compresses spatial information while improving the semantic richness of features, achieving hierarchical feature abstraction from local texture to global semantics.

[0042] This invention employs a hierarchical architecture, aiming to progressively compress spatial dimensions and increase semantic density through stepwise downsampling. This design is similar to the pyramid structure of traditional CNN backbone networks (such as ResNet), but replaces the core operator of each stage from convolution to SS2D-based visual Mamba blocks, thereby obtaining the global receptive field without significantly increasing computational cost.

[0043] Staged Feature Extraction: The hierarchical encoder is divided into four stages, with downsampling performed between each stage using convolutional layers with a stride of 2. In the shallow stages (Stage 1 / 2), the feature map resolution is high, used to preserve fine-grained spatial details (such as object edges and texture direction). In the deep stages (Stage 3 / 4), the feature map resolution gradually decreases while the number of channels gradually increases, used to extract higher-order semantic information (such as object category and scene layout). In the deep stages (Stage 3 / 4), to prevent memory overflow under high channel counts, this invention limits the state dimension d_state to between 8 and 32. .

[0044] Skip connections and scale alignment: The fine-grained edge feature map output from Stage 1 is element-wise added to the depth feature map at the end of the decoder.

[0045] Loss function constraint: Scale-invariant log loss (SILog) combined with local gradient loss is used to constrain the overall consistency of the model when predicting absolute distance.

[0046] The hierarchical encoder utilizes the recursive nature of the Mamba algorithm to directly realize pixel-to-pixel positional awareness. It consists of four cascaded stages, with feature map resolutions for each stage being 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, respectively. Notably, because Mamba's recursive computation inherently possesses a natural sensitivity to sequence position, the hierarchical encoder of this invention can directly perceive the spatial relationships between pixels without introducing additional learnable positional encoding parameters, thereby further reducing the model's parameter count and computational overhead.

[0047] The state dimension within the SS2D module is limited to between 8 and 32, and the internal channel expansion factor is limited to between 1.5 and 2.0. Furthermore, the time step parameter of the dynamic learning is specifically calculated by adding its linear mapping value to the learnable parameter and then processing it through the Softplus activation function.

[0048] S3. Lightweight Feature Decoding and Edge Enhancement: The high-order semantic features output by the encoder (at a resolution of 1 / 32 of the original input) are input into the decoder for progressive upsampling, gradually restoring the spatial resolution to match the input image. During each upsampling stage, a Feature Fusion Module (FFM) with a Channel Attention (CA) mechanism is used to perform channel-by-channel weight calibration on the shallow spatial features of the corresponding scale returned by the encoder via skip connections. This strengthens feature channels sensitive to depth boundaries (such as high-frequency components corresponding to object contours) and suppresses over-response to textured areas (such as brick walls and asphalt pavements), thereby enhancing the contour clarity and sharpness of the depth map at object edges.

[0049] Multi-scale feature decoding and fusion: The decoder employs a coarse-to-fine progressive upsampling strategy, gradually restoring the low-resolution, high-semantic features (1 / 32 resolution) output by the encoder to a high-resolution depth map consistent with the original input resolution through multiple decoding stages. Each decoding stage includes three key operations: feature upsampling, skip connection fusion with features from the corresponding level of the encoder, and channel attention calibration. The steps include: S31. In the decoding stage, the fine-grained spatial features extracted in the first stage of the encoder are passed to the front of the output layer through skip connections. S32. A lightweight channel attention unit (Squeeze-and-Excitation module, SE-Net) is used to recalibrate the channel-by-channel weights of the fused features: First, global average pooling is used to compress the two-dimensional spatial information of each channel into a scalar statistical value (descriptor). Then, two fully connected layers (containing ReLU activation and Sigmoid gating) are used to learn the nonlinear dependencies between channels and generate the importance weight coefficients of each channel. Finally, these weights are multiplied with the original fused features channel by channel to adaptively enhance the response strength of key channels for depth estimation and suppress the interference of irrelevant or noisy channels, thereby significantly improving the clarity and sharpness of the depth map at the edges of objects.

[0050] S4. Depth Map Regression and Joint Loss Constraints: A regression head (composed of several convolutional layers and upsampling operations) maps the multi-channel feature map output by the decoder to a single-channel depth map with the same resolution as the original input image. Each pixel value represents the estimated distance (in meters) of that location relative to the camera. During model training, a hybrid loss function consisting of scale-invariant logarithmic loss (SILog) and local gradient loss is used to optimize model parameters. SILog loss constrains the overall scale consistency of global depth prediction (i.e., the relative error between predicted depth and true depth remains stable across the entire image), while gradient loss constrains the sharpness of depth changes at local object edges (i.e., the gradient at the object contour in the depth map remains consistent with the gradient in the true depth map).

[0051] Model Training and Loss Constraints: This invention employs an end-to-end supervised training paradigm, using pairs of RGB images and their corresponding ground-truth depth maps as training data. The training objective is to minimize the mixture loss function between the predicted and ground-truth depth maps. The training includes the following steps: S41. Set the initial learning rate. (i.e., 0.0001), using the AdamW optimizer (weight decay coefficient set to...). End-to-end training is performed, with a total of 100 epochs. The batch size is set to 8 or 16 based on the GPU memory capacity. A cosine annealing scheduler is used to gradually decay the learning rate from its initial value during training. .

[0052] S42. The loss function is a weighted combination of scale-invariant loss (SILog) and local gradient loss, with a weight ratio of 1:0.5.

[0053] The hybrid loss function The expression is: ; in, For scale-invariant logarithmic loss, This is a local gradient loss, and the ratio of their weight coefficients is... Set to 1:0.5.

[0054] S5. Inference Deployment and Performance Evaluation: The trained model is engineered and deployed, and its inference speed, memory usage, and depth estimation accuracy are systematically evaluated on the target hardware platform.

[0055] The trained PyTorch model is exported to the ONNX (Open Neural Network Exchange) intermediate format. It is then subjected to FP16 half-precision quantization (compressing 32-bit floating-point parameters to 16 bits, halving the model size and inference latency with almost no loss of precision) and computation graph optimization (including vertical layer fusion, horizontal layer fusion, and automatic kernel tuning) through the NVIDIA TensorRT inference engine. Finally, it is deployed on an embedded computing platform (such as NVIDIA Jetson Orin / XavierNX series) to achieve high-performance real-time inference.

[0056] Verification has shown that this model reduces inference latency by approximately 40% while maintaining the same accuracy as the traditional Transformer model, and significantly reduces GPU memory usage.

[0057] A lightweight monocular depth estimation system based on a selective scanning state space model is characterized by comprising a feature extraction module, a feature fusion module, and a depth prediction module. The feature extraction module is used to preprocess the input monocular RGB image, and then input the preprocessed RGB image into a hierarchical encoder without position coding, and extract multi-scale feature maps using cascaded visual Mamba blocks. The feature fusion module is used to perform spatial downsampling of feature maps between different cascaded stages of the encoder using separable convolution; the high-order semantic features output by the encoder are input into the decoder for progressive upsampling to gradually restore the spatial resolution to be consistent with the input image; then the regression head is used to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of the position relative to the camera. The depth prediction module is used to perform engineering optimization and deployment of the trained model, and to systematically evaluate its inference speed, memory usage and depth estimation accuracy on the target hardware platform.

[0058] The system is deployed in an embedded computing platform and uses low-level operators to perform multi-threaded parallel acceleration processing on the four-way parallel scanning process in the SS2D module.

[0059] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the lightweight monocular depth estimation method.

[0060] Example Parameter Configuration: The technical solution of the present invention will be described in detail below through specific examples, including recommended parameter settings, configuration basis and corresponding technical effect verification in different application scenarios.

[0061] In the specification of this invention, by listing parameter configuration tables for different application scenarios, the scope constraints in the claims can be further refined.

[0062] 1. Core Algorithm Hyperparameter Configuration Table; Table 1 below shows the core parameter settings of this invention under different model sizes (lightweight vs. standard), which directly determine the computational complexity and storage space.

[0063] Table 1

[0064] 2. Hierarchical encoder structure parameter table; The present invention provides the following quantitative limitations for the four stages of the encoder: Table 2

[0065] 3. Technical Effect Comparison Table; like Figure 4 As shown, the superiority of this invention is highlighted by comparing it with traditional Transformer architectures (such as Swin-T) and CNN architectures (such as ResNet) at the same accuracy.

[0066] Table 3

[0067] Multi-dimensional implementation case description: For different industrial application scenarios, this invention provides three typical parameter configuration modes to meet the needs ranging from extremely low power consumption to high-precision visual perception.

[0068] Example 1: Extreme Fast Mode – Designed for drones or micro-embedded devices, this mode aims to achieve the ultimate inference speed and the lowest power consumption.

[0069] Parameter configuration: Set the state dimension d_{state} = 8, and the expansion factor E = 1.2. The number of blocks in the encoder cascade stage is reduced to [1, 1, 3, 1].

[0070] Downsampling strategy: The initial Patch Embedding uses a large stride convolution of 8 times 8 to quickly compress the spatial dimensions.

[0071] Technical results: When processing 224 x 224 inputs, the FLOPs are less than 5G, and depth perception of more than 60 FPS can be achieved directly on low computing power MCUs or mobile DSPs.

[0072] First, the state dimension d_state = 8 directly constrains the internal computational complexity of the state-space model in the SS2D module. The core recursive operation complexity of SSM is linearly related to the state dimension. Compressing this parameter to 8 (usually 16 or greater) significantly reduces the computational cost per scan.

[0073] Secondly, the expansion factor E = 1.2 controls the channel width of the projection matrix at each layer. In the Mamba class structure, the input is first linearly projected to the number of channels d_state × E, and then compressed back. E close to 1 means that the matrix multiplication size of the projection layer is minimized, almost to the point of "no expansion".

[0074] Third, the encoder cascade blocks [1, 1, 3, 1] total only 6 blocks, and their distribution across different resolution stages is extremely restrained. In contrast, conventional ViT or Swin-type structures typically stack 6 to 18 blocks in the third stage. This directly eliminates a large amount of redundant attention or SSM computation.

[0075] Fourth, the initial patch embedding uses an 8×8 large stride convolution, which compresses the spatial sequence length to 1 / 64 of its original length in the first layer. This means that the number of tokens processed in all subsequent blocks is extremely low, which is the most critical structural design for reducing overall FLOPs.

[0076] Example 2: Balanced Mode – For Advanced Driver Assistance Systems (ADAS); This mode is the standard implementation of the present invention, balancing long-range accuracy and real-time performance.

[0077] Parameter configuration: Set state dimensions expansion factor The number of blocks in each stage of the encoder is distributed as [2, 2, 9, 2], and the number of reference channels is... .

[0078] Core logic: It adopts standard SS2D four-way scanning logic and introduces a lightweight channel attention mechanism (CA) at the decoder end.

[0079] Technical effects: Processing on the NVIDIA Jetson Orin platform The image latency is stable at around 18ms, which meets the requirements for real-time obstacle avoidance.

[0080] First, the block distribution [2, 2, 9, 2] concentrates the main computation in the third stage (9 blocks), where the feature map has already been downsampled twice, resulting in lower resolution. The marginal computational cost of stacking more blocks is manageable. In the early high-resolution stages (the first two stages), only 2 blocks are placed in each stage, avoiding excessive stacking in high-resolution regions. Second, the decoding end introduces "lightweight channel attention" (CA), rather than computationally O(N). 2 Spatial self-attention or cross-attention. Channel attention only requires global average pooling plus two 1×1 convolutions, with negligible additional overhead, yet provides effective feature relabeling.

[0081] Experimental evaluation and performance comparison: The present invention has undergone rigorous testing of the above embodiments, and the resulting comparative data on technical effects are shown in Table 4 below: Table 4: Performance Comparison of the Invention with Existing Mainstream Architectures

[0082] This invention introduces State Space Models (SSM) and applies the Mamba algorithm. Through a selective scan mechanism, it reduces computational complexity to linear levels (O(N)) while maintaining the global receptive field. By reconstructing the Mamba algorithm in two dimensions and compressing the model, this invention designs a novel lightweight vision architecture that provides a high-precision, low-power depth perception solution for embedded devices.

Claims

1. A lightweight monocular depth estimation method based on a selective scanning state-space model, characterized in that, Includes the following steps: S1: Preprocess the input monocular RGB image, then input the preprocessed RGB image into a hierarchical encoder without position coding, and extract multi-scale feature maps using cascaded visual Mamba blocks. S2: Spatial downsampling of feature maps is performed using separable convolution between different cascaded stages of the encoder; S3: Input the high-order semantic features output by the encoder into the decoder for upsampling step by step, gradually restoring the spatial resolution to be consistent with the input image; S4: Use the regression head to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of that location relative to the camera; S5: Optimize and deploy the trained model in an engineering manner, and systematically evaluate its inference speed, memory usage and depth estimation accuracy on the target hardware platform. In step S1, the visual Mamba block adopts a dual-path parallel structure: the first path is the main path, which includes a depth-separable convolutional layer and a two-dimensional selective scanning module. The two-dimensional selective scanning module is abbreviated as SS2D module, which is used to coordinate local spatial alignment and global context modeling with linear complexity. The second path is a gated path, which includes a linear mapping layer and a gated linear unit, used to perform nonlinear transformation and adaptive information filtering on the feature map; the outputs of the first path and the second path are fused by element-wise multiplication and aggregated with the input of the visual Mamba block via residual connections. The SS2D module processes the input two-dimensional feature map X through three steps: unfolding, scanning, and merging. ； Where X is the output feature map from the layer before the VMB block, and its dimensions are B×C×H×W, which represent the batch size, channel number, feature map height and width, respectively. Unfolding involves flattening the two-dimensional features into a sequence along four preset paths: Sequence 1 from the top left to the bottom right; Sequence 2 from the bottom right to the top left; Sequence 3 from the bottom left to the top right; and Sequence 4 from the top right to the bottom left. Scanning involves inputting the sequences of the four scanning directions into four independent S6 state-space equations, thereby achieving parallel capture of the spatial dependencies of the image in the four directions. Merging involves rearranging the four sequence vectors processed by the S6 state-space equations back to the original spatial topology by performing the inverse mapping operation that is the opposite of the unfolding operation. The feature maps in the four directions are then weighted and fused using learnable weight parameters to obtain the final SS2D output.

2. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: In step S1, the input monocular RGB image is preprocessed by scaling the monocular RGB image to the standard input resolution and normalizing it to the [0, 1] interval. Then, a convolutional layer with a kernel size of 4×4 and a stride of 4 is used to convert the image into an initial feature map, achieving 4x downsampling and obtaining the preprocessed feature map.

3. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: In step S2, the hierarchical encoder is divided into four stages, and each stage is downsampled by a convolutional layer with a stride of 2. In the shallow stage, Stage 1 / 2, fine-grained spatial details are preserved. In the deep stage, Stage 3 / 4, high-order semantic information is extracted. To prevent memory overflow under high channel count, the state dimension d_state is limited to between 8 and 32.

4. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: In step S3, the high-order semantic features output by the encoder are input to the decoder for step-by-step upsampling to gradually restore the spatial resolution to match the input image. This includes the following steps: S31: In the decoding stage, the fine-grained spatial features extracted in the first stage of the encoder are passed to the front of the output layer through skip connections; S32: Lightweight channel attention units are used to recalibrate the weights of the fused features channel by channel. First, global average pooling is used to compress the two-dimensional spatial information of each channel into a scalar statistical value. Then, two fully connected layers with ReLU activation and Sigmoid gating are used to learn the non-linear dependencies between channels and generate importance weight coefficients for each channel. Finally, these weights are multiplied with the original fused features channel by channel to adaptively enhance the response strength of key channels in the depth estimation task.

5. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: In step S4, an end-to-end supervised training paradigm is adopted, using pairs of RGB images and corresponding real depth maps as training data. The training objective is to minimize the mixed loss function between the predicted depth map and the true depth map, which includes the following steps: S41. Set the initial learning rate to 1×10⁻⁶. -4 End-to-end training was performed using the AdamW optimizer, with a total of 100 training epochs. The batch size was set to 8 or 16 based on the GPU memory capacity, and a cosine annealing strategy was used to gradually decay the learning rate from the initial value during training. ; S42. The loss function adopts a weighted combination of scale-invariant loss SILog and local gradient loss, with the weight ratio set to 1:0.5; The hybrid loss function The expression is: ； in, For scale-invariant logarithmic loss, This is a local gradient loss, and the ratio of their weight coefficients is... Set to 1:0.

5.

6. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: In step S5, the trained PyTorch model is exported as an ONNX intermediate format, and then FP16 half-precision quantization and computation graph optimization are performed through the NVIDIA TensorRT inference engine. Finally, it is deployed on an embedded computing platform to achieve high-performance real-time inference.

7. The lightweight monocular depth estimation method based on a selective scanning state-space model according to claim 1, characterized in that: The S6 state-space equation utilizes the discretized state transition matrix. and the discretized input mapping matrix The state transition matrix after discretization is obtained by processing four sequences in parallel. and the discretized input mapping matrix They are defined as follows: ； ； Where A is the original state transition matrix of the continuous system, B is the original input mapping matrix of the continuous system, I is the identity matrix, and Δ is the discretization time step.

8. A lightweight monocular depth estimation system based on a selective scanning state-space model, characterized in that: It includes a feature extraction module, a feature fusion module, and a deep prediction module; The feature extraction module is used to preprocess the input monocular RGB image, and then input the preprocessed RGB image into a hierarchical encoder without position coding, and extract multi-scale feature maps using cascaded visual Mamba blocks. The feature fusion module is used to perform spatial downsampling of feature maps between different cascaded stages of the encoder using separable convolution; the high-order semantic features output by the encoder are input to the decoder for progressive upsampling, gradually restoring the spatial resolution to be consistent with the input image; Then, the regression head is used to map the multi-channel feature map output by the decoder into a single-channel depth map with the same resolution as the original input image, where each pixel value represents the estimated distance of that location relative to the camera. The depth prediction module is used to perform engineering optimization and deployment of the trained model, and to systematically evaluate its inference speed, memory usage and depth estimation accuracy on the target hardware platform.

Images