Lightweight self-supervised monocular depth estimation method based on mamba and temporal enhancement

Abstract

The present application relates to a lightweight self-supervised monocular depth estimation method based on Mamba and timing enhancement, which solves the defects of difficult to realize the direct balance of calculation efficiency and representation ability, and the lack of model precision and generalization ability leading to difficult to land application compared with the prior art. The present application comprises the following steps: acquisition and preprocessing of original image; construction of monocular depth estimation model; training of monocular depth estimation model; obtaining of monocular depth estimation result. The present application realizes the balance between calculation efficiency and representation ability based on the lightweight TinyViM backbone model of Mamba architecture; combines the optical flow interpolation timing enhancement module for enhancing the timing modeling ability, further improves the timing modeling ability in self-supervised monocular depth estimation; and introduces the spatial collaborative attention mechanism ECCA which can improve the scene structure and spatial relationship perception ability, significantly enhances the perception ability of the model to the scene structure and spatial relationship under the premise of almost no increase in calculation burden.

Description

Technical Field

[0001] This invention relates to the field of computer video recognition technology, specifically a lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation. Background Technology

[0002] Monocular depth estimation (MDE) aims to predict the depth information of a scene using only a single RGB image. Data acquisition requires no additional hardware, offering advantages such as low cost, ease of operation, and flexible deployment, attracting numerous researchers to study monocular depth estimation techniques. Depth information plays a crucial role in computer vision and related applications (robotics, autonomous driving, intelligent livestock measurement, etc.), essentially providing the three-dimensional geometric structure of a scene. Compared to two-dimensional RGB images that only contain color and texture information, depth information contributes direct measurements of spatial position and distance to computer vision.

[0003] Early research on monocular depth estimation relied primarily on hand-designed features and prior knowledge, such as methods based on texture gradients, focusing depth cues, shadow variations, and atmospheric scattering models. While these methods performed reasonably well in specific scenarios, their generalization ability was limited, making them ill-suited for complex lighting conditions, texture loss, or occlusion. With the development of Convolutional Neural Networks (CNNs), monocular depth estimation entered a data-driven era. Eigen et al. first applied CNNs to this task in 2014, significantly improving accuracy by simultaneously predicting global coarse depth and local details through a multi-scale network structure. To improve the accuracy of self-supervised monocular depth estimation, existing methods have introduced advanced loss functions, data preprocessing, and semantic segmentation guidance techniques. However, they remain limited by fixed camera viewpoints, making it difficult to synthesize views of occluded or invisible areas, and failing to fully utilize the temporal continuity of video data. In terms of architecture, early CNN-based methods were mainly limited to local features, while recent Transformers, although capable of capturing long-range dependencies and global context, suffer from computational costs that increase quadratically with resolution due to their self-attention mechanism, resulting in significant overhead when processing high-resolution depth estimation inputs. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies, such as the difficulty in achieving a direct balance between computational efficiency and representational capability, as well as the lack of model accuracy and generalization ability leading to difficulties in practical applications. This invention provides a lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation to solve the above problems.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation includes the following steps:

[0007] Acquisition and preprocessing of raw images;

[0008] Construction of a monocular depth estimation model;

[0009] Training of a monocular depth estimation model;

[0010] Obtaining the monocular depth estimation result: After acquiring the image to be estimated and preprocessing it, input it into the trained monocular depth estimation model to obtain the estimation result.

[0011] The construction of the monocular depth estimation model includes the following steps:

[0012] The monocular depth estimation model is defined by including the TAM module, the lightweight TinyViM backbone network based on the Mamba architecture, and the ECCA module;

[0013] Configure the TAM module;

[0014] Configure a lightweight TinyViM backbone network based on Mamba architecture;

[0015] Configure the ECCA module.

[0016] The training of the monocular depth estimation model includes the following steps:

[0017] Pair consecutive input frames with I t I t-1 with I t I t+1 Input to the TAM module respectively, outputting the original target frame I. t Interpolation frame I t* False target frame Multi-view input system;

[0018] Original target frame I t Interpolation frame I t* False target frame Adjacent frames I t-1 I t+1 The constructed multi-view input system and interpolation frame I t* With corresponding source image I s Relative pose transformation Input the lightweight TinyViM backbone network with Mamba architecture and output feature maps. ;

[0019] feature map Input to the ECCA module, the ECCA module outputs the final feature X outEnter Monodepth2 baseline model predicted depth map D pref .

[0020] Setting up the TAM module includes the following steps:

[0021] The TAM module uses a pre-trained video frame interpolation VFI network as a prior and generates intermediate virtual views to help the network learn continuous motion and geometric transformations. It selects the high-quality optical flow-based interpolation network IFRNet as the base model.

[0022] For consecutive input frame pairs I t I t-1 with I t I t+1 Intermediate virtual view interpolation frames, denoted as I, are generated using a video frame interpolation VFI network. t* ,in, ;

[0023] Generated intermediate virtual view interpolation frame I t* The video frame interpolation VFI network is input again to generate a reverse I-frame with the target frame. t Simultaneous pseudo-target frame This allows the construction of the original target frame I via the TAM module. t Interpolation frame I t* and pseudo-target frames Multi-view input system;

[0024] Interpolated frame I t* With corresponding source target frame I s Input the pose estimation network, i.e., t*=t+0.5 corresponds to s=t+1, t*=t-0.5 corresponds to s=t-1, and estimate the relative pose transformation. This allows for the construction of denser time-series monitoring signals.

[0025] Setting up a lightweight TinyViM backbone network with a Mamba architecture includes the following steps:

[0026] A lightweight TinyViM backbone network based on the Mamba architecture is set up as the core feature extractor of the encoder. The lightweight TinyViM backbone network adopts a five-stage hierarchical structure design, extracting semantic information from the input image layer by layer from low level to high level.

[0027] The backbone module is defined, consisting of two consecutive 3×3 standard convolutional layers, with the input image... R is the set of real numbers, indicating that each value in the input image I is a real number. C×H×W: represents a tensor with dimensions C channels, H height, and W width.

[0028] First layer: Set kernel size K=3, stride S=2, padding P=1. This layer downsamples the spatial resolution to... ;

[0029] Second layer: Set kernel size K=3, stride S=1, padding P=1. This layer is above the previous layer. Feature extraction is performed at the scale while maintaining the image resolution.

[0030] The two convolutional layers described above are used to initially extract shallow visual features, including edges and textures. Simultaneously, the spatial resolution is downsampled to half that of the original image to obtain an initial feature map. ,in, Representation of feature map It is a three-dimensional tensor with shape [C, H / 2, W / 2].

[0031] The cascaded processing stage consists of four stages. Within each stage, feature extraction is primarily performed by two types of modules stacked alternately: the Local Block and the TViM Block. At the end of each stage, the feature map is downsampled using the Patch Embed module. The Patch Embed module is typically implemented using a convolutional layer with a stride of 2, which further halves the spatial resolution of the feature map, thereby generating a compact feature representation for the next stage, i.e., the feature map. ;

[0032] Local Block is designed to capture fine image textures and high-frequency edge details. It introduces a reparameterized 3×3 depthwise separable convolution, which has a multi-branch structure to enhance non-linear fitting ability during the training phase and is equivalent to a single convolution kernel during the inference phase, thus significantly reducing inference overhead while ensuring accuracy.

[0033] The TViM Block global module, as the core global context modeling unit, uses a Laplacian mixer to decouple the feature frequencies captured from the low-frequency structure and long-range dependencies of the image. It divides the input features into low-frequency and high-frequency components. The low-frequency components represent the global contour and semantic background of the image and are processed by the two-dimensional selective scanning mechanism unique to the Mamba architecture to efficiently model long-distance spatial correlations. The high-frequency components retain local subtle changes and are processed again by RepDW-3 to maintain the integrity of details.

[0034] Setting up the ECCA module includes the following steps:

[0035] The ECCA algorithm employs a dual-branch parallel structure, introducing bidirectional chained attention in the spatial dimension to efficiently capture global context dependencies, and using the ECA module in the channel dimension to model channel relationships; for a given input feature map... The ECCA module feeds it into the spatial attention branch and the channel attention branch for parallel processing.

[0036] A spatial attention branch is defined: In the spatial branch, a bidirectional chained attention mechanism is introduced to efficiently aggregate global information of the feature map in the horizontal and vertical directions with linear complexity.

[0037] The input feature X is mapped to query Q, key K, and value V tensors through three 1×1 convolutional layers. The number of channels in query Q and key K is set to 1 / 8 of the original number of channels, while the number of channels in value V is kept to preserve information. For feature X, the query vector Q at any position u is iterated. u Define its cross-shaped receptive field Ω u Let X be the set of all key vectors in the same row and column as key K, after removing the central duplicates. This set contains a total of H / 32+W / 32−1 feature vectors, where H and W are the height and width of the original graph, and H / 32 and W / 32 are the height and width of the input feature graph X.

[0038] Subsequently, the query vector Q at position u is calculated. u and Ω u Each key vector K i,u similarity , ,

[0039] Where i is the set Ω u Index in For Q u The transposed matrix, where u is the calculation position;

[0040] Then, the attention weight A assigned to the i-th context element at position u is calculated using Softmax normalization. u,i ;

[0041] Based on attention weight A u,i We perform weighted aggregation on the corresponding features in the same row and column of the value tensor V to generate spatially enhanced features. ;

[0042] Finally, spatial enhancement features for all locations. Re-stack the data according to the original spatial coordinates to form a complete feature map X. s ;

[0043] This operation enables each feature to obtain global context information about its entire row and column, effectively modeling long-range spatial dependencies.

[0044] Channel attention branch: In the channel branch, the ECA module is used to perform global average pooling on the input feature X along the spatial dimension H / 32×W / 32 to obtain a one-dimensional channel descriptor z, where z is the one-dimensional channel description vector obtained after global average pooling.

[0045] To capture local cross-channel interactions while avoiding the side effects of dimensionality reduction, a one-dimensional convolutional kernel is used for information aggregation:

[0046] ,

[0047] in, This represents the feature vector of the middle channel after one-dimensional convolution. This represents a one-dimensional convolution operation, where the subscript k indicates the kernel size.

[0048] The kernel size k is adaptively determined based on the channel dimension C:

[0049] ,

[0050] b are hyperparameters. This indicates taking the nearest odd number;

[0051] Channel attention weights are generated using the Sigmoid activation function σ. The attention weight m is multiplied channel-by-channel with the input feature X to achieve feature recalibration X. c ;

[0052] Feature fusion

[0053] Output X from the spatial attention branch s With channel attention branch output X c The features are fused and combined with the original input features X through a residual connection to obtain the final output features X. out ;

[0054] Predict the depth map, and then use the final feature X out Input Monodepth2 baseline model to predict depth map D pref .

[0055] A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the implementation of the lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement.

[0056] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, implements a lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement.

[0057] Beneficial effects

[0058] The lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement of this invention achieves a balance between computational efficiency and representational capability compared with the existing technology based on the lightweight TinyViM backbone model of Mamba architecture; it combines optical flow interpolation temporal enhancement module (TAM) to enhance temporal modeling capability, further improving the temporal modeling capability in self-supervised monocular depth estimation; and it introduces spatial collaborative attention mechanism ECCA (Efficient Channel-Chain Attention) to enhance the perception of scene structure and spatial relationships, significantly enhancing the model's perception of scene structure and spatial relationships with almost no increase in computational burden.

[0059] Experimental results show that the method described in this invention achieves a value of 0.101 on the core metric Abs Rel, which is a relative improvement of 4.7% compared to MViTDepth and ADDepth (both of which are 0.106). Attached Figure Description

[0060] Figure 1 This is a sequence diagram of the method of the present invention;

[0061] Figure 2 This is a schematic diagram of the self-supervised monocular depth estimation model structure involved in this invention;

[0062] Figure 3 This is a schematic diagram of the TAM module structure involved in the present invention;

[0063] Figure 4 This is a schematic diagram of the lightweight TinyViM backbone network structure based on the Mamba architecture involved in this invention.

[0064] Figure 5 This is a schematic diagram of the ECCA module structure involved in this invention;

[0065] Figure 6 This invention provides a comparison of depth map prediction results with those of other mainstream models.

[0066] Figure 7 To predict the depth map results of pigs using the self-supervised monocular depth estimation model of this invention. Detailed Implementation

[0067] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0068] like Figure 1 As shown, the lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation described in this invention includes the following steps:

[0069] The first step is to acquire and preprocess the original image.

[0070] The second step is to construct a self-supervised monocular depth estimation model.

[0071] (1) such as Figure 2 As shown, the monocular depth estimation model is configured to include the TAM module, the lightweight TinyViM backbone network based on the Mamba architecture, and the ECCA module.

[0072] The core of self-supervised monocular depth estimation lies in utilizing adjacent frames (temporal or spatial dimensions) as geometric supervision signals, eliminating reliance on expensive ground truth LiDAR data. Its construction and technical implementation have distinct characteristics and a high barrier to entry:

[0073] The core challenge in this project lies in achieving geometric consistency and handling occlusion.

[0074] The model is built upon the photometric consistency assumption, which states that pixel brightness remains constant across different viewpoints for the same scene. In practice, dynamic objects, lighting variations, shadows, and low-texture regions need to be addressed. In a monocular setup, when reprojecting using inter-frame pose estimation, object occlusion can cause some pixels to be invisible in the target frame. The model design needs to construct a robust masking mechanism to ignore the loss contribution from these invalid regions; otherwise, it will introduce significant noise supervision.

[0075] The key technical challenge lies in the pitfalls of network design and optimization.

[0076] Distinguishing between "camera motion" and "object motion" is extremely difficult when dealing with dynamic scenes. Without introducing additional semantic branches or optical flow-assisted tasks, a simple rigid body motion model is prone to misestimating moving objects as distant static backgrounds, resulting in overestimated depth values.

[0077] When dealing with edge details and texture replication, early codecs based on skip connections were prone to "texture replication" artifacts, which meant replicating the texture details of the RGB image at the object edges in the depth map instead of outputting smooth, accurate geometric boundaries. Solving this problem requires designing specific surface normal constraints or improving the upsampling / decoding structure (such as replacing convolution with Transformer or multi-scale perceptron).

[0078] Self-supervised loss training is prone to getting stuck in local optima. Carefully designed edge weighting, minimizing reprojection error, and complex multi-scale training strategies are needed to achieve stable convergence.

[0079] Its main technical features are label-free dependency and generalization ability.

[0080] Purely vision-driven geometry: Technically, it does not rely on labeled data. Instead, it uses a Pose Network to predict inter-frame transformation matrices and a Depth Network to predict inverse depth. Through differentiable warehousing, it constructs a loss loop of "predicted view vs. target view". This end-to-end geometry learning enables it to utilize massive amounts of dashcam or video data.

[0081] Lightweight and real-time potential: Compared to traditional SLAM which requires feature matching and optimized mapping, self-supervised models, once trained, only require forward propagation of a single image during the inference phase, making them suitable for mobile deployment and real-time applications.

[0082] Cross-domain adaptability: Because they do not rely on calibration data of specific sensors (such as strict alignment between LiDAR and cameras), these models often perform better than fully supervised models when generalizing across datasets.

[0083] Overall, self-supervised monocular depth estimation lowers the threshold for data acquisition, but shifts the difficulty to the rigor of geometric modeling, the decoupling of dynamic objects, and the ingenuity of loss function design. It is currently the key technical path connecting pure visual perception and 3D geometric understanding.

[0084] Knowledge distillation is frequently used in subsequent research or industrial application models that pursue higher accuracy, better efficiency, or adaptation to specific scenarios, to achieve the following objectives:

[0085] Model compression and lightweighting: Use a high-precision but computationally intensive "teacher model" (such as a deep network pre-trained on multi-frame or binocular data) to guide a "student model" with fewer parameters (a monocular network suitable for edge deployment), reducing inference costs while maintaining good depth, making it more suitable for in-vehicle or mobile applications.

[0086] Mitigating inherent defects of self-supervision: By leveraging a teacher model that has already learned and is more stable in static geometry or weakly textured regions, knowledge can be transferred to the student model to help reduce depth ambiguity or noise problems caused by the breaking of the photometric consistency assumption (such as dynamic objects or occluded areas).

[0087] Cross-domain / multimodal enhancement: For example, using a LiDAR-supervised depth model or a semi-supervised model as a teacher to provide additional geometric prior guidance to a purely self-supervised monocular student model, thereby improving generalization ability.

[0088] (2) Configure the TAM module, such as Figure 3 As shown.

[0089] To further enhance the temporal modeling capability in self-supervised monocular depth estimation, a Temporal Augmentation Module (TAM) based on optical flow interpolation is proposed. The core idea of ​​this module is to construct a denser supervisory signal through high-fidelity view synthesis, thereby alleviating the geometric constraint sparsity problem caused by the large inter-frame intervals in traditional methods. Specifically, TAM utilizes a pre-trained Video Frame Interpolation Network as a prior, generating intermediate virtual views to assist the network in learning continuous motion and geometric transformations.

[0090] A1) The TAM module uses a pre-trained video frame interpolation VFI network as a prior and generates intermediate virtual views to help the network learn continuous motion and geometric transformations. It selects the high-quality optical flow-based interpolation network IFRNet as the base model.

[0091] For consecutive input frame pairs I t I t-1 with I t I t+1 Intermediate virtual view interpolation frames, denoted as I, are generated using a video frame interpolation VFI network. t* ,in, ;

[0092] A2) Generated intermediate virtual view interpolation frame I t* The video frame interpolation VFI network is input again to generate a reverse I-frame with the target frame. t Simultaneous pseudo-target frame This allows the construction of the original target frame I via the TAM module. t Interpolation frame I t* and pseudo-target frames Multi-view input system;

[0093] A3) Interpolate frame I t* With corresponding source target frame I s Input the pose estimation network, i.e., t*=t+0.5 corresponds to s=t+1, t*=t-0.5 corresponds to s=t-1, and estimate the relative pose transformation. This allows for the construction of denser time-series monitoring signals.

[0094] (3) such as Figure 4 As shown, a lightweight TinyViM backbone network based on the Mamba architecture is configured.

[0095] To achieve a balance between computational efficiency and representational capability in multi-scale feature extraction, a lightweight TinyViM backbone network based on the Mamba architecture was designed as the core feature extractor of the encoder. The network adopts a five-stage hierarchical structure, which can extract semantic information from the input image layer by layer from low to high levels, effectively promoting multi-scale feature fusion.

[0096] First, the input image is initialized through a backbone module, where shallow feature extraction is performed using standard convolutional operations, simultaneously compressing the spatial resolution to reduce the computational burden for subsequent deep processing. Then, the network enters four cascaded processing stages. Within each stage, multiple local modules and TViM modules are alternately stacked; between different stages, an embedding module performs feature map downsampling to progressively expand the receptive field and abstract semantic information.

[0097] In this hierarchical structure, the local modules are designed to capture fine image textures and high-frequency details such as edges. Specifically, we introduce a reparameterized 3×3 depthwise separable convolution (RepDW-3), which has a multi-branch structure during the training phase to enhance nonlinear fitting capabilities, while it can be equivalently converted into a single convolution kernel during the inference phase, thereby significantly reducing inference overhead while maintaining accuracy.

[0098] Complementing this is the TViM module, which serves as the core unit for global context modeling, dedicated to capturing the low-frequency structure and long-range dependencies of an image. The TViM module introduces a Laplacian Mixer, which decomposes input features into high-frequency and low-frequency components. The low-frequency components, representing the global contour and semantic background of the image, are processed by the Mamba architecture's unique Selective Scan 2D (SS2D) mechanism to efficiently model long-range spatial correlations. The high-frequency components, preserving subtle local variations, are further processed by RepDW-3 to maintain detail integrity. This parallel processing strategy of high and low-frequency information allows TinyViM to significantly enrich the semantic hierarchy and expressive power of features while strictly controlling computational complexity.

[0099] (1) A lightweight TinyViM backbone network based on the Mamba architecture is set up as the core feature extractor of the encoder. The lightweight TinyViM backbone network adopts a five-stage hierarchical structure design to extract semantic information from the input image from low level to high level layer by layer.

[0100] (2) Set the backbone module, which consists of two consecutive 3×3 standard convolutional layers. Input image R is the set of real numbers, indicating that each value in the input image I is a real number. C×H×W: represents a tensor with dimensions C channels, H height, and W width.

[0101] First layer: Set kernel size K=3, stride S=2, padding P=1. This layer downsamples the spatial resolution to... ;

[0102] Second layer: Set kernel size K=3, stride S=1, padding P=1. This layer is above the previous layer. Feature extraction is performed at the scale while maintaining the image resolution.

[0103] The two convolutional layers described above are used to initially extract shallow visual features, including edges and textures. Simultaneously, the spatial resolution is downsampled to half that of the original image to obtain an initial feature map. ,in, Representation of feature map It is a three-dimensional tensor with shape [C, H / 2, W / 2].

[0104] (3) Set up cascaded processing stages. There are four cascaded processing stages. Within each stage, feature extraction is mainly completed by two types of modules stacked alternately: Local Block and TViM Block. At the end of each stage, the feature map is downsampled by the Patch Embed module. The Patch Embed module is usually implemented by a convolutional layer with a stride of 2, which is responsible for further halving the spatial resolution of the feature map, thereby generating a compact feature representation for the next stage, i.e., the feature map. ;

[0105] Local Block is designed to capture fine image textures and high-frequency edge details. It introduces a reparameterized 3×3 depthwise separable convolution, which has a multi-branch structure to enhance non-linear fitting ability during the training phase and is equivalent to a single convolution kernel during the inference phase, thus significantly reducing inference overhead while ensuring accuracy.

[0106] The TViM Block global module, as the core global context modeling unit, uses a Laplacian mixer to decouple the feature frequencies captured from the low-frequency structure and long-range dependencies of the image. It divides the input features into low-frequency and high-frequency components. The low-frequency components represent the global contour and semantic background of the image and are processed by the two-dimensional selective scanning mechanism unique to the Mamba architecture to efficiently model long-distance spatial correlations. The high-frequency components retain local subtle changes and are processed again by RepDW-3 to maintain the integrity of details.

[0107] (4) Configure the ECCA module.

[0108] In lightweight monocular depth estimation, the global contextual dependencies of the image in the horizontal and vertical directions are effectively modeled by a bidirectional spatial attention mechanism. Combined with lightweight ECA channel attention, the model’s ability to perceive scene structure and spatial relationships is significantly enhanced. Under the premise of almost no increase in computational burden, the prediction accuracy and continuity of the depth map in weak texture regions, edge details and complex structures are improved.

[0109] To efficiently model global contextual priors in resource-constrained monocular depth estimation tasks, the efficient channel-chain attention mechanism ECCA (Efficient Channel-Chain Attention) is employed. Figure 5 As shown, ECCA adopts a dual-branch parallel structure, introducing bidirectional chained attention in the spatial dimension to efficiently capture global context dependencies, and using an efficient ECA module in the channel dimension to model channel relationships. This significantly enhances the model's ability to perceive scene structure and spatial relationships without increasing computational burden.

[0110] (1) Set ECCA to adopt a dual-branch parallel structure, introduce bidirectional chained attention in the spatial dimension to efficiently capture global context dependencies, and use an efficient ECA module to model channel relationships in the channel dimension; for a given input feature map The ECCA module feeds it into the spatial attention branch and the channel attention branch for parallel processing.

[0111] (2) Setting up a spatial attention branch: In the spatial branch, a bidirectional chained attention mechanism is introduced to efficiently aggregate global information of the feature map in the horizontal and vertical directions with linear complexity.

[0112] The input feature X is mapped to query Q, key K, and value V tensors through three 1×1 convolutional layers. The number of channels in query Q and key K is set to 1 / 8 of the original number of channels, while the number of channels in value V is kept to preserve information. For feature X, the query vector Q at any position u is iterated. u Define its cross-shaped receptive field Ω uLet X be the set of all key vectors in the same row and column as key K, after removing the central duplicates. This set contains a total of H / 32+W / 32−1 feature vectors, where H and W are the height and width of the original graph, and H / 32 and W / 32 are the height and width of the input feature graph X.

[0113] Subsequently, the query vector Q at position u is calculated. u and Ω u Each key vector K i,u similarity , ,

[0114] Where i is the set Ω u Index in For Q u The transposed matrix, where u is the calculation position;

[0115] Then, the attention weight A assigned to the i-th context element at position u is calculated using Softmax normalization. u,i ;

[0116] Based on attention weight A u,i We perform weighted aggregation on the corresponding features in the same row and column of the value tensor V to generate spatially enhanced features. ;

[0117] Finally, spatial enhancement features for all locations. Re-stack the data according to the original spatial coordinates to form a complete feature map X. s ;

[0118] This operation allows each feature to obtain global context information about its entire row and column, effectively modeling long-range spatial dependencies.

[0119] (3) Channel attention branch: In the channel branch, the efficient channel attention ECA module is used to perform global average pooling on the input feature X along the spatial dimension H / 32×W / 32 to obtain a one-dimensional channel descriptor z, where z is a one-dimensional channel descriptor vector obtained after global average pooling.

[0120] To capture local cross-channel interactions while avoiding the side effects of dimensionality reduction, a one-dimensional convolutional kernel is used for information aggregation:

[0121] ,

[0122] in, This represents the feature vector of the middle channel after one-dimensional convolution. This represents a one-dimensional convolution operation, where the subscript k indicates the kernel size.

[0123] The kernel size k is adaptively determined based on the channel dimension C:

[0124] ,

[0125] b are hyperparameters. This indicates taking the nearest odd number;

[0126] Channel attention weights are generated using the Sigmoid activation function σ. The attention weight m is multiplied channel-by-channel with the input feature X to achieve feature recalibration X. c .

[0127] (4) Feature fusion,

[0128] Output X from the spatial attention branch s With channel attention branch output X c The features are fused and combined with the original input features X through a residual connection to obtain the final output features X. out .

[0129] (5) Predict the depth map and convert the final feature X out Input Monodepth2 baseline model to predict depth map D pref .

[0130] The third step is to train the monocular depth estimation model.

[0131] (1) Pair consecutive input frames with I t I t-1 with I t I t+1 Input to the TAM module respectively, outputting the original target frame I. t Interpolation frame I t* False target frame A multi-view input system.

[0132] (2) Transfer the original target frame I t Interpolation frame I t* False target frame Adjacent frames I t-1 I t+1 The constructed multi-view input system and interpolation frame I t* With corresponding source image I s Relative pose transformation Input the lightweight TinyViM backbone network with Mamba architecture and output feature maps. .

[0133] (3) Feature map Input to the ECCA module, the ECCA module outputs the final feature X out Enter Monodepth2 baseline model predicted depth map D pref .

[0134] The fourth step is to obtain the monocular depth estimation result: After acquiring the image to be estimated and preprocessing it, input it into the trained monocular depth estimation model to obtain the estimation result.

[0135] Table 1. Error Analysis Comparison of Each Model Trained Using the KITTI Dataset

[0136]

[0137] Table 2 Comparison of computational complexity and inference efficiency of each model trained using the KITTI dataset

[0138]

[0139] Error assessment metrics include relative error (Abs Rel), squared relative error (Sq Rel), root mean square error (RMSE), logarithmic root mean square error (RMSE log), and threshold accuracy (δ1, δ2, δ3), calculated as follows:

[0140] (1),

[0141] (2),

[0142] (3),

[0143] (4),

[0144] (5),

[0145] Where N is the total number of valid pixels in the image, k is the pixel index number, and d k and These are the k-th predicted ground depth and the k-th actual ground depth, respectively. `max()` indicates taking the larger of the two values, with thresholds of 1.25 and 1.25, respectively. 2 1.25 3 , corresponding to δ1, δ2, δ3.

[0146] As shown in Table 1, the errors of the model of this invention are compared with those of other models Monodepth2, R-MSFM6, Lite-Mono, and MViTDepth. The evaluation metrics include: absolute relative error (Abs Rel), squared relative error (SqRel), root mean square error (RMSE), logarithmic root mean square error (RMSE log), and threshold accuracy (δ1, δ2, δ3). Table 1 shows that the method of this invention achieves an Abs Rel of 0.101, a 4.7% improvement compared to the best Abs Rel method, MViTDepth (0.106). The method of this invention also outperforms or matches the compared methods in RMSE, RMSE log, and the three threshold accuracy metrics. In particular, the method of this invention (0.706) improves upon other methods in Sq Rel, a metric more sensitive to larger depth errors, indicating that our model is more robust when dealing with regions with large depth ranges.

[0147] Table 2 shows a quantitative comparison of the method described in this invention with other models Monodepth2, R-MSFM6, Lite-Mono, and MViTDepth in terms of Params (number of parameters), FLOPs (Floating Point Operations), and Speed ​​(inference speed). As can be seen from Table 2, while achieving higher prediction accuracy, the method of this invention also has a significant advantage in computational efficiency. Specifically, the number of parameters in the model of this invention is 6.2M, slightly lower than that of MViTDepth (6.3M). In terms of FLOPs, which better reflects the actual inference cost, our method has the lowest computational cost of only 3.3G, nearly 30% lower than MViTDepth's 4.7G. This indicates that the lightweight TinyViM backbone network and ECCA attention module proposed in this invention, based on the Mamba architecture, achieve a better balance between model performance and computational cost. Therefore, the method proposed in this invention achieves more accurate monocular depth estimation while maintaining a lightweight model.

[0148] like Figure 6 As shown, from Figure 6 As can be seen from the depth estimation results on the KITTI dataset, compared with other models Monodepth2, R-MSFM6, Lite-Mono, and MViTDepth, the model of this invention can retain more details and sharper edges in the depth map. The differences are highlighted within the green boxes.

[0149] like Figure 7 As shown, from Figure 7As can be seen, the model of this invention, when applied to visual analysis in pig farming, can extract richer biological and behavioral information from a three-dimensional perspective, providing an efficient, low-cost, and reliable technical path for the precision, automation, and animal welfare improvement of smart farming. Future work can focus on in-depth exploration in areas such as domain adaptation for specific farming scenarios and deep segmentation and tracking of multiple pig instances.

[0150] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of the original image; 12) Construction of a monocular depth estimation model; 13) Training of the monocular depth estimation model; 14) Obtaining the monocular depth estimation result: After acquiring the image to be estimated and preprocessing it, input it into the trained monocular depth estimation model to obtain the estimation result.

2. The lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation as described in claim 1, characterized in that, The construction of the monocular depth estimation model includes the following steps: 21) The monocular depth estimation model is defined as including the TAM module, the lightweight TinyViM backbone network with Mamba architecture, and the ECCA module; 22) Configure the TAM module; 23) Configure a lightweight TinyViM backbone network based on the Mamba architecture; 24) Configure the ECCA module.

3. The lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation according to claim 1, characterized in that, The training of the monocular depth estimation model includes the following steps: 31) Pair consecutive input frames with I t I t-1 with I t I t+1 Input to the TAM module respectively, outputting the original target frame I. t Interpolation frame I t* False target frame Multi-view input system; 32) Transfer the original target frame I t Interpolation frame I t* False target frame Adjacent frames I t-1 I t+1 The constructed multi-view input system and interpolation frame I t* With corresponding source image I s Relative pose transformation Input the lightweight TinyViM backbone network with Mamba architecture and output feature maps. ; 33) Feature map Input to the ECCA module, the ECCA module outputs the final feature X out Enter Monodepth2 baseline model predicted depth map D pref .

4. The lightweight self-supervised monocular depth estimation method based on Mamba and temporal augmentation according to claim 2, characterized in that, The TAM setting module includes the following steps: 41) The TAM module uses a pre-trained video frame interpolation VFI network as a prior and generates intermediate virtual views to help the network learn continuous motion and geometric transformations. It selects the high-quality optical flow-based interpolation network IFRNet as the base model. For consecutive input frame pairs I t I t-1 with I t I t+1 Intermediate virtual view interpolation frames, denoted as I, are generated using a video frame interpolation VFI network. t* ,in, ; 42) Generated intermediate virtual view interpolation frame I t* The video frame interpolation VFI network is input again to generate a reverse I-frame with the target frame. t Simultaneous pseudo-target frame This allows the construction of the original target frame I via the TAM module. t Interpolation frame I t* and pseudo-target frames Multi-view input system; 43) Interpolate frame I t* With corresponding source target frame I s Input the pose estimation network, i.e., t*=t+0.5 corresponds to s=t+1, t*=t-0.5 corresponds to s=t-1, and estimate the relative pose transformation. This allows for the construction of denser time-series monitoring signals.

5. The lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement according to claim 2, characterized in that, Setting up a lightweight TinyViM backbone network with a Mamba architecture includes the following steps: 51) Set up a lightweight TinyViM backbone network based on the Mamba architecture as the core feature extractor of the encoder. The lightweight TinyViM backbone network adopts a five-stage hierarchical structure design to extract semantic information from the input image from low level to high level layer by layer. 52) Define the backbone module, which consists of two consecutive 3×3 standard convolutional layers. Input image R is the set of real numbers, indicating that each value in the input image I is a real number. C×H×W: represents a tensor with dimensions C channels, H height, and W width. First layer: Set kernel size K=3, stride S=2, padding P=1. This layer downsamples the spatial resolution to... ; Second layer: Set kernel size K=3, stride S=1, padding P=1. This layer is above the previous layer. Feature extraction is performed at the scale while maintaining the image resolution. The two convolutional layers described above are used to initially extract shallow visual features, including edges and textures. Simultaneously, the spatial resolution is downsampled to half that of the original image to obtain an initial feature map. ,in, Representation of feature map It is a three-dimensional tensor with shape [C, H / 2, W / 2]. 53) The cascaded processing stage is set up with four stages. Within each stage, feature extraction is mainly completed by two types of modules stacked alternately: the Local Block and the TViM Block. At the end of each stage, the feature map is downsampled by the Patch Embed module. The Patch Embed module is usually implemented by a convolutional layer with a stride of 2, which is responsible for further halving the spatial resolution of the feature map, thereby generating a compact feature representation for the next stage, i.e., the feature map. ; Local Block is designed to capture fine image textures and high-frequency edge details. It introduces a reparameterized 3×3 depthwise separable convolution, which has a multi-branch structure to enhance non-linear fitting ability during the training phase and is equivalent to a single convolution kernel during the inference phase, thus significantly reducing inference overhead while ensuring accuracy. The TViM Block global module, as the core global context modeling unit, uses a Laplacian mixer to decouple the feature frequencies captured from the low-frequency structure and long-range dependencies of the image. It divides the input features into low-frequency and high-frequency components. The low-frequency components represent the global contour and semantic background of the image and are processed by the two-dimensional selective scanning mechanism unique to the Mamba architecture to efficiently model long-distance spatial correlations. The high-frequency components retain local subtle changes and are processed again by RepDW-3 to maintain the integrity of details.

6. The lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement according to claim 2, characterized in that, Setting up the ECCA module includes the following steps: 61) Set ECCA to use a two-branch parallel structure, introduce bidirectional chained attention in the spatial dimension to efficiently capture global context dependencies, and use the ECA module to model channel relationships in the channel dimension; for a given input feature map The ECCA module feeds it into the spatial attention branch and the channel attention branch for parallel processing. 62) Setting up a spatial attention branch: In the spatial branch, a bidirectional chained attention mechanism is introduced to efficiently aggregate global information of the feature map in the horizontal and vertical directions with linear complexity. The input feature X is mapped to query Q, key K, and value V tensors through three 1×1 convolutional layers. The number of channels in query Q and key K is set to 1 / 8 of the original number of channels, while the number of channels in value V is kept to preserve information. For feature X, the query vector Q at any position u is iterated. u Define its cross-shaped receptive field Ω u Let X be the set of all key vectors in the same row and column as key K, after removing the central duplicates. This set contains a total of H / 32+W / 32−1 feature vectors, where H and W are the height and width of the original graph, and H / 32 and W / 32 are the height and width of the input feature graph X. Subsequently, the query vector Q at position u is calculated. u and Ω u Each key vector K i,u similarity , , Where i is the set Ω u Index in For Q u The transposed matrix, where u is the calculation position; Then, the attention weight A assigned to the i-th context element at position u is calculated using Softmax normalization. u,i ; Based on attention weight A u,i We perform weighted aggregation on the corresponding features in the same row and column of the value tensor V to generate spatially enhanced features. ; Finally, spatial enhancement features for all locations. Re-stack the data according to the original spatial coordinates to form a complete feature map X. s ; This operation enables each feature to obtain global context information about its entire row and column, effectively modeling long-range spatial dependencies. 63) Channel attention branch: In the channel branch, the ECA module is used to perform global average pooling on the input feature X along the spatial dimension H / 32×W / 32 to obtain a one-dimensional channel descriptor z, where z is a one-dimensional channel description vector obtained after global average pooling. To capture local cross-channel interactions while avoiding the side effects of dimensionality reduction, a one-dimensional convolutional kernel is used for information aggregation: ， in, This represents the feature vector of the middle channel after one-dimensional convolution. This represents a one-dimensional convolution operation, where the subscript k indicates the kernel size. The kernel size k is adaptively determined based on the channel dimension C: ， b are hyperparameters. This indicates taking the nearest odd number; Channel attention weights are generated using the Sigmoid activation function σ. The attention weight m is multiplied channel-by-channel with the input feature X to achieve feature recalibration X. c ; 64) Feature fusion, Output X from the spatial attention branch s With channel attention branch output X c The features are fused and combined with the original input features X through a residual connection to obtain the final output features X. out ; 65) Predict the depth map, and then use the final feature X out Input Monodepth2 baseline model to predict depth map D pref .

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement as described in any one of claims 1-6.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it can implement the lightweight self-supervised monocular depth estimation method based on Mamba and temporal enhancement as described in any one of claims 1-6.

Images