Self-supervised depth estimation method based on block-wise detail extraction and multi-level feature aggregation

Abstract

The application discloses a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation, and is specifically implemented according to the following steps: step 1, image feature extraction; step 2, multi-scale feature extraction; step 3, block detail extraction; step 4, long-distance dependence capture; step 5, multi-level feature aggregation; step 6, depth value prediction; step 7, relative pose estimation; step 8, self-supervised signal generation; and step 9, model training optimization. The application does not depend on traditional depth true value labeling, but learns through geometric clues of adjacent image frames as self-supervised signals, and performs well in recovering three-dimensional depth structures of complex scenes and fine local features of multi-scale objects, and the generated depth map has clear boundaries and accurate structures.

Description

Technical Field

[0001] This invention relates to the field of computer vision and deep learning methods, specifically to a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation. Background Technology

[0002] In recent years, computer vision applications have developed rapidly, especially in high-end applications such as robotics, autonomous vehicles, 3D scene reconstruction, and augmented reality. One of the keys to realizing these technologies is accurately extracting the 3D depth information of a scene to achieve accurate perception and understanding of the environment. Although existing depth estimation techniques such as stereo vision and time-series analysis have achieved significant breakthroughs in accuracy, they often rely on images from multiple viewpoints or continuous frame sequences. This reliance limits the widespread deployment of these techniques in resource-constrained applications or those with stringent real-time processing requirements. In contrast, monocular depth estimation techniques predict scene depth information by analyzing visual cues in a single image, making them more attractive in practical applications, especially in resource-limited or real-time-critical scenarios. However, existing monocular depth estimation methods may neglect fine features during model learning, affecting their ability to recover local details and maintain depth continuity when dealing with complex scenes, low-texture regions, or occluded areas. Summary of the Invention

[0003] The purpose of this invention is to provide a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation, which can enhance the model's ability to perceive multi-scale objects and restore the detailed structure of complex scenes.

[0004] The technical solution adopted in this invention is as follows:

[0005] The self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation is implemented in the following steps:

[0006] Step 1, Image Feature Extraction;

[0007] Step 2, multi-scale feature extraction;

[0008] Step 3, extract details from blocks;

[0009] Step 4, long-distance dependent capture;

[0010] Step 5, multi-level feature aggregation;

[0011] Step 6, depth value prediction;

[0012] Step 7, relative pose estimation;

[0013] Step 8, self-supervised signal generation;

[0014] Step 9: Model training and optimization.

[0015] The present invention is characterized in that,

[0016] Step 1 is implemented in the following steps:

[0017] Step 1.1, Input Image Segmentation: Segment the input RGB image. Divide into non-overlapping patches of size p×P, with each patch treated as a token;

[0018] Step 1.2, Linear Projection: Perform a linear projection on each token, mapping it to a higher-dimensional feature space.

[0019] Step 1.3, Feature merging: Use the Patch Merging module to merge adjacent patch features to achieve downsampling of the feature map;

[0020] Step 1.4: Multi-stage feature extraction. Four hierarchical Transformer Blocks are constructed, with the output of each stage serving as the input to the next stage, generating four feature maps at different resolutions.

[0021] Step 1.5: Feature map upsampling and concatenation. Upsampling restores the low-resolution feature map to a high-resolution version, and then concatenates it with the high-resolution features from the decoder via skip connections. This aggregates high-level and low-level feature information, resulting in a final output feature map X with a resolution of [resolution missing].

[0022] Step 2 is implemented in the following steps:

[0023] Step 2.1: Construct three pyramid pooling layers, each using convolutional kernels of sizes 1×1, 3×3, and 6×6 to aggregate features at different scales;

[0024] Step 2.2: Simultaneously perform pooling operations, applying average pooling to the feature map at each scale;

[0025] Step 2.3, Feature Upsampling: Perform an upsampling operation on the feature map at each scale after average pooling, and use bilinear interpolation to restore the size of the feature map to the same level as the original input feature map. Figure 1 Achieved spatial resolution;

[0026] Step 2.4: Output multi-scale feature maps.

[0027] Step 3 is implemented in the following steps:

[0028] Step 3.1, Feature map segmentation, segmenting the input multi-scale feature map. Divide into N equal sizes Each patch is a specific region in the feature map, where and

[0029] Step 3.2, apply channel attention to each segmented patch. i The independent application channel attention mechanism is shown in equation (1) below:

[0030]

[0031] In the formula, p′ i This represents the channel data after adjustment using the channel attention mechanism. The channel attention is represented by the intermediate weight graph obtained through calculation. Weighting is applied to the channel dimension to adjust p. i The contribution of each channel; in addition, γ(·) represents an activation function for nonlinear transformation, α i (·) represents a channel weight mapping function;

[0032] Step 3.3, apply spatial attention to the result p′ of channel attention. i Spatial attention is applied as shown in equation (2):

[0033]

[0034] In the formula, p″ i This represents the channel data after spatial attention processing. This represents channel attention, and the spatial attention function β. i The result p′ of channel attention i Generate a spatial weight graph This weighted graph is used to adjust the contribution of different spatial locations, and δ(·) represents an activation function that performs a nonlinear transformation on the spatial weighted graph;

[0035] Step 3.4, Feature aggregation and refinement, converting the spatial attention result p″ i Compared with the original data p i The features are concatenated along the channel dimension, and then refined using spatial global average pooling to obtain a more detailed representation. The formula is expressed as follows:

[0036] z i =ρ([p″) i ,p i ]), i=1,2,…,N (3)

[0037] In the formula, ρ(·) represents global spatial average pooling, and [·] represents feature concatenation operation;

[0038] Step 3.5, Feature Reorganization: Repeat the above steps for all patches, and rearrange all the obtained refined tokens to form an ordered feature sequence. One-dimensional convolution is used to reduce the dimensionality of the channels of the token sequence Z to adapt to the input requirements of subsequent network layers;

[0039] Step 4 is implemented in the following steps:

[0040] Step 4.1: Apply multi-head attention to each feature sequence to capture long-range dependencies within the sequence. The formula is as follows:

[0041]

[0042] In the formula, Q, K, and V are the query, key, and value matrices, respectively. and Let d represent the learnable parameter matrix, n represent the number of attention heads, the softmax(·) function converts the attention scores into a probability distribution, and d k It is the number of channel dimensions of the key;

[0043] Step 4.2: Apply a feedforward network to further extract features from the output of the multi-head attention. The formula is as follows:

[0044] FFN(E i ′)=ReLU(E i ′·W1+b1)·W2+b2 (7)

[0045] In the formula, E i ′ represents the result of applying multi-head attention in step 4.1, where W1 and W2 are learnable weight matrices, b1 and n2 are the corresponding bias terms, and ReLU(·) is the activation function used to introduce nonlinearity;

[0046] Step 4.3, Layer Normalization and Residual Connections: The output of the feedforward network is normalized, and residual connections are added to enhance the model's learning ability and stability. The formula is as follows:

[0047] E″ i =LN(E′) i +FFN(E′ i (8)

[0048] In the formula, LN(·) is the layer normalization operation, which normalizes each feature channel.

[0049] Step 5 is implemented in the following steps:

[0050] Step 5.1: Input the multi-scale feature sequence and obtain the multi-scale token feature sequence (E1, E2, E3, E4) output in Step 4;

[0051] Step 5.2: Initialize the query sequence. Select the finest feature sequence E1 to retain as much detail as possible, and initialize it as the starting query sequence.

[0052] Step 5.3, Dimension Alignment: Use linear mapping to align the query sequence. Mapped to the same dimension as other token sequences;

[0053] Step 5.4, recursively multi-scale feature interaction, to Starting from this point, it recursively interacts with its lower-scale token sequence E2, E3, E4, E1.

[0054] Step 5.5: Apply multi-head cross-attention for each pair of query sequences. and token sequence E i Multi-head cross-attention is used to integrate cross-scale feature information, as shown in the following formula:

[0055]

[0056]

[0057] In the formula, and It is a learnable parameter matrix used to process the query sequence. and token sequence E i These are mapped to query, key, and value representations, respectively, where n represents the number of attention heads to allow the model to capture information in parallel across multiple subspaces, and d... k It is the channel dimension of the key, and the softmax(·) function converts the attention score into a probability distribution;

[0058] Step 5.6: Apply multi-head self-attention. Apply multi-head self-attention to the output of multi-head cross-attention to capture long-range dependencies within the sequence. The formula is as follows:

[0059]

[0060] In the formula, and It is a learnable parameter matrix, where n represents the number of attention heads, which is consistent with the parameters set in step 5.5, and A(·) represents the attention function, which is consistent with step 5.5.

[0061] Step 5.7, Residual Connection and Layer Normalization: Combine the output of the multi-head self-attention with the original query sequence. After summing and normalizing by layer, the formula is as follows:

[0062]

[0063] In the formula, LN(·) is a layer normalization operation, which is used to normalize the channel features to ensure the stability of the data distribution and reduce the internal covariate bias.

[0064] Step 5.8, update the query sequence and output the updated query sequence. and the token sequence E of the next scale. i+1 Interact, until sequences at all scales are interacted and aggregated;

[0065] Step 5.9: Output global context features. After completing all recursive interactions, output the aggregated global context features.

[0066] Step 6 is implemented in the following steps:

[0067] Step 6.1, pixel-level probability distribution calculation: First, apply a convolution operation to the depth cue map to map the channels to N dimensions; then, apply the softmax activation function to the mapped depth cue map to obtain the probability distribution p of each pixel in the depth interval (bins). i,j,k ,1≤i≤N,1≤j≤h,1≤k≤w,where N,h, andw represent the depth bin dimension, the height of the image, and the width of the image, respectively;

[0068] Step 6.2, calculate the bin center. The bin center value is calculated based on the bin width, using the following formula:

[0069]

[0070] In the formula, d min and d max b refers to the lower and upper limits of the depth range, respectively. i Indicates the width of each depth bin;

[0071] Step 6.3: Calculate the final depth value. For each pixel in the image, calculate the final depth value d based on a linear combination of the softmax score and the bin center value. The formula is as follows:

[0072]

[0073] Step 6.4, Depth Value Normalization: The calculated depth values ​​are normalized to ensure they remain within the predetermined depth range [-d].min ,d max ]Inside.

[0074] Step 7 is implemented in the following steps:

[0075] Step 7.1, Image Feature Extraction, using a shared backbone network with the deep network to extract features from the target image I. t and adjacent frame images I t′ Extract features from;

[0076] Step 7.2, Relative pose estimation: Jointly train an auxiliary pose network, which learns from the target image I. t and reference image I t′ Estimate their relative pose T t→t′ The pose contains rotation and translation information from the target image to the reference image.

[0077] Step 8 is implemented in the following steps:

[0078] Step 8.1, Image Projection, using the depth map D obtained in Step 6. t The relative pose T output in step 7 t→t′ I is calculated using the projection function proj(·). t Each pixel in the adjacent frame image I t′ The corresponding 2D coordinates;

[0079] Step 8.2, Image Synthesis: Based on the 2D coordinates of the projection, the deformed image I is synthesized using the sampling operator <·>. t→t′ The formula is expressed as follows:

[0080] I t→t′ =I t′ <proj(D t ,T t→t′ ,K)> (17)

[0081] In the formula, proj(D) t ,T t→t′ K) Returns the depth map D t in I t′ The 2D coordinates of the projection in the image, <·>, are used to distinguish between adjacent frame images I. t′ The pixel value at the corresponding sampling position is denoted by K, which represents the camera's intrinsic parameter matrix.

[0082] Step 9 is implemented in the following steps:

[0083] Step 9.1, calculate the photometric error, expressed by the following formula:

[0084]

[0085] In the formula, α is the weight used to balance SSIM and L1 norm, usually set to 0.85, and SSIM is the structural similarity index used to measure the visual similarity between two images.

[0086] Step 9.2, calculate the minimum photometric error, expressed by the following formula:

[0087] L p (I t ,I t′ ) = min t′ pe(I t ,I t→t′ (19)

[0088] In the formula, t′∈{t-1,t+1} represents considering the target image I t The previous or next frame;

[0089] Step 9.3 generates an automatic mask to filter out stationary pixels and low-texture areas, reducing interference from moving objects in dynamic scenes during training. The formula is as follows:

[0090] μ = [min t′ pe(I t ,I t′→t ) <min t′ pe(I t ,I t′ (20)

[0091] In the formula, [.] indicates that Iverson brackets are used to convert the expression to a binary value, so μ∈{0,1};

[0092] Step 9.4, calculate the image reprojection error, expressed by the following formula:

[0093] L r (I t ,I t→t′ )=μ·L p (I t ,I t′ ) (twenty one).

[0094] The beneficial effects of this invention are:

[0095] This invention is based on a self-supervised depth estimation method that utilizes block detail extraction and multi-level feature aggregation. It employs a self-supervised monocular depth estimation model that does not rely on external labeled data but instead learns and optimizes itself through the characteristics of its internal data structure. By progressively mining the fine features of local regions and continuously refining and aggregating multi-scale contextual information through cross-layer interaction of multi-level feature sequences, the model's ability to perceive multi-scale objects and its recovery of detailed structures in complex scenes are significantly enhanced. Attached Figure Description

[0096] Figure 1 This is a network architecture diagram of the self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation of the present invention;

[0097] Figure 2 This invention relates to the block detail extraction module of a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation.

[0098] Figure 3 This invention relates to a multi-level feature interaction module for a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation.

[0099] Figure 4 This is a visualization of the results of the self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation of the present invention. Detailed Implementation

[0100] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments;

[0101] Example 1

[0102] This invention is a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation. The overall framework is as follows: Figure 1 As shown, please follow these steps:

[0103] Step 1, Image Feature Extraction;

[0104] Step 2, multi-scale feature extraction;

[0105] Step 3, extract details from blocks;

[0106] Step 4, long-distance dependent capture;

[0107] Step 5, multi-level feature aggregation;

[0108] Step 6, Depth value prediction;

[0109] Step 7, relative pose estimation;

[0110] Step 8, self-supervised signal generation;

[0111] Step 9: Model training and optimization.

[0112] Example 2

[0113] This invention is based on a self-supervised depth estimation method using block detail extraction and multi-level feature aggregation, and is implemented according to the following steps:

[0114] Step 1: Use a Transformer-based hierarchical codec structure to perform in-depth feature extraction on the RGB image;

[0115] Step 2: Input the obtained features into the pyramid pooling module to extract multi-scale visual features;

[0116] Step 3: The multi-scale features are fed into the block detail extraction module, which segments the input feature map into blocks and extracts the fine features of each local region.

[0117] Step 4: All fine feature sequences are fed into the visual Transformer, which captures the complex long-distance dependencies between features through a self-attention mechanism.

[0118] Step 5: Further integrate multi-scale features through a multi-level feature aggregation module to construct a comprehensive global context;

[0119] Step 6: Map each pixel to the actual depth value using the generated depth cue map;

[0120] Step 7: Accurately estimate the relative pose between the target image and adjacent frames using an auxiliary pose network;

[0121] Step 8: Synthesize a new view as a self-supervised signal;

[0122] Step 9: Calculate the loss between the synthesized view and the generated depth map to optimize the network model.

[0123] Example 3

[0124] This invention is based on a self-supervised depth estimation method using block detail extraction and multi-level feature aggregation, and is implemented according to the following steps:

[0125] Step 1 is the feature extraction module for the input image. A hierarchical Transformer module is used to extract features from the input image. The specific steps are as follows:

[0126] Step 1.1, Input Image Segmentation: Segment the input RGB image. Divide into non-overlapping patches of size P×P, with each patch treated as a token;

[0127] Step 1.2, Linear Projection: Perform a linear projection on each token, mapping it to a higher-dimensional feature space.

[0128] Step 1.3, Feature Merging: The Patch Merging module is used to merge adjacent patch features, achieving downsampling of the feature map. During this process, the number of tokens is reduced by a factor of 4, and the channel dimension is increased by a factor of 2 through a linear transformation.

[0129] Step 1.4: Multi-stage feature extraction. Four Transformer Blocks are constructed, with the output of each stage serving as the input for the next stage, generating four feature maps at different resolutions.

[0130] Step 1.5: Feature map upsampling and concatenation. Upsampling restores the low-resolution feature map to a high-resolution version, and then concatenates it with the high-resolution features from the decoder via skip connections. This aggregates high-level and low-level feature information, resulting in a final output feature map X with a resolution of [resolution missing].

[0131] Step 2 involves a pyramid pooling module that extracts multi-scale features from the input feature map. The specific steps are as follows:

[0132] Step 2.1: Construct three pyramid pooling layers, each using convolutional kernels of sizes 1×1, 3×3, and 6×6 to aggregate features at different scales;

[0133] Step 2.2, Parallel pooling operation: Apply average pooling operation to the feature map at each scale;

[0134] Step 2.3, Feature Upsampling: Perform an upsampling operation on the feature map at each scale after average pooling, and use bilinear interpolation to restore the size of the feature map to the same level as the original input feature map. Figure 1 Achieved spatial resolution;

[0135] Step 2.4: Output multi-scale feature maps, which will be used as input to subsequent network modules to support deeper feature learning and information integration.

[0136] Step 3 is the block detail extraction module, which segments the input feature map and extracts local details. Its network structure is as follows: Figure 2 As shown, the specific steps are as follows:

[0137] Step 3.1, Feature map segmentation, segmenting the input multi-scale feature map. Divide into N equal sizes Each patch is a specific region in the feature map, where and

[0138] Step 3.2, apply channel attention to each segmented patch p i The independent application channel attention mechanism is expressed by the following formula:

[0139]

[0140] In the formula, p′ iThis represents the channel data after adjustment using the channel attention mechanism. The channel attention is represented by the intermediate weight graph obtained through calculation. Weighting is applied to the channel dimension to adjust p. i The contribution of each channel; in addition, γ(·) represents an activation function for nonlinear transformation, α i (·) represents a channel weight mapping function;

[0141] Step 3.3, apply spatial attention to the result p′ of channel attention. i The formula for applying spatial attention is as follows:

[0142]

[0143] In the formula, p″ i This represents the channel data after spatial attention processing. This represents channel attention, and the spatial attention function β. i The result p′ of channel attention i Generate a spatial weight graph This weighted graph is used to adjust the contribution of different spatial locations, and δ(·) represents an activation function that performs a nonlinear transformation on the spatial weighted graph;

[0144] Step 3.4, Feature aggregation and refinement, converting the spatial attention result p″ i Compared to the original patchp i The features are concatenated along the channel dimension; then, spatial global average pooling is used to refine the concatenated features, resulting in a more refined image. The formula is expressed as follows:

[0145] z i =ρ([p″) i ,p i ]), i=1,2,…,N (3)

[0146] In the formula, ρ(·) represents global spatial average pooling, and [·] represents feature concatenation operation;

[0147] Step 3.5, Feature Reorganization: Repeat the above steps for all patches, and rearrange all the obtained refined tokens to form an ordered feature sequence. One-dimensional convolution is used to reduce the dimensionality of the channels of the token sequence Z to adapt to the input requirements of subsequent network layers;

[0148] Step 4 involves constructing a visual Transformer module that establishes long-range dependencies within the feature sequences. The specific steps are as follows:

[0149] Step 4.1: Apply multi-head attention to each feature sequence to capture long-range dependencies within the sequence. The formula is as follows:

[0150]

[0151] In the formula, Q, K, and V are the query, key, and value matrices, respectively. and Let d represent the learnable parameter matrix, n represent the number of attention heads, the softmax(·) function converts the attention scores into a probability distribution, and d k It is the number of channel dimensions of the key;

[0152] Step 4.2: Apply a feedforward network to further extract features from the output of the multi-head attention. The formula is as follows:

[0153] FFN(E i ′)=ReLU(E i ′·W1+b1)·W2+b2 (7)

[0154] In the formula, E i ′ represents the result of applying multi-head attention in step 4.1, where W1 and W2 are learnable weight matrices, b1 and b2 are the corresponding bias terms, and ReLU(·) is the activation function used to introduce nonlinearity;

[0155] Step 4.3, Layer Normalization and Residual Connections: The output of the feedforward network is normalized, and residual connections are added to enhance the model's learning ability and stability. The formula is as follows:

[0156] E″ i =LN(E′) i +FFN(E′ i (8)

[0157] In the formula, LN(·) is the layer normalization operation, which normalizes each feature channel;

[0158] Step 5 involves constructing a multi-level feature aggregation module for multi-scale contextual information, the network structure of which is as follows: Figure 3 As shown, the specific steps are as follows:

[0159] Step 5.1: Input multi-scale feature sequences and obtain the multi-scale token feature sequences (E1, E2, E3, E4) output in Step 4. These sequences represent feature representations at different scales.

[0160] Step 5.2: Initialize the query sequence. Select the finest feature sequence E1 to retain as much detail as possible, and initialize it as the starting query sequence.

[0161] Step 5.3, Dimension Alignment: Use linear mapping to align the query sequence. Mapped to the same dimension as other token sequences;

[0162] Step 5.4, recursively multi-scale feature interaction, to Starting from this point, it recursively interacts with its lower-scale token sequence E2, E3, E4, E1.

[0163] Step 5.5: Apply multi-head cross-attention for each pair of query sequences. and token sequence E i Multi-head cross-attention is used to integrate cross-scale feature information, as shown in the following formula:

[0164]

[0165] In the formula, and It is a learnable parameter matrix used to process the query sequence. and token sequence E i Mapped to query (Q), key (K), and value (V) representations respectively, d k is the channel dimension of the key (K), n is the number of attention heads to allow the model to capture information in parallel across multiple subspaces, and the softmax(·) function is used to convert the attention scores into a probability distribution.

[0166] Step 5.6: Apply multi-head self-attention. Apply multi-head self-attention to the output of multi-head cross-attention to capture long-range dependencies within the sequence. The formula is as follows:

[0167]

[0168] In the formula, and It is a learnable parameter matrix, where n represents the number of attention heads, which is consistent with the parameters set in step 5.5, and A(·) represents the attention function, which is consistent with step 5.5.

[0169] Step 5.7, Residual Connection and Layer Normalization: Combine the output of the multi-head self-attention with the original query sequence. After summing and normalizing by layer, the formula is as follows:

[0170]

[0171] In the formula, LN(·) is a layer normalization operation, which is used to normalize the channel features to ensure the stability of the data distribution and reduce the internal covariate bias.

[0172] Step 5.8, update the query sequence and output the updated query sequence. and the token sequence E of the next scale. i+1 Interact, until sequences at all scales are interacted and aggregated;

[0173] Step 5.9: Output global context features. After completing all recursive interactions, output the aggregated global context features, which integrate all information from the finest to the coarsest scale.

[0174] Step 6 involves generating a depth value prediction head module for each pixel of the target image, which is implemented as follows:

[0175] Step 6.1, pixel-level probability distribution calculation: First, apply a convolution operation to the depth cue map to map the channels to N dimensions; then, apply the softmax activation function to the mapped depth cue map to obtain the probability distribution p of each pixel in the depth interval (bins). i,j,k ,1≤i≤N,1≤j≤h,1≤k≤w,where N,h, andw represent the depth bin dimension, the height of the image, and the width of the image, respectively;

[0176] Step 6.2, calculate the bin center. The bin center value is calculated based on the bin width, using the following formula:

[0177]

[0178] In the formula, d min and d max b refers to the lower and upper limits of the depth range, respectively. i Indicates the width of each depth bin;

[0179] Step 6.3: Calculate the final depth value. For each pixel in the image, calculate the final depth value d based on a linear combination of the softmax score and the bin center value. The formula is as follows:

[0180]

[0181] Step 6.4, Depth Value Normalization: The calculated depth values ​​are normalized to ensure they fall within the predetermined depth range [-d]. min ,d max ]Inside;

[0182] Step 7 involves estimating the auxiliary pose network for relative pose, which is implemented as follows:

[0183] Step 7.1, Image Feature Extraction, using a shared backbone network with the deep network to extract features from the target image I. t and adjacent frame images I t′Extract features from;

[0184] Step 7.2, Relative pose estimation: Jointly train an auxiliary pose network, which learns from the target image I. t and reference image I t′ Estimate their relative pose T t→t′ The pose contains rotation and translation information from the target image to the reference image;

[0185] Step 8 is the view synthesis module, which synthesizes new views as self-supervised training signals. The specific steps are as follows:

[0186] Step 8.1, Image Projection, using the depth map D obtained in Step 6. t The relative pose T output in step 7 t→t′ I is calculated using the projection function proj(·). t Each pixel in the adjacent frame image I t′ The corresponding 2D coordinates;

[0187] Step 8.2, Image Synthesis: Based on the 2D coordinates of the projection, the deformed image I is synthesized using the sampling operator <·>. t→t′ The formula is expressed as follows:

[0188] I t→t′ =I t′ <proj(D t ,T t→t′ ,K)> (17)

[0189] In the formula, proj(D) t ,T t→t′ K) Returns the depth map D t in I t′ The 2D coordinates of the projection in the image, <·>, are used to distinguish between adjacent frame images I. t′ The pixel value at the corresponding sampling position is used, and K represents the camera's intrinsic parameter matrix, which we assume is the same for all views;

[0190] Step 9 is to optimize the network model training, and the specific steps are as follows:

[0191] Step 9.1, calculate the photometric error, expressed by the following formula:

[0192]

[0193] In the formula, α is the weight used to balance SSIM and L1 norm, and SSIM is the structural similarity index used to measure the visual similarity between two images.

[0194] Step 9.2, calculate the minimum photometric error, expressed by the following formula:

[0195] L p (I t ,I t′ ) = min t′ pe(I t ,I t→t′ (19)

[0196] In the formula, t′∈{t-1,t+1} represents considering the target image I t The previous or next frame;

[0197] Step 9.3 generates an automatic mask to filter out stationary pixels and low-texture areas, reducing interference from moving objects in dynamic scenes during training. The formula is as follows:

[0198] μ = [min t′ pe(I t ,I t′→t ) <min t′ pe(I t ,I t′ (20)

[0199] In the formula, [·] represents Iverson brackets used to convert the expression to a binary value, so μ∈{0,1};

[0200] Step 9.4, calculate the image reprojection error, expressed by the following formula:

[0201] L r (I t ,I t→t′ )=μ·L p (I t ,I t′ ) (twenty one).

[0202] These steps collectively construct a self-supervised learning framework for training a depth estimation model. The core of this framework lies in guiding the model's learning by minimizing the image reprojection error, thereby ensuring the quality of the synthesized image. t→t′ With target image I t Make the visuals as consistent as possible.

[0203] Figure 4 This paper presents the depth estimation results of selected RGB images on the KITTI and Make3D datasets using the present invention. These visualizations demonstrate that the depth maps generated by the present invention are of high quality, exhibiting excellent detail and depth structure integrity. Upon closer inspection, even minute pedestrian outlines, prominent traffic signs, clearly defined roadblock locations, and details of roadside vegetation are clearly rendered.

[0204] This invention presents a self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation. It eliminates the need for expensive ground truth depth data, significantly reducing data acquisition costs and resolving the dependence on costly ground truth depth data. Through a self-supervised learning strategy, this invention effectively optimizes model parameters from monocular image sequences. Furthermore, in practical applications, depth estimation can be performed using only a single image, making depth estimation more cost-effective. The block detail extraction module divides the abstract feature map into multiple sub-regions and deeply mines and aggregates the fine features of each sub-region, significantly enhancing the model's perception of local depth cues and improving the ability to recover target details in complex scenes, especially in areas lacking texture or with occlusion. The multi-level feature aggregation module extracts and aggregates high-level semantic information at multiple scales from coarse to fine sequences to establish global contextual capabilities, optimizing the model's depth recovery capabilities for complex scene structures and multi-scale objects, achieving more accurate depth estimation. Through these methods, depth estimation techniques not only effectively capture key local details but also better understand the semantic structure of the scene, thereby improving the quality of depth maps. Especially when capturing scene elements such as pedestrians and traffic signs, it can generate depth maps with clear boundaries and precise structure.

Claims

1. A self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation, characterized in that, The specific steps are as follows: Step 1, Image Feature Extraction; Step 2, multi-scale feature extraction; Step 3, extract details from blocks; Step 4, long-distance dependent capture; Step 5, multi-level feature aggregation; Step 6, Depth value prediction; Step 7, relative pose estimation; Step 8, self-supervised signal generation; Step 9, Model training optimization; Step 3 is implemented in the following steps: Step 3.1, Feature map segmentation, segmenting the input multi-scale feature map. Divide into N equal-sized patches Each patch is a specific region in the feature map, where and ; Step 3.2: Apply channel attention to each segmented patch. The independent application channel attention mechanism is shown in equation (1) below: （1） In the formula, This represents the channel data after adjustment using the channel attention mechanism. The channel attention is represented by the intermediate weight graph obtained through calculation. Weighting is applied to the channel dimension to adjust The contribution of each channel; in addition, Let represent an activation function used for nonlinear transformations. This represents a channel weight mapping function; Step 3.3, apply spatial attention to the results of channel attention. Spatial attention is applied as shown in equation (2): （2） In the formula, This represents the channel data after spatial attention processing. Represents channel attention, spatial attention function Results of channel attention Generate a spatial weight graph This weighted map is used to adjust the contribution of different spatial locations. This represents an activation function that performs a nonlinear transformation on the spatial weighted graph; Step 3.4, Feature Aggregation and Refinement, which involves applying the results of spatial attention... Compared to the original patch The tokens are concatenated along the channel dimension, and then refined using spatial global average pooling to obtain a more refined token. The formula is expressed as follows: （3） In the formula, ( ) indicates global space average pooling, [ ] indicates a feature concatenation operation; Step 3.5, Feature Reorganization: Repeat the above steps for all patches, and rearrange all the obtained refined tokens to form an ordered feature sequence. And use one-dimensional convolution on the token sequence The channels are dimensionality reduced to meet the input requirements of subsequent network layers; Step 4 is implemented in the following steps: Step 4.1: Apply multi-head attention to each feature sequence to capture long-range dependencies within the sequence. The formula is as follows: （4） （5） （6） In the formula, , , These are query, key, and value matrices, respectively. and This represents the learnable parameter matrix. Indicates the number of attention heads. The function is to convert attention scores into a probability distribution. It is the number of channel dimensions of the key; Step 4.2: Apply a feedforward network to further extract features from the output of the multi-head attention. The formula is as follows: （7） In the formula, This indicates the result of applying multi-head attention in step 4.

1. and It is a learnable weight matrix. and It is the corresponding bias term, and It is an activation function used to introduce nonlinearity; Step 4.3, Layer Normalization and Residual Connections: The output of the feedforward network is normalized, and residual connections are added to enhance the model's learning ability and stability. The formula is as follows: （8） In the formula, It is a layer normalization operation that normalizes each feature channel; Step 5 is implemented in the following steps: Step 5.1: Input the multi-scale feature sequence and obtain the multi-scale token feature sequence output from Step 4. ; Step 5.2: Initialize the query sequence and select the most refined feature sequence. To retain as much detail as possible and initialize it as the starting query sequence. ; Step 5.3, Dimension Alignment: Use linear mapping to align the query sequence. Mapped to the same dimension as other token sequences; Step 5.4, recursively multi-scale feature interaction, to Starting from this point, gradually integrate it with its low-scale token sequence. Perform recursive interaction; Step 5.5: Apply multi-head cross-attention for each pair of query sequences. and token sequence Multi-head cross-attention is used to integrate cross-scale feature information, as shown in the following formula: （9） （10） （11） In the formula, and It is a learnable parameter matrix used to process the query sequence. and token sequence Mapped to queries respectively ( ),key( ) and value ( )express, It is a key ( The channel dimension, It refers to the number of attention heads, allowing the model to capture information in parallel across multiple subspaces. The function is to convert attention scores into a probability distribution; Step 5.6: Apply multi-head self-attention. Apply multi-head self-attention to the output of multi-head cross-attention to capture long-range dependencies within the sequence. The formula is as follows: （12） （13） In the formula, and It is a learnable parameter matrix. This indicates that the number of attention heads is consistent with the parameters set in step 5.

5. This indicates that the attention function is consistent with step 5.5; Step 5.7, Residual Connection and Layer Normalization: Combine the output of the multi-head self-attention with the original query sequence. After summing and normalizing by layer, the formula is as follows: （14） Step 5.8, update the query sequence and output the updated query sequence. and the token sequence of the next scale. Interact, until sequences at all scales are interacted and aggregated; Step 5.9: Output global context features. After completing all recursive interactions, output the aggregated global context features. Step 6 is implemented in the following steps: Step 6.1, pixel-level probability distribution calculation: First, apply a convolution operation to the depth cue map to map the channels to N dimensions; then, apply the softmax activation function to the mapped depth cue map to obtain the probability distribution of each pixel in the depth interval bins. , and These represent the depth bins dimension, the image height, and the width, respectively. Step 6.2, calculate the bin center. The bin center value is calculated based on the bin width, using the following formula: （15） In the formula, and These refer to the lower and upper limits of the depth range, respectively. This indicates the width of bins in each depth range; Step 6.3: Calculate the final depth value. For each pixel in the image, calculate the final depth value based on a linear combination of the probability of the softmax score and the bin center value. The formula is expressed as follows: （16） Step 6.4, Depth Value Normalization: The calculated depth values ​​are normalized to ensure they remain within the predetermined depth range. Inside; Step 7 is implemented in the following steps: Step 7.1, Image Feature Extraction, using a shared backbone network with the deep network from the target image. and adjacent frame images Extract features from; Step 7.2, Relative pose estimation: Jointly train an auxiliary pose network from the target image. and reference image Estimate their relative poses The pose contains rotation and translation information from the target image to the reference image.

2. The self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation according to claim 1, characterized in that, Step 1 is implemented in the following steps: Step 1.1, Input Image Segmentation: Segment the input RGB image. Divide into sizes Non-overlapping patches, each patch is treated as a token; Step 1.2, Linear Projection: Perform a linear projection on each token, mapping it to a higher-dimensional feature space. ; Step 1.3, Feature merging: Use the Patch Merging module to merge adjacent patch features to achieve downsampling of the feature map; Step 1.4, multi-stage feature extraction, constructing 4 hierarchical Transformer Blocks, each... The output of one stage serves as the input for the next stage, generating feature maps at four different resolutions. ; Step 1.5, Feature Map Upsampling and Concatenation: Upsampling restores the low-resolution feature map to a high-resolution version, and then concatenates it with the high-resolution features in the decoder via skip connections to aggregate high-level and low-level feature information, ultimately outputting the feature map. The resolution is .

3. The self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation according to claim 1, characterized in that, Step 2 is implemented in the following steps: Step 2.1: Construct three pyramid pooling layers, each using... , and Convolutional kernels of varying sizes aggregate features at different scales; Step 2.2: Simultaneously perform pooling operations, applying average pooling to the feature map at each scale; Step 2.3, Feature Upsampling: Perform upsampling operation on the feature map at each scale after average pooling, and use bilinear interpolation to restore the size of the feature map to the same spatial resolution as the original input feature map; Step 2.4: Output multi-scale feature maps.

4. The self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation according to claim 1, characterized in that, Step 8 is implemented in the following steps: Step 8.1, Image projection, using the depth map obtained in step 6. The relative pose output in step 7 Through the projection function To calculate Each pixel in the adjacent frame image The corresponding 2D coordinates; Step 8.2, image synthesis, based on the 2D coordinates of the projection, using the sampling operator Synthesized deformed image The formula is expressed as follows: （17） In the formula, Return to depth map exist 2D coordinates of the projection in the middle. Used to extract images from adjacent frames The pixel value at the corresponding sampling position is sampled. This represents the intrinsic parameter matrix of the camera.

5. The self-supervised depth estimation method based on block detail extraction and multi-level feature aggregation according to claim 1, characterized in that, Step 9 is implemented in the following steps: Step 9.1, calculate the photometric error, expressed by the following formula: （18） In the formula, It is used to balance the weights of SSIM and L1 norm. SSIM is the structural similarity index, which is used to measure the visual similarity between two images. Step 9.2, calculate the minimum photometric error, expressed by the following formula: （19） In the formula, Indicates consideration of the target image The previous or next frame; Step 9.3 generates an automatic mask to filter out stationary pixels and low-texture areas, reducing interference from moving objects in dynamic scenes during training. The formula is as follows: （20） In the formula, Iverson brackets are used to convert expressions to binary values, therefore ; Step 9.4, calculate the image reprojection error, expressed by the following formula: （21）。

Images