Global circulation depth refinement method for single-frame image
By improving the RepLKNet encoder and global feature modulation module, and combining it with an upsampling network, the problem of edge information loss in monocular depth estimation on resource-constrained platforms is solved, and efficient and high-precision depth estimation is achieved.
Patent Information
- Application Number
- CN202511132344.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-13
- Publication Date
- 2025-11-14
AI Technical Summary
Existing monocular depth estimation methods are difficult to deploy on resource-constrained embedded platforms, and deep encoders suffer from edge information loss.
By employing an improved RepLKNet encoder and parameter-sharing decoder, combined with a global feature modulation module and an upsampling network, high-precision depth estimation is generated by explicitly expanding the receptive field and recurrent depth refinement to optimize feature representation and computational efficiency.
While controlling computational overhead, it significantly improves depth estimation accuracy and can be deployed on resource-constrained embedded platforms, preserving image edge and motion boundary information.
Smart Images
Figure CN120953337A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a global cyclic depth thinning method for single-frame images. Background Technology
[0002] Current mainstream monocular depth estimation methods generally employ an encoder-decoder architecture, using deep convolutional networks to achieve coarse-to-fine depth regression. A deep encoder-decoder convolutional neural network can be represented as follows: the encoder extracts features from the input image, while the decoder is responsible for recovering depth information from these features. In a typical process, the encoder expands the receptive field by progressively reducing the feature map resolution and increasing the number of channels; that is, the coverage area of the input image region by points on the feature map in the network. The coverage area and weight distribution of the receptive field directly affect network performance, but deep encoders have inherent drawbacks: a fixed receptive field and a center weight bias that increases with depth easily lead to the loss of edge information. To address this, researchers have introduced improvements such as dilated convolution, multi-scale fusion, and attention mechanisms to enhance encoder performance. While these methods improve baseline performance, their high memory requirements make them difficult to deploy on resource-constrained embedded platforms. Summary of the Invention
[0003] This invention provides a global cyclic depth thinning method for single-frame images to address the problems existing in the prior art. Under the premise of controllable computational overhead, this invention significantly improves the accuracy of depth estimation by explicitly expanding the receptive field of the baseline cyclic thinning model, thereby verifying the effectiveness of the global cyclic depth thinning scheme in depth estimation.
[0004] The technical inventions employed in this invention include:
[0005] A global cyclic depth thinning method for single-frame images includes the following steps:
[0006] S1: Randomly select a single frame image as input;
[0007] S2: A large receptive field encoder is used to extract global features of the single frame image, and a parameter-sharing decoder is used to generate an initial depth map from the global features;
[0008] S3: Optimize the feature representation of the global features based on the global feature modulation module, so that the optimized features can be adapted to the subsequent optimization process;
[0009] S4: At a fixed resolution, the optimized features are combined and a cyclic depth refinement mechanism is used to perform cyclic depth refinement, generating multiple depth prediction results, and the initial depth estimation result is updated according to the refinement direction generated in each refinement process.
[0010] S5: Fuse the depth information generated by the cyclic depth refinement with the initial depth map to output the optimized depth estimate;
[0011] S6: Build an upsampling network, train the upsampling network to learn the mask, and upsample the optimized depth estimate to obtain the full resolution result.
[0012] Furthermore, in S2, an improved RepLKNet is used as the encoder of the backbone network to extract global features of the single frame image, and a parameter-shared decoder is used to generate an initial depth map from the global features.
[0013] The encoder uses an input image downsampled eight times to balance computational efficiency with feature preservation requirements;
[0014] Furthermore, the improved RepLKNet is an improvement on the original RepLKNet structure as follows:
[0015] (1) Adjust the second-stage convolution stride of the original RepLKNet from 2 to 1 so that the 1 / 4 resolution features can undergo more nonlinear transformations of RepLK blocks;
[0016] (2) Add three RepLK blocks to the third stage of the original RepLKNet to enhance feature processing at 1 / 8 resolution.
[0017] After the above improvements, the model only needs the first seven RepLK blocks to complete efficient deep encoding, which significantly improves computational efficiency compared to the original twenty-four-layer structure, thereby obtaining global features.
[0018] Furthermore, in S2, an initial depth map is generated from global features using a parameter-shared decoder. The parameter-shared decoder operates at 1 / 8 of the input resolution and performs feature decoding through five consecutive convolutional layers to obtain the depth prediction result. Finally, the sigmoid function is used to activate the output, so that the depth prediction result is limited to the range of 0 to 1.
[0019] 5. The global cyclic depth thinning method for single-frame images as described in claim 4, characterized in that: the parameter-shared decoder further employs two convolutional layers to re-aggregate depth information, ultimately obtaining the initial depth estimation result. .
[0020] Furthermore, in S3, the feature representation of global features is optimized by designing a global feature modulation module, which modulates the previously activated h using a convolution-based gated recurrent unit. t−1 and the current input x t The content between these elements is used to find the most suitable activation h in each iteration update. t，The process follows the formula:
[0021] ,
[0022] Among them, h t-1 Indicates previously activated, x t Indicates the current input. Indicates that the currently hidden activity is active; z t To update the door, the expression for how much information needs to be remembered to control the current hidden / active state is:
[0023] ,
[0024] in, It is the sigmoid activation function. For concatenation operators; Conv Z It is a single separable convolutional unit, consisting of convolutional layers of 1×3 and 3×1;
[0025] Currently hidden and active Depends on the current input x t and previously activated h t−1 The expression is:
[0026] ,
[0027] Among them, the reset gate r t This indicates the degree to which previous activations have been forgotten, and this degree is calculated using the following formula:
[0028] ,
[0029] Among them, Conv H and Conv R These are separable convolutional units that do not share weights.
[0030] Furthermore, in S3, the global features are modulated to optimize the feature representation, including the following operations:
[0031] The output image features are evenly divided into multiple groups, and interactive self-attention information is calculated within a specified window area to achieve global multi-scale information aggregation.
[0032] Feature fusion is performed by connecting channels and performing joint convolution to obtain a global multi-scale feature sequence for subsequent iterative refinement.
[0033] Furthermore, in S5, through the calculation formula The optimized depth information is fused with the initial prediction result to update the current depth prediction result; and a Sigmoid nonlinear activation is applied to the current depth prediction result, limiting the output result to between 0 and 1.
[0034] in, This represents the depth prediction result after the nth update. Indicates the refinement direction generated during the iterative refinement process; This represents the depth prediction result of the (n-1)th prediction, which is the result of the previous prediction.
[0035] Furthermore, in S6, the upsampling module uses convolutional layers to learn convex masks, treating each depth pixel at full resolution as a convex combination of its 3×3 neighborhood grid at 1 / 8 of the input resolution; after activation by the Softmax function, it controls the weight allocation of 9 neighborhoods at 1 / 8 of the input resolution; depth retrieval is performed at full resolution to achieve full resolution depth prediction, while preserving image edges and motion boundaries.
[0036] The present invention has the following beneficial effects:
[0037] 1. It can significantly improve the accuracy of depth estimation. During the iterative refinement process, the initial depth estimation result is adjusted multiple times using optimized features, thereby obtaining more accurate depth information.
[0038] 2. While improving the accuracy of depth estimation, this method also focuses on controlling computational overhead, enabling deployment on resource-constrained embedded platforms. Through improvements such as the RepLKNet encoder, a balance between computational efficiency and feature preservation is achieved.
[0039] 3. The design of the upsampling network can better preserve image edges and motion boundaries, thus preserving important image details while achieving full-resolution depth prediction. Attached Figure Description
[0040] Figure 1 The input is a single-frame image of autonomous driving;
[0041] Figure 2 For the global loop refinement model;
[0042] Figure 3 The RepLK block structure in the middle;
[0043] Figure 4 A GMSA-RF module is generated for global multi-scale features based on grouped self-attention. Detailed Implementation
[0044] The invention will now be further described with reference to the accompanying drawings.
[0045] This invention provides a global cyclic depth thinning method for single-frame images, comprising the following steps:
[0046] I. For example Figure 1A single frame image is randomly selected as input, denoted as . ;
[0047] 2. A large receptive field encoder is used to extract global features from the input image, resulting in global features for a single frame image:
[0048] In the field of monocular depth estimation, research has shown that initializing model parameters based on ImageNet pre-training can significantly improve the robustness of monocular depth estimation. Therefore, this invention selects RepLKNet as the encoder backbone network, leveraging its powerful feature extraction capabilities to optimize the training convergence of the depth estimation task.
[0049] The RepLKNet encoder downsamples the input image by eight times to balance computational efficiency with feature preservation. While its first two stages achieve this downsampling using only four large convolutional kernels, this shallow structure may weaken the model's non-linear expressive power. Therefore, this invention specifically improves the original RepLKNet structure (i.e., obtains the improved RepLKNet) to compensate for this performance limitation. The specific improvements are as follows:
[0050] (1) Adjust the second-stage convolution stride from 2 to 1 so that the 1 / 4 resolution features can undergo more RepLK block nonlinear transformations;
[0051] (2) Three RepLK blocks are added in the third stage to enhance feature processing at 1 / 8 resolution. The optimized model only needs the first seven RepLK blocks to complete efficient deep encoding, which significantly improves computational efficiency compared to the original twenty-four-layer structure, thereby obtaining the features of the specified input image. .
[0052] like Figure 2 The RepLKNet encoder is used to extract features from a single frame of image, thereby obtaining multi-scale image features: , and Their sizes are divided into 1 / 2, 1 / 4, and 1 / 8 of the input image frame. The process can be formalized as follows: Where H and W represent the input image frames respectively. Height and width, , and Multi-scale image features , and The number of channels, , and They were set to 64, 64 and 128 respectively.
[0053] Third, an initial depth map is generated from global features based on a parameter-shared decoder. This decoder operates with a fixed input resolution of 1 / 8 of the original input resolution, and performs feature decoding through five consecutive convolutional layers to obtain the depth prediction result. Finally, a sigmoid function is used at the output to activate the depth prediction result, limiting it to the range of 0 to 1. Furthermore, the decoder employs two additional convolutional layers to re-aggregate the depth information, promoting the full flow of depth information during the iterative refinement process, ultimately yielding the initial depth estimation result. .
[0054] IV. Figure 3 A global feature modulation module is designed to optimize feature representation and adapt features to subsequent optimization processes. The principle of the global feature modulation module is to modulate the previously activated h using gated recurrent units based on convolution. t−1 and the current input x t The content between these steps finds the most suitable activation h for each iteration update except the first update. t Among them, h t-1 Indicates previously activated, x t Indicates the current input. This indicates that the currently hidden activity is active. This process can be formalized as:
[0055] ,
[0056] In the formula, z t The expression for how much information needs to be remembered to update the currently hidden and active door control is:
[0057] ,
[0058] in, It is the sigmoid activation function. For the concatenation operator, Conv Z For a single separable convolutional unit, Conv Z It consists of two convolutional layers with 1×3 kernels and one 3×1 convolutional layer. Z Its purpose is to reduce model parameters while maintaining accuracy.
[0059] The current hidden / active state depends on the current input x. t and previously activated h t−1 This can be expressed as an expression:
[0060] ,
[0061] In the formula, the reset gate r t This indicates the degree to which previous activations have been forgotten, and this degree is calculated using the following formula:
[0062] ,
[0063] In the formula, Conv H and Conv R These are separable convolutional units that do not share weights.
[0064] like Figure 4 Modulating global features to optimize feature representation includes the following operations: the deep encoder is given output image features. First, divide it evenly into There are 1 group, denoted as .
[0065] For each feature group In the specified window size The self-attention information of interactions is calculated within the region, thereby achieving the aggregation of global multi-scale information. Furthermore, by constructing a symmetric Gaussian space—that is, setting the query vector to be in the same space as the key vector—the problem of interference can be mitigated. The computational burden of feature transformations via convolution.
[0066] Through the above operations, the computational efficiency of the self-attention mechanism can be further improved without affecting the estimation results.
[0067] After obtaining the self-attention information from multiple windows, it is combined through channel connections. Convolutional processing fuses features and aggregates global multi-scale self-attention information. After feature fusion is completed, the feature is split... That is, to obtain global multi-scale information and obtain global multi-scale feature sequences. This is used for subsequent iterations and refinement.
[0068] V. For example Figure 2 At a fixed resolution, joint modulation features and parameter-shared decoders are used for cyclic depth refinement. This cyclic refinement process generates a series of depth prediction results. Each refinement process generates a refinement direction. It can be used to update the initial depth estimation results. ;
[0069] VI. For example Figure 2 The optimized depth information is fused with the initial prediction results to output an optimized depth estimate, which updates the current depth prediction results. The calculation process is as follows: The final depth estimation result, specifically, is the depth prediction result for the current state. Apply a Sigmoid nonlinear activation to limit the output to between 0 and 1. dn represents the final depth estimation result after processing with the Sigmoid activation function.
[0070] VII. For example Figure 2 An upsampling network is constructed, and a mask is learned through training the upsampling network. The upsampling module uses convolutional layers to learn a convex mask, treating each depth pixel at full resolution as a convex combination of its 3×3 neighborhood grid at 1 / 8 of the input resolution. This is activated by the Softmax function to control the weight distribution of 9 neighborhoods at 1 / 8 of the input resolution, and depth retrieval is performed at full resolution. This achieves full-resolution depth prediction while better preserving image edges and motion boundaries. The final depth map is then upsampled to obtain the full-resolution result.
[0071] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.
Claims
1. A global cyclic depth thinning method for single-frame images, characterized in that: Includes the following steps: S1: Randomly select a single frame image as input; S2: Extract the global features of the single-frame image and generate an initial depth map from the global features; S3: Optimize the feature representation of global features so that the optimized features can be adapted to the subsequent optimization process; S4: At a fixed resolution, the jointly optimized features are subjected to iterative depth refinement to generate multiple depth prediction results, and the initial depth estimation results are updated according to the refinement direction generated in each refinement process. S5: Fuse the depth information generated by the cyclic depth refinement with the initial depth map to output the optimized depth estimate; S6: Build an upsampling network, learn the mask by training the upsampling network, and upsample the optimized depth estimate to obtain the full resolution result.
2. The global cyclic depth thinning method for single-frame images as described in claim 1, characterized in that: In S2, an improved RepLKNet is used as the encoder of the backbone network to extract global features of the single frame image, and a parameter-shared decoder is used to generate an initial depth map from the global features.
3. The global cyclic depth thinning method for single-frame images as described in claim 2, characterized in that: The improved RepLKNet is an improvement on the original RepLKNet structure as follows: (1) Adjust the stride of the second-stage convolution in the original RepLKNet from 2 to 1; (2) Add three layers of RepLK blocks to the third stage of the original RepLKNet.
4. The global cyclic depth thinning method for single-frame images as described in claim 1, characterized in that: In S2, an initial depth map is generated from global features using a parameter-shared decoder. The parameter-shared decoder operates at 1 / 8 of the input resolution and performs feature decoding through five consecutive convolutional layers to obtain the depth prediction result. Finally, the sigmoid function is used at the output to activate the depth prediction result, limiting it to the range of 0 to 1.
5. The global cyclic depth thinning method for single-frame images as described in claim 4, characterized in that: The parameter-sharing decoder also employs two additional convolutional layers to re-aggregate depth information, ultimately obtaining the initial depth estimation result. .
6. The global cyclic depth thinning method for single-frame images as described in claim 1, characterized in that: In S3, a global feature modulation module is designed to optimize the feature representation of global features. This global feature modulation module modulates the previously activated h based on a convolutional gated recurrent unit. t−1 and the current input x t The content between these elements is used to find the most suitable activation h in each iteration update. t， The process follows the formula: ， Among them, h t-1 Indicates previously activated, x t Indicates the current input. Indicates that the current hidden activity is active; z t To update the door, the expression for how much information needs to be remembered to control the current hidden / active state is: ， in, It is the sigmoid activation function. For concatenation operators; Conv Z It is a single separable convolutional unit, consisting of convolutional layers of 1×3 and 3×1; Currently hidden and active Depends on the current input x t and previously activated h t−1 The expression is: ， Among them, the reset gate r t This indicates the degree to which previous activations have been forgotten, and this degree is calculated using the following formula: ， Among them, Conv H and Conv R These are separable convolutional units that do not share weights.
7. The global cyclic depth thinning method for single-frame images as described in claim 6, characterized in that: In S3, global features are modulated to optimize feature representation, including the following operations: The output image features are evenly divided into multiple groups, and interactive self-attention information is calculated within a specified window area to achieve global multi-scale information aggregation. Feature fusion is performed by connecting channels and performing joint convolution to obtain a global multi-scale feature sequence for subsequent iterative refinement.
8. The global cyclic depth thinning method for single-frame images as described in claim 1, characterized in that: In S5, through the calculation formula The optimized depth information is fused with the initial prediction results to update the current depth prediction results; A Sigmoid nonlinear activation is applied to the current depth prediction result, limiting the output result to between 0 and 1; in, This represents the depth prediction result after the nth update. Indicates the refinement direction generated during the iterative refinement process; This represents the depth prediction result of the (n-1)th prediction, which is the result of the previous prediction.
9. The global cyclic depth thinning method for single-frame images as described in claim 1, characterized in that: In S6, the upsampling module uses convolutional layers to learn convex masks, treating each depth pixel at full resolution as a convex combination of its 3×3 neighborhood grid at 1 / 8 of the input resolution; after activation by the Softmax function, it controls the weight allocation of 9 neighborhoods at 1 / 8 of the input resolution; depth retrieval is performed at full resolution to achieve full resolution depth prediction, while preserving image edges and motion boundaries.