High-precision circulation depth refining method suitable for embedded platform
By combining shallow feature encoders with multi-branch residual learning techniques and a triple attention mechanism, the problems of high storage consumption and low computational efficiency of multi-branch structures on embedded platforms are solved, achieving high-precision depth estimation that is suitable for deployment on embedded platforms.
Patent Information
- Application Number
- CN202511132354.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-13
- Publication Date
- 2025-11-14
AI Technical Summary
In the existing technology, the monocular depth estimation model with multi-branch structure has high storage consumption and low computational efficiency on embedded platforms, which limits its deployment and application in small devices or mobile devices.
We employ a shallow feature encoder and multi-branch residual learning technique, combined with structural reparameterization, to transform the multi-branch structure into a single-path inference structure. We also utilize a triple attention mechanism to construct a feature modulation module for multi-scale feature extraction and modulation, and gradually improve the depth estimation accuracy through a cyclic refinement process.
It effectively reduces model complexity, improves depth estimation accuracy, enhances computational efficiency, and is suitable for deployment on embedded platforms.
Smart Images
Figure CN120953341A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a high-precision loop depth refinement method suitable for embedded platforms. Background Technology
[0002] In coarse-to-fine monocular depth estimation schemes, researchers typically focus on designing feature extraction networks with deeper and more complex structures—i.e., depth encoders—to improve the accuracy of monocular depth estimation. In this coarse-to-fine approach, the initial design of the depth encoder primarily focuses on improving its performance by constructing multi-branch structures or integrating different high-performance components, aiming to enhance the accuracy of monocular depth estimation. However, this multi-branch structure design has limitations in practical deployment. First, the multi-branch structure requires storing intermediate results during inference, leading to a significant increase in GPU memory usage. Second, considering the computational parallelism and memory access costs during model inference, the multi-branch structure needs to compute the results of each branch separately. Due to uneven computational pressure, the computational speed of different branches varies, causing fast branches to wait for slower branches to perform data fusion. This reduces hardware computational efficiency, increases model inference time, and limits the deployment and application of the model on small or mobile devices. Summary of the Invention
[0003] The present invention provides a high-precision loop depth refinement method suitable for embedded platforms in order to solve the problems existing in the prior art.
[0004] The technical solutions adopted in this invention are as follows:
[0005] A high-precision loop depth refinement method suitable for embedded platforms includes the following steps:
[0006] Step 1: Select any single frame image;
[0007] Step 2: Design a shallow feature encoder that adapts to the cyclic thinning process. Its structure includes eleven convolutional modules. The first five layers downsample the input image by four times, and the last six layers downsample the input image by eight times.
[0008] Step 3: During the training of the shallow feature encoder, feature extraction is performed using multi-branch residual learning technology. During the testing phase, the multi-branch residual structure in the shallow feature encoder is transformed into a single-path inference structure through structural reparameterization technology to reduce the number of parameters.
[0009] Step 4: Use a shallow feature encoder to extract features from the single frame image to obtain multi-scale features;
[0010] Step 5: Design a parameter-shared decoder, whose input is the multi-scale features, outputs the initial depth prediction result, and activates it using the Sigmoid function at the output.
[0011] Step 6: Construct a feature modulation module based on a triple attention mechanism to perform feature modulation on the multi-scale features;
[0012] Step 7: Use the initial depth prediction result output in Step 5 as the input for iterative refinement and perform iterative refinement;
[0013] Step 8: Repeat the refinement process until the set number of refinement iterations is reached or the convergence condition is met, and finally obtain the depth prediction result to be estimated.
[0014] Furthermore, the structural reparameterization technique in step 3 includes the following steps:
[0015] (31) The convolutional layer and BN layer in the multi-branch residual structure are fused together, and the BN parameters of the 3×3 convolution, 1×1 convolution and identity mapping branches are retained respectively.
[0016] (32) Convert the fused convolutional layer and its corresponding BN layer into biased convolutional units;
[0017] (33) Merge the convolution weights and biases of all branches to generate a single 3×3 convolutional layer, thereby transforming the multi-branch training structure into a single-path inference structure.
[0018] Furthermore, the parameter-shared decoder operates at a fixed 1 / 8 input resolution, with the multi-scale features as its input. After convolutional feature decoding, it outputs the initial depth prediction result and activates it at the output using the Sigmoid function to limit the initial depth prediction result to between 0 and 1.
[0019] The parameter-sharing decoder also includes two convolutional layers for the re-aggregation of depth information to facilitate the full flow of depth information during the cyclic refinement process.
[0020] Furthermore, in step 6, a feature modulation module is constructed based on a triple attention mechanism, specifically as follows:
[0021] (51) Input tensor Input the three parallel branches of the triple attention mechanism respectively:
[0022] The first branch reduces the number of channels to two through a ZPool operation, transforming the dimension to 2×H×W. Then, it obtains a 1×C×W attention weight sequence through a 7×7 convolutional layer activated by Sigmoid. ;
[0023] The second and third branches first rotate the input tensor x, performing dimensionality transformations to obtain transformed tensors of H×C×W and W×H×C respectively. Similarly, the ZPool operation is used to reduce the number of channels to two, obtaining aggregated features. and Then, through a 7×7 convolutional layer activated by Sigmoid and without parameter sharing, attention weight sequences with dimensions 1×C×W and 1×H×C are generated. and ;
[0024] (52) Using the obtained attention weight sequence The content between previous deep prediction results and current input features is enriched with semantic content and spatial location information during iterative updates;
[0025] (53) The amount of information that needs to be remembered between the previous depth prediction result and the current depth prediction result is controlled by the learnable weight sequence and the update gate, and the degree of forgetting of the previous depth prediction result is adjusted by the reset gate, so as to finally achieve feature modulation.
[0026] Furthermore, in step 7, the cyclic refinement process includes the following sub-steps:
[0027] (71) Fuse the current depth prediction result with the multi-scale features;
[0028] (72) Modulate the fused features using the feature modulation module;
[0029] (73) Generate updated depth prediction results and apply Sigmoid nonlinear activation to the updated depth prediction results;
[0030] (74) Use the depth prediction result after the previous update as the input for the next iteration of refinement, and repeat steps 71 to 73 until the set number of refinement iterations is reached or the convergence condition is met.
[0031] Furthermore, during the iterative refinement in step 7,
[0032] From the initial depth prediction results Initially, a series of deep prediction results are generated;
[0033] Each refinement process generates an update direction. This is used to update the depth prediction results, and the update formula is: ;
[0034] Updated depth prediction results Apply Sigmoid activation, limiting the output to between 0 and 1: ;
[0035] The image feature sequence is channel-concatenated and fused with the previous depth prediction result through convolutional transformation C1 to obtain the fused feature Yn;
[0036] The fused feature Yn is modulated using the feature modulation module TAFM to obtain the modulated feature Dn;
[0037] The modulated feature Dn is input into the depth decoder and refined cyclically at a fixed input resolution of 1 / 8 to finally obtain the depth estimation result to be estimated.
[0038] The present invention has the following beneficial effects:
[0039] This paper effectively extracts and modulates multi-scale features through multi-branch residual learning and a triple attention mechanism, improving the accuracy of depth estimation. The feature modulation module based on the triple attention mechanism effectively modulates iterative image features during the iterative refinement process, enhancing the semantic content and spatial location information of the features. Structural reparameterization is used to transform the multi-branch residual structure into a single-path inference structure, reducing the number of parameters and model complexity, making it suitable for embedded platforms. Through multiple iterations to optimize the depth prediction results, the paper gradually guides the prediction from coarse to fine, improving the quality of depth estimation. The introduction of additional nonlinear transformation layers during the iterative refinement process reduces computational costs and improves model efficiency. Attached Figure Description
[0040] Figure 1 This is a single-frame image related to autonomous driving input in an embodiment of the present invention.
[0041] Figure 2 This is a structural diagram of the structural reparameterization technique and multi-branch learning technique in the embodiments of the present invention.
[0042] Figure 3 This is a feature modulation module based on an attention mechanism designed in this embodiment of the invention.
[0043] Figure 4 This is a high-precision loop depth refinement method in the embodiments of the present invention.
[0044] Figure 5 This is the final depth estimation result in the embodiments of the present invention. Detailed Implementation
[0045] The invention will now be further described with reference to the accompanying drawings.
[0046] This invention discloses a high-precision loop depth refinement method suitable for embedded platforms, comprising the following steps:
[0047] Step 1: As Figure 1Select a single frame image related to autonomous driving, denoted as... ;
[0048] Step 2: Design a shallow feature encoder that adapts to the cyclic thinning process. Its structure includes eleven convolutional modules. The first five layers downsample the input image by a factor of four, and the last six layers downsample the input image by a factor of eight.
[0049] Step 3: During the training of the shallow feature encoder, feature extraction is performed using multi-branch residual learning technology. During the testing phase, the multi-branch residual structure in the shallow feature encoder is transformed into a single-path inference structure through structural reparameterization technology to reduce the number of parameters.
[0050] like Figure 2 Where (a) is the RepVGG multi-branch residual learning technique, which introduces a convolutional kernel size of [missing value] during the training phase. The convolution and the kernel size are Multi-scale learning of features is performed using convolution and identity transformations to provide multiple paths for gradient flow, enabling more effective ensemble learning using multi-branch structures during the training phase.
[0051] (b) is a structural reparameterization technique that transforms the multi-branch residual structure during runtime into a single-path structure to reduce the computational overhead of the model while maintaining multi-scale feature learning capabilities, thus further adapting it to embedded platforms. This technique can be expressed as:
[0052] Step 1: First, fuse the convolutional layers and BN layers in the multi-branch residual structure. This process can be expressed by the formula: Let This indicates that the number of input and output channels is C1 and C2, and the kernel size is... The convolutional layer branches, This indicates that the input and output channels are C2 and C1, and the kernel size is... The convolutional layer branches.
[0053] They represent Similarly, the mean, standard deviation, scale factor, and offset factor of the BN layer after convolution are calculated. express The mean, standard deviation, scale factor, and offset factor of the BN layer after convolution. These are the mean, standard deviation, scaling factor, and offset factor of the identity mapping branch. and Let C1 and H2 represent the input and output respectively, and * denote the convolution operation. When C1=C2, H1=H2, and W1=W2, then:
[0054] ,
[0055] When identity connections are not included, the BN function for the inference phase can be written as:
[0056] ,
[0057] First, each batch normalization (BN) and its preceding convolutional layer are transformed into a convolution with a bias vector. Let {W', b'} be derived from... The resulting kernel and bias can then be obtained as follows:
[0058] ,
[0059] The final fusion result can be expressed as:
[0060] ,
[0061] Step 2: Convert the merged convolutional layers into single convolutional layers. The convolutional layers. Since the entire residual design may contain... There are two types of branches: convolutional branches and identity connections. Therefore, for The entire transformation process of the convolution branch utilizes Convolutional replacement Convolution, soon The numerical values in the convolution are shifted to The center point of the convolution.
[0062] For the identity connection branch, since it is entirely an identity mapping of the input, it can be... The weights at all 9 positions of the convolution are set to 1.
[0063] Step 3: Merge all elements in the residual branches The parameters in convolution. That is, fusing the weights W from all branches with the bias b to obtain a single... convolution.
[0064] Step 4: Use a shallow feature encoder to extract features from the single frame image to obtain multi-scale features.
[0065] Step 5: Design a parameter-shared decoder, whose input is the multi-scale features, outputs the initial depth prediction result, and activates it using the Sigmoid function at the output.
[0066] Step 6: Construct a feature modulation module based on the triple attention mechanism to perform feature modulation on the multi-scale features.
[0067] like Figure 3 The schematic diagram of the triple attention mechanism module is shown in 3(a), where... This represents a 7×7 convolutional layer activated by Sigmoid. As shown in the figure, the entire module consists of three parallel branches. The first two branches are used to obtain cross-dimensional interactions between channel dimension and spatial location, while the last branch is responsible for constructing global spatial attention, ultimately obtaining global attention across channels and space.
[0068] The ZPool layer in the diagram uses average pooling and max pooling to calculate global features, and then concatenates the results to reduce the input channel dimension to two dimensions. This allows for further optimization of computational efficiency while preserving global detail information to the maximum extent. This process can be specifically described as follows:
[0069] Given input tensor The input tensor x is fed into three parallel branches of the triple attention mechanism. The first branch constructs cross-dimensional interactions in the spatial location dimensions H and W. The second and third branches construct cross-dimensional interactions in the C and W dimensions and H and C dimensions, respectively, through rotation operations. For the first branch, given the input tensor x, it first performs a concatenation of max pooling and average pooling using the ZPool operation, as shown in the following equation, reducing its channel count to two channels, i.e., transforming the dimension to... Then through a 7×7 convolutional layer activated by Sigmoid. ,get attention weights .
[0070] ,
[0071] ,
[0072] in, The branch indexes represent the original feature map and the rotated feature map, and [.,.] represent the concatenation operations along the channel dimensions.
[0073] For the second and third branches, we first rotate the given input tensor x to obtain the dimensions respectively. , The transformation tensor is then used, and subsequently, ZPool operations are employed to combine max pooling and average pooling, reducing its channel count to two channels to obtain aggregated features. , Then, it is passed through a 7×7 convolutional layer that is activated by Sigmoid and does not share parameters. The generation dimension is , attention weights and Finally, the attention weight sequence is obtained. .
[0074] By utilizing attention weight sequences, a feature modulation module based on an attention mechanism can be constructed, as shown in Figure 3(b). Here, 1- indicates subtracting an element from 1. MA represents element-wise multiplication and addition of the sequence, and TA represents a triple attention mechanism module. Indicates that the channels are connected in series. This indicates element-wise addition, and Tanh represents a 1×1 convolutional layer activated by Tanh. Its workflow is as follows:
[0075] Given the previous activation ht−1 and the current input xt, the efficient feature modulation module TAFM proposed in this chapter is used to find the most suitable current activation ht for the current iteration update process. This process can be formulated as follows:
[0076] ,
[0077] in, and It is a learnable weight sequence used to balance cross-dimensional features, and the sum of each sequence is equal to 1. Furthermore, the update gate... An update gate is generated for the Triple Attention (TA) module to control the relationship between the previously activated ht−1 and the current hidden activation. The amount of information that needs to be remembered:
[0078] ,
[0079] ,
[0080] Among them, FZ and FQ are 1×1 convolutional layers with non-shared parameters, used for feature fusion and dimensionality reduction. In contrast , This represents another set of learnable weight sequences, whose sequence sum is also 1. Finally... This indicates the reset gate generated by the Triple Attention Mechanism (TA) module:
[0081] ,
[0082] Used to adjust the degree to which previous activations are forgotten, where FR is relative to FZ, and FQ represents another 1×1 convolution.
[0083] Step 7: Use the initial depth prediction result output in Step 5 as the input for iterative refinement and perform iterative refinement.
[0084] Step 8: Repeat the refinement process until the set number of refinement iterations is reached or the convergence condition is met, and finally obtain the depth prediction result to be estimated.
[0085] Figure 4The diagram illustrates the high-precision cyclic depth thinning method constructed in this invention. To adapt to embedded platforms, the complete depth encoder consists of eleven convolutional layers, downsampling the input image by eight times. The first five layers downsample the input image by four times, and the last six layers downsample the input image by eight times. To accommodate the multi-scale features required in the cyclic thinning process, the convolutional features of the last six layers are used as the input multi-scale features for subsequent parameter-shared decoders for cyclic thinning. It is important to note that this encoder performs multi-branch residual feature learning during training and converts to a single-path structure during testing.
[0086] The depth of this decoder is fixed. The input resolution is processed, and the depth prediction result is obtained through convolutional feature decoding. Finally, the output is activated using the Sigmoid function, thus confining the depth prediction result to the range of 0 to 1. Additionally, the decoder employs two extra convolutional layers to re-aggregate the depth information, ensuring sufficient flow of depth information during the cyclic refinement process. The entire cyclic refinement process starts from the initial depth... Initially, a series of deep prediction results are generated. Each refinement process generates an update direction. This update direction is used to update the previous coarse depth prediction. Guided to the current fine-grained depth prediction Then, for the current fine-grained depth prediction Apply a Sigmoid nonlinear activation to limit the output to between 0 and 1. To facilitate the fusion of depth and image features during the iterative thinning process, a convolutional transformation C1 is used to transform the image feature sequence from the depth encoder. Compared with previous rough depth predictions Channel concatenation and feature fusion are performed. Then, Tanh function activation is applied via convolution, resulting in:
[0087] ,
[0088] in, Yn represents the number of updates, and Yn represents the fusion features of the image and depth at the nth thinning step. This process avoids introducing an additional Tanh nonlinear transformation layer during the cyclic thinning process, thereby improving the flexibility of the cyclic thinning process and reducing its computational cost.
[0089] Then, the fused feature Yn, excluding the first refinement process, is modulated using the efficient feature modulation (TAFM) module, resulting in:
[0090] ,
[0091] Where Dn represents the modulated feature during the nth iteration of refinement. The modulated feature Dn is input into the depth decoder to achieve a fixed... Depth at input resolution The process involves iterative refinement, ultimately yielding the depth estimation result to be estimated, such as... Figure 5 As shown.
[0092] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.
Claims
1. A high-precision loop depth refinement method suitable for embedded platforms, characterized in that: Includes the following steps: Step 1: Select any single frame image; Step 2: Design a shallow feature encoder that adapts to the cyclic thinning process. Its structure includes eleven convolutional modules. The first five layers downsample the input image by four times, and the last six layers downsample the input image by eight times. Step 3: During the training of the shallow feature encoder, feature extraction is performed using multi-branch residual learning technology. During the testing phase, the multi-branch residual structure in the shallow feature encoder is transformed into a single-path inference structure through structural reparameterization technology to reduce the number of parameters. Step 4: Use a shallow feature encoder to extract features from the single frame image to obtain multi-scale features; Step 5: Design a parameter-shared decoder, whose input is the multi-scale features, outputs the initial depth prediction result, and activates it using the Sigmoid function at the output. Step 6: Construct a feature modulation module based on a triple attention mechanism to perform feature modulation on the multi-scale features; Step 7: Using the initial depth prediction result output in Step 5 as the initial input for cyclic refinement, perform cyclic refinement again using the parameter-shared decoder; Step 8: Repeat the refinement process until the set number of refinement iterations is reached or the convergence condition is met, and finally obtain the depth prediction result to be estimated.
2. The high-precision loop depth refinement method for embedded platforms as described in claim 1, characterized in that: Step 3, the structure reparameterization technique, includes the following steps: (31) The convolutional layer and BN layer in the multi-branch residual structure are fused together, and the BN parameters of the 3×3 convolution, 1×1 convolution and identity mapping branches are retained respectively. (32) Convert the fused convolutional layer and its corresponding BN layer into biased convolutional units; (33) Merge the convolution weights and biases of all branches to generate a single 3×3 convolutional layer, thereby transforming the multi-branch training structure into a single-path inference structure.
3. The high-precision loop depth refinement method for embedded platforms as described in claim 1, characterized in that: The parameter-shared decoder operates at a fixed 1 / 8 input resolution. Its input is the multi-scale feature. After the convolutional feature is decoded, the initial depth prediction result is output. The Sigmoid function is used to activate the output to limit the initial depth prediction result to between 0 and 1. The parameter-sharing decoder also includes two convolutional layers for the re-aggregation of depth information to facilitate the full flow of depth information during the cyclic refinement process.
4. The high-precision loop depth refinement method for embedded platforms as described in claim 1, characterized in that: In step 6, a feature modulation module is constructed based on a triple attention mechanism, specifically as follows: (51) Input tensor Input the three parallel branches of the triple attention mechanism respectively: The first branch reduces the number of channels to two through a ZPool operation, transforming the dimension to 2×H×W. Then, it obtains a 1×C×W attention weight sequence through a 7×7 convolutional layer activated by Sigmoid. ; The second and third branches first rotate the input tensor x, performing dimensionality transformations to obtain transformed tensors of H×C×W and W×H×C respectively. Similarly, the ZPool operation is used to reduce the number of channels to two, obtaining aggregated features. and Then, through a 7×7 convolutional layer activated by Sigmoid and without parameter sharing, attention weight sequences with dimensions 1×C×W and 1×H×C are generated. and ; (52) Using the obtained attention weight sequence Feature modulation is performed on the current input features and the previous input features to enrich the semantic content and spatial location information during the iterative update process; (53) The amount of information that needs to be remembered between the previous depth prediction result and the current depth prediction result is controlled by the learnable weight sequence and the update gate, and the degree of forgetting of the previous depth prediction result is adjusted by the reset gate, so as to finally achieve feature modulation.
5. The high-precision loop depth refinement method for embedded platforms as described in claim 1, characterized in that: Step 7, the cyclic refinement process includes the following sub-steps: (71) Fuse the current depth prediction result with the multi-scale features; (72) Modulate the fused features using the feature modulation module; (73) Generate updated depth prediction results and apply Sigmoid nonlinear activation to the updated depth prediction results; (74) Use the depth prediction result after the previous update as the input for the next iteration of refinement, and repeat steps 71 to 73 until the set number of refinement iterations is reached or the convergence condition is met.
6. The high-precision loop depth refinement method for embedded platforms as described in claim 5, characterized in that: During the iterative refinement in step 7, From the initial depth prediction results Initially, a series of deep prediction results are generated; Each refinement process generates an update direction. This is used to update the depth prediction results, and the update formula is: ; Updated depth prediction results Apply Sigmoid activation, limiting the output to between 0 and 1: ; The image feature sequence is channel-concatenated and fused with the previous depth prediction result through convolutional transformation C1 to obtain the fused feature Yn; The fused feature Yn is modulated using the feature modulation module TAFM to obtain the modulated feature Dn; The modulated feature Dn is input into the depth decoder and refined cyclically at a fixed input resolution of 1 / 8 to finally obtain the depth estimation result to be estimated.