Lightweight cyclic refinement method for image dense estimation
By employing a lightweight cyclic refinement method and utilizing multi-scale feature extraction and feature modulation, the error propagation problem in monocular depth estimation is solved, achieving efficient and accurate depth estimation suitable for portable devices.
Patent Information
- Application Number
- CN202511132348.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-13
- Publication Date
- 2025-11-14
AI Technical Summary
Existing monocular depth estimation models suffer from error propagation during training, leading to decreased estimation accuracy. Furthermore, the trade-off between model size and computational efficiency limits their application on portable devices.
A lightweight cyclic refinement method is adopted, which performs initial depth estimation through multi-scale feature extraction, feature modulation and parameter sharing decoder, and combines upsampling module to refine depth, gradually update depth estimation results, reduce model parameters and improve estimation accuracy.
It achieves lightweight depth estimation, reduces the trade-off between model size and computational efficiency, improves the accuracy of depth estimation, is suitable for portable devices, and can adapt to the needs of different scenarios.
Smart Images
Figure CN120953338A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a lightweight cyclic thinning method for image density estimation. Background Technology
[0002] Recovering dense depth information of a target scene pixel-by-pixel from a single image is one of the most challenging tasks in computer vision research. Thanks to the advancements in deep learning, monocular depth estimation techniques have rapidly overcome the bottlenecks in accuracy and robustness of traditional methods, revitalizing research in this area. Current monocular depth estimation research commonly employs coarse-to-fine estimation models. A key problem with this model is error propagation during training; errors from low-resolution stages are transmitted to high-resolution results, affecting the overall accuracy of the estimation. While deeper and more complex network structures can improve model performance, this error propagation problem remains a challenge. Furthermore, as network depth and the number of model parameters increase, the trade-off between model size and computational efficiency becomes more pronounced, limiting the application and scenario expansion possibilities of monocular depth vision technology on portable devices. Summary of the Invention
[0003] The present invention provides a lightweight cyclic thinning method for image density estimation in order to solve the problems existing in the prior art.
[0004] The technical solutions adopted in this invention are as follows:
[0005] A lightweight cyclic thinning method for image density estimation includes the following steps:
[0006] (1) Select any single frame image, extract features from the single frame image, and obtain the multi-scale features of the single frame image;
[0007] (2) Perform initial depth estimation on the multi-scale features;
[0008] (3) Modulate the multi-scale features to obtain modulation features adapted to cyclic refinement;
[0009] (4) The modulation features are cyclically refined, and a refinement direction is generated in each cycle;
[0010] (5) Apply the refinement direction generated in each iteration to the initial depth estimate in turn, and gradually update to obtain the final depth estimate result;
[0011] (6) Construct an upsampling module and learn the upsampling mask;
[0012] (7) Use the upsampling mask learned in step (6) to upsample the final estimated depth result to obtain the full-resolution depth estimation result.
[0013] Further, select any single frame image, denoted as... By using a shallow feature encoder to extract features from a single frame of image, multi-scale image features are obtained: , and Their sizes are 1 / 2, 1 / 4, and 1 / 8 of the input image frame, respectively. The process can be formally expressed as:
[0014] ,
[0015] Where H and W represent the input image frames respectively. Height and width, , and Multi-scale image features , and The number of channels is set to 64, 64 and 128 respectively.
[0016] Furthermore, in step 2, the parameter-shared decoder is used to perform initial depth estimation on the multi-scale features. The parameter-shared decoder operates at a fixed 1 / 8 input resolution, decodes the multi-scale features into depth prediction results through five consecutive convolutional layers, and activates the output using the Sigmoid function to limit the depth prediction results to between 0 and 1.
[0017] The decoder also employs two additional convolutional layers to re-aggregate depth information. These two convolutional layers are not included in the aforementioned five consecutive convolutional layers, ultimately obtaining the initial depth estimation result. .
[0018] Furthermore, in step 3, a feature modulation module is constructed to modulate the multi-scale features. This feature modulation module utilizes convolution-based gated recurrent units to modulate the previously activated h. t−1 and the current input x t The content between these parameters is implemented by the feature modulation module using the following formula:
[0019] ,
[0020] Among them, h t-1 Indicates previously activated, x t Indicates the current input. Indicates that the currently hidden activity is active;
[0021] z t For the update gate, its expression is:
[0022] ,
[0023] in, It is the sigmoid activation function. It is a series operator, Conv Z It is a single separable convolutional unit, consisting of two convolutional layers with 1×3 kernels and a 3×1 convolutional layer;
[0024] The current hidden / active state depends on the current input x. t and previously activated h t−1 :
[0025] ,
[0026] Among them, the reset gate r t The degree to which previous activations have been forgotten is adjusted, and the calculation formula is as follows:
[0027] ,
[0028] Among them, Conv H and Conv R These are separable convolutional units that do not share weights.
[0029] Furthermore, in step 3, the specific process of feature modulation of multi-scale features is as follows:
[0030] The feature X1 is downsampled by 1 / 4 by applying two consecutive 3×3 convolutional layers with a stride of 2;
[0031] The feature X2 is downsampled by 1 / 2 in a 3×3 convolutional layer with a stride of 2;
[0032] A 3×3 convolutional layer with a stride of 1 is applied to feature X3 for feature transformation;
[0033] Finally, Tanh nonlinear activation is applied to X3, X2, and X1 to obtain multi-scale features with consistent size. Perform feature modulation.
[0034] Furthermore, in step 4, additional feature extraction is performed on the image features at each scale, resulting in multiple iterative refinements; each iteration of the refinement process generates a refinement direction. Used to update the initial depth estimation results .
[0035] Furthermore, in step 5: by refining the direction The current depth prediction result is updated using the following formula:
[0036] ;
[0037] Updated depth prediction results Apply a Sigmoid nonlinear activation to limit the output to between 0 and 1. .
[0038] Furthermore, step 6 specifically includes:
[0039] By using convolutional layers to learn an upsampling mask, each depth pixel at full resolution is treated as a convex combination of its neighborhood 3×3 grid at 1 / 8 of the input resolution;
[0040] After activation by the Softmax function, the weight distribution of 9 neighborhoods is controlled at 1 / 8 of the input resolution;
[0041] Depth retrieval is performed at full resolution, enabling full-resolution depth prediction while preserving image edges and motion boundaries.
[0042] The present invention has the following beneficial effects:
[0043] 1. By reducing model parameters, lightweight depth estimation is achieved, reducing the trade-off between model size and computational efficiency, making it more suitable for portable devices.
[0044] 2. It fundamentally solves the error propagation problem caused by the coarse-to-fine estimation method, improves the accuracy of depth estimation, and can adaptively select the number of refinements according to actual needs, effectively adapting to the requirements of various different scenarios. Attached Figure Description
[0045] Figure 1 The input is a single frame image of autonomous driving.
[0046] Figure 2 A lightweight loop refinement model for [the application].
[0047] Figure 3 The multi-scale feature modulation module is used in this module.
[0048] Figure 4 This represents the refinement direction for each iteration in the intermediate loop refinement model.
[0049] Figure 5 This is an upsampling module for learning intermediate parameters.
[0050] Figure 6 The final depth estimate result is given in the figure. Detailed Implementation
[0051] The invention will now be further described with reference to the accompanying drawings.
[0052] This invention provides a lightweight cyclic thinning method for image density estimation, comprising the following steps:
[0053] I. For example Figure 1 Select a single frame image related to autonomous driving, denoted as... ;
[0054] II. Figure 2 A shallow feature encoder is used to extract features from a single frame of image. The encoder accepts a single frame of image as input to obtain multi-scale image features. , and Its size is divided into input image frames. , and Formulating this process into a formal expression, we have:
[0055] ,
[0056] Where H and W represent the input image frames respectively. Height and width, , and Multi-scale image features , and The number of channels is set to 64, 64 and 128 respectively.
[0057] III. Figure 2 We use a parameter-shared decoder to perform initial depth estimation on multi-scale features.
[0058] The decoder is fixed. The input resolution is used for state operation, and features are decoded to a depth prediction result through five consecutive convolutional layers. Finally, the output is activated using the Sigmoid function, thus constraining the depth prediction result to the range of 0 to 1. Additionally, the decoder employs two extra convolutional layers to re-aggregate the depth information, promoting the full flow of depth information during the iterative refinement process, ultimately obtaining the initial depth estimate. .
[0059] IV. Figure 3 Construct a feature modulation module.
[0060] The feature modulation module utilizes convolution-based gated recurrent units to modulate the previous activation h. t−1 and the current input x t The content between them, enriched with semantic content and spatial location information during iterative updates. Among them, h t-1 Indicates previously activated, x t Indicates the current input. This indicates that the current hidden activity is active. The goal of this module is to find the most suitable activation h for each iteration update except the first update. t This process can be formalized as follows:
[0061] ,
[0062] In the formula, z t The expression for how much information needs to be remembered to update the currently hidden and active gate control is as follows:
[0063] ,
[0064] in, It is the sigmoid activation function. It is a series operator, Conv Z It is a single separable convolutional unit, consisting of two convolutional layers with 1×3 kernels and a 3×1 convolutional layer. Its purpose is to reduce model parameters while maintaining accuracy.
[0065] The current hidden / active state depends on the current input x. t and previously activated h t−1 :
[0066] ,
[0067] Among them, the reset gate r t Adjust the degree to which previous activations have been forgotten, which is calculated using the following formula:
[0068] ,
[0069] Among them, Conv H and Conv R These are separable convolutional units that do not share weights.
[0070] V. For example Figure 2 Feature modulation is performed on multi-scale features.
[0071] First, we perform dimensional alignment on the multi-scale features. Specifically:
[0072] Feature X1 is downsampled by 1 / 4 using two consecutive 3×3 convolutional layers with a stride of 2. Then, feature X2 is downsampled by 1 / 2 using a 3×3 convolutional layer with a stride of 2. Next, feature X3 is transformed by a 3×3 convolutional layer with a stride of 1. Finally, Tanh nonlinear activation is applied to X3, X2, and X1 to obtain multi-scale features of consistent size. Perform feature modulation.
[0073] VI. By combining the parameter-sharing decoder with the feature modulation model, additional feature extraction is performed on image features at each scale, thereby enabling multiple iterations of refinement.
[0074] The refinement process generates a series of deep prediction results. Each refinement process generates a refinement direction. ,like Figure 4 Used to update the initial depth estimation results .
[0075] 7. Apply the iterative refinement results to the initial depth estimate to update the current depth prediction results. The calculation formula is: The final depth estimation result is obtained.
[0076] Specifically, this refers to the depth prediction result for the current state. Apply a Sigmoid nonlinear activation to limit the output to between 0 and 1. .
[0077] 8. For example Figure 5 We construct an upsampling module and learn the upsampling convex mask.
[0078] Specifically, this upsampling module uses convolutional layers to learn convex masks, treating each depth pixel at full resolution as a convex combination of its neighborhood 3×3 grid at 1 / 8 of the input resolution. Finally, it is activated using a Softmax function to achieve... The weight allocation of the nine neighborhoods is controlled at the input resolution, and depth retrieval is performed at full resolution, such as... Figure 6 It achieves full-resolution depth prediction while preserving image edges and motion boundaries relatively well;
[0079] 9. The learned upsampling convex mask is used to upsample the final estimated depth result to obtain the full-resolution depth estimation result.
[0080] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.
Claims
1. A lightweight cyclic thinning method for image density estimation, characterized in that: Includes the following steps: (1) Select any single frame image, extract features from the single frame image, and obtain the multi-scale features of the single frame image; (2) Perform initial depth estimation on the multi-scale features; (3) Modulate the multi-scale features to obtain modulation features adapted to cyclic refinement; (4) The modulation features are cyclically refined, and a refinement direction is generated in each cycle; (5) Apply the refinement direction generated in each iteration to the initial depth estimate in turn, and gradually update to obtain the final depth estimate result; (6) Construct an upsampling module and learn the upsampling mask; (7) Use the upsampling mask learned in step (6) to upsample the final estimated depth result to obtain the full-resolution depth estimation result.
2. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: Select any single frame image, denoted as... By using a shallow feature encoder to extract features from a single frame of image, multi-scale image features are obtained: , and Their sizes are 1 / 2, 1 / 4, and 1 / 8 of the input image frame, respectively. The process can be formally expressed as: ， Where H and W represent the input image frames respectively. Height and width, , and Multi-scale image features , and The number of channels is set to 64, 64 and 128 respectively.
3. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: In step 2, a parameter-shared decoder is used to perform initial depth estimation on the multi-scale features. The parameter-shared decoder operates at a fixed 1 / 8 input resolution, decodes the multi-scale features into depth prediction results through five consecutive convolutional layers, and activates the output using the Sigmoid function to limit the depth prediction results to between 0 and 1.
4. The lightweight cyclic thinning method for image density estimation as described in claim 3, characterized in that: The decoder also employs two additional convolutional layers to re-aggregate depth information, ultimately obtaining the initial depth estimation result. .
5. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: In step 3, a feature modulation module is constructed to modulate the multi-scale features. This feature modulation module utilizes convolution-based gated recurrent units to modulate the previously activated h. t−1 and the current input x t The content between these parameters is implemented by the feature modulation module using the following formula: ， Among them, h t-1 Indicates previously activated, x t Indicates the current input. Indicates that the currently hidden activity is active; z t For the update gate, its expression is: ， in, It is the sigmoid activation function. It is a series operator, Conv Z It is a single separable convolutional unit, consisting of two convolutional layers with 1×3 kernels and a 3×1 convolutional layer; The current hidden / active state depends on the current input x. t and previously activated h t−1 : ， Among them, the reset gate r t The degree to which previous activations have been forgotten is adjusted, and the calculation formula is as follows: ， Among them, Conv H and Conv R These are separable convolutional units that do not share weights.
6. The lightweight cyclic thinning method for dense image estimation as described in claim 1, characterized in that: In step 3, the specific process of feature modulation of multi-scale features is as follows: The feature X1 is downsampled by 1 / 4 by applying two consecutive 3×3 convolutional layers with a stride of 2; The feature X2 is downsampled by 1 / 2 in a 3×3 convolutional layer with a stride of 2; A 3×3 convolutional layer with a stride of 1 is applied to feature X3 for feature transformation; Finally, Tanh nonlinear activation is applied to X3, X2, and X1 to obtain multi-scale features with consistent size. Perform feature modulation.
7. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: In step 4, additional feature extraction is performed on the image features at each scale, resulting in multiple iterative thinning processes; each iteration of the thinning process generates a thinning direction. Used to update the initial depth estimation results .
8. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: In step 5: by refining the direction The current depth prediction result is updated using the following formula: ； Updated depth prediction results Apply a Sigmoid nonlinear activation to limit the output to between 0 and 1. .
9. The lightweight cyclic thinning method for image density estimation as described in claim 1, characterized in that: Step 6 specifically involves: By using convolutional layers to learn an upsampling mask, each depth pixel at full resolution is treated as a convex combination of its neighborhood 3×3 grid at 1 / 8 of the input resolution; After activation by the Softmax function, the weight distribution of 9 neighborhoods is controlled at 1 / 8 of the input resolution; Depth retrieval is performed at full resolution, enabling full-resolution depth prediction while preserving image edges and motion boundaries.