A robust bilateral dense feature up-sampling method based on gradient prior and application thereof
By generating a bilateral dense feature upsampling method based on gradient and distance priors, the inconsistency of upsampling operators in dense prediction networks across different tasks is solved, achieving good performance and versatility in multi-task scenarios.
Patent Information
- Application Number
- CN202310740750.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-06-21
AI Technical Summary
In existing dense prediction networks, the upsampling operator exhibits task-specific biases, fails to perform consistently across different dense prediction tasks, and lacks effective utilization of low-resolution feature maps, resulting in insufficient versatility.
By generating a gradient-based bilateral dense feature upsampling method, an upsampling kernel is generated using the gradient and distance prior information of the low-resolution feature map. The gradient kernel and the distance kernel are then combined to perform a convolution operation to achieve feature map upsampling.
It achieves upsampling with multi-task robustness and good performance in different dense prediction tasks, overcomes the dependence on high-resolution features, and improves the versatility and plug-and-play nature of the operator.
Smart Images
Figure CN116863254B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and more particularly relates to a robust bilateral dense feature up-sampling method based on gradient prior and application. BACKGROUND
[0002] Dense Prediction is a hot field in current deep learning tasks, which covers a series of pixel-by-pixel labeling tasks in vision, including Object Detection, Depth Estimation, Semantic Segmentation, Image Matting, Image Super-Resolution, etc. With the emergence of the first end-to-end dense prediction network, a large number of deep learning-based dense prediction networks have emerged. Based on the commonality of these tasks, the current typical dense prediction deep network architecture is mainly based on the form of encoder and decoder: the input features are extracted by the encoder, and then the features are aggregated and processed to obtain the final result through the feature fusion and decoder head modules of the decoder. At present, a large number of related works focus on improving the ability of the encoder to extract features, hoping to optimize the extraction and utilization of features from the input end of the network, so as to improve the prediction effect of the network. These works include but are not limited to: multi-layer stacking to improve the model receptive field, increasing inter-layer skipping, and adding attention mechanism. However, for most dense prediction tasks, the final output of the network is usually the same resolution as the input result, which inevitably leads to the need for feature up-sampling steps at the decoder end: using feature up-sampling operators to up-sample low-resolution features to high-resolution. Therefore, feature up-sampling has become an indispensable part of mainstream dense prediction networks. At present, the common up-sampling operators in dense prediction networks mainly include classical interpolation operators represented by bilinear interpolation, learnable up-sampling operators represented by Deconvolution and Pixel Shuffle, and dynamic up-sampling operators represented by CARAFE and IndexNet.
[0003] Recently, with the rapid development of dense prediction tasks, limited selection of up-sampling operators has become one of the key factors restricting the performance of dense prediction networks. However, most of the up-sampling operators have their specific task preferences except for the most basic interpolation up-sampling operator. For example, the index network (IndexNet) performs well in the image matting task, but performs poorly in the semantic segmentation and object detection tasks. On the contrary, the use of the CARAFE operator in semantic segmentation and object detection brings great performance improvement, but it cannot function in the image matting task, and even brings worse results than the traditional interpolation operator, which makes researchers need to reasonably select the appropriate up-sampling operator when training different dense prediction networks.
[0004] In addition, in recent years, up-sampling operators similar to the index network (IndexNet) have been proposed. Although these operators are dynamic operators suitable for dense prediction tasks and have shown excellent results in specific tasks, their limitation is that they all use high-resolution encoder features for help, which makes the operators have additional auxiliary information in the up-sampling process, thereby obtaining up-sampling results closer to the encoder features. However, the design of such operators ignores the application in tasks such as image super-resolution (Image Super-Resolution) and other dense prediction tasks. Such tasks often lack corresponding high-resolution encoding features, so the above two algorithms cannot be used in related tasks, which greatly reduces their versatility. SUMMARY
[0005] In view of the above defects or improvement needs of the prior art, the present application provides a gradient prior-based robust bilateral dense feature up-sampling method and application, which aims to reasonably utilize the prior information of low-resolution feature maps to help the feature maps to be up-sampled, thereby overcoming the limitation that the operator needs high-resolution feature guidance, so that it can have multi-task robustness while having good performance and plug-and-play capability.
[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, a gradient prior-based robust bilateral dense feature up-sampling method is provided, comprising the following steps:
[0007] S1: extracting image features that need to be up-sampled from a dense prediction network to obtain an input feature map;
[0008] S2: performing mean value processing on the input feature map to obtain a mean value feature map;
[0009] S3-1: performing gradient mapping on the mean value feature map to obtain gradient prior features of the mean value feature map, and arranging the gradient prior of each corresponding point as an up-sampling gradient kernel;
[0010] S3-2: mapping the relative distance between the original pixel and the up-sampled pixel in the up-sampling process to generate a distance prior feature, and arranging the distance prior feature of each corresponding point into an up-sampling distance kernel;
[0011] S4: combining the up-sampling distance kernel and the up-sampling gradient kernel to generate an up-sampling kernel;
[0012] S5: performing convolution operation between the input feature map and the up-sampling kernel to obtain the up-sampled feature map corresponding to the input feature map.
[0013] Further, in S1, the dense prediction network can be any existing convolutional neural network, including but not limited to image matting network, monocular depth estimation network, semantic segmentation network, etc. The image feature that needs to be up-sampled is the intermediate layer feature of the dense prediction network. In the case of default two times up-sampling, given the input feature map the output feature map after the up-sampling operator is
[0014] Further, in S2, given the input feature map performing mean operation on the channel dimension to obtain the corresponding single-channel mean feature map where C, H, and W are the channel number, feature height, and feature width of the input feature map, respectively.
[0015] Further, in S3-1, the gradient of the mean feature map is gradient-mapped using a gradient mapping function g(x) to obtain a gradient prior, and the gradient prior is arranged in the form of a gradient kernel. In particular, the gradient mapping function can be any differentiable smooth function that is monotonically decreasing in [0, +∞), and in this case, the function is selected as the gradient mapping function, and ∈ is an infinitesimal constant to prevent the denominator from being 0. In particular, the gradient of each feature point of the mean feature map is the difference between the pixel value of the point and the pixel values of the 3x3 neighborhood points centered on the point. Specifically, the point on the up-sampled feature map is located at coordinate p=(i,j), and the corresponding point before up-sampling is located at coordinate In X, a 3x3 local region Ω centered on the point is extracted, and the gradient prior of the point is mapped to where q∈Ω. Subsequently, the distance prior is arranged in the form of a 3x3 kernel.
[0016] Further, S3-2 includes: directly mapping the relative distance prior between the points before and after up-sampling, and the mapping function is is a constant infinitesimal to prevent the denominator from being zero. Specifically, similarly as in S2, the distance of the point on the up-sampled feature map is mapped to Subsequently, the distance prior mapping is arranged in the form of a 3x3 kernel. In addition, a distance bias value can be added to the distance prior when producing the distance prior to increase the flexibility of the distance prior. The distance prior with bias is where Δ=f(X), and f is a NN up-sampling layer connected to a 5x5 two-dimensional convolution layer. Subsequently, the distance prior with bias is arranged in the form of a 3x3 kernel.
[0017] Further, in S4, the gradient kernel is multiplied by the corresponding distance kernel, and a normalization operation is performed to obtain an up-sampling kernel. In particular, the normalization operation here selects Softmax normalization. Specifically, the point on the up-sampled feature map is located at coordinate p=(i,j), and the corresponding point before up-sampling is located at coordinate A 3x3 local region Ω centered at the point is extracted in X, and the form of the up-sampling kernel at p is:
[0018]
[0019] When considering the distance bias, the form is:
[0020]
[0021] where q∈Ω.
[0022] Further, in S5, the input feature X is convolved with the up-sampling kernel to obtain the final output X out . Specifically, the convolution process can be represented as:
[0023]
[0024] According to another aspect of the present application, a training method of a dense prediction network is provided, comprising:
[0025] S1: obtaining a new dense prediction network by using the above up-sampling method;
[0026] S2: retraining the dense prediction network under the original conditions to obtain a target dense prediction network.
[0027] According to another aspect of the present application, a dense prediction method is provided, comprising:
[0028] S1: obtaining a target dense prediction network by using the training method of the dense prediction network.
[0029] S2: processing the target dense prediction network on the current image to obtain a target dense prediction result.
[0030] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method.
[0031] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:
[0032] 1. The present application provides a robust bilateral dense feature up-sampling method based on gradient prior, which proposes a bilateral up-sampling kernel idea, generates a gradient kernel using the gradient prior information of the current feature map, and generates a distance kernel using the distance prior information of the relative position distance before and after the up-sampling point, combines the two different prior information kernels together to obtain the up-sampling kernel. Finally, the up-sampling kernel is used for convolution operation with the input feature map to be up-sampled to obtain the required up-sampled feature map.
[0033] 2. The present application can solve the problem of requiring high-resolution feature guidance in the previous up-sampling method, and is a plug-and-play up-sampling operator. By reasonably using the prior information of the low-resolution feature map to help the feature map to be up-sampled, the limitation of the previous operator requiring high-resolution feature guidance is overcome, so that it can have multi-task robustness, good performance and light weight, so that the operator can be widely used in the up-sampling link of various dense prediction networks. BRIEF DESCRIPTION OF DRAWINGS
[0034] Figure 1 is a flowchart of a robust bilateral dense feature up-sampling method based on gradient prior provided by an embodiment of the present application.
[0035] Figure 2 is a schematic diagram of the up-sampling feature generation process in the operator provided by the present application.
[0036] Figure 3 is a schematic diagram of the distance kernel generation process with bias provided by the present application.
[0037] Figure 4 is a comparison diagram of the up-sampling results of other up-sampling operators provided by the present application. DETAILED DESCRIPTION
[0038] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.
[0039] In the present application, the terms "first", "second", etc. (if any) in the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.
[0040] Figure 1 A flowchart of a robust bilateral dense feature up-sampling method based on gradient prior provided for an embodiment of the present application, the method comprising operations S1 to S4.
[0041] Operation S1, extracting image features that need to be up-sampled from a dense prediction network to obtain an input feature map of the up-sampling operator.
[0042] According to an embodiment of the present application, the dense prediction network in S1 can be any existing convolutional neural network, including but not limited to image matting network, monocular depth estimation network, semantic segmentation network, etc. The feature map that needs to be up-sampled is the intermediate layer feature of the dense prediction network. In the case of default two times up-sampling, given an input feature map The output feature map after the up-sampling operator is
[0043] Operation S2, performing mean value processing on the input feature to obtain a mean value feature map of the input feature.
[0044] In this embodiment, as shown in "channel mean value" in Figure 2 for a given input feature map performing mean value operation on the channel dimension to obtain the corresponding single-channel mean value feature map where C, H, W are the channel number, feature height and feature width of the input feature map, respectively.
[0045] Operation S3, performing gradient mapping on the mean value feature map to obtain gradient prior features of the mean value feature map, and arranging the gradient prior of each corresponding point into an up-sampling gradient kernel.
[0046] In this embodiment, the gradient of the feature map is first calculated for the mean value feature map, and then the gradient prior information is mapped into a gradient kernel. As shown in Figure 2As shown in the section on "gradient mapping," the gradient of the mean feature map is mapped using the gradient mapping function g(x) to obtain the gradient prior, which is then rearranged into the form of a gradient kernel. Specifically, the gradient mapping function can be any differentiable smooth function that is monotonically decreasing in [0, +∞). Here, a specific function is chosen. As the gradient mapping function, ∈ is an infinitesimally small constant to prevent the denominator from being zero. Specifically, the gradient of each feature point in the mean feature map is the difference between that pixel value and its 3×3 neighborhood. Specifically, the gradient of points on the upsampled feature map... The point located at coordinates p = (i,j) before upsampling Located at coordinates Extract a point from X. If we consider a 3×3 local region Ω centered at point X, then point X... p out The gradient prior is mapped as Where q∈Ω. Subsequently, the distance prior mapping is rearranged into a 3×3 kernel form.
[0047] Operation S4 maps the relative distances between the original pixels and the upsampled pixels during the upsampling process to generate distance prior features, and organizes the distance prior features of each corresponding point into an upsampled distance kernel.
[0048] According to an embodiment of the present invention, operation S4 includes sub-operations S41 to S42.
[0049] In suboperation S41, such as Figure 2 As shown in the "Distance Mapping" section, the relative distance prior between points before and after upsampling is directly mapped, and the mapping function is: ∈ is an infinitesimally small constant to prevent the denominator from being zero. Specifically, the points on the upsampled feature map The distance prior mapping is Subsequently, the distance prior mapping is rearranged into a 3×3 kernel form.
[0050] In suboperation S42, such as Figure 3 As shown, when producing distance priors, a distance bias value can be added to increase the flexibility of the distance prior mapping. The distance prior mapping with bias is as follows: Here, Δ = f(X), where f is an upsampling layer of an neural network connected to a 5×5 two-dimensional convolutional layer. Subsequently, the biased distance prior mapping is rearranged into a 3×3 kernel form.
[0051] Operation S5 combines the distance kernel from S3 and the gradient kernel from S4 to generate an upsampling kernel.
[0052] In the embodiment, the gradient kernel obtained in S3 and S4 is multiplied with the distance prior kernel corresponding to the multiplication, and a normalization operation is performed to obtain the up-sampling kernel. In particular, the normalization operation here selects Softmax normalization. Specifically, a point on the up-sampled feature map corresponds to a point on the feature map before up-sampling located at coordinate A 3*3 local region Ω centered at point is extracted in X, the form of the up-sampling kernel at point
[0053] is:
[0054] When considering distance bias, the form is:
[0055]
[0056] where q∈Ω.
[0057] In operation S6, a convolution operation is performed between the input feature and the up-sampling kernel in S5 to obtain an up-sampled feature map corresponding to the input feature map.
[0058] In the embodiment, the input feature X is convolved with the up-sampling kernel obtained in S5 to obtain the final output X out . Specifically, the convolution process can be represented as:
[0059]
[0060] According to another aspect of the present application, a training method of a dense prediction network is provided, comprising:
[0061] S1: replacing the original up-sampling operator or up-sampling operation sub-network in the dense prediction network with the above up-sampling method to obtain a new dense prediction network;
[0062] S2: retraining the dense prediction network under the original condition to obtain a target dense prediction network.
[0063] According to another aspect of the present application, a dense prediction method is provided, comprising:
[0064] S1: obtaining a target dense prediction network by using the training method of the dense prediction network;
[0065] S2: processing the current image by using the target dense prediction network to obtain a target dense prediction result.
[0066] As Figure 4As shown, the result of the dense prediction network output using the above-mentioned characteristic up-sampling operator can achieve better visualization effect than other existing methods.
[0067] According to another aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method.
[0068] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A robust bilateral dense feature up-sampling method based on gradient prior, characterized in that, The method comprises the following steps: S1: extracting image features needing up-sampling from a dense prediction network to obtain an input feature map; S2: performing mean value processing on the input feature map to obtain a mean value feature map; S3-1: Perform gradient mapping on the mean feature map to obtain the gradient prior features of the mean feature map, and organize the gradient prior of each corresponding point into an upsampling gradient kernel; including: using the gradient mapping function Gradient mapping is performed on the gradient of the mean feature map to obtain the gradient prior, which is then arranged into the form of a gradient kernel; where, A constant that is infinitesimally small; points on the upsampled feature map Located at coordinates The corresponding point before upsampling Located at coordinates ,exist Extract a point local area centered Then point The gradient prior is mapped as ,in ; S3-2: performing distance mapping on the relative distance between original pixels and up-sampled pixels in the up-sampling process to generate distance prior features, and arranging the distance prior features of each corresponding point into an up-sampling distance kernel; the method comprises the following steps: The relative distance prior of the points before and after upsampling is mapped, and the mapping function is wherein, is an infinitesimal constant; the distance prior of the points on the feature map after upsampling is mapped as ; the distance prior considering the distance bias is mapped as wherein , is an NN upsampling layer connected with a two-dimensional convolution layer; S4: combining the up-sampling distance kernel and the up-sampling gradient kernel to generate an up-sampling kernel; S5: performing convolution operation between the input feature map and the up-sampling kernel to obtain a feature map after up-sampling of the corresponding input feature map.
2. The robust bilateral dense feature up-sampling method based on gradient prior of claim 1, wherein, The dense prediction network is any convolutional neural network, and the image features needing up-sampling are intermediate layer features of the dense prediction network.
3. The gradient prior based robust bilateral dense feature up-sampling method of claim 1, wherein, The step S2 comprises: for a given input feature map , a channel dimension mean operation is performed thereon to obtain a corresponding single-channel mean feature map .
4. The gradient prior based robust bilateral dense feature up-sampling method of claim 1, wherein, The step S4 includes: multiplying the up-sampling gradient kernel with the up-sampling distance kernel corresponding to each other, and performing a normalization operation to obtain an up-sampling kernel; The form of the up-sampling kernel at the step S4 is: The form when the distance bias is considered is: wherein .
5. The gradient prior based robust bilateral dense feature up-sampling method of claim 1, wherein, The step S5 includes: performing a convolution operation on the input features and the up-sampling kernel, thereby obtaining the final output : wherein .
6. A method of training a dense prediction network, the method comprising: The method comprises the following steps: S1: using the up-sampling method in any one of claims 1 to 5 to obtain a new dense prediction network; S2: retraining the dense prediction network under the original condition to obtain a target dense prediction network.
7. A method of dense prediction, characterized by, The method comprises the following steps: S1: using the training method of the dense prediction network in claim 6 to obtain a target dense prediction network; S2: processing a current image by using the target dense prediction network to obtain a target dense prediction result.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to realize the steps of the method in any one of claims 1 to 5 or claim 6 or claim 7.