Feature map upsampling method and application
By mapping feature maps to location bias maps and interpolating them, combined with linear networks and dynamic weight adjustment, the high computational complexity of existing feature map upsampling methods is solved, achieving low-complexity and efficient feature map upsampling, which is suitable for a variety of dense prediction tasks.
Patent Information
- Application Number
- CN202311088025.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-28
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2043-08-28
AI Technical Summary
Existing feature map upsampling methods are computationally complex and slow, and the dynamic upsampling operator structure is complex, which limits their application scenarios. In particular, the FADE and SAPA algorithms have a large amount of computation under high resolution guidance, resulting in low efficiency.
By mapping the input feature map to a location bias map and interpolating based on the sampled location map, combined with a linear network and dynamic weight matrix adjustment, feature map upsampling is achieved, bypassing the kernel-based upsampling paradigm and using bilinear interpolation to reduce computational cost.
It achieves low computational complexity and fast feature map upsampling, improving computational efficiency and expanding the applicability of application scenarios, making it suitable for various dense prediction networks.
Smart Images
Figure CN117152455B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of image processing, and more particularly relates to a feature map upsampling method and application. BACKGROUND
[0002] Image dense prediction task is a task of making classification prediction for each pixel of a picture, and plays a very important role in many visual tasks such as salient object detection, semantic segmentation, depth estimation, edge detection and key point detection. Feature map upsampling is an indispensable part of image dense prediction task, which gradually restores the resolution of the feature map.
[0003] Common feature map upsampling methods include nearest neighbor interpolation and bilinear interpolation, which follow fixed rules to calculate the upsampling values. In order to increase flexibility, some tasks introduce learnable upsampling operators, such as deconvolution in instance segmentation and depth-to-space in image super-resolution. However, deconvolution can cause checkerboard effect, and depth-to-space is not suitable for high-level visual tasks.
[0004] With the popularity of dynamic networks, some dynamic upsampling methods have shown great potential in some dense prediction tasks, such as CARAFE algorithm, FADE algorithm and SAPA algorithm. Among them, the CARAFE algorithm first generates a content-dependent upsampling kernel, and then uses dynamic convolution for upsampling. The FADE algorithm and the SAPA algorithm propose to combine high-resolution and low-resolution features to generate an upsampling kernel, so that the upsampling process can be guided by high-resolution structures. However, the structure of these dynamic upsampling operators is generally complex, requires the installation of a special CUDA code package, and has considerable time consumption compared to bilinear interpolation. Especially for the FADE algorithm and the SAPA algorithm, the high-resolution guidance will introduce more computational complexity and limit their application scenarios. SUMMARY
[0005] In view of the above defects or improvement needs of the prior art, the present application provides a feature map upsampling method and application, which aims to provide a feature map upsampling method with low computational complexity and fast calculation speed on the basis of ensuring upsampling performance.
[0006] In order to achieve the above purpose, in a first aspect, the present application provides a feature map upsampling method, comprising:
[0007] S1, mapping the input feature map to a position bias map; wherein the size of the position bias map is sH*sW*2, and the element (Δx i ,Δy j) represents a position bias at a position of the input feature map ([i / s], [j / s]); [·] is a rounding symbol; i=1, 2, …, sH; j=1, 2, …, sW; s is a scaling ratio; H and W are respectively a length and a width of the input feature map;
[0008] S2, adding an element (Δx i ,Δy j ) in the i-th row and the j-th column of the position bias map and a position ([i / s], [j / s]) to obtain a sampling position map of the input feature map; i ,Δy j ) in the i-th row and the j-th column of the position bias map and a position ([i / s], [j / s]) to obtain a sampling position map of the input feature map;
[0009] S3, performing interpolation on the input feature map based on the sampling position map to obtain a high-resolution feature map, thereby realizing the up-sampling operation on the input feature map.
[0010] Further preferably, the input feature map is bilinearly interpolated based on the sampling position map.
[0011] Further preferably, the input feature map is mapped into the position bias map by using a linear network.
[0012] Further preferably, after obtaining the position bias map, the position bias map is further adjusted by way one or way two;
[0013] In way one, the position bias map is adjusted to be a product of the position bias map and a preset weight value α; wherein the preset weight value α is greater than 0.2 and less than 0.3;
[0014] In way two, the position bias map is adjusted to be a Hadamard product of the position bias map and a dynamic weight matrix M; wherein, the preset weight value β is greater than 0.4 and less than 0.6; f(·) is a mapping function; is the input feature map.
[0015] Further preferably, the preset weight value α is 0.25; and the preset weight value β is 0.5.
[0016] In a second aspect, the present application provides a feature map up-sampling method, comprising:
[0017] grouping the input feature map in a channel dimension to obtain a plurality of feature sub-maps;
[0018] mapping each feature sub-map into a corresponding position bias map; the size of the position bias map is sH*sW*2, and an element (Δx i ,Δy jrepresents a position bias at a position corresponding to a feature subgraph ([i / s], [j / s]); [·] is a rounding symbol; i = 1, 2, …, sH; j = 1, 2, …, sW; s is a scaling ratio; H and W are respectively a length and a width of the input feature map;
[0019] An element (Δx i ,Δy j ) in the i-th row and the j-th column of each position bias map is added to the position ([i / s], [j / s]) to obtain a sampling position map of the corresponding feature subgraph;
[0020] Based on the sampling position map, the corresponding feature subgraph is interpolated to obtain a high-resolution feature subgraph.
[0021] The high-resolution feature subgraphs are spliced in the channel dimension to obtain a high-resolution feature map, thereby realizing the up-sampling operation on the input feature map.
[0022] Further preferably, based on the sampling position map, the corresponding feature subgraph is bilinearly interpolated to obtain a high-resolution feature subgraph.
[0023] Further preferably, a linear network is used to respectively map each feature subgraph to a corresponding position bias map.
[0024] Further preferably, after obtaining the position bias map, the position bias map is further adjusted by way one or way two.
[0025] In way one, the position bias map is adjusted to a product of the position bias map and a preset weight value α; wherein the preset weight value α is greater than 0.2 and less than 0.3.
[0026] In way two, the position bias map is adjusted to a Hadamard product of the position bias map and a dynamic weight matrix M; wherein M = βsigmoid(f(Z)); the preset weight value β is greater than 0.4 and less than 0.6; f(·) is a mapping function; Z is a feature subgraph corresponding to the position bias map.
[0027] Further preferably, the preset weight value α is 0.25; and the preset weight value β is 0.5.
[0028] In a third aspect, the present application provides a feature map up-sampling system, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the feature map up-sampling method provided in the first aspect or the second aspect of the present application.
[0029] In a fourth aspect, the present application provides an image processing method, comprising: inputting an image to be processed into a dense prediction network to make the image perform any one of semantic segmentation, object detection, instance segmentation, panorama segmentation and depth estimation;
[0030] The dense prediction network comprises an up-sampling layer configured to perform up-sampling on the input feature map by using the feature map up-sampling method according to the first aspect or the second aspect.
[0031] In a fifth aspect, the present application provides a computer readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls a device in which the storage medium is located to perform one or more methods according to the first aspect, the second aspect or the fourth aspect.
[0032] In general, the above technical solutions of the present application can achieve the following beneficial effects:
[0033] 1. The feature map up-sampling method according to the first aspect maps the input feature map to a position bias map, each element of the position bias map representing a position bias at a corresponding position of the input feature map, thereby adjusting each position in the input feature map to obtain a sampling position map, and then performing interpolation on the input feature map based on the sampling position map to realize up-sampling of the input feature map. The present application bypasses the kernel-based up-sampling paradigm and returns to the fundamental point of sampling, and bypasses the previous dynamic convolution-based method, thereby realizing up-sampling with lower complexity and faster calculation speed while ensuring up-sampling performance.
[0034] 2. The feature map up-sampling method according to the second aspect further improves the calculation efficiency by grouping the input feature map according to the channel, performing up-sampling on each group respectively, and then recombining the results of each group along the channel dimension.
[0035] 3. Further, the feature map up-sampling method according to the first aspect and the second aspect maps the input feature map or the feature sub-map to the position bias map using a linear network, thereby further reducing the calculation amount.
[0036] 4. Further, the feature map up-sampling method according to the first aspect and the second aspect performs bilinear interpolation on the input feature map or the feature sub-map based on the sampling position map, thereby further reducing the calculation amount.
[0037] 5、Further, considering that the sampling points may deviate from the correct range after the position bias adjustment, the feature map upsampling method provided by the first aspect and the second aspect of the present application adjusts the position bias map to the product of the position bias map and a fixed preset weight value a of the range between 0.2 and 0.3 after obtaining the position bias map, so as to adjust the position to the correct range, thereby further improving the performance of the feature map upsampling.
[0038] 6、Further, considering that the sampling points may deviate from the correct range after the position bias adjustment, the feature map upsampling method provided by the first aspect and the second aspect of the present application adjusts the position bias map to the product of the position bias map and a fixed preset weight value a of the range between 0.2 and 0.3 after obtaining the position bias map, so as to adjust the position to the correct range, thereby further improving the performance of the feature map upsampling. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The feature map upsampling method flowchart provided for the first embodiment of the present application;
[0040] Figure 2 The entire upsampling process schematic diagram provided for the first embodiment of the present application;
[0041] Figure 3 The upsampling process schematic diagram under the fixed sampling range provided for the first embodiment of the present application;
[0042] Figure 4 The upsampling process schematic diagram under the dynamic sampling range provided for the first embodiment of the present application;
[0043] Figure 5 The result diagram for comparing the semantic segmentation performance of the improved model after the same semantic segmentation model is improved by using the feature map upsampling method provided by the present application, and the existing CARAFE algorithm, FADE algorithm, SAPA-B algorithm, respectively, provided for the first embodiment of the present application;
[0044] Figure 6 The comparative result schematic diagram of the dense prediction network constructed based on the CARAFE algorithm, IndexNet algorithm, A2U algorithm, FADE algorithm, SAPA algorithm and the upsampling method provided by the present application in different image dense prediction tasks provided for the fourth embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.
[0046] Embodiment 1
[0047] A feature map upsampling method, the flow chart of which is shown in Figure 1 , and the entire upsampling process is shown in Figure 2 . Specifically, the above-mentioned feature map upsampling method comprises the following steps:
[0048] S1, mapping the input feature map into a position bias map , wherein the size of the position bias map is sH*sW*2, and the element in the i-th row and the j-th column of the position bias map represents the position bias at the position ([i / s], [j / s]) of the input feature map; [·] is the rounding symbol; i=1, 2, …, sH; j=1, 2, …, sW; s is the scaling ratio, which is 2 in the present embodiment; H and W are the length and width of the input feature map, respectively;
[0049] Specifically, the input feature map can be mapped into the position bias map by using a linear network or a convolution layer; in order to further reduce the calculation amount, the input feature map is preferably mapped into the position bias map by using a linear network, i.e.
[0050] Further, after obtaining the position bias map, the position bias map is also adjusted by way one or way two;
[0051] In way one, as shown in Figure 3 , the position bias map is adjusted to the product of the position bias map and a preset weight value α, i.e. , wherein the preset weight value α is greater than 0.2 and less than 0.3, and the preferred value is 0.25.
[0052] In way two, as shown in Figure 4 , the position bias map is adjusted to the Hadamard product of the position bias map and a dynamic weight matrix M, i.e. , wherein The preset weight value β is greater than 0.4 and less than 0.6, and the preferred value is 0.5; f(·) is a mapping function, which can be a convolution function, a linear mapping function, etc. is the input feature map.
[0053] S2, add the element (Δx i , Δy j ) in the i-th row and j-th column of the position offset map to the position ([i / s], [j / s]) to obtain a sampling position map of the input feature map
[0054] Specifically, add the position offset map to the original grid coordinate map to obtain a sampling position map That is
[0055] S3, interpolate the input feature map based on the sampling position map to obtain a high-resolution feature map, thereby realizing the up-sampling operation on the input feature map.
[0056] Specifically, the input feature map can be bilinearly interpolated or bicubically interpolated based on the sampling position map. In order to further reduce the amount of calculation, it is preferred to bilinearly interpolate the input feature map, and the obtained high-resolution feature map The initial sampling position is selected to be the same as the bilinear interpolation.
[0057] The present application assumes that the input feature map is a continuous feature map after bilinear interpolation, and then generates content-related sampling positions, and uses these positions to resample the input feature map to obtain an up-sampled feature map. In implementation, the present application maps the input feature map into a sampling position map, and then uses a grid sampling function to resample based on these positions. The present application bypasses the kernel-based up-sampling paradigm and returns to the essence of up-sampling - point sampling. It bypasses the previous dynamic convolution-based method and achieves the purpose of low computational complexity and low latency. At the same time, the present application does not need to guide the feature map, and is more widely used, has a wider practical application space, so that the operator can be widely used in the up-sampling link of various dense prediction networks.
[0058] In order to further illustrate the performance of the feature map up-sampling method provided by the embodiment 1 of the present application, the following will be described in detail in combination with specific experiments:
[0059] The present application tests the inference delay input using a random feature map with a size of 256*120*120 (if necessary, a guided feature map with a size of 256*240*240 is also provided). The present application makes the upsampling layer of a semantic segmentation deep neural network model SegFormer-B1 use the feature map upsampling method (DySample) provided by the present application and the existing CARAFE algorithm, FADE algorithm and SAPA-B algorithm for upsampling respectively, and then compares the semantic segmentation performance based on each upsampling method, including accuracy (mIoU), required test time (latency), training memory occupation (memory), training time (training time), number of floating point operations (GFLOPs) and parameter quantity (parameters), to obtain the quantitative results as shown in Figure 5 From the figure, it can be seen that the feature map upsampling method provided by the present application achieves the best performance, the highest efficiency and the lowest complexity. For inference time, the upsampling method provided by the present application can complete the upsampling of the 256*120*120 feature map in 6.2ms, close to bilinear interpolation (1.6ms), and much lower than the time required by other methods. In addition, due to the use of highly optimized PyTorch built-in functions, the back propagation of DySample is quite fast, and the increased training time can be ignored.
[0060] Embodiment 2
[0061] A feature map upsampling method, comprising:
[0062] A1, grouping the input feature map in the channel dimension to obtain a plurality of feature subgraphs;
[0063] A2, respectively mapping each feature subgraph into a corresponding position bias graph; the size of the position bias graph is sH*sW*2, and the element (Δx i ,Δy j ) of the i-th row and the j-th column of the position bias graph represents the position bias at the position of the corresponding feature subgraph ([i / s], [j / s]); [·] is the rounding symbol; i=1, 2, …, sH; j=1, 2, …, sW; s is the scaling ratio; H and W are the length and width of the input feature map, respectively;
[0064] Specifically, a linear network or a convolution layer can be used to map each feature subgraph into a corresponding position bias graph. To further reduce the calculation amount, a linear network is preferably used.
[0065] After obtaining the position bias graph, the position bias graph is adjusted by method one or method two;
[0066] In the first mode, the position bias map is adjusted as a product of the position bias map and a preset weight value α; wherein the preset weight value α is greater than 0.2 and less than 0.3, and preferably takes a value of 0.25.
[0067] In the second mode, the position bias map is adjusted as a Hadamard product of the position bias map and a dynamic weight matrix M; wherein M = βsigmoid(f(Z)); a preset weight value β is greater than 0.4 and less than 0.6, and preferably takes a value of 0.5; f(·) is a mapping function, which can be a convolution function, a linear mapping function, etc.; and Z is a feature subgraph corresponding to the position bias map.
[0068] A3, adding an element (Δx i ,Δy j ) in the i-th row and the j-th column of each position bias map and a position ([i / s], [j / s]) to obtain a sampling position map of the corresponding feature subgraph;
[0069] A4, based on the sampling position map, interpolating the corresponding feature subgraph to obtain a high-resolution feature subgraph;
[0070] Specifically, the corresponding feature subgraph can be bilinearly interpolated or bicubicly interpolated based on the sampling position map. To further reduce the calculation amount, the bilinear interpolation method is preferably adopted.
[0071] A5, splicing the high-resolution feature subgraphs in the channel dimension to obtain a high-resolution feature map, thereby realizing the up-sampling operation on the input feature map.
[0072] The method provided in Embodiment 2 further improves the calculation efficiency by grouping the input feature map according to the channel and then respectively performing the up-sampling operation, and finally re-pasting the results of each group along the channel dimension on the basis of Embodiment 1.
[0073] The related technical solution is the same as that in Embodiment 1, which will not be repeated here.
[0074] Embodiment 3
[0075] A feature map up-sampling system, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the feature map up-sampling method provided in Embodiment 1 or Embodiment 2 of the present application.
[0076] The related technical solution is the same as that in Embodiment 1 or Embodiment 2, which will not be repeated here.
[0077] Embodiment 4
[0078] An image processing method, comprising: inputting an image to be processed into a dense prediction network to make the dense prediction network perform any one of semantic segmentation, object detection, instance segmentation, panoptic segmentation and depth estimation on the image;
[0079] The dense prediction network comprises an up-sampling layer configured to perform up-sampling on the feature map in the dense prediction network by using the feature map up-sampling method provided in Embodiment 1 or Embodiment 2.
[0080] For the semantic segmentation task, after the dense prediction network is trained by using a semantic segmentation dataset (such as a PASCAL VOC dataset), an image to be subjected to semantic segmentation is input into the dense prediction network to predict the label of each pixel point in the image, thereby obtaining a semantic segmentation result.
[0081] For the object detection task, after the dense prediction network is trained by using an object detection dataset (such as an ImageNet dataset), an image to be subjected to object detection is input into the dense prediction network, thereby obtaining a bounding box of each target in the image.
[0082] For the instance segmentation task, after the dense prediction network is trained by using an instance segmentation dataset (such as a COCO dataset), an image to be subjected to instance segmentation is input into the dense prediction network, thereby obtaining the contour of each target in the image.
[0083] For the panoptic segmentation task, after the dense prediction network is trained by using a panoptic segmentation dataset (such as a KITTI dataset), an image to be subjected to panoptic segmentation is input into the dense prediction network, thereby predicting the semantic label and instance of each pixel point in the image, and obtaining a panoptic segmentation result.
[0084] For the depth estimation task, after the dense prediction network is trained by using a depth estimation dataset (such as a KITTI dataset), an image to be subjected to depth estimation is input into the dense prediction network, thereby obtaining the depth information of the image.
[0085] Specifically, as shown in Figure 6 The results of the dense prediction network constructed based on the CARAFE algorithm, the IndexNet algorithm, the A2U algorithm, the FADE algorithm, the SAPA algorithm and the up-sampling method provided in the present application in different image dense prediction tasks are shown in the table, and the results of the semantic segmentation, the object detection, the instance segmentation, the panoptic segmentation and the depth estimation tasks are shown from the first column to the last column. As can be seen from the table, the performance of the dense prediction network constructed based on the up-sampling method provided in the present application is better in each task.
[0086] The related technical solutions are the same as those in Embodiment 1 or Embodiment 2, and thus are not described herein.
[0087] Embodiment 5
[0088] A computer readable storage medium comprising a stored computer program which, when executed by a processor, controls a device in which the storage medium is present to perform one or more of the methods provided by embodiments 1, 2, 4.
[0089] The related technical solutions are the same as those of embodiments 1-4, which are not described herein.
[0090] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method of feature map upsampling, the method comprising: The method comprises the following steps: S1, map the input feature map to a position bias map; wherein the position bias map has a size of s H* s W*2, an element in the i-th row and the j-th column of the position bias map represents a position bias at a position of the input feature map i j is a rounding symbol; ; ; s is a scaling ratio; H and W are a length and a width of the input feature map, respectively. After obtaining the position bias map, the position bias map is adjusted by the first method or the second method; In the first mode, the position bias map is adjusted as a product of the position bias map and a preset weight value ; wherein the preset weight value is greater than 0.2 and less than 0.
3. In the second manner, the position bias map is adjusted as a Hadamard product of the position bias map and a dynamic weight matrix ; wherein, ; the preset weight value is greater than 0.4 and less than 0.6; is a mapping function; is an input feature map; S2, the position offset diagram of the first i Line number j Column elements With position The samples are added together to obtain the sampling location map of the input feature map; S3, based on the sampling position map, the input feature map is interpolated to obtain a high-resolution feature map, thereby realizing the up-sampling operation of the input feature map.
2. The feature map upsampling method of claim 1, wherein, Based on the sampling position map, the input feature map is bilinearly interpolated; the input feature map is mapped into a position bias map by using a linear network.
3. The feature map upsampling method of claim 1 or 2, wherein, The preset weight value is 0.25; the preset weight value is 0.
5.
4. A feature map upsampling method, characterized by, The method comprises the following steps: Group the input feature map in the channel dimension to obtain a plurality of feature sub-maps; Each feature sub-map is mapped into a corresponding position bias map; The size of the position bias map is s H* s W*2, the element in the i-th row and the j-th column of which represents the position bias at the corresponding feature sub-map i ; j ; ; ; is a rounding symbol; ; ; s is a scaling ratio; H and W are the length and the width of the input feature map respectively; after obtaining the position bias map, the position bias map is adjusted by way one or way two; in the way one, the position bias map is adjusted as the product of the position bias map and a preset weight value ; wherein, the preset weight value is greater than 0.2 and less than 0.3; in the way two, the position bias map is adjusted as the Hadamard product of the position bias map and a dynamic weight matrix ; wherein, ; the preset weight value is greater than 0.4 and less than 0.6; is a mapping function; is a feature sub-map corresponding to the position bias map; adding the element in the i-th row and j-th column of each position bias map to the position i of the i-th row and j-th column of the feature sub-map j of the i-th row and j-th column of the feature sub-map of the i-th row and j-th column of the feature sub-map of the i-th row and j-th column of the feature sub-map Based on the sampling position map, the corresponding feature sub-map is interpolated to obtain a high-resolution feature sub-map; Each high-resolution feature sub-map is spliced in the channel dimension to obtain a high-resolution feature map, thereby realizing the up-sampling operation of the input feature map.
5. The feature map upsampling method of claim 4, wherein, Based on the sampling position map, the corresponding feature sub-map is bilinearly interpolated to obtain a high-resolution feature sub-map; each feature sub-map is mapped into a corresponding position bias map by using a linear network.
6. The feature map upsampling method of claim 4 or 5, wherein, The preset weight value is 0.25; the preset weight value is 0.
5.
7. A feature map upsampling system, comprising: The method comprises the following steps: A memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the feature map up-sampling method of any one of claims 1-3, or the feature map up-sampling method of any one of claims 4-6.
8. An image processing method characterized by, The method comprises the following steps: Input the image to be processed into the dense prediction network to make it perform any one of the following operations on the image: semantic segmentation, object detection, instance segmentation, panorama segmentation and depth estimation; The dense prediction network comprises an up-sampling layer, which is used to perform the up-sampling operation on the feature map input therein by using the feature map up-sampling method of any one of claims 1-3, or the feature map up-sampling method of any one of claims 4-6.
Citation Information
Patent Citations
Apparatus and method for compressing dynamic range ofimage
KR1020060031519A
Center-biased machine learning techniques to determine saliency in digital images
US20210012201A1