Feature map upsampling method, small target semantic segmentation method and imaging method
By using feature map upsampling method in the semantic segmentation model, the problem of poor training effect on small target images is solved, and higher semantic segmentation accuracy and reliability are achieved.
Patent Information
- Application Number
- CN202210288172.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2025-04-15
- Estimated Expiration
- 2042-03-22
AI Technical Summary
The existing semantic segmentation model has poor training results on small target images, mainly due to data domain differences and feature map down sampling, which are difficult to identify small targets in feature maps.
The feature map upsampling method is adopted, and the upsampling operator prediction module and the scaling factor prediction module are constructed, the upsampling kernel and scaling factor are calculated, and feature reorganization is performed to complete the upsampling of the feature map.
The accuracy and reliability of semantic segmentation of small targets is improved, allowing the model to more effectively identify and segment small targets.
Smart Images
Figure CN114677510B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of semantic segmentation, and in particular relates to a feature map upsampling method, a small target semantic segmentation method and an imaging method. Background Art
[0002] With the development of economy and technology and the improvement of people's living standards, deep learning technology has been widely used in people's production and life, bringing endless convenience to people's production and life. Therefore, improving the accuracy and reliability of deep learning technology has become the focus of people's research.
[0003] At present, although the semantic segmentation technology of small targets has been widely used, the existing semantic segmentation methods of small targets still have some problems. Most of the existing semantic segmentation models are designed for natural image datasets. When migrating to the field of small target semantic segmentation, the training effect of the semantic segmentation model on small target images will be greatly affected because there are large differences in the data domain. The size of an original image for small target semantic segmentation is relatively large. Due to the limited memory of the graphics card, it is impossible to send a complete original image to the neural network, so the input image size must be reduced and then input into the model, which makes the scale of the small target smaller, making it difficult for the model to recognize. The general semantic segmentation model regards each input pixel as a sample, outputs a multi-classification result for each pixel, and then calculates the cross entropy loss with the annotation on the corresponding label. However, this pixel-level loss treats each pixel equally, regardless of whether it is a small target to be identified; since the number of background pixels is much larger than the number of small target pixels, such a loss will cause the loss function of the model to be unbalanced. More importantly, this loss gives the same penalty weight to each pixel. So for small targets, when the number of labeled pixels of a small target is quite small, the feedback it gives to the model is insufficient, which makes it difficult for the model to recognize small targets.
[0004] Generally speaking, semantic segmentation models will downsample feature maps. For example, the backbone network used by FCN is a classification network with good performance at the time, such as AlexNet, VGG, and GoogLeNet; these backbone networks will downsample the feature map to 1 / 4 of the original image in the first block, and each of the following three blocks will have a downsampling operation, and finally downsample the feature map to 1 / 32 of the original image. Some of the latest networks will downsample the feature map to 1 / 16 of the original image; taking DeepLabV3+ as an example, the backbone network used by DeepLabV3+ is ResNet, and after downsampling by 1 / 4 in the first stage, it only downsamples in the next two blocks, thereby downsampling the original image to 1 / 16. The purpose of designing the semantic segmentation network in this way is to take into account the computational complexity of the model and ensure the performance of the model. However, when such a network is migrated to the field of small target semantic segmentation, downsampling by 1 / 16 will make the originally small small targets almost invisible in the feature map, which makes these models perform poorly in the field of small target semantic segmentation. Summary of the invention
[0005] One of the purposes of the present invention is to provide a feature map upsampling method which is suitable for semantic segmentation of small objects and has high reliability and good practicality.
[0006] A second objective of the present invention is to provide a small object semantic segmentation method including the feature map upsampling method.
[0007] A third object of the present invention is to provide an imaging method including the small target semantic segmentation.
[0008] The feature map upsampling method provided by the present invention comprises the following steps:
[0009] The step of obtaining a feature map to be upsampled;
[0010] Steps for constructing the upsampling operator prediction module: by channel compression, encoding feature maps and pixel replay, the upsampling kernel corresponding to the feature map to be upsampled is calculated;
[0011] Steps for constructing a scaling factor prediction module: Calculate the scaling factor corresponding to the upsampling kernel through channel compression, pixel replay and scaling factor encoding;
[0012] According to the obtained upsampling kernel and scaling factor, the feature map to be upsampled is reorganized to complete the upsampling process of the feature map.
[0013] The channel compression in the step of constructing the upsampling operator prediction module and the channel compression in the step of constructing the scaling factor prediction module are the same channel compression operations that are shared.
[0014] The channel compression specifically includes first performing channel compression using a convolution layer with a convolution kernel of 1×1; and then sequentially connecting a batch normalization layer and a ReLU activation layer after the convolution layer to complete the final channel compression.
[0015] The steps of constructing the upsampling operator prediction module are specifically as follows: firstly, channel compression is performed; then, a feature map is encoded through a convolution layer; and finally, an upsampling kernel corresponding to the feature map to be upsampled is obtained through a pixel replay operation.
[0016] The step of constructing the upsampling operator prediction module specifically includes the following steps:
[0017] First, perform channel compression;
[0018] Then, the output channel is α 2 ×k 2 , the convolution layer with a convolution kernel of 3×3 is used for processing, so that the size of the output feature map is α 2 ×k 2 ×H×W; α is the upsampling multiple; k is the size of the upsampling kernel; H is the height of the original feature map; W is the width of the original feature map;
[0019] Finally, a pixel replay operation is performed to convert a channel number α 2 ×k 2 The pixels become α×α channels with the number of k 2 pixels, and perform the same pixel replay operation on each pixel, and finally get k 2 ×αH×αW upsampling operator feature map, each pixel on which is an upsampling kernel W l .
[0020] The steps of constructing the scaling factor prediction module are as follows: first, channel compression is performed; then, a pixel replay operation is performed so that the length of the feature map is equal to the length of the upsampling operator feature map, and the width of the feature map is equal to the width of the upsampling operator feature map; finally, a scaling factor encoder composed of a convolutional layer, a batch normalization layer, an activation layer, a parallel pooling layer, a feature map stack and a convolutional layer is used for calculation to obtain the final scaling factor.
[0021] The step of constructing the scaling factor prediction module specifically includes the following steps:
[0022] First, perform channel compression;
[0023] Then, a pixel replay operation is performed so that the length of the feature map is equal to the length of the upsampling operator feature map, and the width of the feature map is equal to the width of the upsampling operator feature map;
[0024] Finally, the scaling factor encoder is constructed: first, a 3×3 convolution layer with an output channel of 64 is used for feature extraction; then it is processed in sequence through a batch normalization layer and a ReLU activation layer; next, the parallel pooling layer is a parallel maximum pooling layer and an average pooling layer, and the output of the activation layer is connected to the input of the maximum pooling layer and the input of the average pooling layer at the same time. The output result of the maximum pooling layer is the maximum value of all channels above the corresponding position, and the output result of the average pooling layer is the average value of all channels above the corresponding position; then, the feature map with channel 1 output by the maximum pooling layer and the feature map with channel 1 output by the average pooling layer are stacked together to obtain a feature map with channel 2; finally, the obtained feature map is processed by a convolution layer with a convolution kernel size of 7×7, and a ReLU activation function is used to filter the negative results, and finally the scaling factor p corresponding to the upsampling kernel is obtained.
[0025] According to the obtained upsampling operator and scaling factor, the feature map to be upsampled is reorganized to complete the upsampling process of the feature map. Specifically, each upsampling kernel on the upsampling operator feature map is first scaled using the scaling factor to obtain the scaled upsampling operator feature map; then, for the sampling kernel at each position on the scaled upsampling feature map, the corresponding position on the original feature map is found, and weighted summation is performed on the area centered on the found position on the original feature map, thereby completing the upsampling process of the feature map.
[0026] The feature reorganization of the feature map to be upsampled is performed according to the obtained upsampling operator and scaling factor, thereby completing the upsampling process of the feature map, which specifically includes the following steps:
[0027] First, use the scaling factor to scale each upsampling kernel on the upsampling operator feature map; specifically, the following formula is used to calculate and obtain the scaled upsampling operator W l ':
[0028] W l '=softmax(p×W l )
[0029] Where p is the scaling factor; W l is the upsampling kernel at position l on the upsampling operator feature map; softmax() is the normalized exponential function;
[0030] Then, for the scaled upsampling operator W l 'Sampling kernel at each position (i, j) in the image, find the corresponding position on the original feature map Then to The weighted summation is performed on the k×k region centered to obtain the value of the upsampled feature map at (i, j); specifically, the following formula is used for calculation:
[0031]
[0032] Where X (i,j) is the pixel at (i, j) in the upsampled feature map; r is the range of the upsampling kernel, and k is the size of the upsampling kernel; W l'(s,t) is the upsampling kernel W l' The weight at the position (s, t); X (i+s,j+t) is the pixel of the original feature map located at (i+s,j+t);
[0033] Finally, the calculation result obtained is the upsampling result corresponding to the feature map to be upsampled.
[0034] The present invention also provides a small object semantic segmentation method including the above-mentioned feature map upsampling method, which specifically includes the following steps:
[0035] S1. Obtain the original small target image and perform pixel-level annotation to obtain a training data set;
[0036] S2. Use existing technology to build an initial semantic segmentation model;
[0037] S3. Replace the upsampling method in the initial semantic segmentation model constructed in step S2 with the above-mentioned feature map upsampling method, thereby obtaining an initial small target semantic segmentation model;
[0038] S4. Use the training data set to train the initial small object semantic segmentation model to obtain the final small object semantic segmentation model;
[0039] S5. Use the small target semantic segmentation model obtained in step S4 to perform semantic segmentation on the target image to complete the semantic segmentation of the small target in the target image.
[0040] The present invention also provides an imaging method including the above-mentioned small target semantic segmentation, comprising the following steps:
[0041] A. Obtain the original image for small object semantic segmentation;
[0042] B. Use the above-mentioned small object semantic segmentation method to perform small object semantic segmentation on the original image obtained in step A;
[0043] C. The small target semantic segmentation result obtained in step B is annotated and imaged on the original image obtained in step A to obtain a target image with the small target annotation result;
[0044] D. Output the target image obtained in step C to complete the imaging.
[0045] The feature map upsampling method, small target semantic segmentation method and imaging method provided by the present invention innovate the feature map upsampling process to address the existing problems of small target semantic segmentation, and creatively propose a feature map upsampling process with a scaling factor; therefore, the method of the present invention is extremely suitable for small target semantic segmentation, and has high reliability and good practicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 Schematic diagram of the method flow of the upsampling method of the present invention.
[0047] Figure 2 Schematic diagram of a model of the upsampling method of the present invention.
[0048] Figure 3 The figure is a flowchart of the method for semantic segmentation of small targets of the present invention.
[0049] Figure 4 The figure is a schematic diagram of the method flow of the imaging method of the present invention. DETAILED DESCRIPTION
[0050] like Figure 1 FIG. 4 is a schematic diagram of a method flow of an upsampling method of the present invention. Figure 2 The figure shows a schematic diagram of a model of the upsampling method of the present invention: the feature map upsampling method provided by the present invention comprises the following steps:
[0051] The step of obtaining a feature map to be upsampled;
[0052] The steps of constructing the upsampling operator prediction module are as follows: by channel compression, encoding feature maps and pixel replay, the upsampling kernel corresponding to the feature map to be upsampled is calculated; specifically, channel compression is first performed; then the feature map is encoded through a convolution layer; finally, the upsampling kernel corresponding to the feature map to be upsampled is obtained through pixel replay operation;
[0053] The specific implementation includes the following steps:
[0054] First, perform channel compression;
[0055] Then, the output channel is α 2 ×k 2 , the convolution layer with a convolution kernel of 3×3 is used for processing, so that the size of the output feature map is α 2 ×k 2×H×W; α is the multiple of upsampling; k is the size of the upsampling kernel; H is the height of the original feature map; W is the width of the original feature map; the size of the convolution kernel of the convolution layer will not affect the size of the output feature map, but will affect the amount of calculation. Generally speaking, if the size of the convolution kernel is larger, the receptive field of the convolution is wider and the effect is better, but more calculation will be introduced; therefore, the preferred value given by the present invention is 3×3;
[0056] Finally, a pixel replay operation is performed to convert a channel number α 2 ×k 2 The pixels become α×α channels with the number of k 2 pixels, and perform the same pixel replay operation on each pixel, and finally get k 2 ×αH×αW upsampling operator feature map, each pixel on it is an upsampling kernel W l ;The pixel replay operation is used to adjust the dimensions of the feature map;
[0057] Steps for constructing a scaling factor prediction module: Calculate the scaling factor corresponding to the upsampling kernel through channel compression, pixel replay and scaling shadow coding; Specifically, first perform channel compression; Then, perform pixel replay operation to make the length of the feature map equal to the length of the upsampling operator feature map, and the width of the feature map equal to the width of the upsampling operator feature map; Finally, calculate through a scaling factor encoder composed of a convolutional layer, a batch normalization layer, an activation layer, a parallel pooling layer, a feature map stack and a convolutional layer to obtain the final scaling factor;
[0058] The specific implementation includes the following steps:
[0059] First, perform channel compression;
[0060] Then, a pixel replay operation is performed to make the length of the feature map equal to the length of the upsampling operator feature map, and the width of the feature map equal to the width of the upsampling operator feature map; the pixel replay of this part is the same as the pixel replay in the upsampling operator prediction module;
[0061] Finally, the scaling factor encoder is constructed: first, a 3×3 convolution layer with 64 output channels is used for feature extraction. Considering that the feature map of the final predicted scaling factor has only 1 channel, there is no need for too many channels to reduce the amount of calculation, so the number of output channels of this convolution is set to 64; then it is processed in sequence through a batch normalization layer and a ReLU activation layer; next, the scaling factor of the feature map is calculated in a way similar to the spatial attention mechanism: the parallel pooling layer is a parallel maximum pooling layer and an average pooling layer, and the output of the activation layer is connected to the input of the maximum pooling layer at the same time. The input of the maximum pooling layer and the average pooling layer, the output result of the maximum pooling layer is the maximum value of all channels above the corresponding position, and the output result of the average pooling layer is the average value of all channels above the corresponding position; then, the feature map with channel 1 output by the maximum pooling layer and the feature map with channel 1 output by the average pooling layer are stacked (concat) together to obtain a feature map with channel 2; finally, the obtained feature map is processed by a convolution layer with a convolution kernel size of 7×7, and a ReLU activation function is used to filter negative results, and finally the scaling factor p corresponding to the upsampling kernel is obtained. Because the channels of the feature map are very small, even if a large-sized convolution kernel is used, there will not be a lot of calculations, but a large-sized convolution kernel will bring a large receptive field, so a 7×7 convolution kernel is used here to make the scaling factor better; since p is a scaling factor, its value cannot be negative, so a ReLU activation function is used to filter out negative numbers;
[0062] According to the obtained upsampling kernel and scaling factor, the feature map to be upsampled is reorganized to complete the upsampling process of the feature map; specifically, each upsampling kernel on the upsampling operator feature map is scaled using the scaling factor to obtain the scaled upsampling operator feature map; then, for the sampling kernel at each position on the scaled upsampling feature map, the corresponding position on the original feature map is found, and the area centered on the found position on the original feature map is weighted summed to complete the upsampling process of the feature map;
[0063] The specific implementation includes the following steps:
[0064] First, use the scaling factor to scale each upsampling kernel on the upsampling operator feature map; specifically, the following formula is used to calculate and obtain the scaled upsampling operator W l ':
[0065] W l '=softmax(p×W l )
[0066] Where p is the scaling factor; W lis the upsampling kernel at position l on the upsampling operator feature map; softmax() is the normalized exponential function;
[0067] Then, for the scaled upsampling operator W l 'Sampling kernel at each position (i, j) in the image, find the corresponding position on the original feature map Then to The weighted summation is performed on the k×k region centered to obtain the value of the upsampled feature map at (i, j); specifically, the following formula is used for calculation:
[0068]
[0069] Where X (i,j) is the pixel at (i, j) in the upsampled feature map; r is the range of the upsampling kernel, and k is the size of the upsampling kernel; W l'(s,t) is the upsampling kernel W l' The weight at the position (s, t); X (i+s,j+t) is the pixel of the original feature map located at (i+s,j+t);
[0070] Finally, the calculation result obtained is the upsampling result corresponding to the feature map to be upsampled.
[0071] In the specific implementation, the channel compression in the step of constructing the upsampling operator prediction module and the channel compression in the step of constructing the scaling factor prediction module share the same channel compression operation; the channel compression is specifically to first use a convolution layer with a convolution kernel of 1×1 to perform channel compression; after the convolution layer, a batch normalization layer and a ReLU activation layer are connected in sequence to complete the final channel compression.
[0072] During channel compression, because the number of channels of feature maps in different layers is different, for example, the number of channels in some layers is 64, the number of channels in some layers is 128, and the number of channels in some layers is 256, etc.; therefore, it is necessary to compress the channels so that the number of channels input to the module is a constant value for subsequent processing. Generally, 64 is selected as the number of channels after compression. Another benefit of channel compression is that it can reduce the amount of calculation: if a feature map with a channel number of 256 is subsequently processed, then the subsequent modules will operate on a very large feature map, which takes up a lot of memory and consumes computing power; by compressing the channels to a smaller 64 channels, it can reduce a lot of calculation.
[0073] like Figure 3 The method flow chart of the method for semantic segmentation of small objects of the present invention is shown as follows: The method for semantic segmentation of small objects provided by the present invention, which includes the above-mentioned feature map upsampling method, specifically includes the following steps:
[0074] S1. Obtain the original small target image and perform pixel-level annotation to obtain a training data set;
[0075] S2. Use existing technology to build an initial semantic segmentation model;
[0076] S3. Replace the upsampling method in the initial semantic segmentation model constructed in step S2 with the above-mentioned feature map upsampling method, thereby obtaining an initial semantic segmentation model for a small target;
[0077] S4. Use the training data set to train the small target initial semantic segmentation model to obtain the final small target semantic segmentation model;
[0078] S5. Use the small target semantic segmentation model obtained in step S4 to perform semantic segmentation on the target image to complete the semantic segmentation of the small target in the target image.
[0079] In specific implementation, the small target semantic segmentation method provided by the present invention can be used for the segmentation of microaneurysms, hemorrhage points, hard exudates and soft exudates in fundus images; correspondingly, the small target semantic segmentation method of fundus images includes the following steps:
[0080] S1. Obtain the original fundus image and perform pixel-level annotation (including the annotation of each small target in the fundus image, including microaneurysms, bleeding points, hard exudates and soft exudates, etc.), and perform data enhancement on the image (including flipping, rotation, color perturbation, etc.) to obtain a training data set;
[0081] S2. Use existing technology to build an initial semantic segmentation model, specifically build a HRNetV2 semantic segmentation model;
[0082] S3. Replace the upsampling method in the HRNetV2 semantic segmentation model constructed in step S2 with the feature map upsampling method provided by the present invention, thereby obtaining an initial model for small target segmentation of fundus images;
[0083] S4. Using the training data set to train the initial model for small object segmentation of fundus images, to obtain the final semantic segmentation model for small objects in fundus images;
[0084] S5. Use the fundus image small object semantic segmentation model obtained in step S4, input the target image into the model, obtain the labeling result of the small objects in the target image, and thus complete the semantic segmentation of the small objects in the target fundus image.
[0085] In addition, the small target semantic segmentation method provided by the present invention can also be used for the semantic segmentation of small targets (pedestrians, vehicles, houses, etc.) in aerial images; correspondingly, the small target semantic segmentation method for aerial images includes the following steps:
[0086] S1. Obtain the original aerial image and perform pixel-level annotation (including annotation of each small target in the aerial image, including pedestrians, vehicles, houses, etc.), and perform data enhancement on the image (including flipping, rotation, color perturbation, etc.) to obtain a training data set;
[0087] S2. Use existing technology to build an initial semantic segmentation model; specifically, build an HRNetV2 semantic segmentation model;
[0088] S3. Replace the upsampling method in the HRNetV2 semantic segmentation model constructed in step S2 with the feature map upsampling method provided by the present invention, thereby obtaining an initial model for small target segmentation in aerial images;
[0089] S4. Using the training data set to train the initial model for small object segmentation in aerial images, and obtain the final small object segmentation model for aerial images;
[0090] S5. Use the aerial image small target segmentation model obtained in step S4, input the target aerial image into the model, obtain the labeling results of the small targets in the target aerial image, and thus complete the semantic segmentation of the small targets in the target aerial image.
[0091] Finally, the small target semantic segmentation method provided by the present invention can also be used for the semantic segmentation of small targets (white safety helmets, red safety helmets, etc.) worn by safety helmets at construction sites; the corresponding small target semantic segmentation method of construction site images includes the following steps:
[0092] S1. Obtain the original construction site image and perform pixel-level annotation (including annotation of various colored helmets in the construction site image), and perform data enhancement on the image (including flipping, rotation, color perturbation, etc.) to obtain a training data set;
[0093] S2. Use existing technology to build an initial semantic segmentation model; specifically, build an HRNetV2 semantic segmentation model;
[0094] S3. Replace the upsampling method in the HRNetV2 semantic segmentation model constructed in step S2 with the feature map upsampling method provided by the present invention, thereby obtaining an initial model for small target segmentation in aerial images;
[0095] S4. Using the training data set to train the initial model for small target segmentation of the construction site image, to obtain the final small target segmentation model of the construction site image;
[0096] S5. Use the small target segmentation model of the construction site image obtained in step S4, input the target construction site image into the model, obtain the labeling result of the small target in the target construction site image, and thus complete the semantic segmentation of the safety helmet in the target construction site image.
[0097] Through the above small target semantic segmentation method, it is possible to perform long-distance image semantic segmentation of the situation of people wearing safety helmets on the construction site, thereby realizing the corresponding construction site safety helmet wearing detection.
[0098] like Figure 4 The figure shows a schematic flow chart of the imaging method of the present invention: the imaging method provided by the present invention including the above-mentioned small target semantic segmentation comprises the following steps:
[0099] A. Obtain the original image for small object semantic segmentation;
[0100] B. Use the above-mentioned small target semantic segmentation method to perform semantic segmentation on the original image obtained in step A;
[0101] C. The small target semantic segmentation result obtained in step B is annotated and imaged on the original image obtained in step A to obtain a target image with the small target annotation result;
[0102] D. Output the target image obtained in step C to complete the imaging.
[0103] The imaging method provided by the present invention can be directly applied to existing imaging devices. When applied, firstly, the corresponding original image is acquired by using the existing imaging device, and then the small target semantic segmentation method mentioned above is used to semantically segment the small target in the acquired original image; after the segmentation is completed, the segmentation result is annotated on the original image (including highlighting, rectangular frame annotation, etc.) and secondary imaging is performed, and finally an image with the small target annotation result is output. At this point, the imaging device using the present invention becomes an imaging device with small target semantic segmentation function.
[0104] In specific implementation, if the imaging method of the present invention is applied to an existing fundus image device, then after the existing fundus image device is integrated with the imaging method of the present invention, it can output a fundus image device with small target annotation results (including annotation results of microaneurysms, bleeding points, hard exudates, soft exudates, etc.); the function of the device will be better enriched;
[0105] Alternatively, when the imaging method of the present invention is applied to an aerial photography device, the existing aerial photography device can output an aerial photography device with small target annotation results (including annotation results of pedestrians, vehicles, houses, etc.) after integrating the imaging method of the present invention; the functions of the device will also be better enriched;
[0106] Alternatively, when the imaging method of the present invention is applied to construction site monitoring equipment, the existing monitoring equipment can output monitoring equipment with small target labeling results (including labeling results of white safety helmets, red safety helmets, etc.) after integrating the imaging method of the present invention; the functions of the equipment will also be better enriched.
Claims
1. A feature map upsampling method, comprising the following steps: The step of obtaining a feature map to be upsampled; Steps for constructing the upsampling operator prediction module: by channel compression, encoding feature maps and pixel replay, the upsampling kernel corresponding to the feature map to be upsampled is calculated; Steps to construct the scaling factor prediction module: Calculate the scaling factor corresponding to the upsampling kernel through channel compression, pixel replay and scaling shadow coding; According to the obtained upsampling operator and scaling factor, the feature map to be upsampled is reorganized to complete the upsampling process of the feature map; specifically, the steps include: First, use the scaling factor to scale each upsampling kernel on the upsampling operator feature map; specifically, the following formula is used to calculate and obtain the scaled upsampling operator W l ': W l '=softmax(p×W l ) Where p is the scaling factor; W l is the upsampling kernel at position l on the upsampling operator feature map; Softmax() is a normalized exponential function; Then, for the scaled upsampling operator W l 'Sampling kernel at each position (i, j) in the image, find the corresponding position on the original feature map Then to The weighted summation is performed on the k×k region centered to obtain the value of the upsampled feature map at (i, j); specifically, the following formula is used for calculation: Where X (i,j) is the pixel at (i, j) in the upsampled feature map; r is the range of the upsampling kernel, and k is the size of the upsampling kernel; W l'(s,t) is the upsampling kernel W l' The weight at the position (s, t); X (i+s,j+t) is the pixel of the original feature map located at (i+s,j+t); α is the upsampling multiple; k is the size of the upsampling kernel; Finally, the calculation result obtained is the upsampling result corresponding to the feature map to be upsampled.
2. The feature map upsampling method according to claim 1, characterized in that The channel compression in the step of constructing the upsampling operator prediction module and the channel compression in the step of constructing the scaling factor prediction module share the same channel compression operation; the channel compression specifically comprises firstly performing channel compression using a convolution layer with a convolution kernel of 1×1; and then sequentially connecting a batch normalization layer and a ReLU activation layer after the convolution layer to complete the final channel compression.
3. The feature map upsampling method according to claim 2, characterized in that The steps of constructing the upsampling operator prediction module are specifically as follows: firstly, channel compression is performed; then, a feature map is encoded through a convolution layer; and finally, an upsampling kernel corresponding to the feature map to be upsampled is obtained through a pixel replay operation.
4. The feature map upsampling method according to claim 3, characterized in that The step of constructing the upsampling operator prediction module specifically includes the following steps: First, perform channel compression; Then, the output channel is α 2 ×k 2 , the convolution layer with a convolution kernel of 3×3 is used for processing, so that the size of the output feature map is α 2 ×k 2 ×H×W; α is the upsampling multiple; k is the size of the upsampling kernel; H is the height of the original feature map; W is the width of the original feature map; Finally, a pixel replay operation is performed to convert a channel number α 2 ×k 2 The pixels become α×α channels with the number of k 2 pixels, and perform the same pixel replay operation on each pixel, and finally get k 2 ×αH×αW upsampling operator feature map.
5. The feature map upsampling method according to claim 3, characterized in that The steps of constructing the scaling factor prediction module are as follows: first, channel compression is performed; then, a pixel replay operation is performed so that the length of the feature map is equal to the length of the intermediate feature map, and the width of the feature map is equal to the width of the intermediate feature map; finally, a scaling factor encoder composed of a convolutional layer, a batch normalization layer, an activation layer, a parallel pooling layer, a feature map stack and a convolutional layer is used to calculate to obtain the final scaling factor.
6. The feature map upsampling method according to claim 5, characterized in that The step of constructing the scaling factor prediction module specifically includes the following steps: First, perform channel compression; Then, a pixel replay operation is performed to make the length of the feature map equal to the length of the intermediate feature map, and the width of the feature map equal to the width of the intermediate feature map; Finally, the scaling factor encoder is constructed: first, a 3×3 convolutional layer with 64 output channels is used for feature extraction; then it is processed in sequence through a batch normalization layer and a ReLU activation layer; next, the parallel pooling layer is a parallel maximum pooling layer and an average pooling layer, and the output of the activation layer is connected to the input of the maximum pooling layer and the input of the average pooling layer at the same time. The output result of the maximum pooling layer is the maximum value of all channels above the corresponding position, and the output result of the average pooling layer is the average value of all channels above the corresponding position; Then, the feature map with channel 1 output by the maximum pooling layer and the feature map with channel 1 output by the average pooling layer are stacked together to obtain a feature map with channel 2; finally, the obtained feature map is processed by a convolution layer with a convolution kernel size of 7×7, and a ReLU activation function is used to filter the negative results, and finally the scaling factor p corresponding to the upsampling kernel is obtained.
7. A small object semantic segmentation method comprising the feature map upsampling method according to any one of claims 1 to 6, characterized in that The specific steps include: S1. Obtain the original small target image and perform pixel-level annotation to obtain a training data set; S2. Use existing technology to build an initial semantic segmentation model; S3. Replace the upsampling method in the initial semantic segmentation model constructed in step S2 with the feature map upsampling method described in one of claims 1 to 6, thereby obtaining a small target semantic segmentation model; S4. Use the training data set to train the small object semantic segmentation model to obtain the final small semantic segmentation model; S5. Use the small object semantic segmentation model obtained in step S4 to perform semantic segmentation on the target image and complete the semantic segmentation of small objects in the target image.
8. An imaging method comprising the small object semantic segmentation method according to claim 7, characterized in that The steps include: A. Obtain the original image for small object semantic segmentation; B. performing semantic segmentation on the original image obtained in step A using the small object semantic segmentation method described in claim 7; C. The semantic segmentation result of the small target obtained in step B is annotated and imaged on the original image obtained in step A to obtain a target image with the small target annotated result; D. Output the target image obtained in step C to complete the imaging.
Citation Information
Patent Citations
Image semantic segmentation method based on full convolution neural network
CN109101975A
Airfield pavement underground structure disease automatic detection method based on deep learning
CN112700429A