Small target detection algorithm based on full convolution

Abstract

This invention provides a small target detection algorithm based on fully convolutional modeling, comprising a convolutional network model, a loss calculation part, and a training and parameter tuning part. The convolutional network model is used to extract image features and predict targets. The loss calculation part is used to calculate the prediction loss during training to obtain gradients to guide the network model in weight learning. The training and parameter tuning part is used to feed labeled data into the network model for forward inference, perform backpropagation using the gradient of the loss function, and adjust the network learning rate and dataset based on the validation accuracy to obtain the optimal model weights. This invention uses multi-scale feature fusion to improve the ability to extract target features at different scales, uses dual-scale target prediction to address the issue that small targets have a small information content in the feature map and are therefore easily affected by the features of large targets, and uses the true boundary of the target to calculate the loss to guide the network to learn target boundary features more accurately.

Description

Technical Field

[0001] This invention belongs to the field of small target detection technology, specifically relating to a small target detection algorithm based on full convolution. Background Technology

[0002] Image-based object detection is a key research area in computer vision. In the COCO object detection dataset, targets with a pixel area smaller than 32×32 are considered small targets. Due to the limited information content of this type of target, it has always been a research challenge. In recent years, thanks to advancements in computing devices and deep learning theory, the accuracy of object detection tasks has been greatly improved, and research on small target detection has gradually shown promising results.

[0003] Researchers have focused on four main research directions related to the characteristics of small targets. First, for small targets with a low pixel count, data augmentation strategies are used to increase their presence in images, such as random cropping, random scaling, target region copying and pasting, and GAN generation of small targets. Second, for multi-scale feature extraction of small targets with small area sizes, feature matrices of different depths and scales are fused into the backbone network to improve the ability to extract information at different scales of the image, such as FPN and its various derivatives. Third, for global feature fusion, researchers believe that there are usually correlations between targets and scenes, and between targets themselves; for example, a fish is more likely to be in water than in the sky. Methods typically use channel attention and spatial attention mechanisms to learn and utilize global feature information. Fourth, for anchor-box-less mechanisms for small targets, anchor-box-based target detection methods typically set the anchor box size based on prior experience and calculate the contents within the anchor box for classification or regression. In this type of method, positive and negative samples need to be separated. Small targets can cause large fluctuations in accuracy due to slight deviations in the intersection-union calculation, making it difficult for small targets to learn. After the emergence of target detection methods without anchor boxes, researchers have made improvements to enhance the detection effect of small targets. For example, they use enhanced feature extraction networks to directly predict the center point and size of the target bounding box.

[0004] Methods based on data augmentation, multi-scale feature extraction, and global feature fusion demonstrate general performance improvements in the field of object detection and can be used as plug-in modules in various object detection models.

[0005] The anchorless object detection algorithm breaks the definition of pre-made anchor boxes and alleviates the imbalance problem of positive samples of small targets in the prediction mechanism. However, since the anchorless detection method usually still uses the largest outer rectangle of the target to calculate the intersection-union ratio of the target box to generate the loss and guide the network learning. Summary of the Invention

[0006] This invention provides a small target detection algorithm based on full convolution. It uses multi-scale feature fusion to improve the ability to extract target features at different scales, uses dual-scale target prediction to address the issue that small target features have a small proportion of information and are easily affected by large target features, and uses the true boundary of the target to calculate the loss to guide the network to learn target boundary features more accurately.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A small object detection algorithm based on full convolution includes a convolutional network model, a loss calculation part, and a training and parameter tuning part;

[0009] The convolutional network model includes a backbone network module, a multi-scale feature fusion module, and a dual-scale prediction module. The backbone network module uses the backbone network to extract features from the image at different scales sequentially. The multi-scale feature fusion module fuses feature matrix information from different scales. The dual-scale prediction module predicts large-scale targets and small-scale targets respectively, and finally outputs the results.

[0010] The convolutional network model is used to extract image features and predict targets; the loss calculation part is used to calculate the prediction loss during training to obtain gradients to guide the network model in weight learning; the training and parameter tuning part is used to feed labeled data into the network model for forward inference, perform backpropagation using the gradient of the loss function, and finally adjust the network learning rate and dataset based on the validation accuracy; the specific algorithm steps are as follows:

[0011] Step S1: Construct the network model;

[0012] Step S2: Construct a loss function based on the target boundary distance;

[0013] Step S3: Training and parameter tuning.

[0014] Furthermore, the backbone network module is specifically as follows:

[0015] Input the batch image pixel matrix I into the backbone network module:

[0016] I = [B, C, H, W]

[0017] Where B is the number of images in the batch, C is the number of channels, and the input is usually a 3-channel image with R being red, G being green, and B being blue color features, H is the image height, and W is the image width;

[0018] After passing through the backbone module, three features of different scales, C3, C4, and C5, are output, where 3, 4, and 5 represent the feature matrix scale downsampled to the power of 2.

[0019] Furthermore, the multi-scale feature fusion module is specifically as follows:

[0020] The input feature matrix is ​​divided into two parts along the channel dimension. The first part is subjected to convolution and other operations, and the second part is directly shorted to the end of the module output and concatenated with the result of the first part. Finally, feature matrices P3 and P4 at two scales are obtained.

[0021] Furthermore, the dual-scale prediction module is specifically as follows:

[0022] Using a general semantic segmentation prediction structure on P3 and P4, prediction results at two different scales are obtained, as specifically implemented as follows:

[0023] Targets with a connected component area greater than 32×32 are considered large targets, and the P4 feature matrix is ​​responsible for prediction, resulting in a large target result R4; targets with a connected component area less than 32×32 are considered small targets, and the P3 feature matrix is ​​responsible for prediction, resulting in a small target result R3.

[0024] When calculating the loss by comparing the prediction results with the labels, different label maps are generated based on the size of the target:

[0025] When calculating the loss of the P3 prediction result, the small target label is used for calculation, while when calculating the loss of the P4 prediction result, the large target label is used for calculation.

[0026] In the small target label map, small targets are used as the first-level label area, and large targets are used as the second-level label area; the opposite is true in the large target label map.

[0027] During the training phase, the loss of each pixel in the first-level label region is calculated normally, while the loss of pixels predicted as background regions is not calculated in the second-level label region. It does not restrict the prediction of targets that do not belong to this scale, thus preventing the impact of feature conflicts.

[0028] Extract the connected components in the small target result graph R3 that conform to the small target rule, and cover them onto the large target result graph R4 to obtain the final prediction result R.

[0029] Furthermore, step S2 specifically involves: Since the fully convolutional network performs class prediction for each pixel, it is more prone to fragmented blob errors. Therefore, the loss function based on the target boundary distance is:

[0030]

[0031] Where L pixelloss y represents the loss for each pixel, classes represents the set of all classes, and y represents the loss for each pixel. true This indicates the label of the pixel in a certain category, y pred This represents the predicted score of the pixel in a certain category. The weighting coefficient is determined by the distance between the predicted category and the nearest connected component of that category in the label. The calculation formula is as follows:

[0032]

[0033] The above formula Indicates pixel i in y true In the label area of ​​the category, This indicates that pixel i is not in y. true In the label area of ​​the category, dis represents the distance of pixel i to the nearest y. true The distance to the category label region, this weight can further constrain erroneous predictions that are far from the correct label region.

[0034] Furthermore, step S3 is as follows:

[0035] Step S31: Collect the target image data required for the task, and assign labels to the data according to the label format of semantic segmentation to obtain the dataset required for training.

[0036] Step S32: Divide the dataset into training set, validation set and test set according to the ratio. The general ratio is 7:1:2, which can be modified as appropriate according to the amount of data.

[0037] Step S33: Feed the training set into the network model constructed in step S1 for forward computation, obtain the prediction results, use the loss function constructed in step S2 to calculate the gradient and backpropagate to adjust the model parameters.

[0038] Step S34: After training multiple batches, adjust the learning rate parameter based on the accuracy performance on the validation set, and observe whether the decreasing trend of model loss is positively correlated with the increasing trend of validation set accuracy to avoid overfitting.

[0039] Step S35: Finally, based on the training results of multiple training rounds, test the network model using the test set, select the optimal network model as the result model and store it for use in the next small object detection inference.

[0040] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0041] 1. The small target detection algorithm based on full convolution provided by this invention uses dual-scale prediction. The network model focuses more on the detection of small targets at a small scale, reducing interference between large and small targets. At the same time, it constructs label maps at different scales, effectively improving the accuracy of small targets.

[0042] 2. The small target detection algorithm based on full convolution provided in this invention proposes a loss function targeting the target boundary in order to reduce the speckled prediction errors that occur when semantic segmentation networks are transferred to target detection tasks. This strengthens the loss weight for predicting errors outside the target and guides the network to focus on the target region, thereby reducing speckled prediction errors. Attached Figure Description

[0043] Figure 1 This is a flowchart of the present invention;

[0044] Figure 2 This is a diagram of the network model structure of the present invention;

[0045] Figure 3 This is a diagram of the multi-scale information fusion structure of the present invention;

[0046] Figure 4 These are different label images of the present invention. Detailed Implementation

[0047] The present invention is described below based on embodiments, but the invention is not limited to these embodiments. In the detailed description of the invention below, certain specific details are described in detail. Those skilled in the art will fully understand the invention even without these details. To avoid obscuring the essence of the invention, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0048] This invention consists of three parts: first, a convolutional network model that extracts image features and predicts targets; second, a loss calculation part that calculates the prediction loss during training to obtain gradients to guide the network model in weight learning; and third, a training and parameter tuning part that feeds labeled data into the network model for forward inference, performs backpropagation using the gradient of the loss function, and finally adjusts the network learning rate and dataset based on the validation accuracy. Ultimately, the optimal weight parameters are selected for use in target detection inference.

[0049] The network model structure diagram in this algorithm is as follows: Figure 2 As shown:

[0050] Figure 2 The model is divided into three modules: the first is the backbone network module, which uses the backbone network to extract features from the image at different scales in sequence; the second is the multi-scale feature fusion module, which fuses feature matrix information at different scales; and the third is the dual-scale prediction module, which predicts large-scale targets and small-scale targets respectively, and finally outputs the results.

[0051] The backbone network module allows for flexible use of different general-purpose backbone network modules to suit varying task difficulty and device computation levels. Different backbone networks may compute different numbers of feature scales; the feature matrices of the last three scales can be used as output.

[0052] In this invention, ESNet proposed in PP-PicoDet is selected. Alternatively, backbone networks such as ResNet, SENet, MobileNet, and EfficientNet can be selected as alternatives. Different backbone networks have different computational complexity and feature extraction capabilities.

[0053] The backbone network uses advanced network search algorithms and channel information fusion structures to improve the feature extraction capability of the backbone network, and uses the lightweight network architecture ShuffleNet and Ghost blocks to effectively reduce computational complexity.

[0054] Input the batch image pixel matrix I into the backbone network module:

[0055] I = [B, C, H, W]

[0056] Where B is the number of images in the batch, C is the number of channels, and the input is usually a 3-channel image with R (red), G (green), and B (blue) color features, H is the image height, and W is the image width.

[0057] After passing through the backbone module, three features of different scales, C3, C4, and C5, are output, where 3, 4, and 5 represent the feature matrix scale downsampled to the power of 2.

[0058] Three feature matrices were obtained, downsampled by 8 times (C3), 16 times (C3), and 32 times (C5). To further fuse feature information, this algorithm employs the following... Figure 3 Feature fusion structure shown:

[0059] The CSP structure, derived from CSPNet, divides the input feature matrix into two parts along the channel dimension. The first part undergoes convolution and other operations, while the second part is directly shorted to the end of the module output and concatenated with the first part. This reduces computational cost while enriching the diversity of gradient paths.

[0060] Finally, feature matrices P3 and P4 at two scales were obtained.

[0061] After obtaining feature matrices P3 and P4 at two scales, to reduce the loss imbalance caused by targets at different scales, a general semantic segmentation prediction structure is used on P3 and P4 to obtain prediction results at the two different scales. The specific implementation is as follows:

[0062] In this algorithm, targets with a connected region area greater than 32×32 are considered large targets, and the P4 feature matrix is ​​responsible for prediction, resulting in the large target result image R4; targets with a connected region area less than 32×32 are considered small targets, and the P3 feature matrix is ​​responsible for prediction, resulting in the small target result image R3.

[0063] In real-world object detection tasks, the perceived size of an object in an image can vary due to imaging conditions. This can result in an object being classified as large in one image and small in another. Forcing the network to learn this discrepancy will negatively impact its performance. Therefore, when calculating the loss between the prediction results and the labels, different label maps are generated based on the size of the object, such as... Figure 4 As shown:

[0064] When calculating the loss of the P3 prediction result, the small target label is used for calculation, while when calculating the loss of the P4 prediction result, the large target label is used for calculation.

[0065] In the small target label map, small targets are designated as primary label areas, and large targets as secondary label areas; the opposite is true in the large target label map.

[0066] During the training phase, the loss for each pixel is calculated normally for the primary label region, while the loss for pixels predicted as background regions is not calculated for the secondary label region. It does not restrict predictions to targets not belonging to its scale, preventing the impact of feature conflicts.

[0067] In the prediction phase, we extract the connected components in the small target result graph R3 that conform to the small target rules and cover them onto the large target result graph R4 to obtain the final prediction result R.

[0068] Construct a loss function based on target boundary distance.

[0069] Since fully convolutional networks perform class prediction for each pixel, they are more prone to sporadic bloat errors. To further constrain this type of error within the network model, this algorithm proposes a loss function based on the target boundary distance:

[0070]

[0071] Where L pixelloss y represents the loss for each pixel, classes represents the set of all classes, and y represents the loss for each pixel. true This indicates the label of the pixel in a certain category, y pred This represents the predicted score of the pixel in a certain category. The weighting coefficient is determined by the distance between the predicted category and the nearest connected component of that category in the label. The calculation formula is as follows:

[0072]

[0073] The above formula Indicates pixel i in y true In the label area of ​​the category, This indicates that pixel i is not in y. true In the label area of ​​the category, dis represents the distance of pixel i to the nearest y. true The distance to the category label region. This weight can further constrain erroneous predictions that are far from the correct label region.

[0074] The training and parameter tuning steps are as follows:

[0075] Step S31: Collect the target image data required for the task, and assign labels to the data according to the label format of semantic segmentation to obtain the dataset required for training.

[0076] Step S32: Divide the dataset into training set, validation set and test set according to the ratio. The general ratio is 7:1:2, which can be modified as appropriate according to the amount of data.

[0077] Step S33: Feed the training set into the network model constructed in step S1 for forward computation, obtain the prediction results, use the loss function constructed in step S2 to calculate the gradient and backpropagate to adjust the model parameters.

[0078] Step S34: After training multiple batches, adjust the learning rate parameter based on the accuracy performance on the validation set, and observe whether the decreasing trend of model loss is positively correlated with the increasing trend of validation set accuracy to avoid overfitting.

[0079] Step S35: Finally, based on the training results of multiple training rounds, test the network model using the test set, select the optimal network model as the result model and store it for use in the next small object detection inference.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A small target detection algorithm based on fully convolution, characterized in that, It includes the convolutional network model, the loss calculation, and the training and parameter tuning. The convolutional network model includes a backbone network module, a multi-scale feature fusion module, and a dual-scale prediction module. The backbone network module uses the backbone network to extract features from the image at different scales sequentially. The multi-scale feature fusion module fuses feature matrix information from different scales. The dual-scale prediction module predicts large-scale targets and small-scale targets respectively, and finally outputs the results. The convolutional network model is used to extract image features and predict targets; the loss calculation part is used to calculate the prediction loss during training to obtain gradients to guide the network model in weight learning; the training and parameter tuning part is used to feed labeled data into the network model for forward inference, perform backpropagation using the gradient of the loss function, and finally adjust the network learning rate and dataset based on the validation accuracy; the specific algorithm steps are as follows: Step S1: Construct the network model; Step S2: Construct a loss function based on target boundary distance: Since the fully convolutional network performs class prediction for each pixel, it is more prone to fragmented blob errors. Therefore, the loss function based on target boundary distance is: Where L pixelloss y represents the loss for each pixel, classes represents the set of all classes, and y represents the loss for each pixel. true This indicates the label of the pixel in a certain category, y pred This represents the predicted score of the pixel in a certain category. The weighting coefficient is determined by the distance between the predicted category and the nearest connected component of that category in the label. The calculation formula is as follows: The above formula Indicates pixel i in y true In the label area of ​​the category, This indicates that pixel i is not in y. true In the label area of ​​the category, dis represents the distance of pixel i to the nearest y. true The distance to the category label region, this weight further constrains incorrect predictions that are far from the correct label region; Step S3, Training and Parameter Tuning: Step S31: Collect the target image data required for the task, and assign labels to the data according to the label format of semantic segmentation to obtain the dataset required for training. Step S32: Divide the dataset into training set, validation set and test set according to the ratio. The general ratio is 7:1:2, which can be modified as appropriate according to the amount of data. Step S33: Feed the training set into the network model constructed in step S1 for forward computation, obtain the prediction results, use the loss function constructed in step S2 to calculate the gradient and backpropagate to adjust the model parameters. Step S34: After training multiple batches, adjust the learning rate parameter based on the accuracy performance on the validation set, and observe whether the decreasing trend of model loss is positively correlated with the increasing trend of validation set accuracy to avoid overfitting. Step S35: Finally, based on the training results of multiple training rounds, test the network model using the test set, select the optimal network model as the result model and store it for use in the next small object detection inference.

2. The small target detection algorithm based on fully convolution as described in claim 1, characterized in that, The backbone network module is as follows: Input the batch image pixel matrix I into the backbone network module: I = [B, C, H, W] Where B is the number of images in the batch, C is the number of channels, and the input is usually a 3-channel image with R being red, G being green, and B being blue color features, H is the image height, and W is the image width; After passing through the backbone module, three features of different scales, C3, C4, and C5, are output, where 3, 4, and 5 represent the feature matrix scale downsampled to the power of 2.

3. The small target detection algorithm based on fully convolution as described in claim 1, characterized in that, The multi-scale feature fusion module is specifically as follows: The input feature matrix is ​​divided into two parts along the channel dimension. The first part is subjected to convolution and other operations, and the second part is directly shorted to the end of the module output and concatenated with the result of the first part. Finally, feature matrices P3 and P4 at two scales are obtained.

4. The small target detection algorithm based on fully convolution as described in claim 1, characterized in that, The dual-scale prediction module is as follows: Using a general semantic segmentation prediction structure on P3 and P4, prediction results at two different scales are obtained, as specifically implemented as follows: Targets with a connected component area greater than 32×32 are considered large targets, and the P4 feature matrix is ​​responsible for prediction, resulting in a large target result R4; targets with a connected component area less than 32×32 are considered small targets, and the P3 feature matrix is ​​responsible for prediction, resulting in a small target result R3. When calculating the loss by comparing the prediction results with the labels, different label maps are generated based on the size of the target: When calculating the loss of the P3 prediction result, the small target label is used for calculation, while when calculating the loss of the P4 prediction result, the large target label is used for calculation. In the small target label map, small targets are used as the first-level label area, and large targets are used as the second-level label area; the opposite is true in the large target label map. During the training phase, the loss of each pixel in the first-level label region is calculated normally, while the loss of pixels predicted as background regions is not calculated in the second-level label region. It does not restrict the prediction of targets that do not belong to this scale, thus preventing the impact of feature conflicts. Extract the connected components in the small target result graph R3 that conform to the small target rule, and cover them onto the large target result graph R4 to obtain the final prediction result R.

Images