A small target detection method of asymmetric modulation fusion features

Abstract

The application discloses a small target detection method based on asymmetric modulation fusion features, and specifically comprises the following steps: 1: four feature maps of an input image are extracted, namely feature map 1, feature map 2, feature map 3 and feature map 4; 2: the feature map 1 is input into a first spatial attention network, and the feature map 3 is input into a first channel attention network; the feature maps output by the two attention networks are fused through element addition to obtain a fusion feature map 1; the feature map 2 is input into a second spatial attention network, and the feature map 4 is input into a second channel attention network; the feature maps output by the two attention networks are fused through element addition to obtain a fusion feature map 2; 3: a region of interest of the fusion feature map is extracted by using an RPN network; 4: the region of interest is amplified; 5: the amplified region of interest is sent into a classification network, and the original region of interest is sent into a regression network; and the application improves the accuracy of classification and regression.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and object detection. BACKGROUND

[0002] Object detection is a challenging task in the field of computer vision, and is also one of the hot topics discussed by scholars in recent years. The definition of object detection is to detect the position and size of the target object in the given image or video, and to perform related tasks such as classification or recognition. Compared with image classification, object recognition and other tasks, object detection needs to find the position of the target in the image, so it is more difficult and requires higher precision. Small target detection has wide application in remote sensing image analysis, medical disease detection, agricultural product detection and other fields, so the attention of scholars on small target detection is gradually increasing. Object detection is roughly divided into two categories: the first is traditional machine learning-based object detection, and the other is neural network-based object detection.

[0003] With the booming of deep neural network, target detection algorithm based on deep learning has become one of the most advanced target detection algorithms. The current research shows that in the target detection model using deep learning, the detection accuracy of two-stage detection algorithm is generally better than that of single-stage algorithm, so the two-stage detection algorithm can be used in the task which does not have strict requirement on detection speed. RCNN (Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation [C] / / Proc of the 2014 IEEE / CVF International Conference on Computer Vision and Pattern Recognition, 2014: 580-587.) first introduced convolutional neural network into the target detection task, but due to the use of unsupervised selective search and repeated convolution, the detection speed is slow and the memory occupation is large. As a representative of two-stage algorithm, Faster RCNN (Ren S Q, He K M, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137-1149.) constructs RPN network, which combines the candidate frame generation network and the candidate frame classification and regression network to train together, and breaks through in speed and accuracy, but it only generates candidate regions on single layer feature map without considering the feature difference of different levels and the loss of small targets in deep convolution process. Most of the previous models including the above two algorithms do not consider the fusion of different levels of features or the single way of feature fusion, so the detection performance cannot reach the ideal effect. Therefore, only relying on single layer feature or single way of feature fusion cannot effectively improve the small target detection performance. SUMMARY

[0004] The present application provides a small target detection method based on asymmetric modulation fusion features.

[0005] The technical scheme is characterized in that the small target detection method based on asymmetric modulation fusion features comprises the following steps:

[0006] Step 1: input the image into the Resnet50 network, and the image is outputted by the Resnet50 network through four stages to obtain four feature maps from shallow to deep: feature Figure 1 , feature Figure 2 , feature Figure 3 and feature Figure 4 ;

[0007] Step 2: input the feature Figure 1 into the first spatial attention network, and input the feature Figure 3 into the first channel attention network; the feature map outputted by the first spatial attention network and the feature map outputted by the first channel attention network are fused by element addition to obtain the fused feature Figure 1 ; input the feature Figure 2 into the second spatial attention network, and input the feature Figure 4 into the second channel attention network; the feature map outputted by the second spatial attention network and the feature map outputted by the second channel attention network are fused by element addition to obtain the fused feature Figure 2 ; the first and second spatial attention networks have the same structure of the spatial channel attention modulation structure from bottom to top, and the first and second channel attention networks have the same structure of the global channel attention modulation structure from top to bottom;

[0008] Step 3: the region of interest of the fused feature map is extracted by using the RPN network;

[0009] Step 4: the region of interest is augmented;

[0010] Step 5: the augmented region of interest is sent into the classification network to output the category label; and the original region of interest is sent into the regression network to output the predicted detection frame.

[0011] Further, the structures of the first and second channel attention networks are as follows:

[0012]

[0013] Wherein, X represents the input of the channel attention network, X' represents the output of the channel attention network, x is the feature obtained by performing global average pooling on X, σ(.) is the Sigmoid function, В(.) represents normalization, f (c,c / γ) represents convolution with the channel number from C to C / γ, represents convolution with the channel number from C / γ to C, and × represents element multiplication;

[0014] The structures of the first and second spatial attention networks are as follows:

[0015]

[0016] wherein Y is the input of the spatial attention network, Y' is the output of the spatial attention network, represents a convolution whose channel number is changed from 2 to 1; + is concatenation by channel; y max represents the maximum value taken by any point in space in the channel direction; y mean represents the average value taken by any point in space in the channel direction.

[0017] Further, the step 4 of expanding the region of interest is specifically: according to the size of the region of interest, 8 interest blocks are expanded around the region of interest with the region of interest as the center, and the coordinates of the vertex farthest from the origin of the expanded region of interest are (X c ,Y c ), wherein X c =X max +w, Y c =Y max +h; wherein (X max ,Y max ) is the coordinates of the vertex farthest from the origin of the original region of interest, w=X max -X min , h=Y max -Y min , wherein (X min ,Y min ) is the coordinates of the vertex closest to the origin of the original region of interest; the coordinates of the vertex farthest from the origin of the 8 interest blocks are taken as the vertex coordinates of the 8 interest blocks, if X C <E and Y C <H, all the interest blocks are retained; if X C >W and Y C >H, the interest block whose horizontal coordinate value of the vertex coordinate is X C and the interest block whose vertical coordinate value of the vertex coordinate is Y C are discarded; if X C >W and Y C <H, the interest block whose horizontal coordinate value of the vertex coordinate is X C is discarded; if X C <W and Y C >H, the interest block whose vertical coordinate value of the vertex coordinate is Y C is discarded, wherein W and H are the size of the feature map in step 1.

[0018] Further, when the model composed of the Resnet50 network, the first and second spatial attention networks, the first and second channel attention networks, the RPN network, the classification network and the regression network is trained, the loss function used is as follows:

[0019] L=L cls +Lreg

[0020] wherein L cls is a classification loss, L reg is a regression loss; L reg The expression of L

[0021] L reg = 1-IoU+R DIoU

[0022] wherein IoU is the ratio of the intersection area of the predicted box and the real box to the total area of the predicted box and the real box, R DIoU is a penalty term, and the expression is as follows:

[0023]

[0024] wherein b and b gt represent the center points of the predicted box and the real box, c is the diagonal length of the minimum bounding box containing the predicted box and the real box, and p(.) represents the calculation of the Euclidean distance.

[0025] Beneficial effects: the two different attention modulation modes constructed by the application integrate the information lacking in each other between high and low layers, can make up for the shortcomings of small target size and small amount of information contained, and fully highlight the outline and details of small targets; in addition, the application designs a flexible expansion size of the region of interest, flexibly introduces background information in the region of interest classification task, and for small targets, the amount of information contained is small and easy to be ignored, and increasing the background information can increase the classification accuracy of small targets. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the structural diagram of feature fusion proposed by the application,

[0027] Figure 2 is the top-down global channel attention modulation structural diagram proposed by the application,

[0028] Figure 3 is the bottom-up spatial attention modulation structural diagram proposed by the application,

[0029] Figure 4 is the flexible expansion diagram of the region of interest proposed by the application,

[0030] Figure 5 is the second stage flowchart of Faster RCNN,

[0031] Figure 6The images show a comparison of the detection examples of the proposed method and the original Faster R-CNN algorithm on the DIOR dataset. (a) shows the detection results of the original Faster R-CNN algorithm for overpass targets; (b) shows the detection results of the original algorithm of this invention for overpass targets; (c) shows the detection results of the original Faster R-CNN algorithm for bridge targets; (d) shows the detection results of the algorithm of this invention for bridge targets; (e) shows the detection results of the original Faster R-CNN algorithm for ship targets; and (f) shows the detection results of the algorithm of this invention for ship targets. Detailed Implementation

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the following further explanation of the invention is provided in conjunction with the accompanying drawings and specific embodiments. The GPU used in the entire experiment of this invention is an RTX 3060, the CPU is an i7 11700f, the operating system is Windows 10, the CUDA version is 11.6, the deep learning framework used is PyTorch 1.12.0, and the Python version is 3.7.3. This invention specifically includes the following operational steps:

[0034] Step 1: As Figure 1 As shown, the image is input into the ResNet50 network. After passing through four stages, the ResNet50 network outputs four feature maps from shallow to deep: Feature Maps Figure 1 ,feature Figure 2 ,feature Figure 3 and characteristics Figure 4 ;

[0035] This invention uses an improved ResNet50 as the backbone network for feature extraction. Compared to the VGG network, the ResNet network of the same depth has fewer parameters, and because it uses a residual structure, it does not cause model degradation even if the number of network layers is too deep. This invention uses a ResNet50 network with the last linear layer removed, divided into four layers according to its depth. After the image passes through four convolutional blocks, four feature maps from shallow to deep are obtained.

[0036] Step 2: Construct an asymmetric high-low layer modulation structure. After Step 1, the image yields four feature maps from shallow to deep. This invention uses spatial attention to encode the rich detail contour information of the shallow feature maps, while simultaneously using channel attention to encode the high-level semantic information. Fusing these two attention-encoded features results in a rich feature map that contains both high-level semantic information and low-level detail information. Predicting candidate regions from the fused feature map is then performed. The asymmetric modulation structure is as follows: Figure 2 As shown in Figure 3. Specifically, the features Figure 1 The feature map contains the most detailed information, and the information about small targets is most clearly displayed on the feature map. Figure 4 The semantic information is the strongest, and it is effective for detecting large targets. This invention uses an interval fusion method to combine features. Figure 1 Input to a spatial attention network, features Figure 3 The input is fed into the channel attention network, and the feature maps output by the spatial attention network and the channel attention network are fused through element-wise addition; the features are then... Figure 2 Input to spatial attention network and features Figure 4 The input is fused with the channel attention network, and the feature maps output by the spatial attention network and the channel attention network are fused by element-wise addition; this results in two fused feature maps. Figure 1 It contains both rich detailed contour information and high semantic information, mainly for detecting small and medium-sized targets. The anchor box size is set to 32×32, 64×64, and 128×128, and features are fused. Figure 2 It contains advanced semantic information while also taking into account strong detailed information, and mainly plays the role of detecting large targets. The anchor box size is set to 128×128, 256×256, and 512×512.

[0037] Assuming the feature map has C channels and a size of H×W, and the channel attention network is a top-down global channel attention modulation structure, such as... Figure 2 As shown, the expression for this structure is:

[0038]

[0039] Where X represents the input of the spatial attention network, X′ represents the output of the spatial attention network, and x is the feature obtained by applying global average pooling to X. σ(.) is the Sigmoid function, and В(.) represents normalization. This indicates a convolution with channel number C from C / γ. This indicates a convolution from C / γ to C channels, and × indicates element-wise multiplication.

[0040] The spatial attention network is a bottom-up spatial attention modulation structure, as shown in the following formula: Figure 3

[0041]

[0042] wherein Y is the input of the channel attention network, Y' is the output of the channel attention network, represents a convolution in which the number of channels is changed from 2 to 1; + represents channel-wise concatenation; y max represents the maximum value taken by any point in space in the channel direction, y max = max(Y[i,:,:]); y mean represents the average value taken by any point in space in the channel direction,

[0043] Therefore, the fused feature obtained by asymmetric modulation is Z = X' + Y' = P(X) x X + L(Y) x Y.

[0044] Step 3: Extract the region of interest of the fused feature map using the RPN network.

[0045] Step 4: Expand the region of interest of the positive and negative samples used for training after screening by the RPN network by 8 blocks one by one. For the blocks that cross the boundary of the feature map, they are directly discarded, and those that do not cross the boundary are retained. The new region of interest obtained after a series of expansion operations is only used for classification tasks, while the original region of interest without expansion is used for position regression. This is equivalent to adding an effective context window to the region of interest classification task. At this time, only classification information is added, and regression information is not added, which can flexibly use background information to improve classification accuracy, and also does not introduce prior box regression error.

[0046] Specifically, according to the size of the region of interest, 8 interest blocks are expanded around the region of interest with the region of interest as the center. The coordinates of the vertex of the expanded region of interest c farthest from the origin are (X c ,Y c ), wherein X c = X max + w, Y c = Y max + h; wherein (X max ,Y max ) are the coordinates of the vertex of the original region of interest farthest from the origin, w = X max -X min , and h = Y max -Y min , wherein (X min ,Y min ​) Coordinates of the vertex of the original region of interest that is closest to the origin; the coordinates of the vertices of the 8 interest blocks that are farthest from the origin are used as the vertex coordinates of the 8 interest blocks. If X C < W and Y C < H, then all interest blocks are retained; if X C > W and Y C > H, discard the interest blocks whose abscissa value of the vertex coordinates is X C , and the interest blocks whose ordinate value of the vertex coordinates is Y C ; if X C > W and Y C < H, then discard the interest blocks whose abscissa value of the vertex coordinates is X C ; if X C < W and Y C > H, then discard the interest blocks whose ordinate value of the vertex coordinates is Y C , where W and H are the sizes of the feature maps in step 1. Specifically, as Figure 4 shown: A is the upper left vertex of the candidate region (i.e., the vertex closest to the origin), and the coordinates are (X min , Y min ), B is the lower right vertex of the candidate region (i.e., the vertex farthest from the origin), and the coordinates are (X max , Y max ), C is the lower right vertex of the expanded block8 (i.e., the vertex farthest from the origin), and the coordinates are (X c , Y c ), the width of the ROI is w = X max - X min , and the height is h = Y max - Y min , and the coordinates of point C are (X max + w, Y max + h). W×H is the size of the feature map. When X C = X max + w > W and Y C = Y [[ID=​​​​​​​​​​​​​​​​​​​max +w < W and Y C = Y max +h < H, all will be retained. The above is for the right lower corner vertex of the enlarged region, and other vertex cases are similar.

[0047] Step 5: as shown in the figure, the flexible amplified region of interest NewROIpooling is sent into the classification network, and the original region of interest Original ROIpooling is directly sent into the regression network. Figure 5

[0048] Step 6: training the network. In the experiment, the hyperparameter γ is set to 8, the Batchsize size is 2, SGD (Stochastic Gradient Descent) is used as the optimizer, the initial learning rate is set to 0.01, and the learning rate is reduced in a fixed step attenuation manner during the training process, and is reduced to 0.33 times of the previous value every 3 rounds, and a total of 50 rounds are trained. In the training stage, the loss function of the model is L = L cls + L reg , wherein L cls is the classification loss of the algorithm, and L reg is the regression loss of the algorithm. The original FasterRCNN detection algorithm uses SmoothL1 as the regression loss function, and there is a obvious problem: when multiple detection boxes have the same SmoothL1 loss value, their IoU with the true box may have a large difference. The true evaluation detection box regression index is to use IoU, and at this time, the calculation of the SmoothL1 loss value cannot reflect the overlapping degree of the detection box and the true box.

[0049] The DIoU loss used in the application adds a penalty term to the IoU loss, which minimizes and normalizes the distance between the centers of the two boundary boxes, so that the convergence process is accelerated, and R is defined as follows:

[0050]

[0051] Where b and b gt represent the center points of the predicted box and the true box, ρ(b,b gt ) is the Euclidean distance between the two points, and c is the diagonal length of the minimum boundary box containing the two boxes, so the loss function of DIoU can be defined as:

[0052]

[0053] ​Step five: detection test is carried out online. In the test stage of the present application, the image is first scaled and filled to make the size the same so as to be packaged into the prediction network, the weight transmitted into the prediction network is the best weight obtained by training, the main part of the prediction network extracts the features and then the asymmetric modulation module increases and fuses the features. The feature map after fusion generates a series of regions of interest through the RPN network, the size of the region of interest is expanded, and the expanded block exceeding the boundary is discarded. Finally, the candidate region after expansion is sent into the classification network, and the original region is sent into the regression network to obtain the score and boundary box offset of the candidate region, and finally the detection result is obtained.

[0054] In order to compare the detection performance of each model, the present application adopts AP (Average Precision) and mAP (mean Average Precision) as evaluation indexes, and the calculation methods of AP and mAP are as follows:

[0055]

[0056]

[0057] AP = ∫PdR

[0058]

[0059] Where T P is a true positive example, F P is a false positive example, F N is a false negative example, AP i is the detection performance value corresponding to each category, mAP is the average detection performance value, and C is the total number of categories.

[0060] DIOR data set is adopted as experimental data in the embodiment, and the distribution of each type of target of the DIOR data set is shown in Table 1:

[0061] Table 1

[0062]

[0063]

[0064] Based on the adopted DIOR data set, the present application is 15.9%, 9.0%, 5.6%, 9.8%, 6.5% and 16.3% higher than the algorithms SSD, FSSD300, FSSD512, TDFSSD300, TDFSSD512 in accuracy, respectively.

[0065] Figure 6The detection examples of the method proposed in the application and the original Faster RCNN algorithm on the DIOR data set are compared, Figure 6 (a) and (b) are the detection comparison of the original Faster RCNN and the algorithm of the application on the overpass target, it can be seen that the original algorithm has missed detection, but the algorithm of the application accurately detects the overpass; Figure 6 (c) and (d) are the detection comparison of the original Faster RCNN and the algorithm of the application on the bridge target, although both algorithms detect the bridge target, the original algorithm does not detect the vehicle on the bridge, but the algorithm of the application can accurately detect the vehicle; Figure 6 (e) and (f) are the detection comparison of the original Faster RCNN and the algorithm of the application on the ship target, the original algorithm has missed detection and mis-detection, but the algorithm of the application accurately detects the ship; it can be seen that the classification of the application is more accurate and comprehensive.

[0066] In addition, it should be noted that various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combinations are not described again in the application.

Claims

1. A small target detection method of asymmetric modulation fusion features, characterized in that, The method specifically comprises the following steps: Step 1: inputting an image into a Resnet50 network, and outputting four feature maps from shallow to deep by the four stages of the Resnet50 network, namely, feature map 1, feature map 2, feature map 3 and feature map 4; Step 2: inputting the feature map 1 into a first spatial attention network and inputting the feature map 3 into a first channel attention network; fusing the feature map output by the first spatial attention network and the feature map output by the first channel attention network through element addition to obtain a fused feature map 1; inputting the feature map 2 into a second spatial attention network and inputting the feature map 4 into a second channel attention network; fusing the feature map output by the second spatial attention network and the feature map output by the second channel attention network through element addition to obtain a fused feature map 2; the first and second spatial attention networks have the same structure, which is a spatial channel attention modulation structure from bottom to top, and the first and second channel attention networks have the same structure, which is a global channel attention modulation structure from top to bottom; Step 3: extracting a region of interest of the fused feature map by using an RPN network; Step 4: amplifying the region of interest; Step 5: inputting the amplified region of interest into a classification network to output a category label, and inputting the original region of interest into a regression network to output a predicted detection frame; The structures of the first and second channel attention networks are as shown in the following formula: ； wherein, represents an input of the channel attention network, represents an output of the channel attention network, is to be features obtained by global average pooling, is a Sigmoid function, represents normalization, represents that the number of channels is to convolution, represents that the number of channels is to convolution, represents element multiplication; The structures of the first and second spatial attention networks are as shown in the following formula: ； wherein, is the input to the spatial attention network, is the output of the spatial attention network, represents a convolution that changes the number of channels from 2 to 1; is a concatenation along the channel dimension; represents the maximum value taken by any point in space in the channel direction; represents the average value taken by any point in space in the channel direction; The step 4 of expanding the region of interest is specifically: according to the size of the region of interest, 8 interest blocks are expanded around the region of interest with the region of interest as the center, and the coordinates of the vertex of the expanded region of interest farthest from the origin are , wherein ; wherein is the coordinates of the vertex of the original region of interest farthest from the origin, , , wherein is the coordinates of the vertex of the original region of interest closest to the origin; the coordinates of the vertex of the 8 interest blocks farthest from the origin are taken as the coordinates of the vertex of the 8 interest blocks, if and , all interest blocks are retained; if and , the interest block with the horizontal coordinate value of the vertex coordinate being is discarded, and the interest block with the vertical coordinate value of the vertex coordinate being is discarded; if and , the interest block with the horizontal coordinate value of the vertex coordinate being is discarded; if and , the interest block with the vertical coordinate value of the vertex coordinate being is discarded, wherein and are the size of the feature map in step 1.

2. The method according to claim 1, wherein, When the model composed of the Resnet50 network, the first and second spatial attention networks, the first and second channel attention networks, the RPN network, the classification network and the regression network is trained, the loss function used is as shown in the following formula: ； wherein is a classification loss, is a regression loss. The expression of is as follows: ； wherein, is the ratio of the area of the intersection of the predicted and true boxes to the total area of the predicted and true boxes, is a penalty term, expressed as follows: ； wherein, and represent the center points of the predicted and true bounding boxes, is the diagonal length of the minimum bounding box containing the predicted and true bounding boxes, denotes the computation of the Euclidean distance.

Images