An improved object detection algorithm based on feature pyramid network and attention mechanism

Abstract

The present invention discloses an improved target detection algorithm based on feature pyramid network and attention mechanism. The method combines the principle of feature pyramid network to fuse the six multi-scale feature maps extracted by the basic network in the original SSD algorithm. The new feature map formed after fusion also contains rich context information to improve the detection ability; and adds an attention model to the fused feature map to effectively extract the feature information of small targets. The missed detection situation is improved, the robustness of the algorithm is improved, and the real-time requirements are still met in terms of detection speed.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, relates to target detection, and in particular to an improved target detection algorithm based on a feature pyramid network and an attention mechanism. Background Art

[0002] The task of target detection is to find the targets of interest in the image and determine their categories and locations. It is one of the core issues in the field of computer vision and is widely used in infrared detection technology, intelligent video surveillance, remote sensing image target detection, medical diagnosis, and fire and smoke detection in intelligent buildings. Target detection algorithms can be divided into traditional target detection algorithms and target detection algorithms based on deep learning; representative algorithms of traditional target detection algorithms include SIFT algorithm and VJ detection algorithm, etc., but this method has high time complexity and lacks good robustness. Based on deep learning target detection algorithms, classic algorithms include R-CNN algorithm, Fast R-CNN algorithm, Faster R-CNN algorithm, YOLO algorithm, SSD algorithm, etc. Although there are many excellent target detection algorithms at this stage, the detection performance still has many shortcomings, which leads to problems such as missed detection and false detection. Summary of the invention

[0003] In view of the defects or shortcomings of the above-mentioned prior art, the object of the present invention is to provide an improved target detection algorithm based on feature pyramid network and attention mechanism.

[0004] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0005] An improved target detection algorithm based on feature pyramid network and attention mechanism, characterized in that it includes the following steps:

[0006] Step 1) Combining the principle of feature pyramid network, the basic network VGG-16 in the original SSD algorithm extracts 6 multi-scale feature maps of the input image, and performs feature fusion in order from small to large; obtains feature maps of different layers, and the fused feature maps contain rich semantic information and detail information at the same time;

[0007] Among them, in the original SSD algorithm, the scale of the feature map extracted from the input image by the basic network VGG-16 decreases from large to small, wherein the bottom feature map has a larger resolution and contains more detailed information, and the high-level feature map has a smaller resolution and contains more abstract semantic information. Therefore, the original SSD algorithm uses the bottom feature map to detect small targets, and the high-level feature map to detect medium and large targets;

[0008] Step 2) introduces the channel attention mechanism, and adds an attention model to the two feature maps that have richer detail information and semantic information after feature fusion and are more sensitive to small target detection; that is, the attention mechanism is implemented by adding a mask to the feature map, and the features of the region of interest are identified. Through continuous training of the network, the network learns the regions of interest that need to be focused on in each image, suppresses the influence of other interference areas, and thus enhances the algorithm's detection ability for small target objects.

[0009] According to the present invention, the input image size in step 1) is 300×300, and the feature map sizes obtained after the basic network VGG-16 for detection are 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 respectively. According to the principle of the feature pyramid network, the feature maps are sequentially fused in order of size from small to large, and 6 feature maps of size 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 are obtained.

[0010] Furthermore, in step 2), an attention model is added to the feature maps fused according to the feature pyramid principle in step 1). Because the fusion process is performed in the order of feature map size from small to large, the feature maps with the richest information after fusion are (38, 38) and (19, 19). Compared with other feature maps, these two feature maps have richer detail information and semantic information, and are more sensitive to small target detection. In order to maintain the detection speed of the algorithm and reduce the calculation amount of the algorithm, the attention model is only added to the two feature maps (38, 38) and (19, 19) after fusion. The specific detection algorithm process is as follows:

[0011] a) Object detection based on a single-stage network model uses the idea of ​​regression to directly regress the category and bounding box of the object on the input image through a convolutional neural network. First, combined with the principle of feature pyramid network, the multi-scale feature maps extracted by the original SSD algorithm are sequentially fused in order of size from small to large; the multi-scale feature maps extracted by the original SSD algorithm from the basic network VGG-16 have sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1. According to the principle of feature pyramid network, the features are fused in order of size from small to large, and 6 feature maps with sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 are obtained. These feature maps contain rich semantic information and detail information.

[0012] b) Combining the principle of attention mechanism, channel attention is introduced to add an attention model to the feature map after feature fusion; the attention model is added to the feature map after feature fusion in 1a). Since the fused 38×38 and 19×19 feature maps contain the richest information, and in order to maintain the real-time performance of the algorithm, the attention model is only added to these two feature maps.

[0013] c) For the 6 multi-scale feature maps obtained in steps a) and b), candidate boxes of different sizes and aspect ratios are set in each unit. The scale of the candidate box is calculated according to the following formula (1):

[0014]

[0015] Among them, m represents the number of feature layers; s k Indicates the ratio of the candidate box to the image; s max and min The maximum and minimum values ​​representing the ratio are 0.9 and 0.2 respectively. The scale of each candidate box is obtained using the above formula (1);

[0016] For the aspect ratio, the general value is The width and height of the candidate box are calculated according to the following formula (2):

[0017]

[0018] For the candidate box with an aspect ratio of 1, a scale is added. The candidate box, the center coordinates of the candidate box are where |f k | represents the size of the feature layer;

[0019] d) Use a 3×3 convolution kernel to detect the category and confidence of the multi-scale feature map through convolution operation, and train the target detection algorithm; the loss function during model training is defined as the weighted sum of the position loss (localization loss, loc) and the confidence loss (conf), and the calculation formula is as follows:

[0020]

[0021] Where N is the number of matched candidate boxes; x∈{1,0} indicates whether the candidate box matches the real box. If it matches, x=1, otherwise x=0; c is the category confidence prediction value; g is the position parameter of the real box; l is the position prediction value of the predicted box; α is the weight coefficient, which is set to 1.

[0022] For the position loss function in SSD, Smooth L1 loss is used to regress the center (cx, cy) and width (w) and height (h) offsets of the candidate box. The formula is as follows:

[0023]

[0024]

[0025]

[0026] For the confidence loss function in SSD, the typical softmax loss is used, and its formula is:

[0027]

[0028] The improved target detection algorithm based on feature pyramid network and attention mechanism of the present invention is based on the single-stage target detection algorithm SSD algorithm. Taking into account the influence of feature map resolution on target detection performance, the original algorithm is improved. Combined with the idea of ​​feature pyramid network, the multi-scale feature maps extracted by the original SSD algorithm are fused to form a feature map with rich semantic information and rich detail information; combined with the principle of attention mechanism, an attention model is added to the two feature maps with sizes of 38×38 and 19×19 after fusion to enhance the recognition effect of small target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the network structure of the target detection algorithm that combines the feature pyramid network and the attention mechanism;

[0030] Figure 2 These are comparison pictures of the detection effects of the original SSD algorithm and the improved target detection algorithm. Among them, Figure a1, Figure a2, Figure a3, Figure a4 and Figure a5 on the left are all detection pictures of the original SSD algorithm; Figure b1, Figure b2, Figure b3, Figure b4 and Figure b5 on the right are all detection pictures of the improved target detection algorithm.

[0031] The present invention is further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0032] The improved target detection algorithm based on feature pyramid network and attention mechanism of the present invention adopts the technical idea of ​​analyzing the shortcomings of the SSD algorithm based on the single-stage target detection algorithm SSD, and proposing to improve the SSD target detection algorithm. The principle of the integrated feature pyramid network is used to fuse the six feature maps extracted by the original SSD algorithm to form a new feature map with rich semantic information and detail information; then the attention model is added to the fused feature map, but in order to maintain the real-time performance of the algorithm, the attention model is only added to the 38×38 and 19×19 feature maps that contain the richest information and are more sensitive to small target detection. By improving the algorithm, the detection capability of the target detection algorithm can be improved and problems such as missed detection can be improved.

[0033] This embodiment provides an improved target detection algorithm based on a feature pyramid network and an attention mechanism, including the following steps:

[0034] Step 1) Combining the principle of feature pyramid network, the basic network VGG-16 in the original SSD algorithm extracts 6 multi-scale feature maps of the input image, and performs feature fusion in order from small to large; obtains feature maps of different layers, and the fused feature maps contain rich semantic information and detail information at the same time;

[0035] Among them, in the original SSD algorithm, the scale of the feature map extracted from the input image by the basic network VGG-16 decreases from large to small, wherein the bottom feature map has a larger resolution and contains more detailed information, and the high-level feature map has a smaller resolution and contains more abstract semantic information. Therefore, the original SSD algorithm uses the bottom feature map to detect small targets, and the high-level feature map to detect medium and large targets;

[0036] Step 2) introduces the channel attention mechanism, and adds an attention model to the two feature maps that have richer detail information and semantic information after feature fusion and are more sensitive to small target detection; that is, the attention mechanism is implemented by adding a mask to the feature map, and the features of the region of interest are identified. Through continuous training of the network, the network learns the regions of interest that need to be focused on in each image, suppresses the influence of other interference areas, and thus enhances the algorithm's detection ability for small target objects.

[0037] In step 1), the size of the input image is 300×300, and the sizes of the feature maps extracted by the basic network VGG-16 are 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 respectively. Combined with the idea of ​​the feature pyramid network, the 6 extracted feature maps are fused in ascending order of size, i.e., 1×1 and 3×3, 3×3 and 5×5, 5×5 and 10×10, 10×10 and 19×19, and 19×19 and 38×38. The sizes of the fused feature maps are still 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1.

[0038] In step 2), combined with the principle of attention mechanism, an attention model is added to the fused feature map. Since the 38×38 and 19×19 feature maps after feature fusion contain the richest information, and in order to maintain the real-time performance of the detection algorithm and reduce the amount of calculation, the attention model is only added to these two feature maps. After adding the attention model, the extraction of features of small target objects can be enhanced.

[0039] The detection process of the improved target detection algorithm is as follows:

[0040] a) Object detection based on a single-stage network model uses the idea of ​​regression to directly regress the category and bounding box of the object on the input image through a convolutional neural network. First, combined with the principle of feature pyramid network, the multi-scale feature maps extracted by the original SSD algorithm are sequentially fused in order of size from small to large; the multi-scale feature maps extracted by the original SSD algorithm from the basic network VGG-16 have sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1, respectively. According to the principle of feature pyramid network, feature fusion is performed in order of size from small to large. Take the two feature maps (1, 1) and (3, 3) as examples:

[0041] First, the feature map of size (1, 1) is upsampled. Based on the original image pixels, a suitable interpolation algorithm is used to insert new elements between pixels to expand the size of the feature map, so that the size of the feature map is consistent with that of the (3, 3) feature map after expansion. Then, a 1*1 convolution operation is performed on the (3, 3) feature map to change its number of channels so that the number of channels is the same as that of the feature map obtained after upsampling. Finally, feature fusion is performed. After fusion, a 3*3 convolution kernel is used to perform a convolution operation on the fused feature map to eliminate the aliasing effect of upsampling. The fusion between other adjacent feature maps is consistent with the above method. The fusion results in 6 feature maps of sizes 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1. These feature maps contain rich semantic information and detail information.

[0042] b) Combining the principle of attention mechanism, channel attention is introduced to add attention model to the feature map after feature fusion; the attention model is added to the feature map after feature fusion in step a). Since the fused 38×38 and 19×19 feature maps contain the richest information, and in order to maintain the real-time performance of the algorithm, the attention model is only added to these two feature maps. The process of adding the attention model is divided into three steps: squeezing, excitation, and attention.

[0043] The formula for the extrusion operation is as follows:

[0044]

[0045] Among them, H and W represent the height and width of the input respectively, U represents the input, Y represents the output, and C is the number of input channels;

[0046] The function of formula (1) is to convert the H*W*C input into the 1*1*C output, which is equivalent to performing a global average pooling operation.

[0047] The formula for the incentive operation is as follows:

[0048] S=h-Swish(W2×ReLU6(W1Y))(2)

[0049] Among them, Y represents the output of the squeeze operation, S represents the output of the excitation operation, the dimension of W1 is C / r*C, the dimension of W2 is C*C / r, r is a scaling parameter, and the value of this paper is 4. W1 multiplied by Y represents a full connection operation, and then passes through the ReLU6 activation function; then multiplied by W2, it also represents a full connection operation, and finally passes through the hard-Swish activation function to complete the excitation operation. The formulas of ReLU6 and hard-Swish activation functions are shown in equation (3).

[0050]

[0051] The operation of Attention is shown as follows:

[0052] X=S×U (4)

[0053] In the formula, X represents the feature map after adding the attention mechanism, U represents the original input, S represents the output of the excitation operation, and the weight of each feature map is multiplied by the feature of the feature map.

[0054] c) For the 6 multi-scale feature maps obtained in step a) and step b), candidate boxes of different sizes and aspect ratios must be set in each unit. The scale of the candidate box is calculated according to the following formula:

[0055]

[0056] Among them, m represents the number of feature layers; s k Indicates the ratio of the candidate box to the image; s max and min represents the maximum and minimum values ​​of the ratio, which are 0.9 and 0.2 respectively;

[0057] Use the above formula (5) to get the scale of each candidate box;

[0058] For the aspect ratio, the general value is The width and height of the candidate box are calculated according to the following formula (6):

[0059]

[0060] For the candidate box with an aspect ratio of 1, a scale is added. The candidate box, the center coordinates of the candidate box are where |f k | represents the size of the feature layer;

[0061] d) Use a 3×3 convolution kernel to detect the category and confidence of the multi-scale feature map through convolution operation, and train the target detection algorithm; the loss function during model training is defined as the weighted sum of the position loss (localization loss, loc) and the confidence loss (conf), and the calculation formula is as follows:

[0062]

[0063] Where N is the number of matched candidate boxes; x∈{1,0} indicates whether the candidate box matches the real box. If it matches, x=1, otherwise x=0; c is the category confidence prediction value; g is the position parameter of the real box; l is the position prediction value of the predicted box; α is the weight coefficient, which is set to 1.

[0064] For the position loss function in SSD, Smooth L1 loss is used to regress the center (cx, cy) and width (w) and height (h) offsets of the candidate box. The formula is as follows:

[0065]

[0066]

[0067]

[0068] For the confidence loss function in SSD, the typical softmax loss is used, and its formula is:

[0069]

[0070] Then the improved target detection algorithm model is trained.

[0071] In this embodiment, the PASCAL VOC2007 dataset and the PASCAL VOC2012 dataset are used as training sets for model training, and data augmentation technology is used to expand the training set images by performing operations such as horizontal flipping and random cropping on the datasets.

[0072] The data used in the experiment: PASCAL VOC dataset is a standardized dataset for image recognition and classification. The dataset contains 20 categories, namely people, birds, cats, cows, dogs, horses, sheep, airplanes, bicycles, boats, buses, cars, motorcycles, trains, bottles, chairs, dining tables, potted plants, sofas, and televisions.

[0073] This embodiment uses the above-mentioned VOC2007 dataset and VOC2012 dataset for training, and uses the VOC2007 dataset for testing. Stochastic gradient descent (SGD) is used for training, batch size is set to 32, initial learning rate is set to 0.001, momentum parameter momentum is set to 0.9, learning rate is reduced by 90% when the number of iterations is 100000 and 150000, and a total of 200000 training times are performed.

[0074] In order to verify the detection effect of the improved target detection algorithm based on the single-stage network model of this embodiment, the applicant selected the test set in the PASCAL VOC2007 data set for detection, and used mAP (mean Average Precision) as the evaluation index of the detection algorithm. Each detected category will obtain a curve composed of precision and recall, that is, a PR curve. The area under the curve is the AP value. The AP values ​​of all detected categories are averaged to obtain the mAP value. The detection effect is compared with other mainstream target detection models from both subjective and objective aspects (see Table 1 and Table 2).

[0075] Table 1

[0076]

[0077] Table 2

[0078]

[0079]

[0080] In the subjective evaluation of the detection effect, the original SSD algorithm and the improved detection algorithm are compared (such as Figure 2 As shown in the figure, a1, a2, a3, a4, and a5 are all pictures detected by the original SSD algorithm; b1, b2, b3, b4, and b5 are all pictures detected by the target detection algorithm). As can be seen from the figure, the improved target detection algorithm has significantly improved the problems of missed detection in the original algorithm compared with the original SSD algorithm, and has better detection capabilities for densely distributed small targets, and can detect more targets. The detection effect has been significantly improved compared with the original SSD algorithm.

Claims

1. An improved target detection algorithm based on feature pyramid network and attention mechanism, characterized in that: The following steps are involved: Step 1) Combining the principle of feature pyramid network, the six multi-scale feature maps extracted by the basic network VGG-16 from the input image in the original SSD algorithm are fused in order from small to large; the feature maps of different layers are fused, and the fused feature maps contain rich semantic information and detail information at the same time; Among them, in the original SSD algorithm, the scale of the feature map extracted from the input image by the basic network VGG-16 decreases from large to small, wherein the bottom feature map has a larger resolution and contains more detailed information, and the high-level feature map has a smaller resolution and contains more abstract semantic information. Therefore, the original SSD algorithm uses the bottom feature map to detect small targets, and the high-level feature map to detect medium and large targets; Step 2) introduce the channel attention mechanism, add the attention model to the two feature maps that have richer detail information and semantic information after feature fusion and are more sensitive to small target detection; that is, implement the attention mechanism by adding a mask to the feature map, identify the features of the region of interest, and through continuous network training, let the network learn the region of interest that needs to be focused on in each image, suppress the influence of other interference areas, thereby enhancing the algorithm's detection ability for small target objects; In step 2), the attention model is added to the feature map fused according to the feature pyramid principle in step 1. Because the fusion process is carried out in the order of feature map size from small to large, the feature maps with the richest information after fusion are (38, 38) and (19, 19). Compared with other feature maps, these two feature maps have richer detail information and semantic information, and are more sensitive to small target detection. In order to maintain the detection speed of the algorithm and reduce the calculation amount of the algorithm, the attention model is only added to the two feature maps (38, 38) and (19, 19) after fusion. The detection process of the target detection algorithm is as follows: a) Based on the single-stage network model, target detection uses the idea of ​​regression to directly regress the target category and bounding box on the input image through a convolutional neural network. First, combined with the principle of feature pyramid network, the multi-scale feature maps extracted by the original SSD algorithm are sequentially fused in order of size from small to large. The original SSD algorithm extracts multi-scale feature maps of the input image from the basic network VGG-16, with sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1. According to the principle of feature pyramid network, the features are fused in order of size from small to large, and 6 feature maps with sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 are obtained. These feature maps contain rich semantic information and detail information. b) Combining the principle of attention mechanism, channel attention is introduced to add an attention model to the feature map after feature fusion; the attention model is added to the feature map after feature fusion in 1a). Since the fused 38×38 and 19×19 feature maps contain the richest information, and in order to maintain the real-time performance of the algorithm, the attention model is only added to these two feature maps; c) For the 6 multi-scale feature maps obtained in steps a) and b), candidate boxes of different sizes and aspect ratios are set in each unit. The scale of the candidate box is calculated according to the following formula (1): Among them, m represents the number of feature layers; s k Indicates the ratio of the candidate box to the image; s max and min represents the maximum and minimum values ​​of the ratio, which are 0.9 and 0.2 respectively; Use the above formula (1) to get the scale of each candidate box; For the aspect ratio, the value is The width and height of the candidate box are calculated according to the following formula (2): For the candidate box with an aspect ratio of 1, a scale is added. The candidate box, the center coordinates of the candidate box are where |f k | represents the size of the feature layer; d) Use a 3×3 convolution kernel to detect the category and confidence of the multi-scale feature map through convolution operation, and train the target detection algorithm; the loss function during model training is defined as the weighted sum of position loss and confidence loss, and the calculation formula is as follows: Where N is the number of matched candidate boxes; x∈{1,0} indicates whether the candidate box matches the real box. If it matches, x=1, otherwise x=0; c is the category confidence prediction value; g is the position parameter of the real box; l is the position prediction value of the predicted box; α is the weight coefficient, which is set to 1; L conf is the confidence loss; L loc is the position loss; For the position loss function in SSD, Smooth L1 loss is used to regress the center (cx, cy) of the candidate box and the offset of width w and height h; the formula is as follows: For the confidence loss function in SSD, the typical softmax loss is used, and its formula is:

2. The algorithm according to claim 1, characterized in that The input image size in step 1) is 300×300, and the feature map sizes obtained after the basic network VGG-16 for detection are 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 respectively; According to the principle of feature pyramid network, the feature maps used for detection are fused in order from small to large size, and 6 feature maps with sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1 are obtained.

Images