Weak and small target recognition method based on MS-CAM improved feature fusion module

Abstract

A weak small target recognition method based on an MS-CAM improved feature fusion module, comprising: constructing a data set; constructing a weak small target recognition model; training the weak small target recognition model; inputting a test set into the trained weak small target recognition model for weak small target recognition, and outputting a target category, a bounding box and a confidence degree; the target recognition algorithm based on YOLOv8 effectively alleviates the feature fusion problem of inconsistent semantics and scales by integrating the feature fusion module CCF of the multi-scale attention channel module MS-CAM, improves the accuracy of weak small target recognition, and adds an attention mechanism in the initial feature fusion stage of the MS-CAM module in different receptive fields in an iterative manner to fuse information, which is helpful to feature extraction of the model for weak small targets and improves the recognition accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target recognition of computer vision, and particularly relates to a weak and small target recognition method. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology and computer vision, unmanned aerial vehicles deploying target recognition algorithms gradually become the main carriers of intelligent traffic systems (ITS). The pictures transmitted by unmanned aerial vehicles generally have the characteristics of wide field of view and rich information, and can provide a large amount of information at a high-altitude perspective. However, in the case of constant image size, as the perspective is pulled up, the number of pixels occupied by the target in the image gradually decreases, accompanied by image blurring and target deformation caused by the displacement and jitter of the unmanned aerial vehicle, greatly increasing the difficulty of processing the unmanned aerial vehicle image. Therefore, how to process the weak and small target image has become an important research topic in the field of computer vision.

[0003] As a single-stage target recognition method, the YOLO (You Only Look Once) algorithm has high accuracy and recognition rate, and plays an important role in real-time application scenarios. YOLOv1 first abandoned the generation of candidate frames in the entire YOLO series, and directly predicted objects through precise grids or anchor frames, only performed one forward propagation, and directly output the detection results of all targets. Although the positioning accuracy is relatively insufficient, it provides a novel direction for the development of subsequent target recognition algorithms. With the rise and development of deep learning, the YOLO algorithm has begun to try to introduce multi-scale feature fusion, adjust the network structure, optimize the residual module, and improve the loss function and other upgrading strategies, gradually improving the detection speed and accuracy of the target. Among them, YOLOv8 is developed by Ultralytics, which not only makes major breakthroughs in speed and accuracy, but also greatly meets the needs of industrialization and life scenarios, and becomes a popular basic network for target recognition tasks in computer vision at present.

[0004] However, YOLOv8 still faces problems such as insufficient feature fusion capability and inaccurate boundary frame positioning in the weak and small target recognition task, making it difficult to guarantee the accuracy and speed of weak and small target recognition. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide a weak and small target recognition method based on an MS-CAM improved feature fusion module with high recognition accuracy and high accuracy.

[0006] The technical solution adopted to solve the above technical problems is: a weak and small target recognition method based on an MS-CAM improved feature fusion module, comprising the following steps:

[0007] Step 1. Constructing a data set

[0008] An unmanned aerial vehicle aerial vehicle pedestrian image data set is obtained, the data set is re-labeled, color enhanced and mirrored, and the data set is divided into a training set and a test set according to a proportion;

[0009] Step 2. Constructing a weak and small target recognition model

[0010] A weak and small target recognition model is obtained by replacing the C2f module of the feature extraction network Backbone in the YOLOv8 basic model with a feature fusion module CCF;

[0011] The feature fusion module CCF is sequentially connected by a first convolutional block, a feature decomposition point Split module, a bottleneck module and a second convolutional block; the first convolutional block is used to perform preliminary feature extraction on the input image to obtain a feature map; the feature decomposition point Split module is used to split the feature map into two parts, one part is input into the bottleneck module for deep processing, and the other part is directly spliced with the feature map output by the bottleneck module; the bottleneck module includes 3-6 Bottleneck layers connected in a stacking manner, which gradually extracts high-level feature information in the input feature map by reducing the spatial dimension and increasing the number of channels; the second convolutional block is used to perform the last convolution on the input feature map and compress it to output a feature map with a target number of channels;

[0012] The Bottleneck layer is sequentially connected by a 1x1 convolutional block, a 3x3 convolutional block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM and a residual connection unit;

[0013] The 1x1 convolutional block is used to reduce the number of channels of the initial input feature map;

[0014] The 3x3 convolutional block is used to extract local spatial features of the feature map output by the 1x1 convolutional block;

[0015] The first multi-scale attention module MS-CAM is used to extract multi-scale features from the input feature map, calculate the weight of each feature channel, emphasize important features and suppress unimportant features, and focus on global context aggregation;

[0016] The second multi-scale attention module MS-CAM is used to further calculate the channel weight of the feature processed by the first multi-scale attention module MS-CAM, helping the model to distinguish different attributes and details of the target;

[0017] The residual connection unit is configured to fuse the initial input feature and the feature output by the second multi-scale attention module, so that the features of the shallow layer and the deep layer can be shared and reused.

[0018] Step 3. Training the weak and small target recognition model

[0019] The weak and small target recognition model is trained using the training set, a loss function Inner-IoU is used to introduce auxiliary boxes of different scales for calculation, and the intersection over union of the real box and the auxiliary box of the predicted box is calculated to enhance the attention to small regions.

[0020] Step 4. Inputting the test set into the trained weak and small target recognition model to recognize the weak and small target, and outputting the target category, bounding box and confidence.

[0021] As a preferred technical solution, the first multi-scale attention module MS-CAM includes a global channel, a local channel and an activation function Sigmoid; the global channel calculates global channel information G of the input feature map F / / , the local channel calculates local channel information L of the input feature map F / , and the activation function Sigmoid combines the global channel information G / / and the local channel information L / to obtain the attention weight of each channel and apply it to the original input feature map F for channel weighting to obtain the attention-modulated feature map F / ; the second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

[0022] As a preferred technical solution, the step of calculating the global channel information G of the input feature map F / / is as follows:

[0023] S1: globally average-pooling the input feature map F according to the following formula

[0024]

[0025] In the formula, G is the global channel feature, the size is 1x1xC, C is the number of channels of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F.

[0026] S2: The global average-pooling result is first reduced in dimension by two fully connected layers, then increased in dimension by using the ReLU function, and finally normalized by batch processing to obtain the global channel information G / /

[0027]

[0028] In the formula, is a dimension reduction layer, and the kernel size is , r is a channel reduction ratio, is a dimension increase layer, and the kernel size is , is a linear function ReLU, and B is batch normalization.

[0029] As a preferred technical solution, the formula for calculating the local channel information L / of the input feature map F is:

[0030]

[0031] In the formula, is a point convolution operation, and the kernel sizes are and , respectively.

[0032] The beneficial effects of the present application are as follows:

[0033] The target recognition algorithm based on YOLOv8 of the present application effectively alleviates the feature fusion problem of inconsistent semantics and scales by integrating the feature fusion module CCF of the multi-scale attention channel module MS-CAM, improves the accuracy of weak and small target recognition, adds an attention mechanism in the initial feature fusion stage of MS-CAM module in different receptive fields, and iteratively fuses information, which is helpful for the model to extract features of weak and small targets and improve recognition accuracy.

[0034] The present application uses Inner-IoU as a regression loss function, introduces auxiliary boxes of different scales to calculate loss, more accurately evaluates the positioning error of weak and small targets, and reduces the missed detection rate and false detection rate.

[0035] The present application is suitable for processing weak and small targets in unmanned aerial vehicle images, and can improve the processing difficulty even in the case of unchanged image size, high viewing angle, reduced target pixel points, image blur and target deformation caused by unmanned aerial vehicle displacement and jitter. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a network structure schematic diagram of the weak and small target recognition model of the present application.

[0037] Figure 2 is a structure schematic diagram of the feature fusion module CCF.

[0038] Figure 3 is a structure schematic diagram of the Bottleneck layer.

[0039] Figure 4 is a structure schematic diagram of the first multi-scale attention module MS-CAM.

[0040] Figure 5 is a loss function Inner-IoU box diagram structure schematic diagram. DETAILED DESCRIPTION

[0041] The application will be further described in detail below in combination with the drawings and examples, but the application is not limited to the following embodiments.

[0042] The weak small target recognition method of the embodiment based on the MS-CAM improved feature fusion module comprises the following steps:

[0043] Step 1. Constructing a data set

[0044] An unmanned aerial vehicle aerial vehicle pedestrian image data set is obtained, and the Visdrone2019 data set is used in the embodiment. The data set is collected by the AISKYEYE team of the machine learning and data mining laboratory of Tianjin University using different unmanned aerial vehicle models, and contains vehicle and pedestrian images in different environments, different illuminations and different times. The data set is re-labeled, color enhanced and mirror operated to obtain 9275 images, and the image size is 640*640. In the labeling process, only four classes of car, pedestrian, truck and bus are labeled. The data set is divided into a training set, a test set and a validation set according to the ratio of 10:1:1.

[0045] Step 2. Constructing a weak small target recognition model

[0046] The feature fusion module CCF is used to replace the C2f module of the feature extraction network Backbone in the YOLOv8 basic model to obtain a weak small target recognition model, as shown in Figure 1 .

[0047] The feature fusion module CCF is composed of a first convolutional block, a feature decomposition point Split module, a bottleneck module and a second convolutional block connected in sequence, as shown in Figure 2 .

[0048] The first convolutional block is used to preliminarily extract features of the input image to obtain a feature map. The feature decomposition point Split module is used to split the feature map into two parts. One part is input into the bottleneck module for deep processing, and the other part is directly spliced with the feature map output by the bottleneck module. The bottleneck module includes 3-6 Bottleneck layers connected in a stacking manner. By reducing the spatial dimension and increasing the number of channels of the input feature map, high-level feature information in the input feature map is gradually extracted. The second convolutional block is used to perform the last convolution on the input feature map and compress it to output a feature map with a target channel number.

[0049] In Figure 3In the embodiment, the bottleneck layer is composed of a 1x1 convolution block, a 3x3 convolution block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM, and a residual connection unit connected in sequence. The 1x1 convolution block is used to reduce the channel number of the initial input feature map; the 3x3 convolution block is used to extract the local spatial features of the feature map output by the 1x1 convolution block; the first multi-scale attention module MS-CAM is used to extract multi-scale features of the input feature map, calculate the weight of each feature channel, emphasize important features and suppress unimportant features, and focus on global context aggregation; and the second multi-scale attention module MS-CAM is used to further calculate the channel weight in detail on the features processed by the first multi-scale attention module MS-CAM, to help the model distinguish different attributes and details of the target.

[0050] The first multi-scale attention module MS-CAM includes a global channel, a local channel, and a Sigmoid activation function, as shown in the following formula: Figure 4 The global channel calculates the global channel information G of the input feature map F according to the following formula: / / The specific steps are as follows:

[0051] S1: Perform global average pooling on the input feature map F according to the following formula:

[0052]

[0053] In the formula, G is the global channel feature with a size of 1x1xC, C is the channel number of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F.

[0054] S2: The global average pooling result is first reduced in dimension by two fully connected layers, then increased in dimension by using a ReLU function, and finally normalized by batch processing to obtain the global channel information G / /

[0055]

[0056] In the formula, is a dimension reduction layer with a kernel size of , r is a channel reduction ratio, is a dimension increasing layer with a kernel size of , is a linear function ReLU, and B is batch normalization. The linear function ReLU increases the nonlinear ability of the model, thereby solving the gradient vanishing problem. Batch normalization reduces the internal covariate shift, so that the input data distribution of each layer in the model is more stable.

[0057] The local channel calculates the local channel information L of the input feature map F according to the following formula: / ​

[0058]

[0059] In the formula, is a point convolution operation, and the kernel sizes are and .

[0060] The activation function Sigmoid combines the global channel information G / / and the local channel information L / to obtain the attention weight of each channel and applies it to the original input feature map F to perform channel weighting, obtaining the feature map F / after attention modulation, specifically:

[0061]

[0062] In the formula, () is a sigmoid activation function, represents a broadcast addition operation, represents multiplication of corresponding elements of two feature maps.

[0063] The second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

[0064] The multi-scale attention distribution mechanism described above captures image features at different spatial scales, effectively improving the recognition ability of small weak targets. By assigning weights to the importance of each channel, the response of important feature channels for small weak target recognition is strengthened, thereby improving the detection accuracy.

[0065] The residual connection unit is used to fuse the initial input feature and the feature output by the second multi-scale attention module, so that the shallow and deep features can be shared and reused, enhancing the representation ability of the model.

[0066] Step 3. Training a small weak target recognition model

[0067] The small weak target recognition model is trained using the training set, and the loss function Inner-IoU is used to introduce auxiliary boxes of different scales for calculation. The intersection over union is calculated between the real box and the predicted box, and the attention to small regions is enhanced.

[0068] Wherein, the training parameters are set as follows: the batch size batch is set to 16, the training rounds epochs are set to 300, the early stopping rounds are set to 50, the stochastic gradient descent SGD is selected as the optimizer, and the learning rate is set to 0.01.

[0069] The calculation steps of the loss function Inner-IoU are as follows:

[0070] A1: define the upper, lower, left and right boundaries of the real bounding box GT and the anchor box Anchor, as follows Figure 5 ;

[0071]

[0072]

[0073]

[0074]

[0075] In the formula, , , , The upper, lower, left and right boundaries of the real bounding box GT are in turn, is the vertical coordinate of the center of the real bounding box GT, is the horizontal coordinate of the center of the real bounding box GT, , is the width and height of the real bounding box GT, and ratio is the scaling factor, , , , The upper, lower, left and right boundaries of the anchor box Anchor are in turn, is the vertical coordinate of the center of the anchor box Anchor, is the horizontal coordinate of the center of the anchor box Anchor, and w, h are the width and height of the anchor box Anchor.

[0076] A2: find the overlapping boundaries of the real bounding box GT and the anchor box Anchor in the horizontal and vertical directions, and calculate the intersection area in of the two boxes according to the following formula:

[0077]

[0078] In the formula, min() is the minimum value, and max() is the maximum value.

[0079] A3: get the union area of the real bounding box GT and the anchor box Anchor according to the following formula:

[0080]

[0081] A4: get the value of the loss function Inner-IoU by the intersection area in and the union area union of the real bounding box GT and the anchor box Anchor according to the following formula ,

[0082]

[0083] Step 4. Input the test set into the trained weak and small target recognition model for weak and small target recognition, and output the target category, bounding box and confidence.

[0084] Experiment

[0085] In order to verify the beneficial effects of the present application, the inventors respectively used the YOLOv8 benchmark model (model 1), the YOLOv8 model with the introduction of the CCF module (model 2), and the weak and small target recognition model of the present application (model 3) on the validation set of the present embodiment to perform an ablation experiment, and the evaluation indexes obtained are as shown in Table 1.

[0086] Table 1. Evaluation indexes of three models

[0087]

[0088] In the table, P(%) represents the tracking accuracy, R(%) represents the recall rate, and mAP@50 represents the average precision of the IOU value between the predicted bounding box of each category and the real bounding box being greater than 0.5. The higher the values of the above evaluation indexes, the better the performance of the model.

[0089] From the experimental result data, it can be known that in the small target recognition task, the three evaluation indexes of the YOLOv8 as a basic model are respectively 68.1%, 51.1% and 0.569. The YOLOv8 with the introduction of the CCF module performs better than the benchmark model due to the enhanced extraction ability of weak and small targets, but the regression loss is sensitive to the deviation of the bounding box in the calculation process, and the convergence effect on the weak and small target features is not particularly ideal. The weak and small target recognition model of the present application is improved on the basis of the model of experiment 2, and the loss function is replaced by Inner-IoU, and the accuracy and mAP@.5 are further improved to 72.2% and 60.0%, which fully shows the effectiveness of the present application method for the vehicle and pedestrian recognition task in the unmanned aerial vehicle intelligent traffic scene.

Claims

1. A weak and small target recognition method based on MS-CAM improved feature fusion module, characterized in that, Comprising the following steps: Step 1. Constructing a data set Obtain an unmanned aerial vehicle aerial vehicle pedestrian image data set, re-label, color enhance and mirror the data set, and divide the data set into a training set and a test set according to a proportion; Step 2. Constructing a weak and small target recognition model Use a feature fusion module CCF to replace the C2f module of the feature extraction network Backbone in the YOLOv8 basic model to obtain a weak and small target recognition model; The feature fusion module CCF is sequentially connected by a first convolutional block, a feature decomposition point Split module, a bottleneck module and a second convolutional block; the first convolutional block is used to preliminarily extract features of an input image to obtain a feature map; the feature decomposition point Split module is used to split the feature map into two parts, one part is input into the bottleneck module for deep processing, and the other part is directly spliced with the feature map output by the bottleneck module; the bottleneck module includes 3-6 Bottleneck layers connected in a stacking manner, which gradually extracts high-level feature information in the input feature map by reducing the spatial dimension and increasing the number of channels; the second convolutional block is used to perform the last convolution on the input feature map and compress it to output a feature map with a target number of channels; The Bottleneck layer is sequentially connected by a 1x1 convolutional block, a 3x3 convolutional block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM and a residual connection unit; The 1x1 convolutional block is used to reduce the number of channels of the initial input feature map; The 3x3 convolutional block is used to extract local spatial features of the feature map output by the 1x1 convolutional block; The first multi-scale attention module MS-CAM is used to extract multi-scale features of the input feature map, calculate the weight of each feature channel, emphasize important features and suppress unimportant features, and focus on global context aggregation; The second multi-scale attention module MS-CAM is used to further calculate the channel weight of the feature processed by the first multi-scale attention module MS-CAM, helping the model to distinguish different attributes and details of the target; The residual connection unit is used to fuse the initial input feature and the feature output by the second multi-scale attention module, so that the shallow and deep features can be shared and reused; Step 3. Training the weak and small target recognition model The training set is used to train the weak and small target recognition model, and the loss function Inner-IoU is used to introduce auxiliary boxes of different scales for calculation, and the intersection over union of the real box and the predicted box is calculated through the auxiliary box, so as to enhance the attention to small areas; Step 4. Input the test set into the trained weak and small target recognition model for weak and small target recognition, and output the target category, bounding box and confidence.

2. The weak and small target recognition method based on the MS-CAM improved feature fusion module of claim 1, characterized in that: The first multi-scale attention module MS-CAM includes a global channel, a local channel and an activation function Sigmoid; the global channel calculates global channel information G of an input feature map F / / ; the local channel calculates local channel information L of the input feature map F / ; the activation function Sigmoid combines the global channel information G / / and the local channel information L / to obtain attention weights of each channel and apply the attention weights to the original input feature map F for channel weighting, so as to obtain a feature map F / after attention modulation; the second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

3. The weak and small target recognition method based on MS-CAM improved feature fusion module of claim 2, characterized in that: the step of calculating global channel information G of the input feature map F / / is: S1: Perform global average pooling on the input feature map F according to the following formula In the formula, G is a global channel feature with a size of 1x1xC, C is the number of channels of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F; S2: The global average pooling result is first reduced in dimension by two fully connected layers, then increased in dimension by using a ReLU function, and finally batch normalization is performed to obtain global channel information G / / wherein, is a dimension reduction layer with kernel size , r is the channel reduction ratio, is a dimension increase layer with kernel size , is a linear function ReLU, B is batch normalization.

4. The weak and small target recognition method based on the MS-CAM improved feature fusion module of claim 2, characterized in that: The local channel information L of the calculation input feature map F / The formula is: In the formula, is a point convolution operation, and the kernel sizes are and .

Images