Weak and small target identification method based on MS-CAM improved feature fusion module

Abstract

The invention discloses a weak and small target identification method based on an MS-CAM improved feature fusion module. The method comprises the following steps: constructing a data set; constructing a weak and small target recognition model; training a weak and small target recognition model; and inputting the test set into the trained weak and small target recognition model for weak and small target recognition, and outputting a target category, a bounding box and confidence. According to the target recognition algorithm based on YOLOv8, the feature fusion problem of semantic and scale inconsistency is effectively relieved by integrating a feature fusion module CCF of a multi-scale attention channel module MS-CAM, the accuracy of weak and small target recognition is improved, a layer of attention mechanism is added in the initial feature fusion stage of different receptive fields by using the MS-CAM module, and the recognition accuracy of weak and small targets is improved. Information is fused in an iteration mode, feature extraction of the model on weak and small targets is facilitated, and the recognition accuracy is improved.

Description

Technical Field

[0001] The invention belongs to the field of target recognition in computer vision, and in particular relates to a method for recognizing weak targets. Background Art

[0002] With the rapid development of drone technology and computer vision, drones equipped with target recognition algorithms have gradually become the main carrier of the Intelligent Traffic System (ITS). The images transmitted by drones generally have the characteristics of wide field of view and rich information, and can provide a large amount of information from a high camera angle. However, when the image size remains unchanged, as the angle of view increases, the pixel points occupied by the target in the image gradually decrease. At the same time, the image blur and target deformation caused by the displacement and jitter of the drone greatly increase the difficulty of processing drone images. Therefore, how to process weak target images has become an important research topic in the field of computer vision.

[0003] As a single-stage target recognition method, the YOLO (You Only Look Once) algorithm has both high accuracy and recognition rate, and plays an important role in real-time application scenarios. For the first time in the entire YOLO series, YOLOv1 abandons candidate box generation and directly predicts objects through precise grids or anchor boxes, performs only one forward propagation, and directly outputs the detection results of all targets. Although the positioning accuracy is relatively insufficient, it provides a novel direction for the development of subsequent target recognition algorithms. With the rise and development of deep learning, the YOLO algorithm has kept pace with the times and has begun to try to introduce many upgrade strategies such as multi-scale feature fusion, adjusting network structure, optimizing residual modules, and improving loss functions, gradually improving the detection speed and accuracy of targets. Among them, YOLOv8 was developed by Ultralytics. It not only made major breakthroughs in speed and accuracy, but also can meet the needs of industrialization and life scenarios to a great extent, becoming a popular basic network for target recognition tasks in current computer vision.

[0004] However, YOLOv8 still faces the problems of insufficient feature fusion capability and inaccurate bounding box positioning in the task of weak and small target recognition, making it difficult to ensure the accuracy and speed of weak and small target recognition. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a small target recognition method based on an improved feature fusion module of MS-CAM with high recognition precision and accuracy.

[0006] The technical solution adopted to solve the above technical problems is: a small target recognition method based on MS-CAM improved feature fusion module, comprising the following steps:

[0007] Step 1. Build the dataset

[0008] Obtain a dataset of vehicle and pedestrian images taken by drones, relabel, color enhance, and mirror the dataset, and divide the dataset into a training set and a test set according to the ratio;

[0009] Step 2. Build a small target recognition model

[0010] Use the feature fusion module CCF to replace the C2f module of the feature extraction network Backbone in the YOLOv8 basic model to obtain a weak target recognition model;

[0011] The feature fusion module CCF is composed of a first convolution block, a feature decomposition point Split module, a bottleneck module, and a second convolution block connected in sequence; the first convolution block is used to perform preliminary feature extraction on the input image to obtain a feature map; the feature decomposition point Split module is used to divide the feature map into two parts, one part is input to the bottleneck module for deep processing, and the other part is directly spliced ​​with the feature map output by the bottleneck module; the bottleneck module includes 3-6 layers of Bottleneck layers connected in a stacked manner, and the high-level feature information in the input feature map is gradually extracted by reducing the spatial dimension of the input feature map and increasing the number of channels; the second convolution block is used to perform the last convolution on the input feature map and compress it, and output the feature map of the target number of channels;

[0012] The Bottleneck layer is composed of a 1×1 convolution block, a 3×3 convolution block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM, and a residual connection unit connected in sequence;

[0013] The 1×1 convolution block is used to reduce the number of channels of the initial input feature map;

[0014] The 3×3 convolution block is used to extract local spatial features of the feature map output by the 1×1 convolution block;

[0015] The first multi-scale attention module MS-CAM is used to perform multi-scale feature extraction on the input feature map, calculate the weight of each feature channel, emphasize important features and suppress unimportant features, and focus on global context aggregation;

[0016] The second multi-scale attention module MS-CAM is used to further perform refined channel weight calculation on the features processed by the first multi-scale attention module MS-CAM to help the model distinguish different attributes and details of the target;

[0017] The residual connection unit is used to fuse the initial input features with the features output by the second multi-scale attention module, so that the shallow and deep features can be shared and reused;

[0018] Step 3. Train the small target recognition model

[0019] Use the training set to train the small target recognition model, use the loss function Inner-IoU to introduce auxiliary boxes of different scales for calculation, and calculate the intersection-over-union ratio between the real box and the auxiliary box of the predicted box to enhance the focus on small areas;

[0020] Step 4. Input the test set into the trained small target recognition model for small target recognition and output the target category, bounding box, and confidence level.

[0021] As a preferred technical disclosure, the first multi-scale attention module MS-CAM includes a global channel, a local channel and an activation function Sigmoid; the global channel calculates the global channel information G of the input feature map F / / , the local channel, calculates the local channel information L of the input feature map F / , the activation function Sigmoid, the global channel information G / / and local channel information L / Combined, the attention weight of each channel is obtained and applied to the original input feature map F for channel weighting to obtain the feature map F after attention modulation / ; The second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

[0022] As a preferred technical disclosure document, the global channel information G of the input feature map F is calculated / / The steps are:

[0023] S1: Perform global average pooling on the input feature map F according to the following formula

[0024]

[0025] Where G is the global channel feature with a size of 1×1×C, C is the number of channels of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F;

[0026] S2: The global average pooling result is first reduced in dimension through two fully connected layers, then increased in dimension using the ReLU function, and finally batch normalized to obtain the global channel information G / /

[0027] G / / =B(T2δ(B(T1(G))))

[0028] In the formula, T1 is the dimension reduction layer, and the kernel size is r is the channel restoration ratio, T2 is the dimension increase layer, the kernel size is C×1×1, δ is the linear function ReLU, and B is batch normalization.

[0029] As a preferred technical disclosure, the local channel information L of the input feature map F is calculated / The formula is:

[0030] L / =B(PW2δ(B(PW1(F))))

[0031] Where PW1 and PW2 are point convolution operations, and the kernel sizes are and C×H×W.

[0032] The beneficial effects of the present invention are as follows:

[0033] The target recognition algorithm of the present invention is based on YOLOv8. By integrating the feature fusion module CCF of the multi-scale attention channel module MS-CAM, the problem of feature fusion with inconsistent semantics and scales is effectively alleviated, and the accuracy of weak target recognition is improved. The MS-CAM module is used to add a layer of attention mechanism in the initial feature fusion stage of different receptive fields to fuse information in an iterative manner, which is helpful for the model to extract the features of weak targets and improve the recognition accuracy.

[0034] The present invention adopts Inner-IoU as the regression loss function and introduces auxiliary frames of different scales to calculate the loss, so as to more accurately evaluate the positioning error of small targets and reduce the missed detection rate and false detection rate.

[0035] The present invention is particularly suitable for processing small targets in drone images, and can improve the processing difficulty even when the image size remains unchanged, the viewing angle is raised, the pixel points occupied by the target are reduced, and the image is blurred and the target is deformed due to the displacement and shaking of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the network structure of the weak target recognition model of the present invention.

[0037] Figure 2 It is a structural diagram of the feature fusion module CCF.

[0038] Figure 3 It is a structural diagram of the Bottleneck layer.

[0039] Figure 4 It is a structural diagram of the first multi-scale attention module MS-CAM.

[0040] Figure 5It is a schematic diagram of the Inner-IoU block diagram of the loss function. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0042] A small target recognition method based on an improved feature fusion module of MS-CAM in this embodiment includes the following steps:

[0043] Step 1. Build the dataset

[0044] Obtain a drone aerial vehicle and pedestrian image dataset. This embodiment uses the Visdrone2019 dataset, which was collected by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University using different drone models. It contains vehicle and pedestrian images in different environments, different lighting, and different times. The dataset is relabeled, color enhanced, and mirrored to obtain 9275 images with a size of 640×640. Only four categories, car, pedestrian, truck, and bus, are labeled during the labeling process. The dataset is divided into a training set, a test set, and a validation set according to a ratio of 10:1:1.

[0045] Step 2. Build a small target recognition model

[0046] Use the feature fusion module CCF to replace the C2f module of the feature extraction network Backbone in the YOLOv8 basic model to obtain a weak target recognition model, such as Figure 1 ;

[0047] The feature fusion module CCF is composed of the first convolution block, the feature decomposition point Split module, the bottleneck module, and the second convolution block connected in sequence, such as Figure 2 .

[0048] The first convolution block is used to perform preliminary feature extraction on the input image to obtain a feature map; the feature decomposition point Split module is used to divide the feature map into two parts, one part is input to the bottleneck module for deep processing, and the other part is directly spliced ​​with the feature map output by the bottleneck module; the bottleneck module includes 3 to 6 layers of Bottleneck layers connected in a stacked manner, which gradually extracts high-level feature information from the input feature map by reducing the spatial dimension of the input feature map and increasing the number of channels; the second convolution block is used to perform the final convolution and compression on the input feature map, and output the feature map of the target number of channels;

[0049] exist Figure 3In the embodiment, the Bottleneck layer is composed of a 1×1 convolution block, a 3×3 convolution block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM, and a residual connection unit connected in sequence. The 1×1 convolution block is used to reduce the number of channels of the initial input feature map; the 3×3 convolution block is used to extract the local spatial features of the feature map output by the 1×1 convolution block; the first multi-scale attention module MS-CAM is used to perform multi-scale feature extraction on the input feature map, calculate the weight of each feature channel, emphasize important features to suppress unimportant features, and pay attention to global context aggregation; the second multi-scale attention module MS-CAM is used to further perform refined channel weight calculations on the features processed by the first multi-scale attention module MS-CAM to help the model distinguish different attributes and details of the target.

[0050] Among them, the first multi-scale attention module MS-CAM includes a global channel, a local channel and an activation function Sigmoid, such as Figure 4 . Global channel, calculate the global channel information G of the input feature map F / / , the specific steps are:

[0051] S1: Perform global average pooling on the input feature map F according to the following formula

[0052]

[0053] Where G is the global channel feature with a size of 1×1×C, C is the number of channels of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F;

[0054] S2: The global average pooling result is first reduced in dimension through two fully connected layers, then increased in dimension using the ReLU function, and finally batch normalized to obtain the global channel information G / /

[0055] G / / =B(T2δ(B(T1(G))))

[0056] In the formula, T1 is the dimension reduction layer, and the kernel size is r is the channel reduction ratio, T2 is the dimension-raising layer, the kernel size is C×1×1, δ is the linear function ReLU, and B is batch normalization. The linear function ReLU is used to increase the nonlinearity of the model, thereby solving the gradient vanishing problem. Batch normalization is used to reduce internal covariate shift, making the input data distribution of each layer in the model more stable.

[0057] The local channel information L of the input feature map F is calculated according to the following formula / ,

[0058] L / =B(PW2δ(B(PW1(F))))

[0059] Where PW1 and PW2 are point convolution operations, and the kernel sizes are and C×H×W.

[0060] The activation function Sigmoid converts the global channel information G / / and local channel information L / Combined, the attention weight of each channel is obtained and applied to the original input feature map F for channel weighting to obtain the feature map F after attention modulation / , specifically:

[0061]

[0062] Where σ() is the sigmoid activation function, represents a broadcast addition operation, Represents the multiplication of corresponding elements of two feature maps.

[0063] The second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

[0064] The above-mentioned multi-scale attention distribution mechanism captures image features at different spatial scales, effectively improving the recognition ability of weak targets. By allocating weights to the importance of each channel, it strengthens the response of important feature channels for weak target recognition, thereby improving the detection accuracy.

[0065] The residual connection unit is used to fuse the initial input features with the features output by the second multi-scale attention module, so that the shallow and deep features can be shared and reused, enhancing the representation ability of the model.

[0066] Step 3. Train the small target recognition model

[0067] Use the training set to train the small target recognition model, use the loss function Inner-IoU to introduce auxiliary boxes of different scales for calculation, and calculate the intersection-over-union ratio between the real box and the auxiliary box of the predicted box to enhance the focus on small areas;

[0068] The training parameters are set as follows: batch size is set to 16, number of training epochs is set to 300, early stopping rounds are set to 50, stochastic gradient descent SGD is selected as the optimizer, and the learning rate is set to 0.01;

[0069] The calculation steps of the loss function Inner-IoU are:

[0070] A1: Define the upper, lower, left, and right boundaries of the real bounding box GT and the anchor box Anchor, such as Figure 5 ;

[0071]

[0072]

[0073]

[0074]

[0075] In the formula, These are the top, bottom, left, and right boundaries of the real bounding box GT. is the ordinate of the center of the ground-truth bounding box GT, is the horizontal coordinate of the center of the ground-truth bounding box GT, w gt 、h gt is the width and height of the ground-truth bounding box GT, ratio is the scaling factor, and b t , b b , b l , b r These are the upper, lower, left, and right boundaries of the anchor box Anchor, respectively. c is the ordinate of the center of the anchor box, x c is the horizontal coordinate of the center of the anchor box Anchor, w and h are the width and height of the anchor box Anchor.

[0076] A2: Find the overlapping boundaries of the ground truth bounding box GT and the anchor box Anchor in the horizontal and vertical directions, and calculate the intersection area in of the two boxes according to the following formula:

[0077]

[0078] In the formula, min() is the minimum value and max() is the maximum value.

[0079] A3: The union area of ​​the ground truth bounding box GT and the anchor box Anchor is obtained according to the following formula;

[0080] union=w gt gt ratio 2 +w·h·ratio 2 -in

[0081] A4: The value of the loss function Inner-IoU IoU is obtained by the intersection area in and the union area union of the real bounding box GT and the anchor box Anchor according to the following formula Inner ,

[0082]

[0083] Step 4. Input the test set into the trained small target recognition model for small target recognition and output the target category, bounding box, and confidence level.

[0084] experiment

[0085] In order to verify the beneficial effects of the present invention, the inventors conducted ablation experiments on the verification set of this embodiment using the YOLOv8 benchmark model (model 1), the YOLOv8 model with the CCF module introduced (model 2), and the small target recognition model of the present invention (model 3). The evaluation indicators obtained are shown in Table 1 below.

[0086] Table 1. Evaluation indicators of the three models

[0087]

[0088] In the table: P(%): tracking accuracy; R(%): recall rate; mAP@50: the average accuracy of the IOU value between the predicted bounding box and the true bounding box of each category greater than 0.5. The higher the value of the above evaluation indicators, the better the model performance.

[0089] It can be seen from the experimental results that the three evaluation indicators of YOLOv8 as the basic model in the small target recognition task are: 68.1%, 51.1%, and 0.569. The YOLOv8 with the CCf module has better performance than the baseline model due to its enhanced ability to extract small targets. However, the regression loss is more sensitive to the bounding box deviation during the calculation process, and the convergence effect on the small target features is not particularly ideal. The small target recognition model of the present invention is an improvement made on the basis of the model of Experiment 2. The loss function is replaced by Inner-IoU, and the accuracy and mAP@.5 are further improved to 72.2% and 60.0%. This fully demonstrates the effectiveness of the method of the present invention in the vehicle and pedestrian recognition task in the UAV intelligent transportation scenario.

Claims

1. A small target recognition method based on MS-CAM improved feature fusion module, characterized in that: The following steps are involved: Step 1. Build the dataset Obtain a dataset of vehicle and pedestrian images taken by drones, relabel, color enhance, and mirror the dataset, and divide the dataset into a training set and a test set according to the ratio; Step 2. Build a small target recognition model Use the feature fusion module CCF to replace the C2f module of the feature extraction network Backbone in the YOLOv8 basic model to obtain a weak target recognition model; The feature fusion module CCF is composed of a first convolution block, a feature decomposition point Split module, a bottleneck module, and a second convolution block connected in sequence; the first convolution block is used to perform preliminary feature extraction on the input image to obtain a feature map; the feature decomposition point Split module is used to divide the feature map into two parts, one part is input to the bottleneck module for deep processing, and the other part is directly spliced ​​with the feature map output by the bottleneck module; the bottleneck module includes 3-6 layers of Bottleneck layers connected in a stacked manner, and gradually extracts high-level feature information in the input feature map by reducing the spatial dimension of the input feature map and increasing the number of channels; the second convolution block is used to perform the last convolution on the input feature map and compress it, and output a feature map with a target number of channels; The Bottleneck layer is composed of a 1×1 convolution block, a 3×3 convolution block, a first multi-scale attention module MS-CAM, a second multi-scale attention module MS-CAM, and a residual connection unit connected in sequence; The 1×1 convolution block is used to reduce the number of channels of the initial input feature map; The 3×3 convolution block is used to extract local spatial features of the feature map output by the 1×1 convolution block; The first multi-scale attention module MS-CAM is used to perform multi-scale feature extraction on the input feature map, calculate the weight of each feature channel, emphasize important features and suppress unimportant features, and focus on global context aggregation; The second multi-scale attention module MS-CAM is used to further perform refined channel weight calculation on the features processed by the first multi-scale attention module MS-CAM to help the model distinguish different attributes and details of the target; The residual connection unit is used to fuse the initial input features with the features output by the second multi-scale attention module, so that the shallow and deep features can be shared and reused; Step 3. Train the small target recognition model Use the training set to train the small target recognition model, use the loss function Inner-IoU to introduce auxiliary boxes of different scales for calculation, and calculate the intersection-over-union ratio between the real box and the auxiliary box of the predicted box to enhance the focus on small areas; Step 4. Input the test set into the trained small target recognition model for small target recognition and output the target category, bounding box, and confidence level.

2. The small target recognition method based on the improved feature fusion module of MS-CAM according to claim 1 is characterized in that: The first multi-scale attention module MS-CAM includes a global channel, a local channel and an activation function Sigmoid; the global channel calculates the global channel information G of the input feature map F / / , the local channel, calculates the local channel information L of the input feature map F / , the activation function Sigmoid, the global channel information G / / and local channel information L / Combined, the attention weight of each channel is obtained and applied to the original input feature map F for channel weighting to obtain the feature map F after attention modulation / ; The second multi-scale attention module MS-CAM has the same structure as the first multi-scale attention module MS-CAM.

3. The small target recognition method based on the improved feature fusion module of MS-CAM according to claim 1 is characterized in that: The global channel information G of the input feature map F is calculated / / The steps are: S1: Perform global average pooling on the input feature map F according to the following formula Where G is the global channel feature with a size of 1×1×C, C is the number of channels of the input feature map F, H is the height of the input feature map F, and W is the width of the input feature map F; S2: The global average pooling result is first reduced in dimension through two fully connected layers, then increased in dimension using the ReLU function, and finally batch normalized to obtain the global channel information G / / G / / =B(T2δ(B(T1(G)))) Where T1 is the dimension reduction layer and the kernel size is r is the channel restoration ratio, T2 is the dimension increase layer, the kernel size is C×1×1, δ is the linear function ReLU, and B is batch normalization.

4. The small target recognition method based on the improved feature fusion module of MS-CAM according to claim 2 is characterized in that: The local channel information L of the input feature map F is calculated / The formula is: L / =B(PW2δ(B(PW1(F)))) Where PW1 and PW2 are point convolution operations, and the kernel sizes are and C×H×W.

Images