Small target detection method based on YOLOv8 convolution module and attention mechanism improvement
By improving the convolutional module and attention mechanism of the YOLOv8 model, the problem of insufficient accuracy in small object detection was solved, and the detection speed and small object recognition effect of the model were improved, especially the detection performance in complex backgrounds.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2024-11-21
- Publication Date
- 2026-05-22
Smart Images
Figure CN122073032A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of small target detection technology in target detection, and in particular to a small target detection method based on YOLOv8 convolutional modules and attention mechanisms. Background Technology
[0002] Object detection and recognition is an important research direction in computer vision, with wide-ranging applications in areas such as defect detection, condition monitoring, traffic monitoring, autonomous driving, and remote sensing. With the rapid development of deep learning, neural network-based object detection and recognition methods have made significant progress. YOLO is one of the more advanced object detection and recognition algorithms currently available, characterized by its speed and accuracy, and has therefore received widespread attention in practical applications. As a crucial branch of computer vision tasks, the main purpose of object detection is to identify the location and category of specific objects in images or videos. With the development of computer hardware and open-source frameworks, object detection has begun to be increasingly widely used in fields such as autonomous driving, security monitoring, smart homes, and medical image analysis.
[0003] This research aims to study object detection and recognition based on the YOLO algorithm. Through theoretical analysis and experimental verification, it explores the algorithm's application effects in different scenarios, and proposes improvements and optimizations to enhance its performance and efficiency. This research is of great significance for promoting the development of object detection and recognition technology and improving the intelligence level of computer vision systems.
[0004] Since 2012, with the emergence of convolutional neural networks (CNNs), deep learning technology has rapidly developed in the field of object detection. These algorithms utilize CNNs to extract image features and combine them with classifiers and regressors for detection and localization. Compared to traditional methods, deep learning-based methods do not require separate feature design; instead, they extract deeper information through CNNs. The application of CNNs laid the foundation for object detection. Therefore, since 2012, the development of object detection has gradually shifted towards deep learning algorithms. This shift has injected new vitality into object detection algorithms and provided more efficient solutions, especially in practical applications for small object detection tasks. Since Hinton proposed using neural networks to autonomously learn high-dimensional features in image data, deep learning-based object detection has become a major research hotspot in the field of computer vision. Deep learning methods are further divided into two-stage and one-stage approaches. Networks using the two-stage approach, such as R-CNN and Faster R-CNN, have lower frame rates (fps) for recognizing data and often can only process single images. In contrast, networks using the one-stage approach, such as YOLO, have extremely high efficiency and can process data such as videos.
[0005] All YOLO object detection models prior to YOLOv3 were written in C and used the Darknet framework. Ultralytics released the first YOLO implementation using the PyTorch framework (YOLOv3). Following YOLOv3, Ultralytics released YOLOv5, and in January 2023, they released YOLOv8, which includes five models for detection, segmentation, and classification. As the latest version of the YOLO series, YOLOv8, through model structure optimization, anchored box or anchorless scheme optimization, and various data augmentation techniques, can improve detection speed while maintaining high accuracy, making it a current research hotspot.
[0006] While deep learning-based object detection methods have made significant progress in large object detection, they still face many challenges and limitations in small object detection: small objects occupy fewer pixels in an image, have lower resolution, and relatively less feature information. This makes small objects easily overlooked during feature extraction, especially in complex backgrounds, where their feature information is more easily obscured.
[0007] To overcome the aforementioned limitations, several improvement methods have emerged. For example, fusing feature maps from different levels enhances the feature representation of small targets. Methods such as Feature Pyramid Networks (FPN) and Bidirectional Feature Pyramid Networks (Bi-FPN) improve small target detection performance through multi-level feature fusion. Introducing attention mechanisms allows the model to focus more intently on small target regions, improving detection accuracy. For instance, channel attention and spatial attention mechanisms can effectively enhance the feature representation of small targets. Adding a dedicated detection layer for small targets to the detection head, and fusing deep and shallow features, further improves the detection accuracy of small targets. Summary of the Invention
[0008] The technical problem this invention aims to solve is that modules that improve the accuracy of YOLOv8 generally increase the number of model parameters, thus reducing image recognition speed and leading to a decrease in FPS. Since small-scale objects typically constitute a small proportion of datasets and contain less semantic information compared to large objects, their detection performance is relatively poor. Improving the accuracy of small object detection is a current challenge in the field. Small objects occupy fewer pixels, have lower resolution, and less feature information in images, making them easily overlooked and susceptible to noise. This can lead to missed detections.
[0009] A small object detection method based on YOLOv8 convolutional modules and attention mechanisms, characterized by the following steps:
[0010] S1. Collect image data of tiny objects from a large number of aerial images, label and preprocess them, and construct a dataset.
[0011] S2. Construct an object detection model based on the YOLOv8 model. This model includes an input layer, a backbone network, a neck network, and a head output layer.
[0012] S3. Train the built object detection model using the labeled dataset.
[0013] S4. Collect images of small objects in the aerial images to be detected, and input them into the trained target detection model to perform target detection, and evaluate the performance of the model.
[0014] Furthermore, in step S1, during the dataset construction process, the size of the collected images is converted into RGB images of 640*640 pixels.
[0015] Furthermore, in step S2, the YOLO model built in step S2 follows the following process.
[0016] Step 11: The Backbone network is improved based on the original backbone network. The ordinary convolutions of the original backbone network are replaced with spatial depth transformation convolutions. Spatial depth transformation convolutions rearrange the input feature maps, grouping spatially adjacent elements into separate depth channels, thus effectively reducing spatial dimensionality. Using spatial depth transformation convolutions, the original 640*640 image size is continuously reduced, while transferring this reduced information to the channels, ensuring no information loss. This type of convolution is more lightweight than ordinary convolutions, which can improve model training and detection speeds.
[0017] Step 12: The Neck network adopts a Bi-FPN structure at the Neck end, improved through a fast normalization approach, as follows: During feature fusion, resizing is required due to different resolutions. However, since different feature inputs have different resolutions, their contributions to the final feature network output will also differ. Therefore, the network needs to learn these weights. A common weight learning formula is as follows: O = ∑ i W i I i
[0018] wi could be a scalar based on each feature, a vector for each channel, or a multidimensional tensor for each pixel. Since wi is unconstrained, it can lead to instability and difficulty in convergence during training. Fast normalization can facilitate faster convergence during training, as shown in the following formula.
[0019]
[0020] The ReLU function is used to ensure that the value is greater than or equal to 0, as shown in the following formula:
[0021] f(x) = max(0,x)
[0022] Then, the values are normalized to between 0 and 1, with the ∈ sign in the denominator set to 0.0001 to avoid numerical instability and improve the speed of normalization. Simultaneously, the information obtained after convolution is passed to the neck at layers p5 and p7, respectively, to integrate more functions and obtain more information without increasing cost.
[0023] The c2f module is modified into the CDM module. The CDM module replaces the ordinary convolution in the bottleneck part of the c2f module with deformable convolution, allowing the model to effectively learn small targets of different shapes. At the same time, an MLCA attention module is added after the bottleneck to combine channel information, spatial information, local information and global information to improve the network's expressive performance.
[0024] Step 13: Add a small target detection layer to the Head output. An additional 160x160 feature map scale layer is added to the head. This layer, through the transfer of deep features and fusion with shallow features, contains more contour and position information of small targets, which is beneficial for the localization and recognition of small targets and reduces the false positives and false negatives of small targets.
[0025] Furthermore, in step 12, the CDM module's process is as follows:
[0026] Step 21: The data enters the CDM module. The input data first undergoes a convolution operation (Conv), doubling the number of channels in the output feature map to enhance the model's feature representation capability. It is then split into two parts, which enter two parallel paths.
[0027] Step 22: The first portion of data enters the first Bottleneck layer and continues to be input into multiple Bottleneck layers. The segmented feature maps are processed layer by layer in multiple Bottleneck modules to extract deeper features. The Bottleneck module configuration uses shortcut connections to enhance gradient propagation and information flow.
[0028] Step 23: The second part of the data also enters another Bottleneck layer, but this path includes a conditional check (Ifshortcut) to decide whether to skip the concatenation operation and directly participate in the subsequent concatenation process. Regardless of whether the concatenation operation is skipped, this part of the data will then undergo two DCN convolutions to better capture features under spatial transformations. Deformable convolutions can make the network spatially invariant, increasing the positional offset of the input feature map at the sampling grid points without adding additional supervised learning. This offset, along with the filter parameters W and b, is obtained through gradient descent parameter updates. The output feature map Y is obtained by multiplying and adding the input features and the filter parameters W and b, as shown in the following formula:
[0029]
[0030] DCN improves upon this by adding a position offset term Δp. n The formula is as follows:
[0031]
[0032] 24: The results from the upper and lower paths are merged before MLCA and input into the MLCA attention module. MLCA is an attention module used to aggregate multi-level contextual information. The MLCA attention module enhances the performance of the object detection network while maintaining low computational complexity. MLCA combines channel information and spatial information, as well as local and global information, to improve the network's expressive power. Finally, the outputs of the upper and lower paths are concatenated at the CONCAT module to generate the final CDM module output.
[0033] Further, in step S3, the built object detection model is trained using the labeled dataset. The input image is set to 640x640 pixels, the total number of training rounds is 100, and the batch size is 4. Mosaic data augmentation is turned off in the first 10 training rounds, and the optimizer is set to SGD.
[0034] Furthermore, in step S4, in addition to the commonly used mAP metric, the model evaluation stage also pays special attention to the detection accuracy and recall of small targets to ensure the effectiveness of the model in practical applications.
[0035] The beneficial effects of this invention are as follows: All convolutional operations in the backbone are replaced with SPDConv. To address the problem of insufficient feature information, a CDM module integrating DCN (Deformable Convolutional Network) and MLCA (Multi-Level Contextual Attention) is proposed, replacing the C2f module in the backbone and neck of YOLOv8 for feature extraction and feature fusion. Simultaneously, the original PANFPN in YOLOv8 is modified based on the BIFPN feature pyramid structure, and the fast normalization concept from BIFPN is introduced to improve the concat operation. To better detect small targets, an additional small target detection head is added to the detection head. Attached Figure Description
[0036] Figure 1 This is a flowchart illustrating an embodiment of the present invention;
[0037] Figure 2 This is a model architecture diagram of an embodiment of the present invention. Detailed Implementation
[0038] The present invention will now be described in further detail with reference to the accompanying drawings.
[0039] The technical problem to be solved by this invention is that modules that improve the accuracy of YOLOv8 generally increase the number of model parameters, which in turn reduces the image recognition speed and leads to a decrease in FPS.
[0040] Small-scale targets typically constitute a small proportion of datasets and contain less semantic information compared to larger targets, resulting in relatively poor detection performance. Improving the accuracy of small-target detection remains a challenge in the field. Small targets occupy fewer pixels, have lower resolution, and less feature information in images, making them easily overlooked and susceptible to noise. This can lead to missed detections.
[0041] This invention proposes an improved small target detection method based on YOLOv8 convolutional modules and attention mechanisms, according to... Figure 1 The process shown for building the system includes the following steps:
[0042] Step 1: Collect image data of tiny objects from a large number of aerial images, label and preprocess them to construct a training set;
[0043] Step 11: Acquire aerial images using a drone or other high-resolution camera. Label small objects (vehicles, buildings, trees, etc.) in the images to generate label files.
[0044] Step 12: Preprocess the acquired image, including cropping, scaling, grayscale conversion, normalization, etc., to ensure that the image data is a 640*640 pixel RGB image.
[0045] Step 13: Divide the collected dataset into training set, validation set and test set according to a size of 8:1:1.
[0046] Step 2: Construct an object detection model based on the YOLOv8 model. This model includes an input layer, a backbone network, a neck network, and a head output layer.
[0047] Step 21: The input layer receives image data.
[0048] Step 22: The Backbone network is improved based on the original backbone network. The ordinary convolutions of the original backbone network are replaced with spatial depth transformation convolutions. Spatial depth transformation convolutions rearrange the input feature maps, grouping spatially adjacent elements into separate depth channels, thus effectively reducing spatial dimensionality. Using spatial depth transformation convolutions, the original 640*640 image size is continuously reduced, while transferring this reduced information to the channels, ensuring no information loss. This type of convolution is more lightweight than ordinary convolutions, which can improve model training and detection speeds.
[0049] Step 23: The Neck network adopts a Bi-FPN structure at the Neck end. The Bi-FPN structure improves upon the original concat operation through fast normalization, and replaces the c2f module with a CDM module. The CDM module replaces the ordinary convolutions in the bottleneck part of the c2f module with deformable convolutions, allowing the model to effectively learn small targets of different shapes. Simultaneously, an MLCA attention module is added after the bottleneck to simultaneously combine channel information, spatial information, and local and global information to improve the network's expressive performance.
[0050] Step 24: Add a small target detection layer to the Head output. An additional 160x160 feature map scale micro-target detection layer is added to the head. This layer, through the transfer of deep features and fusion with shallow features, contains more contour and position information of small targets, which is beneficial for the localization and recognition of small targets and reduces the false negatives and false positives of small targets.
[0051] Step 3: Train the built object detection model using the labeled dataset. Set the input image to 640x640 pixels, the total number of training rounds to 100, and the batch size to 4. In the first 10 training rounds, disable mosaic data augmentation and set the optimizer to SGD.
[0052] Step 4: Model evaluation. In addition to the commonly used mAP metric, the model evaluation stage pays special attention to the detection precision and recall of small objects to ensure the model's effectiveness in practical applications. The experimental datasets are the Satalite dataset and the AI-TOD dataset. The Satalite dataset contains 6685 640*640 pixel images in three categories. AI-TOD is a dataset for detecting small objects in aerial images. AI-TOD provides 700621 object instances in 28036 aerial images, covering 8 categories. The experimental equipment consisted of a 3090-24G GPU, an AMD EPYC7551P 9-core CPU, and 72GB of memory. The experimental results of the model on these two datasets are shown in Tables 1 and 2. mAP represents the average precision across multiple categories. mAP50 represents the mAP value at a 50% IOU threshold. mAP50-90 is a more stringent evaluation metric, which calculates the mAP values within the 50%–95% IOU threshold range and then averages them. This allows for a more accurate evaluation of the model's performance at different IOU thresholds. FPS, in the image processing domain, refers to the number of frames per second. Recall, relative to the original sample, represents the probability of a sample being predicted as positive among samples that are actually positive. As can be seen, the improved model shows improvements in mAP, FPS, and recall.
[0053] Table 1 shows the model's performance on the Satalite dataset.
[0054]
[0055] Table 2 shows the model's performance on the AI-TOD dataset.
[0056]
[0057] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A small target detection method based on YOLOv8 convolutional modules and attention mechanisms, characterized in that, Includes the following steps: S1. Collect image data of tiny objects from a large number of aerial images and perform annotation and preprocessing; S2. Construct an object detection model based on the YOLOv8 model. This model includes an input layer, a backbone network, a neck network, and a head output. S3. Train the constructed object detection model using a labeled dataset. S4. Collect images of small objects in aerial images to be detected and input them into the trained object detection model for object detection, then evaluate the model's performance.
2. The small target detection method based on YOLOv8 convolutional modules and attention mechanisms as described in claim 1, characterized in that, The process of building the YOLO model in step S2 is as follows: Step 21: The Backbone network is improved based on the original backbone network. The ordinary convolutions of the original backbone network are replaced with spatial depth transformation convolutions. Spatial depth transformation convolutions rearrange the input feature maps, grouping spatially adjacent elements into separate depth channels, thus effectively reducing spatial dimensionality. Using spatial depth transformation convolutions, the original 640*640 image size is continuously reduced, while transferring this reduced information to the channels, ensuring no information loss. This type of convolution is more lightweight than ordinary convolutions, which can improve model training and detection speeds. Step 22: The Neck network adopts a Bi-FPN structure at the Neck end. The Bi-FPN structure improves upon the original concat operation through fast normalization, and replaces the c2f module with a CDM module. The CDM module replaces the ordinary convolutions in the bottleneck part of the c2f module with deformable convolutions, allowing the model to effectively learn small targets of different shapes. Simultaneously, an MLCA attention module is added after the bottleneck to simultaneously combine channel information, spatial information, and local and global information to improve the network's expressive performance. Step 23: Add a small target detection layer to the Head output. An additional 160x160 feature map scale micro-target detection layer is added to the head. This layer, through the transfer of deep features and fusion with shallow features, contains more contour and position information of small targets, which is beneficial for the localization and recognition of small targets and reduces the false negatives and false positives of small targets.
3. The small target detection method based on YOLOv8 convolutional modules and attention mechanisms as described in claim 2, characterized in that, In step 22, the CDM module's process is as follows: Step 31: The data enters the CDM module. The input data first undergoes a convolution operation (Conv), doubling the number of channels in the output feature map to enhance the model's feature representation capability. It is then split into two parts, which enter two parallel paths. Step 32: The first portion of data enters the first Bottleneck layer and continues to be input into multiple Bottleneck layers. The segmented feature maps are processed layer by layer in multiple Bottleneck modules to extract deeper features. The Bottleneck module configuration uses shortcut connections to enhance gradient propagation and information flow. Step 33: The second part of the data also enters another Bottleneck layer, but this path includes a conditional check (Ifshortcut) to determine whether to skip the concatenation operation and directly participate in the subsequent concatenation process. Regardless of whether the concatenation operation is skipped, this part of the data will undergo two DCNv4 convolutions to better capture features under spatial transformations. Step 34: The results of the upper and lower paths are merged before MLCA. MLCA is an attention module used to aggregate multi-level contextual information. Finally, the outputs of the upper and lower paths are concatenated at the CONCAT module to generate the final CDM module output.
4. The small target detection method based on YOLOv8 convolutional modules and attention mechanisms as described in claim 1, characterized in that, In S4, specifically, in the model evaluation phase, in addition to the commonly used mAP metric, special attention is paid to the detection accuracy and recall of small targets to ensure the effectiveness of the model in practical applications.