Multi-head prediction algorithm for small target detection based on yolov5 fusion spatial information

Abstract

The application discloses a kind of fusion space information multi-head prediction small target detection algorithm based on Yolov5, belong to target detection field, including the following steps: S1, data acquisition;S2, construct new research and development based on the fusion space information multi-head prediction small target detection algorithm of Yolov5;S3, detection algorithm model is trained, and optimal model is saved;S4, optimal model is predicted, and prediction result is saved, obtains evaluation index, finally carries out result comparison;Small target detection still has great difficulty after years of development, common small target detection method has the information quantity of increasing small target feature, in fact, it is not treated as small target to him in actuality.The difficulty of small target detection is actually caused by the uncertainty of positioning small target in the picture, so the application will improve the position of positioning small target and increase the prediction layer of small target.The experimental verification of the application shows that increasing position information features and increasing prediction layers can improve the detection capability of small targets.

Description

Technical Field

[0001] This invention relates to a deep neural network, and more particularly to an algorithm that improves the detection accuracy of small targets by fusing features of spatial information with the output of multi-layer prediction heads, belonging to the field of target detection. Background Technology

[0002] In object detection tasks, state-of-the-art small object detection algorithms generally suffer from low accuracy in detecting small objects. To address this, we propose a lightweight, easily deployable YOLOv5 small object detection algorithm, based on an attention mechanism, coordinate information localization, and a multi-layer prediction head. Utilizing the model's coordinate convolution criterion to achieve model lightweighting, we modify the YOLOv5 neck layer and combine it with FPN to fuse different semantic information into four feature maps, thereby improving the quality of feature extraction and the performance of small object detection. We modify the main feature extraction network of YOLOv5. Based on the outputs of the original three feature maps, we add a new output and port it to our newly researched YOLOv5 algorithm. We then validate this algorithm and allow experiments on the Visdrone dataset to compare with other YOLO algorithms. The results show that our innovative YOLOv5 algorithm is highly effective for small object detection.

[0003] Inspired by people's visual judgment of small objects, we found that the main problem in small object detection is determining the object's location before classifying it. Therefore, we incorporate coordinate information by performing multi-scale coordinate convolutions on image features, adding W / H dimensions and feature degree to enhance their positional awareness within the feature map, thereby improving the model's accuracy in small object detection. Based on the coordinate feature scheme and the dense prediction architecture (YOLO), we designed a lightweight one-stage detector for small object detection, and simultaneously proposed different lightweight models suitable for feature maps of different sizes, incorporating an attention method.

[0004] Deep learning-based object detection is adept at detecting and recognizing large or medium-sized targets. However, it often falls short when dealing with very small targets (less than 32 pixels). Common problems include incorrect location identification, incorrect category identification, or both.

[0005] The main reason for these undesirable predictions is that images lose some information during convolution. Small targets are fewer in number in the original image, resulting in fewer features extracted by the detector, leading to poor detection performance for small targets. Furthermore, the loss during image feature extraction further exacerbates the problem. Neural networks are drawn to large targets during learning, making it easy for small targets to be lost throughout the learning process, thus resulting in unsatisfactory detection performance for small targets.

[0006] In image convolution, the features of small objects are compressed quadratically, which is beneficial for the final computation of the detector, but the loss of feature information is very severe. Downsampling also leads to a significant loss of information from the source image, especially spatial information. Pixels discarded by downsampling do not have a significant impact on medium to large objects because they already occupy a sufficient number of pixels; in fact, recognizing large objects requires even higher downsampling compression ratios to expand the receptive field of CNNs. However, those discarded pixels are fatal for small objects: small objects occupy very few pixels in the original image, and after multiple downsampling operations, these pixels are almost compressed to the point of "disappearing," and the detector cannot predict the existence of objects based on non-existent pixels. In fact, both CNN-based and Transformer-based detectors need to use downsampling to filter out noise and retain more prominent features; therefore, downsampling is still necessary. However, for small objects, a direct approach is to use feature maps with a small downsampling scaling ratio as the material for the detector's prediction, because the information about small objects retained on these feature maps is richer. While more semantic information slightly reduces the difficulty of classifying small objects, it doesn't alleviate another thorny problem in small object detection: the difficulty of accurate location regression. Two-stage detectors perform well in small object detection due to sparse detection; however, these detectors are not industry-friendly, as their algorithms are more complex, training costs are higher, and deployment is more difficult. Conversely, one-stage detectors are more popular in industry because they often use simpler network architectures and more direct prediction processes, but their dense prediction mechanisms more or less limit their performance in detecting small objects. Recently, Transformer-based detectors have received more attention, but their inference latency is almost unacceptably high. Due to the massive computational cost of self-attention methods on mobile devices, CNN-based detectors are still more commonly used on mobile devices.

[0007] Humans are often more sensitive to the positional information of small objects: if we want to distinguish certain small objects that are farther away from us in our field of vision, we need to first focus our gaze on their location and then try to identify what they are, rather than identifying what they are first and then finding where they are. After image data is filtered multiple times by convolutional kernels, its original spatial and channel information is irreversibly lost, although this filtering mechanism preserves the "useful" features needed by deep networks. To address the challenge of small object position regression, we embed position information awareness into a convolutional network-based CSP structure, named CoordCSP, which is similar to the positional encoding technique used by Transformer-based detectors. We note that position regression is the main challenge in small object detection tasks because while most deep learning-based classifiers achieve good classification accuracy on the 32*32 resolution CIFAR10 dataset, most detectors cannot effectively detect objects smaller than 30 pixels in detection tasks. Summary of the Invention

[0008] To address the issue of extremely low real-time detection accuracy for small targets in object detection, this invention improves the detection rate algorithm for small targets by fusing features of spatial coordinates. This method effectively enhances the detection accuracy of small targets and exhibits excellent real-time performance for detecting small targets. The specific solution of this invention is as follows:

[0009] An algorithm to improve the detection accuracy of small objects by fusing features of spatial coordinates includes the following steps:

[0010] S1, Data Acquisition;

[0011] S2. Construct a newly developed Yolov5-based multi-head prediction algorithm for small target detection based on spatial information;

[0012] S3. Construct the loss function and train the object detection algorithm model, then save the optimal model;

[0013] S4. Use the optimal model to make predictions, save the prediction results, obtain evaluation indicators, and finally compare the results.

[0014] Furthermore, the experimental dataset in step S1 utilizes camera-equipped drones (or general-purpose drones) that have been rapidly deployed to a wide range of applications, including agriculture, aerial photography, rapid delivery, and surveillance. Therefore, the demand for automated understanding of the visual data collected from these platforms is increasing, making the relationship between computer vision and drones increasingly close. We are pleased to present a large benchmark for various important computer vision tasks, carefully annotated with ground truth, named VisDrone, bringing vision and drones together. The VisDrone2019 dataset was collected by the AISKYEYE team at the Machine Learning and Data Mining Laboratory of Tianjin University. The benchmark dataset consists of 288 video clips, comprising 261,908 frames and 10,209 still images, captured by various drone cameras, covering a wide range of locations (from 14 different cities thousands of kilometers apart in China), environments (urban and rural), objects (pedestrians, vehicles, bicycles, etc.), and densities (sparse and crowded scenes). Note that the dataset was collected using different drone platforms (i.e., different models of drones) under different scenes, weather, and lighting conditions. These frames are hand-annotated with over 2.6 million frequently used bounding boxes of interest, such as pedestrians, cars, bicycles, and tricycles. Several important attributes, including scene visibility, object class, and occlusion, also provide better data utilization. Note that the dataset was collected using different drone platforms (i.e., different drone models) under varying scenes, weather, and lighting conditions.

[0015] Furthermore, step S2 will primarily include the structural design of coordinate convolution and coordinate modules, as well as the design of a four-layer detection head architecture. We adopted a simple 3x3 kernel to accelerate data flow: we extensively used 3x3 convolution kernels, which are more suitable for GPU computing, and minimized the use of branch-less structures in bottleneck sections.

[0016] S21. Embed the location features into the feature map through a CoordConv layer;

[0017] S22. The data is fed into two filters, a bottleneck layer, and a coordinate convolution, and then concatenated along the channel dimension. A 1x1 and 3x3 vanilla convolution and an addition from the shortcut are used in the bottleneck section. The shortcut then feeds in traditional channel feature information. The 1x1 and 3x3 vanilla convolutions serve as the backbone to better fit the model and perceive a wider field of view. The pixel addition of the matrix serves as data points to fuse higher-level semantic information that has been filtered out in the channel and spatial information.

[0018] S23 and CoordConv will be used to assign positional information of the original features at the places where the CoordCSP module is connected, and the features with clear position numbers that are perfectly matched by these positional information will be concatenated and output on the channel, and finally the pixel-level spatial and channel information will be fused by a 1*1 vanilla convolution before being passed to the next layer of the network.

[0019] Furthermore, in step S21, CoordCSP uses a large number of 1x1 and 3x3 convolutions primarily to enhance the noise filtering capability of the bottleneck module and better adapt to GPU computation. 1x1 convolutions do not alter the image's spatial structure; fully connected layers, for example, concatenate dimensions into a one-dimensional array. In contrast, 1x1 convolutional layers do not destroy the image's spatial structure, preserving local image features, and convolution possesses spatial invariance. This is done to reduce the number of network parameters and computational complexity (a large number of 1x1 convolutions are used on some lightweight networks designed for mobile devices).

[0020] Furthermore, in step S22, the final fusion method of the coordinate features and semantic features of CoordCSP is the balance of spatial information. This alignment operation maintains the validity of the coordinate feature labels without being offset or biased.

[0021] Furthermore, a four-layer prediction head is used to accurately predict small targets. The VisDrone2021 dataset itself contains many very small targets, so our algorithm adds a prediction head specifically for detecting tiny objects. Combined with the other three prediction heads, the four prediction heads effectively address certain difficulties in detecting small targets and the challenges posed by changes in target scale. The added prediction head (Head 1) is generated from a low-level, high-resolution feature map, making it more sensitive to tiny objects. Although adding the detection head increases computational and storage costs, it significantly improves the detection performance for tiny objects.

[0022] Furthermore, in step S23, the IoU loss is used for bounding box regression in object detection algorithms because it is scale-invariant compared to L1 Loss and L2 Loss and directly quantifies the overlap between the predicted and ground truth boxes. CIoU, based on the IoU loss function, is a widely used bounding box regression loss function in anchor-based detectors, and its formula is as follows:

[0023]

[0024] Furthermore, in step S23, we find that the penalty terms for w and h are completely independent of each other, meaning that the gradient update of w does not affect the update of h. The characteristics of EIoU allow the detector to predict the sizes of a wide variety of real-world objects well, resulting in higher accuracy compared to CIoU.

[0025] Furthermore, in step S23, when using CIoU Loss to regress and predict the bounding box of an object, an increase in the w dimension will lead to a decrease in the h dimension, or vice versa. This is unreasonable because not all objects in an image can be exactly "short" or "wide"; "square" objects are also allowed. We need a regression loss function that does not interfere with each other in the w and h dimensions to truly reflect the size of objects in the real world. Through investigation, we discovered a loss function that satisfies the above theory, EIoU, with the following expression:

[0026]

[0027] Further, in step S24, an object detection algorithm is constructed and the model is trained. The dataset is loaded for training, and then the optimal model is saved. The trained model is used to predict the test set of the data.

[0028] The beneficial effects of this invention are as follows:

[0029] This invention discloses an algorithm for improving the detection accuracy of small targets by fusing spatial coordinate features. The YOLO series of real-time detectors are favored by industry due to their real-time performance. In this section, we compare the performance of similarly sized YOLO detectors, including yolov3-tiny, yolov4-tiny, and yolov5s, on the VisDrone 2019 target detection task to demonstrate the effectiveness of the proposed method on the challenging small target detection problem. Table 4 presents the results of comparative experiments: compared to the three lightweight YOLO detectors (v3-v5), the method using coordinate features achieves the highest validation and test accuracies on the VisDrone small target challenge dataset with similar inference latency.

[0030] To summarize, inspired by biological visual perception, we propose an alternative approach to alleviate the challenge of small object detection: embedding independent coordinate features (wh). This differs from the mainstream approach of increasing the information content of texture features for small objects, yet it achieves equally satisfactory results in improving detection accuracy. This method is so simple that it can be easily and directly applied to improve existing small object detection algorithms, exhibiting similar good generalization to models of different structures at the same size. However, a slight drawback is that our proposed scheme seems to contribute differently to models of different sizes, and it increases inference latency, although this negative impact is almost imperceptible in actual inference. In future work, we will explore the inconsistency in the performance contribution of coordinate features to models of different sizes and accelerate model inference to provide a more robust solution for small object detection.

[0031] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0033] Figure 1 A schematic diagram of the overall architecture with coordinate convolution and a four-layer prediction head;

[0034] Figure 2 A schematic diagram of the CoordCSP module;

[0035] Figure 3 This is a graphical representation of the CoordConv method. The width and height coordinates are the same size as the input features and are concatenated with them. These features are then fused together by a hypothetical convolutional layer.

[0036] Figure 4 A visual comparison of the SOTA Yolo series and Coord-Yolo (ours); Detailed Implementation

[0037] The following detailed description provides further details on specific implementation methods.

[0038] like Figure 3The YOLOv5-based multi-head prediction algorithm model for small target detection, which incorporates spatial information, includes the following steps:

[0039] S1, Data Acquisition

[0040] S11, Data Acquisition Phase:

[0041] We use VisDrone2019, a large-scale benchmark collected using drones, to evaluate the performance of different models. The VisDrone2019 dataset utilizes camera-equipped drones (or general-purpose drones) that have been rapidly deployed to a wide range of applications, including agriculture, aerial photography, rapid delivery, and surveillance. Therefore, the demand for automated interpretation of visual data collected from these platforms is increasing, making the relationship between computer vision and drones increasingly close. We are excited to present a large-scale benchmark for various important computer vision tasks, carefully annotated with ground truth, named VisDrone, bringing vision and drones together. The VisDrone2019 dataset was collected by the AISKYEYE team at the Machine Learning and Data Mining Laboratory of Tianjin University. The benchmark dataset consists of 288 video clips, comprising 261,908 frames and 10,209 still images, captured by various drone cameras, covering a wide range of locations (from 14 different cities thousands of kilometers apart in China), environments (urban and rural), objects (pedestrians, vehicles, bicycles, etc.), and densities (sparse and crowded scenes). Please note that the dataset was collected using different drone platforms (i.e., different drone models) under various scenes, weather, and lighting conditions. These frames are manually labeled with over 2.6 million frequently used bounding boxes of interest, such as pedestrians, cars, bicycles, and tricycles. Increasing the size of the input image can directly alleviate this problem, but due to the secondary increase in computational cost, this solution is extremely unfriendly to edge devices (used on drones). Therefore, we still use a strict input image size (640×640) to investigate the extent to which coordinate convolution and the four-layer prediction head modules contribute to small object detection.

[0042] S2. Construct a small target detection algorithm based on Yolov5 that integrates spatial information through multi-head prediction. The specific steps are as follows:

[0043] S21. Our goal is to design a deployment-friendly detector for small targets without significantly increasing additional costs; therefore, a simple backbone structure was considered. Firstly, CoordCSP uses a large number of 1x1 and 3x3 convolutions primarily to enhance the noise filtering capability of the bottleneck module and better adapt to GPU computation. 1x1 convolutions do not alter the image's spatial structure; fully connected layers can destroy the image's spatial structure, for example, concatenating dimensions into a one-dimensional array. 1x1 convolutional layers, however, do not destroy the image's spatial structure, preserving local image features, and convolution possesses spatial invariance. To reduce the number of network parameters and computational complexity (a large number of 1x1 convolutions are used on some lightweight networks designed for mobile devices), Table 1 shows a set of ablation experiments to analyze the performance improvement brought by CoordConv combined with CoordCSP. The results of this experiment show an average accuracy improvement of 0.3% on the test set.

[0044] S22, such as Figure 2 We designed a location-aware CSP module, CoordCSP, which embeds location features into the representation features through a CoordConv layer. In the CoordCSP module, features are first fed simultaneously to two filters, a Bottleneck and a CoordConv, and then concatenated and fused along the channel dimension. 1x1 and 3x3 vanilla convolutions and an element-wise addition from the shortcut are used by the Bottleneck. The shortcut forwards the original channel feature information, the 1x1 and 3x3 vanilla convolutions serve as the backbone to enable the model to perceive spatial correlations across different viewpoints, and the element-wise addition of the matrix serves as a data flow node to merge the filtered higher-level semantic information across both spatial and channel dimensions.

[0045] S23 and CoordConv are used on the cross-stage local connections of the CoordCSP module to assign coordinate encoding to the initial features. These features, which perfectly match the initial feature size and have explicit positional encodings, are concatenated with the output of Bottleneck in the channel dimension. Finally, element-wise spatial-channel feature information is fused through 1*1 vanilla convolutions before being fed to the next layer. CoordCSP uses a large number of 1*1 convolutions primarily to enhance the noise filtering capability of the bottleneck module. 1*1 convolutions do not expand the receptive field of the network but enhance its non-linear expressive power through linear filtering of the convolution kernels and non-linear mapping of the activation function, and the fusion of gradient flows from different pipelines at the element-wise level, rather than to reduce the number of network parameters and computational complexity (a large number of 1*1 convolutions are used in some lightweight networks designed for mobile devices). The final concatenated coordinate features in CoordCSP are combined with high-level semantic features in a spatially aligned manner. This alignment operation maintains the validity of the coordinate feature labels without offset or deviation.

[0046] S24. Further, a four-layer prediction head is used to accurately predict small targets. The VisDrone2021 dataset contains many very small targets, so our algorithm adds a prediction head specifically for detecting tiny objects. Combined with the other three prediction heads, the four prediction heads effectively address the difficulties in detecting small targets and the challenges posed by changes in target scale. The added prediction head (Head 1) is generated from a low-level, high-resolution feature map, making it more sensitive to tiny objects. Although adding the detection head increases computational and storage costs, it significantly improves the detection performance for tiny objects.

[0047] S3. Construct a hybrid loss function and train the segmentation model, saving the optimal model.

[0048] IoU loss is used in bounding box regression tasks in object detection algorithms. The L1 norm-based bounding box loss function calculates the absolute differences between the x-coordinates and y-coordinates of the four points in the predicted bounding box and the corresponding four points in the ground truth bounding box, then sums them. The L2 norm-based bounding box loss function calculates the sum of the squares of the four points in the predicted bounding box and the corresponding four points in the ground truth bounding box, then sums them. However, it does not consider the correlation between coordinates. The bounding box regression loss function (IoU_Loss) treats the box formed by the four points of the bounding box as a whole for regression, taking into account the correlation between coordinates. Its formula is as follows:

[0049]

[0050] The coordinate convolution is involved. This property means that when using CIoU Loss to regress and predict the bounding box of an object, increasing the w dimension will lead to a decrease in the h dimension, or vice versa. This is an unreasonable situation because not all objects in an image can be exactly "short" or "wide"; "square" objects are also allowed. We need a regression loss function that does not interfere with each other in the w and h dimensions to truly reflect the size of objects in the real world. Through investigation, we found a loss function that satisfies the above theory, EIoU, with the following expression:

[0051]

[0052] However, it's worth noting that we observed an interesting phenomenon: the impact of coordinate features differs significantly across models of different sizes. For smaller models, coordinate features improve recall (31.9% to 33.5%), while for medium-sized models, they improve prediction accuracy (43.7% to 47.6%). This demonstrates the benefits of using lightweight convolutions in the Neck stage for edge computing devices. Table 2 reports the gains from combining coordinate features with EIoU: a 0.5%–1.1% improvement in prediction accuracy at the cost of only 0.02 million parameters.

[0053] Table 1. Ablation study results of different skeletons: uncoordinated skeleton and coordinated CSP skeleton.

[0054]

[0055] Table 2. Ablation study results of Loss EIoU employment with coordinate characteristics (baseline Yolov5s).

[0056]

[0057] Table 3. Competitive experimental results for the lightweight Yolo series real-time detectors

[0058]

[0059] YOLO series real-time detectors are favored by industry due to their real-time performance. In this section, we compare the performance of similarly sized YOLO detectors, including yolov3-tiny, yolov4-tiny, and yolov5s, on the VisDrone 2019 object detection task to demonstrate the effectiveness of the proposed method on the challenging small object detection problem. Table 4 presents the results of comparative experiments: compared to five lightweight YOLO detectors (v3-v5), the method using coordinate features achieves the highest validation and test accuracies on the VisDrone small object challenge dataset with similar inference latency. Figure 4 In the diagram, we present a visualization of the test results for the different detectors mentioned above.

Claims

1. A multi-head prediction method for small target detection based on Yolov5 fusion spatial information, characterized in that, The method comprises the following steps: S1, data acquisition; S2, constructing an improved target detection algorithm model based on a YOLOv5 model; S3, constructing a loss function and training the target detection algorithm model, and saving the optimal model; S4, using the optimal model for prediction, saving the prediction results, obtaining evaluation indexes, and finally comparing the results; The step S2 comprises structure design of a CoordCSP module and design of four-layer prediction layers based on the CoordCSP module; the structure design of the CoordCSP module specifically comprises the following steps: S21, embedding position features into a feature map through a CoordConv layer to increase two dimensions of the feature map on the basis of the original channel dimension; S22, inputting the feature map processed in the step S21 into two modules, a bottleneck module and a position convolution module, respectively, then splicing and fusing the features processed by the bottleneck module and the position convolution module in the channel dimension; wherein the bottleneck module is specifically implemented as follows: using 1*1 vanilla convolution and 3*3 vanilla convolution as a backbone network, and combining a shortcut addition operation, the shortcut is used for transmitting original channel feature information forward, the 1*1 vanilla convolution and the 3*3 vanilla convolution are used for being more suitable for the model and widening the feature perception field, and the pixel addition of the matrix is used as a data point to fuse higher-level semantic information filtered out from the channel information and the spatial information; S23, the CoordConv is used to assign position information of original features to the place connected by the CoordCSP module, and the position information completely matched with the features having explicit position numbers and the output of the neck are connected in the channel, and finally the pixel-level spatial information and channel information fusion is completed through 1*1 vanilla convolution, and the fused feature map is transmitted to the next layer network; The design of the four-layer prediction layers based on the CoordCSP module specifically comprises: adding a prediction head for micro object detection, combining the other three prediction heads, and using four prediction heads to accurately predict small targets, wherein the added prediction head is generated from a low-level and high-resolution feature map, and is more sensitive to micro objects.

Images