Small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structure
By introducing non-near jump feature pyramid structure and multi-scale residual edge profile feature extraction strategy on the YOLOv5s model, the missed detection and missed detection of small target vehicles in aerial photography scenarios is solved, and higher detection accuracy and real-time sex.
Patent Information
- Application Number
- CN202310485268.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-04-28
AI Technical Summary
Existing models find it difficult to accurately detect small-target vehicles in complex environments (occlusion, lighting, long distance, small targets, etc.), resulting in missed detection, missed detection and blurred target profile features, especially in aerial photography scenarios.
Using YOLOv5s as the benchmark model, combined with non-near jump connection and multi-scale residual structure, the NHN-YOLOv5s-MREFE model is designed. Through the non-near jump connection feature pyramid structure and multi-scale residual edge profile feature extraction strategy, the feature layer is enhanced. Addition and fusion and information interaction to improve detection accuracy and real-time.
It significantly improves the accuracy and real-time detection of small-target vehicles, reduces the missed and false detection rates, improves the detection accuracy, and can better adapt to the complex environment in aerial photography scenarios.
Smart Images

Figure CN116630904B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision target detection, and in particular relates to a small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structures. Background Art
[0002] Vehicle detection is a computer vision task that classifies and locates real-time changing targets in complex scenes. The implementation methods include multi-step traditional detection methods and end-to-end deep learning detection methods. Traditional detection methods require manual design of target features, and the traditional classifier outputs the results through a cumbersome sliding window strategy. Not only is the recognition rate low, but it is also quite time-consuming and cannot meet the requirements of vehicle detection accuracy and real-time performance. The deep learning detection method automatically learns target features through a deep network, and obtains results through candidate boxes or direct regression. It is both accurate and time-saving, including two-stage detectors such as RCNN, SPP-Net, Fast RCNN, Faster RCNN and single-stage detectors such as YOLO and SSD. Compared with the two-stage detector, the single-stage YOLO algorithm has high accuracy, fast convergence, and can directly locate and classify targets, which better meets the real-time requirements. However, in the detection of small targets, there are still serious problems of missed detection, false detection, and blurred target contour features.
[0003] In response to the above problems, relevant studies have analyzed the main reasons and proposed corresponding solutions: (1) Since the existing model lacks an efficient feature fusion strategy, it is difficult for information between different levels to interact in time. To this end, Li et al. proposed a densely nested attention network (DNA-Net), which achieves progressive interaction between high-level and low-level features by densely nesting interactive modules, and combines cascade channels and spatial attention modules to enhance multi-level features. This method has significantly improved the detection accuracy under different scales, shapes and signal-to-noise ratio conditions. (2) Occlusion, lighting, long distance and other scenes seriously affect the model's extraction of image features. In order to enhance the model's use of vehicle contour information, Du Wenhan et al. proposed a moving target edge extraction method based on an improved Canny operator, which extracts the moving target edge of the current frame to solve the problem of missed recognition due to missing edge details. (3) Downsampling will cause the loss of image features, affecting the detection results. Xianbao et al. adopted an image double cutting method to fuse the upper-level semantics of the segmented image with the lower-level position information to reduce the loss of feature values. In addition, the study found that the size and number of the prior box sizes and the mismatch between the receptive field sizes of small target objects in the feature map and the original image are also important reasons for the high missed detection rate and false detection rate.
[0004] Based on the above, the present invention proposes a small target vehicle detection algorithm that integrates non-adjacent skip connections and multi-scale residual structures by studying the visual characteristics of small target vehicles in aerial photography scenes. Summary of the invention
[0005] Purpose of the invention: The purpose of the present invention is to provide a small target vehicle detection method that integrates non-adjacent skip connections and multi-scale residual structures. The existing model is difficult to generalize directly to the small target detection task in the aerial photography scene because the target detection model based on the deep convolutional neural network is easily affected by complex environments (occlusion, illumination, long distance, small targets, etc.), resulting in missed detection, false detection and blurred target contour features. To this end, the present invention takes YOLOv5s as the benchmark model, uses theoretical analysis, image processing and ablation experiments, and considers multi-scale detection, deconvolution, parallel strategy, clustering algorithm, non-adjacent feature fusion and edge contour feature extraction strategy, etc., and innovatively proposes a new small target vehicle detection model (NHN-YOLOv5s-MREFE). The effects of increasing the field of view, expanding the detailed information and promoting the interaction of non-adjacent hierarchical information on the detection effect of small target vehicles in aerial photography scenes are analyzed. Following the principle of gradual detailing, the feature layer is added and fused to obtain rich edge information, and the backbone network is assisted to extract image features, so that the NHN-YOLOv5s-MREFE model has both detection accuracy and real-time performance.
[0006] Technical solution: A small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structures, comprising the following steps:
[0007] (1) Obtain target data and preprocess it to complete the preparation of the data set;
[0008] (2) The network structure of the benchmark model YOLOv5s, including the backbone network (Backbone), neck network (Neck) and detection head (Head), is optimized and improved to construct a new network model NHN-YOLOv5s-MREFE;
[0009] (3) Set network model parameters and conduct training according to the experimental environment;
[0010] (4) Set up ablation experiments and comparative experiments to verify the detection performance of the model, and evaluate indicators such as precision (P), recall (R), mean average precision (mAP), number of image detections per unit time (frames per second, FPS) and model parameters (Params).
[0011] Furthermore, the step (1) comprises the following steps:
[0012] (1.1) Data acquisition: To make the experimental results more rigorous, the present invention uses aerial videos in different scenes as data sources, and the pre-processed data is used for training and testing of the model;
[0013] (1.2) Data preprocessing: To achieve the diversification of sample data, vehicle videos of different scenes, different environments, and different densities are screened and frame-processed;
[0014] (1.3) Data labeling: Use open source software to label data, and define small private cars and large vehicles such as buses and tankers as vehicles and bigvehicles respectively;
[0015] (1.4) Dataset division: The optimized dataset is divided into training set, validation set and test set in a ratio of 8:1:1.
[0016] Furthermore, the step (2) comprises the following steps:
[0017] (2.1) Design four detection layers of different scales. The model can detect vehicles of different sizes according to the size of its receptive field.
[0018] (2.2) Drawing on the idea of dense skip connection of DenseNet, a non-adjacent skip connection feature pyramid structure (Non-adjacent Hop Network, NHN) is constructed. Through the skip addition strategy, while strengthening the interaction of non-adjacent layer information, more unaffected original information is integrated to solve the problem of gradual dilution of position information during the transmission process, effectively reducing the false detection rate of the model;
[0019] (2.3) Design a multi-scale residual edge contour feature extraction strategy (MREFE), follow the principle of gradual refinement of features, construct a multi-scale residual structure, use a dual-branch parallel method to capture multi-scale information at different levels, and extract image edge features by subtracting high semantic information at multiple scales from initial shallow information pixel by pixel, thereby assisting the network model to complete target classification;
[0020] (24) The K-Means++ algorithm is used to decentralize the cluster centers, which makes the results reach the global optimum and accelerates the convergence of the model.
[0021] Furthermore, the implementation process of step (21) is as follows:
[0022] In the NHN-YOLOv5s-MREFE model, the 640×640 input image is downsampled by 4, 8, 16, and 32 times to generate feature maps of four different scales. The anchor size increases with the increase of the downsampling multiple, and the number of anchors increases from 9 to 12.
[0023] Furthermore, the implementation process of step (22) is as follows:
[0024] Drawing on the idea of dense skip connection of DenseNet, this paper constructs a new feature pyramid network structure, Non-adjacent Hop Network (NHN), which is used for real-time detection of small target vehicles in aerial photography scenes. Figure 4 As shown. Based on the inherent pyramid structure of convolutional neural networks (such as Figure 4 A top-down (as shown in a) is constructed through parallel strategies. Figure 4 b in the figure) and a propagation path from left to right (as shown in Figure 4 c in the figure), so that the semantic information of the deep layer is transferred to the shallow layer, and the position information of the shallow layer is integrated with the deep layer. In addition, the skip-addition strategy (such as Figure 4 (As shown in the figure) not only strengthens the information interaction at all levels, but also integrates more unaffected original information, solves the problem of location information being gradually diluted during the transmission process, and effectively reduces the false detection rate of the model.
[0025] Furthermore, the implementation process of step (2.3) is as follows:
[0026] Considering the real-time performance of the algorithm and combining the purpose of strengthening feature extraction, this paper proposes a multi-scale residual edge contour feature extraction strategy (MREFE). Different from the traditional grayscale transformation and threshold control method, the MREFE strategy follows the principle of gradual feature refinement and integrates parallel strategies to construct a multi-scale residual structure. It mainly realizes image edge feature extraction through the difference between high semantic information and initial shallow information at multiple scales, such as Figure 7 As shown, the specific steps are as follows:
[0027] S1: Feature map F output by NHN through 2D convolution and deconvolution i,j Adjust the size and number of channels;
[0028] S2: Subtract the high-level semantic information at multiple scales from the initial shallow information to extract the vehicle contour information. The formula is as follows:
[0029]
[0030] In the formula, E i,j Indicates the extracted edge contour information; F i,j It means that the high semantic information feature map obtained by executing step S1 has the same size and number of channels as X i,j Same; X i,j It represents the feature map output by the backbone network in the feature extraction stage; W and H represent the grid cells corresponding to the feature map.
[0031] S3: Edge contour information E i,j With rich high semantic information F i,j Add and fuse multi-scale features to make the vehicle outline clearer. The formula is as follows:
[0032]
[0033] In the formula, Out i,j Indicates edge contour information E i,j With the feature map F i,j The output after addition and fusion.
[0034] S4: Out is processed by SiLu function and 1×1 convolution respectively. i,j After completing the activation operation and feature compression, the results are output.
[0035] Furthermore, the implementation process of step (2.4) is as follows:
[0036] Table 1 K-Means++ clustering results
[0037]
[0038] The NHN-YOLOv5s-MREFE model uses the K-Means++ clustering algorithm to disperse the cluster centers as much as possible, generating 12 groups of anchors with different aspect ratios, which makes the results reach the global optimal. The results are shown in Table 1.
[0039] Furthermore, the step (3) comprises the following steps:
[0040] In order to verify the effectiveness of the NHN-YOLOv5s-MREFE model, this experiment uses the YOLOv5s-6.0 version as the basic framework to design an ablation experiment, and compares and analyzes it with several common mainstream algorithms in the field of target detection. The experimental hardware environment is as follows:
[0041] Table 2 Experimental environment configuration
[0042]
[0043] Before training, the model parameters are configured. After many experiments, the following configuration is the best: the input image size is 640×640, the number of iterations is 400 rounds, the Batch_Size is 10, the initial learning rate is set to 0.01, the learning rate momentum is 0.937, and the weight decay coefficient is 0.0005. The initial learning rate is set to 0.01, the learning rate momentum is 0.937, and the weight decay coefficient is 0.0005. The IoU is set to 0.5. When the intersection over union ratio of the predicted box and the true box is greater than or equal to 0.5, it is considered that the target position is successfully predicted. In order to promote the model to achieve global optimization, the first 10 epochs are preheated using warm up training. After the preheating training, the cosine annealing algorithm is used to adjust the learning rate again. In addition, the rest are the default parameters of version 6.0.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] In the field of small target vehicle detection algorithm practice, the main problems existing in existing research are: ① The existing model lacks an efficient feature fusion strategy, and information between different levels is difficult to interact in time. ② Scenes such as occlusion, lighting, and long distance seriously affect the model's extraction of image features, and the blurred edge details between targets and non-targets lead to a high model false detection rate. ③ Downsampling leads to loss of image features, affecting the detection results. The above situation is very likely to cause problems such as missed detection, false detection, and blurred target contour features of small target vehicles. In addition, the size and number of the prior box size and the inability to correspond to the receptive field size of small target objects in the feature map and the original image are also important reasons for the high missed detection rate and false detection rate. Therefore, the present invention proposes a small target vehicle detection model (NHN-YOLOv5s-MREFE) that integrates non-adjacent skip connections and multi-scale residual structures by studying the visual features of small target vehicles in aerial photography scenes. The specific innovations are as follows:
[0046] (1) In view of the problem that the target vehicles in the aerial photography scene are tiny and the YOLOv5s algorithm misses detection seriously, four detection layers of different scales are designed. According to the size of their own receptive fields, the receptive field of each grid unit corresponding to the original image is made smaller, which is specially used for the detection of tiny targets. At the same time, in multi-scale target detection, the large difference in the size of the objects being detected will affect the detection accuracy of the model, which will lead to the problem of serious false positive rate. Drawing on the ideas of feature pyramid network FPN, path aggregation network PANet and DenseNet dense network, a new feature pyramid structure is innovatively constructed - Non-adjacent Hop Network (NHN), which is used to promote the feature fusion of deep and shallow layers of the network, so that the location information is more abundant in the deep layer and the semantic information is more complete in the shallow layer. The skip addition strategy is used to strengthen the information interaction between non-adjacent layers, reduce feature loss, and effectively reduce the false positive rate of the model.
[0047] (2) To address the problem of image feature loss caused by downsampling, deconvolution and parallel strategies are introduced to expand the detailed information of small target vehicles by increasing the image resolution and breaking through the amount of information in each dimension, thereby improving the robustness of the detector and effectively solving the problem that the detector cannot extract detailed features in complex scenes.
[0048] (3) In order to solve the problem of high false detection rate caused by inaccurate edge detail features between targets and non-targets in complex scenes, a multi-scale residual edge contour feature extraction strategy (MREFE) is designed. Following the principle of gradual refinement of features, a multi-scale residual structure is constructed. A dual-branch parallel method is used to capture multi-scale information at different levels. Image edge features are extracted by pixel-by-pixel difference between high semantic information at multiple scales and initial shallow information, thereby assisting the network model to complete target classification. At the same time, in order to match the detection model, the K-Means++ clustering algorithm is used to generate 12 groups of prior boxes that match the data set, which enables the results to reach the global optimal and improves the detection accuracy of the model.
[0049] Finally, through experimental verification in two typical application scenarios, intersections and lanes along the way, the results show that the NHN-YOLOv5s-MREFE algorithm has good detection accuracy and adaptability, and the mAP value is improved by 10.5%, 6.4%, 22.6% and 8.1% compared with Faster RCNN, YOLOv3, YOLOv4 and YOLOv5s respectively. In addition, the size and position distribution of the prediction box of the algorithm are more accurate, and it also shows a more satisfactory recognition effect for the problems of missed detection and false detection that exist in other mainstream algorithms. At the same time, it can better balance the detection speed and accuracy, which meets the requirements of real-time and accuracy of small target vehicle detection in aerial photography scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a flow chart of the present invention;
[0051] Figure 2 Schematic diagram of the NHN-YOLOv5s-MREFE model structure;
[0052] Figure 3 Schematic diagram of multi-scale detection structure;
[0053] Figure 4 It is a schematic diagram of the structure of a non-adjacent hop network (NHN);
[0054] Figure 5 Schematic diagram of upsampling, deconvolution and parallel strategy, Concat structure;
[0055] Figure 6 It is a schematic diagram of the structure of the multi-scale residual edge contour feature extraction strategy (MREFE);
[0056] Figure 7 This is an effect identification diagram of each comparison algorithm in different scenarios; DETAILED DESCRIPTION
[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings.
[0058] The purpose of the present invention is to provide a small target vehicle detection method that integrates non-adjacent skip connections and multi-scale residual structures. The existing models are difficult to generalize directly to small target detection tasks in aerial photography scenarios, and are easily affected by complex environments (occlusion, lighting, long distance, small targets, etc.), resulting in missed detections, false detections, and blurred target contour features. The present invention uses multi-scale detection, non-adjacent feature fusion, multi-scale residual structure, and cluster center decentralization as entry points, so that the model can effectively reduce missed detection rates and false detection rates while being able to accurately and quickly detect target vehicles with variable shapes, blurred edges, and small sizes. See for details. Figure 1 , the specific steps are as follows:
[0059] Step 1: Data preprocessing and dataset creation.
[0060] The present invention takes into account three aspects: rigor, scientificity and the hardware level of experimental equipment. Aerial videos in different scenarios are used as data sources. The intersections and lanes along the way are photographed vertically and at an angle of 45°. The video resolution is 1080p, and the absolute pixel mean of the vehicle to be detected is approximately 30×15, which meets the definition of small targets in the MS COCO dataset.
[0061] In order to achieve the diversification of sample data, vehicle videos of different scenes, different environments, and different densities were screened and framed. One picture was taken every 30 frames as a sample, and a total of 1,900 pictures were recorded. They were annotated using open source software. When the target occlusion exceeded 90% or the target pixel was less than 8×8, it was discarded and not annotated. Small private cars and large vehicles such as buses and tankers were defined as vehicle and bigvehicle respectively, and the training set, validation set, and test set were set in a ratio of 8:1:1.
[0062] Step 2: Optimize and improve the network structure of the benchmark model YOLOv5s, including the backbone network (Backbone), neck network (Neck) and detection head (Head), and build a new network model NHN-YOLOv5s-MREFE, such as Figure 2 shown.
[0063] (1) YOLOv5s can specifically predict large, medium and small targets through three detection layers of different scales. However, since the vehicles in aerial photography scenes are even smaller, YOLOv5s has a serious missed detection problem and can no longer meet the detection requirements. Therefore, it is necessary to add a detection layer to be responsible for the detection of tiny vehicles. The present invention designs four detection layers of different scales, and the model can be responsible for the detection of vehicles of different sizes according to the size of its own receptive field. In the NHN-YOLOv5s-MREFE model, the input image of 640×640 is downsampled by 4 times, 8 times, 16 times and 32 times respectively to generate feature maps of four different scales. The anchor size also increases with the increase of the downsampling multiple, and the number increases from the original 9 to 12, such as Figure 3 shown.
[0064] In the process of forward inference, the four detection layers output the corresponding prediction information, including the center point coordinates (x, y), width w, height h and confidence parameter c of the prediction box. By comparing with the label information, the loss between the predicted value and the true value is calculated, which guides the adjustment of parameters in the back propagation, thereby optimizing the model performance in the process of repeated training. The entire loss includes three parts: positioning loss Loss ciou , classification loss Losscls and confidence loss Loss obj , the formula is as follows:
[0065] Loss total =Loss ciou +Loss cls +Loss obj
[0066] In the formula, the positioning loss Loss ciou Used to calculate the error between the predicted box and the calibrated box; classification loss Loss cls Used to determine whether the classification of anchor boxes and calibration boxes is correct; Loss obj Used to calculate the network confidence loss.
[0067] (2) Multi-scale object detection has always been a difficult problem in computer vision tasks. To address the problem that large differences in the sizes of the objects being detected can lead to a decrease in model accuracy, the feature pyramid network FPN and its improved method PANet use multi-scale splicing and fusion to enhance shallow semantic information and deep position information, thereby improving model performance. YOLOv5s uses a feature pyramid structure constructed by a path aggregation network, which improves the detection accuracy and reasoning speed of the model. However, due to the influence of splicing and fusion, in the process of information transmission (such as Figure 4 As shown in c in the figure, the shallow position information will be gradually diluted, resulting in a deviation in the position of the prediction result. At the same time, the sequential integration feature method pays more attention to the feature extraction of adjacent levels, ignoring the information interaction with other levels, which is easy to cause misidentification of the target.
[0068] To solve the above problems, the present invention draws on the idea of dense skip connection of DenseNet to construct a new feature pyramid network structure, the non-adjacent skip connection network NHN (Non-adjacent Hop Network), which is used for real-time detection of small target vehicles in aerial photography scenes. Figure 4 As shown. Based on the inherent pyramid structure of convolutional neural networks (such as Figure 4 A top-down (as shown in a) is constructed through parallel strategies. Figure 4 b in the figure) and a propagation path from left to right (as shown in Figure 4 c in the figure), so that the semantic information of the deep layer is transferred to the shallow layer, and the position information of the shallow layer is integrated with the deep layer. In addition, the skip-addition strategy (such as Figure 4 (As shown in the figure) not only strengthens the information interaction at all levels, but also integrates more unaffected original information, solves the problem of location information being gradually diluted during the transmission process, and effectively reduces the false detection rate of the model.
[0069] Path (a) corresponds to the backbone network of the model structure, which consists of 5 stages, each of which is a combination operation and has the same size, such as stage 1 = Conv + BN + SiLU, stage 2 = Conv + BN + SiLU + CSP1_1. The different stages are scaled by 2 times downsampling, and the last layer with the strongest features is used as the output of the stage. Finally, a 1×1 convolution operation is used to control the number of channels in stage 5, and path (b) is constructed with the help of horizontal addition.
[0070] In path (b), drawing on the ideas of FPN and PANet, the NHN network uses deconvolution to reshape the feature map size between each stage. Compared with the upsampling interpolation algorithm, deconvolution not only achieves pixel filling, but also increases the parameter learning process. While increasing the resolution of the feature map, it restores more small target information, such as Figure 5 As shown in the figure, the output of each stage is horizontally added to the corresponding features in path (a), which strengthens the expression ability of semantic information in the shallow layer. However, since multi-scale target detection requires separate predictions at each stage, a path (c) is constructed by performing 1×1 convolution on the bottom layer and combining downsampling and horizontal addition to strengthen the deep position information.
[0071] By adding feature layers horizontally, the amount of information in each dimension of the feature map is increased. Compared with Concat splicing and fusion, the parallel strategy changes the inherent form of increasing the dimension of the feature map and focuses more on the breakthrough of the amount of information in each dimension, such as Figure 5 At the same time, since the feature maps of the corresponding channels have similar semantics, the parallel strategy allows them to share a convolution kernel, which greatly reduces the number of parameters. If we only look at the output of a single channel, let the channels of the two inputs be A1, A2, ...A c and B1, B2…B c , the formula is as follows:
[0072]
[0073]
[0074] Where c is the number of channels; A i , B i There are two input channels; K i is the convolution kernel of the corresponding channel.
[0075] (3) The quality of vehicle contour information extraction is the key to achieving excellent results, especially in the presence of occlusion or extreme environments (light, rain, snow). At the same time, since edge contour information mainly exists between targets and targets, and between targets and backgrounds. Therefore, most existing algorithms use Sobel operators or Canny operators to detect pixel neighborhoods one by one and quantize them through grayscale transformation to determine the edge position. The principle is roughly divided into the following steps: denoising, gradient calculation, non-maximum suppression, threshold setting and filtering. It can indeed achieve good results, but it does not consider the complexity of the algorithm itself and its compatibility with deep neural network models, and is lacking in algorithm complexity and model inference speed.
[0076] Considering the real-time performance of the algorithm and combining the purpose of strengthening feature extraction, this paper proposes a multi-scale residual edge contour feature extraction strategy (MREFE). Different from the traditional grayscale transformation and threshold control method, the MREFE strategy follows the principle of gradual feature refinement and integrates parallel strategies to construct a multi-scale residual structure. It mainly realizes image edge feature extraction through the difference between high semantic information and initial shallow information at multiple scales, such as Figure 6 shown.
[0077] The feature map F is enhanced by the non-adjacent skip-connection feature pyramid network NHN i,j It has rich detail information, and the backbone network has low sensitivity to vehicle contour information, which makes the feature map X i,j Therefore, MREFE uses a dual-branch parallel method to capture multi-scale information at different levels. i,j With X i,j Perform pixel-by-pixel subtraction to extract the edge features of the vehicle. This helps the NHN-YOLOv5s-MREFE model to achieve target classification, improve the detection accuracy of the model, and reduce the missed detection rate and false detection rate. The specific steps are as follows:
[0078] S1: Feature map F output by NHN through 2D convolution and deconvolution i,j Adjust the size and number of channels;
[0079] S2: Subtract the high-level semantic information at multiple scales from the initial shallow information to extract the vehicle contour information. The formula is as follows:
[0080]
[0081] In the formula, E i,j Indicates the extracted edge contour information; F i,jIt means that the high semantic information feature map obtained by executing step S1 has the same size and number of channels as X i,j Same; X i,j It represents the feature map output by the backbone network in the feature extraction stage; W and H represent the grid cells corresponding to the feature map.
[0082] S3: Edge contour information E i,j With rich high semantic information F i,j Add and fuse multi-scale features to make the vehicle outline clearer. The formula is as follows:
[0083]
[0084] In the formula, Out i,j Indicates edge contour information E i,j With the feature map F i,j The output after addition and fusion.
[0085] S4: Out is processed by SiLu function and 1×1 convolution respectively. i,j After completing the activation operation and feature compression, the results are output.
[0086] (4) Selecting a suitable prior frame can accelerate model convergence and improve detection speed and accuracy. However, with the change of scenes and the increase of detection layers, the number and size of anchors determined based on the MS COCO dataset can no longer meet the detection needs. However, YOLOv5s still uses the same K-Means algorithm as YOLOv2 to randomly initialize the cluster centers, resulting in the initial values being too concentrated, and then a local optimal solution appears. To this end, the NHN-YOLOv5s-MREFE model uses the K-Means++ clustering algorithm to make the cluster centers as dispersed as possible, generating 12 groups of anchors with different aspect ratios, which makes the results reach the global optimal, as shown in Table 1.
[0087] Table 1 K-Means++ clustering results
[0088]
[0089] The results show that compared with the original sizes [5, 6, 8, 14, 15, 11], [10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326], the re-clustered results are more concentrated and the aspect ratio is more in line with the distribution characteristics of the small target vehicle dataset.
[0090] Step 3: Set the network model parameters and perform training according to the experimental environment.
[0091] In order to verify the effectiveness of the NHN-YOLOv5s-MREFE model, this experiment uses the YOLOv5s-6.0 version as the basic framework to design an ablation experiment, and compares and analyzes it with several common mainstream algorithms in the field of target detection. The experimental hardware environment is as follows:
[0092] Table 2 Experimental environment configuration
[0093]
[0094] During the training and testing phases of the NHN-YOLOv5s-MREFE model, the input images were scaled to 640×640, and the backbone network (CSPDarkNet-53) extracted image features. Because the dataset contains small target vehicles and contains a large amount of label information, direct training can achieve the desired effect. Due to the change in the model structure, the official pre-training weights will no longer be applicable to this model. The model parameters were configured before training. After multiple experiments, the following configuration was found to be the best:
[0095] For the scientific nature of the experiment, the pre-trained weights were not loaded in the ablation experiment and the comparative test. After many experiments, it was found that 400 iterations were optimal, and 10 samples were set to be input each time (batch size = 10). To accelerate model convergence, the Adam optimizer was used instead of the traditional stochastic gradient descent method to adaptively adjust the model parameters, and the initial learning rate was set to 0.01, the learning rate momentum was set to 0.937, and the weight decay coefficient was set to 0.0005. The IoU was set to 0.5. When the intersection over union ratio of the predicted box and the true box was greater than or equal to 0.5, it was considered that the target position was successfully predicted. In order to promote the model to achieve global optimality, the first 10 epochs were preheated using warm up training, and the cosine annealing algorithm was used to adjust the learning rate again after the preheating training. In addition, the rest are the default parameters of version 6.0.
[0096] Step 4: In order to verify whether the NHN, deconvolution, parallel strategy, MREFE and K-means++ clustering algorithm mentioned in the NHN-YOLOv5s-MREFE model are conducive to improving the detection accuracy of small target vehicles in aerial photography scenarios, an ablation experiment is designed to objectively evaluate the network model and analyze it in combination with the experimental results.
[0097] In order to accurately evaluate the performance of the NHN-YOLOv5s-MREFE model, the present invention uses precision (P), recall (R), average precision (AP), mean average precision (mAP), number of image detections per unit time (frames per second, FPS) and model parameters (Params) as evaluation indicators. The formula is as follows:
[0098]
[0099]
[0100]
[0101]
[0102] In the formula, TP (True Positive) indicates the accurate determination of positive samples; FP (False Positive) indicates the wrong determination of negative samples; FN (False Negative) indicates the wrong determination of positive samples; k indicates the sample data category.
[0103] Table 3 NHN-YOLOv5s-MREFE ablation experiment
[0104]
[0105] Note: NHN is a non-adjacent skip feature pyramid network, MREFE is a multi-scale residual edge contour feature extraction strategy, and ★ indicates the application of the corresponding strategy.
[0106] Table 3 shows the ablation experiment results of NHN-YOLOv5s-MREFE, showing the accuracy, recall and mAP values of the combined model with YOLOv5s as the benchmark and different strategies after 400 iterations. Among them, the performance of model 7 after introducing various improved strategies, that is, the NHN-YOLOv5s-MREFE model proposed in this paper, has the most significant improvement. Compared with model 1 (YOLOv5s), the precision (P), recall (R) and mean average precision (mAP) have increased by 13.7%, 1.6% and 8.1% respectively. The effectiveness of this algorithm in small target vehicle recognition in aerial photography scenarios has been verified.
[0107] As can be seen from Table 3, the application of NHN (non-adjacent skip feature pyramid network) has a positive impact on the performance improvement of model 1 (baseline model). Through the skip addition of NHN, the model's ability to fuse deep semantics and shallow position information is enhanced, and the features of non-adjacent layers are reused in the spatial dimension, which effectively solves the problem of prediction box position deviation caused by inefficient feature fusion. Compared with model 1 (YOLOv5s), the precision (P) of NHN is improved by 3.3 percentage points. In addition, in the NHN structure, the addition of the small target detection layer at the prediction end realizes the correspondence between the receptive field and the small-sized targets of the original image, which improves the recall rate (R) of the model by 3.1 percentage points and reduces the problem of missed detection of small vehicles. However, due to the complex scene information and fewer imaging pixels, it brings great difficulties to the subsequent feature extraction of the model. To this end, this paper applies deconvolution and parallel strategies in models 3 and 4 respectively.
[0108] The ablation data shows that compared with model 2 (NHN), all indicators of model 3 (deconvolution) have declined. Model 4 (parallel strategy) sacrificed 3% recall rate in exchange for 6.7% improvement in accuracy. Model 5, which introduced both deconvolution and parallel strategy, improved the model accuracy by 3.6% while ensuring that the recall rate remained basically unchanged. It can be seen that although deconvolution can adaptively learn an optimal upsampling method through parameter learning, when the kernel size cannot divide strides or the convolution kernel weights are uneven, the image will produce a checkerboard effect, and this effect cannot be eradicated, which seriously affects the performance of the model. The parallel strategy can significantly improve the detection accuracy of the model, and by expanding the amount of information in each dimension of the feature map, it weakens the influence of the checkerboard effect to a certain extent.
[0109] In order to further improve the performance of the model and reduce the impact of occlusion or extreme environments, the present invention applies the MREFE strategy (multi-scale residual edge contour feature extraction strategy) and introduces K-means++ to re-cluster the annotation boxes on the basis of model 5 (NHN+deconvolution+parallel strategy). The data shows that the MREFE strategy can greatly improve the detection accuracy by integrating the image contour information, and only reduce the recall rate by a small amount. The K-means++ algorithm can re-cluster according to the complex scene information, determine the number and size of the Anchor, accelerate the convergence of the model, and improve the detection speed and accuracy.
[0110] Table 4 Performance comparison of this algorithm with other mainstream algorithms
[0111]
[0112] Note: The bold words represent the experimental results of the algorithm in this paper.
[0113] In order to further verify the superiority of the NHN-YOLOv5s-MREFE model performance, the number of image detections per unit time (FPS) and the number of model parameters (Params) are introduced to compare the detection speed and model size. And compared with several common mainstream algorithms in the field of object detection, including One-Stage v3, v4, v5s and Two-Stage Faster R-CNN. For the selection of backbone network, Faster R-CNN is ResNet-50, v3 is DarkNet-53, and v4, v5s and NHN-YOLOv5s-MREFE all use CSPDarkNet-53 as the backbone network. In addition, the input image size of Faster R-CNN is the largest, which is 800×800. The image sizes of (v3, v4) and v5s are uniformly scaled to 416×416 and 640×640, and three image features of different scales are output respectively.
[0114] Table 5 Comparison of model misdetection and missed detection
[0115]
[0116] Note: The unit of experimental data is (vehicles), a represents the intersection (vertical), b represents the lane along the way (inclined), and c represents the lane along the way (vertical).
[0117] In order to objectively evaluate the results, the experiment used the same hyperparameters (the same as the ablation experiment) to train the model under the same configuration. The test results are detailed in Table 4.
[0118] In the specific application process, in addition to the camera, other sensors such as lidar, infrared sensors, etc. can also be installed on the drone to obtain more scene information. First, we control the drone to capture scene images at a specific height, angle, and position through a remote control or an automatic driving system. Secondly, a base station device is built in the take-off area of the drone to obtain the image data sent back by the drone. Finally, the image data is uploaded to the cloud platform, and the NHN-YOLOv5s-MREFE model is used to detect and identify small target vehicles, and the visualization results are displayed in real time. Table 5 and Figure 7 The false detection, missed detection data and visualization results of each algorithm in different scenarios are compared. The specific data are shown in Table 5.
[0119] Combining Table 4 and Figure 7Analysis shows that Faster R-CNN, as a representative algorithm of Two-Stage target detection, has a certain ability to detect small target vehicles in aerial photography scenes. Compared with the One-Stage YOLO algorithm, although its real-time performance is far from meeting the needs of practical applications, in most scenes, the positioning and size of various prediction boxes are correct and the category judgment is basically correct. This shows that the size of the input image resolution can affect the detection performance of the model. The larger the resolution, the more conducive it is to extracting image features.
[0120] Compared with Faster R-CNN, the overall performance of One-Stage's YOLO algorithm has been greatly optimized, and all indicators have been improved to a certain extent. Among them, the detection speed has been improved most significantly. Compared with Faster R-CNN, YOLOv3 and YOLOv4 have increased by 4 times, and YOLOv5s has increased by more than 8 times. However, the problems of missed detection and false detection are serious, and it cannot meet the accuracy requirements of small target vehicle detection.
[0121] The NHN-YOLOv5s-MREFE proposed in this paper has the best mAP value performance among the above algorithms, which is 10.5%, 6.4%, 22.6% and 8.1% higher than Faster RCNN, YOLOv3, YOLOv4 and YOLOv5s respectively. The size and position distribution of the prediction box of this algorithm are more accurate, and it also shows a more satisfactory recognition effect for the missed detection and false detection problems existing in other mainstream algorithms. Figure 7 It can be seen that due to the long shooting distance, small target and inaccurate edge detail features between the target and the guardrail, other algorithms have false detections, such as: unable to accurately judge the type of vehicle (Faster RCNN-c, YOLOv3-bc, YOLOv5s-c), generating multiple frames for one object (Faster RCNN-b, YOLOv3-a, YOLOv5s-ac) or judging the guardrail as a vehicle (YOLOv4-c, YOLOv5s-ab). Only the NHN-YOLOv5s-MREFE algorithm can accurately detect all targets and has a lower missed detection rate.
[0122] Since the non-adjacent skip feature pyramid network (NHN) and the multi-scale residual edge contour feature extraction strategy (MREFE) consume a certain amount of computing time, the model detection speed is slower than that of YOLOv5s, but it is still 6 times that of FasterRCNN, 1.5 times that of YOLOv3 and YOLOv4, which can meet the real-time requirements. In addition, the model parameters are only 1 / 3 of Faster RCNN, 1 / 6 of YOLOv3, and 1 / 5 of YOLOv4, which is easy to embed into small devices such as drones.
[0123] Based on the comprehensive experimental results, the NHN-YOLOv5s-MREFE algorithm can better balance the detection speed and accuracy, slightly reduce the detection speed, but significantly improve the detection accuracy, and can meet the real-time requirements of target detection, and is more suitable for detection tasks in aerial photography scenarios.
Claims
1. A small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structures, characterized in that: The following steps are involved: (1) Obtain target data and preprocess it to complete the preparation of the data set; (2) The network structure of the benchmark model YOLOv5s, including the backbone network (Backbone), neck network (Neck) and detection head (Head), is optimized and improved to construct a new network model NHN-YOLOv5s-MREFE, which includes: (2.1) Design four detection layers of different sizes. The model is responsible for detecting vehicles of different sizes according to the size of its own receptive field. Implementation process: In the NHN-YOLOv5s-MREFE model, the 640×640 input image is downsampled by 4, 8, 16, and 32 times to generate feature maps of four different scales. The anchor size increases with the increase of the downsampling multiple, and the number of anchors increases from 9 to 12. (2.2) Drawing on the idea of dense skip connection of DenseNet, a non-adjacent skip connection feature pyramid structure (Non-adjacent Hop Network, NHN) is constructed. Through the skip addition strategy, while strengthening the interaction of non-adjacent layer information, more unaffected original information is integrated to solve the problem of gradual dilution of position information during the transmission process, effectively reducing the false detection rate of the baseline model. Implementation process: S1: Based on the inherent pyramid structure of convolutional neural networks, a top-down and a left-to-right propagation path are constructed through a parallel strategy, so that the semantic information of the deep layer is transmitted to the shallow layer, and the position information of the shallow layer is integrated with the deep layer; S2: Adding a skip-add strategy, while strengthening the information interaction at each level, it integrates more unaffected original information, solves the problem of the location information being gradually diluted during the transmission process, and effectively reduces the false detection rate of the baseline model; (2.3) Design a multi-scale residual edge contour feature extraction strategy, follow the principle of gradual feature refinement, construct a multi-scale residual structure, and use a dual-branch parallel method to capture multi-scale information at different levels. The image edge feature is extracted by pixel-by-pixel difference between high semantic information at multi-scale and initial shallow information, and then assist the NHN-YOLOv5s-MREFE network model to complete target classification. Implementation process: S1: Feature map F output by NHN through 2D convolution and deconvolution i,j Adjust the size and number of channels; S2: Subtract the high semantic information at multiple scales from the initial shallow information to extract the vehicle contour information. Formula: In the formula, E i,j Indicates the extracted edge contour information; F i,j represents the high semantic information feature map obtained by executing step S1, whose size and number of channels are the same as X i,j Same; X i,j It represents the feature map output by the backbone network in the feature extraction stage; W and H represent the grid cells corresponding to the feature map respectively; S3: Edge contour information E i,j With rich high semantic information F i,j Add and fuse multi-scale features to make the vehicle outline clearer. The formula is: In the formula, Out i,j Indicates edge contour information E i,j With the feature map F i,j Output after addition and fusion; S4: Out is processed by SiLu function and 1×1 convolution respectively. i,j After completing the activation operation and feature compression, output the results; (2.4) The K-Means++ algorithm is used to decentralize the cluster centers, which makes the results reach the global optimal level and accelerates the convergence of the NHN-YOLOv5s-MREFE model. Implementation process: The NHN-YOLOv5s-MREFE model uses the K-Means++ clustering algorithm to disperse the cluster centers as much as possible, generating 12 groups of anchors with different aspect ratios, which makes the results reach the global optimal. (3) Set network model parameters and conduct training according to the experimental environment; (4) Set up ablation experiments and comparative experiments to verify the detection performance of the model. The evaluation indicators include: precision (Precision, P), recall (Recall, R), mean average precision (mean average precision, mAP), number of image detections per unit time (frames per second, FPS) and model parameters (Params).
2. The small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structure according to claim 1 is characterized in that: The step (1) comprises the following steps: (1.1) Data acquisition: In order to make the experimental results more rigorous, aerial videos in different scenes are used as data sources, and the preprocessed data is used for training and testing of this model; (1.2) Data preprocessing: To achieve the diversification of sample data, vehicle videos of different scenes, different environments, and different densities are screened and frame-processed; (1.3) Data labeling: Use open source software to label data, and define small private cars and large vehicles as vehicle and bigvehicle respectively; (1.4) Dataset division: The optimized dataset is divided into training set, validation set and test set in a ratio of 8:1:
1.
3. The small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structure according to claim 1 is characterized in that: The implementation process of step (3) is as follows: Before training, the model parameters were configured. After many experiments, the following configuration was found to be the best: the input image size was 640×640, the number of iterations was 400 rounds, the Batch_Size was 10, the initial learning rate was set to 0.01, the learning rate momentum was 0.937, and the weight decay coefficient was 0.0005; the initial learning rate was set to 0.01, the learning rate momentum was 0.937, and the weight decay coefficient was 0.0005; the IoU was set to 0.
5. When the intersection over union ratio of the predicted box and the true box was greater than or equal to 0.5, it was considered that the target position was successfully predicted. In order to promote the model to achieve global optimization, the first 10 epochs were preheated using warm up training. After the preheating training, the cosine annealing algorithm was used to adjust the learning rate again. In addition, the rest were the default parameters of version 6.
0.
4. The small target vehicle detection method integrating non-adjacent skip connections and multi-scale residual structure according to claim 1 is characterized in that: The implementation process of step (4) is as follows: In order to accurately evaluate the performance of the NHN-YOLOv5s-MREFE model, the precision (Precision, P), recall (Recall, R), average precision (average precision, AP), mean average precision (mean average precision, mAP), the number of image detections per unit time (frames per second, FPS) and the number of model parameters (Params) are used as evaluation indicators. The formula is: In the formula, TP (True Positive) indicates the accurate determination of positive samples; FP (False Positive) indicates the wrong determination of negative samples; FN (False Negative) indicates the wrong determination of positive samples; k indicates the sample data category.
