A target detection method based on deep learning to improve YOLOv3

Abstract

The present invention relates to a target detection method based on deep learning improved YOLOv3, belonging to the technical field of target detection. The method comprises: obtaining image data to be detected, and inputting the image data into a trained improved YOLOv3 detection model to obtain a detection result; the backbone network of the detection model comprises a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer: the first feature extraction layer comprises a first convolution module, a second convolution module, a first residual module, a third convolution module, a second residual module, and a fourth convolution module; the second feature extraction layer comprises a third residual module and a fifth convolution module; the third feature extraction layer comprises a fourth residual module and a sixth convolution module; the fourth feature extraction layer comprises a fifth residual module. The backbone network increases the input size of the image to 640*640, increases the image size, and thus allows the image to contain more small target information, thereby improving the detection accuracy.

Description

Technical Field

[0001] The present invention relates to a target detection method based on deep learning to improve YOLOv3, and belongs to the technical field of target detection. Background Art

[0002] Airport surface aircraft detection is an important part of smart airports and remote control towers. It can automatically identify aircraft in airport images and assist tower controllers to ensure flight safety. It has important application value in surface aircraft conflict warning, runway intrusion prevention, and remote command.

[0003] Aircraft images are usually collected by cameras in terminals and control towers. Compared with remote sensing aircraft images with complete outlines and single postures, aircraft images have problems such as mutual occlusion of aircraft and small size of aircraft due to long shooting distance, which limits the traditional method based on sliding windows.

[0004] In recent years, with the rapid development of deep learning technology, deep learning-based methods have been widely used in the field of target detection. At present, deep learning-based methods are mainly divided into two categories: one is the candidate region-based method, such as RCNN, Fast-RCNN, Faster-RCNN and other algorithms; the other is the regression-based target detection method, such as YOLO, SSD, YOLO9000, YOLOv3 and other algorithms. Among them, although the candidate region-based method has high detection accuracy, the detection speed is slow and is not suitable for field aircraft detection. The regression-based target detection method is fast and can meet the real-time requirements of field aircraft detection. The YOLOv3 algorithm is considered to be the best and is widely used in field aircraft detection.

[0005] The model of the YOLOv3 algorithm is as follows Figure 1 As shown, Figure 1DBL is the basic component of YOLOv3, which is convolution + batch normalization - BN layer (Batch Normalization) + activation function (Leaky relu). For v3, BN and leaky relu are inseparable from the convolution layer (except the last layer of convolution), and together they constitute the smallest component. The basic component of Resi_unit is also DBL. Residual: Indicates how many Res_units are contained in this Residual. It is a large component of YOLOv3. YOLOv3 began to borrow the residual structure of ResNet. Using this structure can make the network structure deeper (from yolov2's darknet-19 to yolov3's darknet-53, the former has no residual structure). Concat is tensor splicing, which splices the upsampling of the middle layer of darknet and the subsequent layer. The splicing operation is different from the residual layer add operation. Splicing will expand the dimension of the tensor, while add is just a direct addition and will not cause the dimension of the tensor to change.

[0006] The backbone convolutional network of YOLOv3 is Darknet-53, which integrates the skip connection layer structure of ResNet and uses 1×1 and 3×3 convolutional layers to expand and compress feature channels, respectively, and enhance the performance of network feature extraction. The network structure of Darknet-53 is shown in Table 1:

[0007] Table 1 Network structure of Darknet-53

[0008]

[0009] However, the input image in the above Darknet-53 network is small, and it is easy to lose the feature information of small targets, resulting in reduced detection accuracy. Summary of the invention

[0010] The purpose of this application is to provide a target detection method based on deep learning to improve YOLOv3, so as to solve the problem of low detection accuracy of existing detection methods.

[0011] To achieve the above objectives, this application proposes a technical solution for a target detection method based on deep learning to improve YOLOv3. The detection method includes the following steps:

[0012] 1) Obtain the image to be detected;

[0013] 2) Inputting the image to be detected into the trained detection model to obtain the detection result; the detection model is an improved YOLOv3 detection model;

[0014] The backbone network of the detection model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer arranged in sequence from input to output:

[0015] The first feature extraction layer includes a first convolution module, a second convolution module, a first residual module, a third convolution module, a second residual module and a fourth convolution module which are sequentially arranged in a direction from input to output;

[0016] The second feature extraction layer includes a third residual module and a fifth convolution module arranged in sequence from input to output;

[0017] The third feature extraction layer includes a fourth residual module and a sixth convolution module arranged in sequence from input to output;

[0018] The fourth feature extraction layer includes a fifth residual module.

[0019] The technical solution of the target detection method based on deep learning to improve YOLOv3 of the present invention has the beneficial effect that: the detection model of the present invention is improved on the basis of the original YOLOv3 detection model, and the backbone network in the original YOLOv3 detection model is replaced by the backbone network with four feature extraction layers in the present invention, and the feature extraction layer is composed of a convolution module and / or a residual module. The backbone network increases the input size of the image to 640*640, increases the image size, and can thus include more small target information in the image, thereby improving the detection accuracy.

[0020] Furthermore, in order to improve the detection speed, the residual module is a bottleneck residual module.

[0021] Furthermore, the bottleneck residual module includes a 1*1 convolution submodule, a 3*3 convolution submodule, and a 1*1 convolution submodule, which are arranged in sequence from input to output.

[0022] Furthermore, each convolution module is a 3*3 convolution module.

[0023] Furthermore, the expansion factor of each residual module is 1.5.

[0024] Furthermore, in the first feature extraction layer, the number of output channels of the first convolution module is 32, the number of repetitions is 1, and the step size is 1; the number of output channels of the second convolution module is 64, the number of repetitions is 1, and the step size is 2; the number of output channels of the first residual module is 64, the number of repetitions is 1, and the step size is 1; the number of output channels of the third convolution module is 128, the number of repetitions is 1, and the step size is 2; the number of output channels of the second residual module is 128, the number of repetitions is 2, and the step size is 1; the number of output channels of the fourth convolution module is 256, the number of repetitions is 1, and the step size is 2.

[0025] Further, in the second feature extraction layer, the output channel number of the third residual module is 256, the repetition number is 3, and the stride is 1; the output channel number of the fifth convolution module is 512, the repetition number is 1, and the stride is 2.

[0026] Further, in the third feature extraction layer, the output channel number of the fourth residual module is 512, the repetition number is 4, and the stride is 1; the output channel number of the sixth convolution module is 1024, the repetition number is 1, and the stride is 2.

[0027] Further, in the fourth feature extraction layer, the output channel number of the fifth residual module is 1024, the repetition number is 3, and the stride is 1.

[0028] Further, before training the detection model, it also includes the step of augmenting the data of the training dataset. Description of the Drawings

[0029] Figure 1 is the structural diagram of the detection model based on the YOLOv3 algorithm in the prior art;

[0030] Figure 2 is the structural diagram of the detection model of the present invention;

[0031] Figure 3a is the structural diagram of the residual module in the Darknet-53 backbone network in the prior art;

[0032] Figure 3b is the structural diagram of the residual module of the present invention;

[0033] Figure 4 is the structural diagram of the dataset directory during the training of the detection model of the present invention;

[0034] Figure 5a is the comparison curve graph of the training loss curves of the detection method of the present invention and the detection method in the prior art;

[0035] Figure 5b is the partial enlarged curve graph of the comparison of the training loss curves of the detection method of the present invention and the detection method in the prior art;

[0036] Figure 5c is the comparison curve graph of the mAP of the detection method of the present invention and the detection method in the prior art when IoU = 0.5;

[0037] Figure 5d is the comparison curve graph of the mAP of the detection method of the present invention and the detection method in the prior art when 0.5 < IoU < 0.95;

[0038] Figure 6a is the effect diagram of Example 1 of the detection result of the present invention;

[0039] Figure 6b It is a rendering of the detection result example 2 of the present invention;

[0040] Figure 6c It is a rendering of the detection result example 3 of the present invention;

[0041] Figure 6d It is a rendering of the detection result example 4 of the present invention. DETAILED DESCRIPTION

[0042] Implementation example of target detection method based on deep learning to improve YOLOv3:

[0043] The main idea of ​​the present invention is that, based on the YOLOv3 algorithm, in order to improve the detection speed, the original Darknet-53 backbone network of the YOLOv3 algorithm is replaced with the backbone network of the present invention, and the backbone network of the present invention includes four feature extraction layers, each of which is provided with a convolution module and a residual module. The image to be detected is input into the backbone network of the present invention to achieve target detection.

[0044] Specifically, the target detection method based on deep learning improved YOLOv3 of the present invention is described with the aircraft on the scene as the target. The target detection method includes the following steps:

[0045] 1) Build a detection model.

[0046] The difference between the detection model and the detection model of the YOLOv3 algorithm lies in the backbone network and the residual module used in the backbone network. The architecture of the overall detection model is the same as that of the detection model of the YOLOv3 algorithm, which will not be described here. Figure 2 As shown, it includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer which are sequentially arranged from input to output:

[0047] The first feature extraction layer includes a first convolution module, a second convolution module, a first residual module, a third convolution module, a second residual module and a fourth convolution module which are sequentially arranged in a direction from input to output;

[0048] The second feature extraction layer includes a third residual module and a fifth convolution module arranged in sequence from input to output;

[0049] The third feature extraction layer includes a fourth residual module and a sixth convolution module arranged in sequence from input to output;

[0050] The fourth feature extraction layer includes a fifth residual module.

[0051] In the above backbone network, each convolution module is a 3*3 convolution module (i.e., conv(3*3)).

[0052] The residual module is a bottleneck residual module. The residual module used in the original Darknet-53 backbone network is as follows Figure 3a As shown, it includes conv1*1 (i.e., 1*1 convolution submodule) and conv3*3 (i.e., 3*3 convolution submodule). The residual module makes the number of layers of the main pole network deeper, which is conducive to extracting the deep semantic information of the target, but it will lose the feature information of small targets, reduce the detection accuracy, double the calculation amount, and reduce the detection rate. In order to improve the detection accuracy and rate of the aircraft on the ground, the present invention introduces a depth separable convolution structure, using Figure 3b The bottleneck residual module shown in the figure includes a 1*1 convolution submodule (conv1*1), a 3*3 convolution submodule (Dwise3*3), and a 1*1 convolution submodule (conv1*1) arranged in sequence from input to output, which not only reduces the loss of target information in the main pole network feature extraction, but also reduces the calculation amount of the detection model and improves the detection speed of the detection model. Of course, if only the detection accuracy needs to be guaranteed, an ordinary residual module can also be used, and the present invention does not limit this.

[0053] The difference between the Dwise 3*3 convolution submodule and the ordinary 3*3 convolution submodule is:

[0054] The ordinary 3*3 convolution submodule performs a convolution on all channels. For example, an RGB three-channel image with a size of 640*640 is input as 640*640*3. After an ordinary 3*3 convolution, the result is 640*640*1, and the output channel becomes 1.

[0055] Dwise3*3 convolution submodule: performs separate convolution operations on all channels, and the number of input channels and output channels is the same: for example: an RGB three-channel image with a size of 640*640, the input here is recorded as 640*640*3, and after one Dwise3*3 convolution, the result is 640*640*3, the output channel is still 3, and the number of channels has not changed. The Dwise3*3 convolution here is calculated in combination with the subsequent 1*1 convolution and is not used alone.

[0056] Specifically, the parameters of each convolution module and residual module are set as follows:

[0057] The output channel number of the first convolution module is 32, the number of repetitions is 1, and the step length is 1; the output channel number of the second convolution module is 64, the number of repetitions is 1, and the step length is 2; the output channel number of the first residual module is 64, the number of repetitions is 1, and the step length is 1; the output channel number of the third convolution module is 128, the number of repetitions is 1, and the step length is 2; the output channel number of the second residual module is 128, the number of repetitions is 2, and the step length is 1; the output channel number of the fourth convolution module is 256, the number of repetitions is 1, and the step length is 2;

[0058] The output channel number of the third residual module is 256, the number of repetitions is 3, and the step size is 1; the output channel number of the fifth convolution module is 512, the number of repetitions is 1, and the step size is 2;

[0059] The output channel number of the fourth residual module is 512, the number of repetitions is 4, and the step size is 1; the output channel number of the sixth convolution module is 1024, the number of repetitions is 1, and the step size is 2;

[0060] The fifth residual module has an output channel number of 1024, a repetition number of 3, and a step size of 1.

[0061] The summary is shown in Table 2:

[0062] Table 2 Structure and parameter settings of the backbone network

[0063]

[0064]

[0065] In Table 2, Layer is the feature extraction layer, Input is the size of the input image, Operator is the operation, e is the expansion factor, c is the output channel, n is the number of repetitions, and s is the stride. It can be seen that the input image size of the backbone network of the present invention is 640*640, and the input image size of the original Darknet-53 backbone network is 416*416, which increases the size of the input image and can reduce the loss of small target features.

[0066] 2) Train the detection model according to the data set to obtain a trained detection model.

[0067] In this step, since the present invention adopts the bottleneck residual module in the depth separable convolution structure, and the depth separable convolution reduces the model parameters and improves the detection rate on the basis of losing a certain detection accuracy, in order to improve the detection accuracy, it is necessary to preprocess the data set, and the preprocessing method is data amplification.

[0068] The main methods of data augmentation are Mosaic and Cutout. In addition to these two methods, image perturbation, changing brightness, contrast, saturation, hue, adding noise, random scaling, random cropping, flipping, rotation, random erasing, etc. are also used.

[0069] 3) Obtain the scene aircraft image to be detected, input the scene aircraft image to be detected into the trained detection model, and obtain the detection result.

[0070] The detection method of the present invention is verified by a specific example below.

[0071] First, establish the detection model and determine the experimental environment and data set.

[0072] The experimental environment is Ubuntu 20.04 operating system, Intel Core TM i9-9900K processor, 32GB memory, RTX 2080Ti*2 GPU, deep learning framework using Pytorch1.8 and general parallel computing architecture CUDA11.1. The data set is shown in Table 3:

[0073] Table 3 Experimental datasets

[0074] source Resolution date Quantity (sheets) category Training set (sheets) Validation set (sheets) Test set (sheets) An airport 1920*1080 2020-12-3 13700 airplane 8220 2740 2740

[0075] The dataset in Table 3 above comes from airplane images at an airport. It contains images of single passenger planes of various models, images of multiple occluded passenger planes, and images of small targets, totaling 13,700 images, covering daytime weather conditions such as sunny days, foggy days, and rainy days. It is divided into training set, validation set, and test set according to the ratio of 6:2:2. The image annotation tool labelimg is used to annotate the images, labeling them as airplanes and other objects. The directory structure of the data is as follows: Figure 4 As shown, Annotations is a label file in XML format, and each XML corresponds to a picture in JPEGImage. You can use labelImg to annotate and view; ImageSets stores the file address path of the training set, validation set, and test set; JPEGImages stores all pictures, including training set, validation set, and test set pictures; SegmengtationClass and SegmengtationObject are used for image segmentation, which we don't use here.

[0076] Secondly, the detection model is trained using the dataset.

[0077] The batch training data size (batch_size) is set to 8, the input image scale is set to 640×640, the weight decay value (Decay) is 0.0005, the training momentum is configured to 0.9, and the entire training process is completed in 300 epochs. The experiment uses the small batch gradient descent method to optimize the parameters.

[0078] Finally, the experimental results are analyzed to evaluate the performance of the detection model.

[0079] In order to effectively evaluate the performance of the network model, precision P, recall R, and average precision (mAP) are selected to evaluate the detection ability of the network model.

[0080] The formula is as follows:

[0081]

[0082]

[0083]

[0084]

[0085] TP: the number of detection boxes with IoU>0.5 (the same manually annotated target box is only counted once); IOU: the intersection over union ratio of the predicted box and the manually annotated bounding box;

[0086] FP: detection boxes with IoU <= 0.5, or the number of redundant detection boxes that detect the same manually annotated target box;

[0087] FN: The number of manually annotated target boxes that were not detected;

[0088] AP: Area under the PR curve (PR curve: Precision-Recall curve); an evaluation indicator for a single target;

[0089] mAP: mean Average Precision, which is the average value of AP of each category; C is the category of the detected target.

[0090] The detection method of the present invention is compared with the detection methods of Faster-RCNN, SSD, and YOLOv3 in the prior art, and the comparison results are shown in Table 4:

[0091] Table 4 Comparison results

[0092]

[0093] In Table 4, mAP val0.5 meaning: Select IoU = 0.5, the average precision detection index of the validation set data; mAP val 0.5:0.95 meaning: Select 0.5 < IoU < 0.95, the average precision detection index of the validation set data; mAP test 0.5:0.95 meaning: Select 0.5 < IoU < 0.95, the average precision detection index of the test set data; FPS (f / s) meaning: How many frames of pictures can be detected per second; Params (M) meaning: The size of the detection model calculation parameters, in M (megabytes) as the unit. It can be seen from Table 4 that Faster-RCNN has the highest accuracy, up to 98.9%, with an FPS of 2.2, and the detection speed is the slowest, unable to meet the requirements of real-time detection at the airport; The detection speed of SSD has a certain improvement compared to Faster-RCNN, with an FPS reaching 2.5, but its performance for small objects is poor and the recognition accuracy is too low, only 67.7%; YOLOV3 has a faster speed, reaching 8.4 FPS, and the accuracy can reach 94.9%. The detection model in this paper has the fastest detection speed, with an FPS reaching 15.6 f / s, the accuracy is on par with Faster-RCNN, and at the same time, it has the smallest parameters and the fastest speed in the end-to-end model detection. At the same time, it has a good detection effect on airplanes at night and can meet the requirements of real-time detection of airport airplanes.

[0094] Specifically, the comparison of the detection method of the present invention with the detection methods of the prior art in terms of training loss, and map0.5, map0.5:0.9 is as Figure 5a 、 Figure 5b 、 Figure 5c 、 Figure 5d shown.

[0095] Figure 5a It is a comparison curve graph of train_loss (training loss). The horizontal axis represents the number of training iterations, and the vertical axis represents the training loss value. The training loss of the ssd model is between 2.3 - 2.5, and the loss value cannot be reduced any further, and the detection accuracy reaches a certain bottleneck. The loss values of the faster-rcnn, yolov3 (i.e., YOLOv3), and ours detection models are lower, and the detection accuracy is further improved. Ours, that is, the model in this paper, has the closest loss and the highest detection accuracy.

[0096] Figure 5b is Figure 5a a partial enlarged view of Figure 5bThe purpose is to amplify the loss values of the Faster-RCNN (i.e., Faster-RCNN), YOLOv3, and our detection models, so as to clearly show the changes in the loss values of these three models. The training loss value of Faster-RCNN is close to but does not reach 0.04. The loss value of YOLOv3 is lower than that of the Faster-RCNN model, and the loss value of our detection model is the lowest, and the detection accuracy is also the best.

[0097] Figure 5c It is a comparison curve of the detection accuracy (map, equivalent to mAP) index when IoU = 0.5. The horizontal axis represents the number of training iterations, and the vertical axis represents the map value. The detection accuracies of Faster-RCNN and our detection models are the highest, approaching 1.0. YOLOv3 is the second, and the accuracy of SSD is the lowest.

[0098] Figure 5d It is a comparison curve of the detection accuracy (map) index when 0.5 < IoU < 0.95 (which can be considered as selecting more stringent detection conditions). Under such more stringent screening conditions, it can be seen that the detection accuracy of ours is the highest.

[0099] After detecting using the detection model of the present invention, the detection results as shown in Figure 6a 、 Figure 6b 、 Figure 6c 、 Figure 6d are obtained.

[0100] The setting of the backbone network of the present invention increases the size of the picture, reduces the loss of small target features, greatly improves the detection accuracy, and at the same time reduces the model parameters and improves the detection speed, achieving the best effect of both detection speed and detection accuracy. At the same time, the object detection method of the present invention is not only applicable to the object detection of aircraft on the scene, but also applicable to other object detections, and the present invention does not limit the detected objects.

[0101] The detection method of the present invention can realize functions such as flight boarding, aircraft landing and takeoff tracking and monitoring, and converting video monitoring data into track data for blind area filling:

[0102] (1) Flight boarding

[0103] Connect to ADS-B information, superimpose flight information on the aircraft in the panoramic video to realize aircraft boarding and achieve the effect of enhanced reality in the flight area.

[0104] (2) Aircraft landing and takeoff tracking and monitoring

[0105] The automatic tracking camera can automatically track the aircraft during takeoff and landing. At the same time, the camera will automatically focus according to the distance between the camera and the aircraft to ensure that the tracking camera image always displays the most suitable observation effect.

[0106] (3) Converting video surveillance data into track data to fill in the gaps

[0107] For some flights without ADS-B signals or with unreliable signals, video surveillance data is converted into track data to fill in the gaps, thereby establishing a low-cost surface surveillance system and enhancing the ability to operate in low-visibility conditions.

Claims

1. A target detection method based on deep learning to improve YOLOv3, characterized in that: The following steps are involved: 1) Obtain the image to be detected; 2) Inputting the image to be detected into the trained detection model to obtain the detection result; the detection model is an improved YOLOv3 detection model; The backbone network of the detection model includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer arranged in sequence from input to output: The first feature extraction layer includes a first convolution module, a second convolution module, a first residual module, a third convolution module, a second residual module and a fourth convolution module which are sequentially arranged in a direction from input to output; The second feature extraction layer includes a third residual module and a fifth convolution module arranged in sequence from input to output; The third feature extraction layer includes a fourth residual module and a sixth convolution module arranged in sequence from input to output; The fourth feature extraction layer includes a fifth residual module; Each convolution module is a 3*3 convolution module, and the residual module is a bottleneck residual module. The bottleneck residual module includes a 1*1 convolution submodule, a 3*3 convolution submodule, and a 1*1 convolution submodule arranged in sequence from input to output. The 3*3 convolution submodule is a Dwise 3*3 convolution submodule.

2. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: The expansion factor of each residual module is 1.

5.

3. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: In the first feature extraction layer, the number of output channels of the first convolution module is 32, the number of repetitions is 1, and the step size is 1; the number of output channels of the second convolution module is 64, the number of repetitions is 1, and the step size is 2; the number of output channels of the first residual module is 64, the number of repetitions is 1, and the step size is 1; the number of output channels of the third convolution module is 128, the number of repetitions is 1, and the step size is 2; the number of output channels of the second residual module is 128, the number of repetitions is 2, and the step size is 1; the number of output channels of the fourth convolution module is 256, the number of repetitions is 1, and the step size is 2.

4. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: In the second feature extraction layer, the output channel number of the third residual module is 256, the number of repetitions is 3, and the step size is 1; the output channel number of the fifth convolution module is 512, the number of repetitions is 1, and the step size is 2.

5. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: In the third feature extraction layer, the output channel number of the fourth residual module is 512, the number of repetitions is 4, and the step size is 1; the output channel number of the sixth convolution module is 1024, the number of repetitions is 1, and the step size is 2.

6. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: In the fourth feature extraction layer, the number of output channels of the fifth residual module is 1024, the number of repetitions is 3, and the step size is 1.

7. The target detection method based on deep learning improved YOLOv3 according to claim 1, characterized in that: Before the detection model is trained, a step is also included to amplify the training data set data.

Images