A lightweight airport scene monitoring method based on YOLOv4

Abstract

The present application relates to a kind of light airport scene monitoring method based on improved YOLOv4, comprising the following steps: the scene monitoring video and photo collected in airport scene are made into data set using labeling tool, and are labeled;IEN-YOLO target detection model is built;The prepared airport scene training set is used to train the IEN-YOLO detection model built, and the test set built is used to test the performance of IEN-YOLO model;The picture to be detected is input into the trained IEN-YOLO detection model to obtain detection recognition result.The method only needs to deploy the monitoring camera with lower hardware cost in airport scene, and the monitoring of airport scene can be realized, the deployment cost can be saved while maintaining high efficient detection recognition precision.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection, and more particularly to a lightweight airport surface surveillance method based on YOLOv4. Background Technology

[0002] Currently, visual, radar, and signal positioning equipment are mainly used for airport surface monitoring to ensure its normal operation. However, with the increasing complexity and scale of airport environments, the deployment cost and monitoring pressure of these traditional monitoring methods are growing. Traditional surface surveillance radar, multi-point positioning, and automatic broadcast correlation (ADR) technologies can only effectively monitor cooperative targets such as aircraft, but cannot effectively monitor non-cooperative targets such as vehicles and personnel. Furthermore, for smaller airports, large-scale surface surveillance radar and numerous ground signal processing base stations are too expensive and complex to deploy. Additionally, radar signals can experience blind spots and signal interference due to the presence of certain building structures. Video-based monitoring of airport surfaces is relatively cheaper and simpler, and most existing airports are required to install surveillance cameras, along with corresponding computing equipment. Therefore, video-based monitoring of airport surfaces has become a new direction. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a lightweight airport surface monitoring method based on YOLOv4 that can effectively improve the monitoring efficiency of airport surfaces and save on the cost of deploying equipment.

[0004] The technical solution to achieve the purpose of this invention is to provide a lightweight airport surface surveillance method based on YOLOv4, characterized by the following steps:

[0005] Step 1: Collect surveillance video from airport surface surveillance cameras, and save each frame as an image by extracting keyframes from the surveillance video. Then, use the open-source software LabelImg to annotate the saved images. The annotation includes target location information and category information. The category information includes three categories: airplane, man, and car, thereby constructing an airport surface dataset.

[0006] Step 2: Using the YOLOv4 algorithm as the basic target detection model, the following three improved modules are used to construct the IEN-YOLO detection model: (1) Using the lightweight EfficientNet network as the backbone feature extraction network, and replacing the SE attention mechanism with the ECA attention mechanism in the backbone network; (2) Adding a 2x deconvolution upsampling and weighted feature fusion structure to the YOLOv4 algorithm; (3) Adding a high-resolution detection layer to the YOLOv4 algorithm; After adding the above three different improved modules to the YOLOv4 algorithm, the IEN-YOLO detection model is obtained, which realizes the detection and recognition of airplanes, people and cars on the airport scene;

[0007] Step 3: Using the constructed airport scene dataset and the IoU of the labeled boxes in the dataset as the clustering metric, calculate the anchor box parameters that the IEN-YOLO detection model needs to design in advance;

[0008] Step 4: Use the constructed airport surface dataset and a small target data augmentation strategy to train the IEN-YOLO detection model. Obtain the optimal weight parameters through training, and then use the optimal weight parameters to construct the final IEN-YOLO airport surface detection model.

[0009] Step 5: Use the constructed IEN-YOLO airport surface detection model to detect and identify all targets in the input airport surface monitoring image, including the location information and category score information of each target. Finally, use the Soft-NMS algorithm to process the overlap problem of the obtained detection boxes to obtain the final detection and recognition results.

[0010] Furthermore, in step ②, the original CSPDarknet53 backbone network of the YOLOv4 model is replaced with the lightweight EfficientNet network. The EfficientNet network consists of 8 modules. The first module is a 3x3 convolutional module, and the remaining 7 modules are MBConv modules. The basic components of each MBConv module are: depthwise separable convolutional layer, switch activation function, SE attention mechanism and residual connection.

[0011] Furthermore, in step ②, the SE attention mechanism in the original EfficientNet network is replaced with the lighter ECA attention mechanism, which performs better in detecting small objects, resulting in an improved EfficientNet network. The ECA attention mechanism uses a 1D adaptive convolution operation of size K and a non-dimensionality reduction feature extraction operation to improve feature extraction capability and reduce the computational cost of the model. The ECA attention mechanism first performs global average pooling (Avgpool) on the input feature F, then uses a 1D convolution operation (Convk) of size K to obtain attention weights, and then multiplies the attention weights with the input feature F to obtain a new feature F′. The expression of the ECA attention mechanism is as follows:

[0012] F′=F*Sigmoid(Conv k (Avgpool(F)))

[0013] In the formula, F′ represents the final output feature, F represents the input feature, and Conv k Represents a 1D convolution operation of size K, and Avgpool represents the average pooling operation.

[0014] Furthermore, in step ②, a 2x deconvolution upsampling and weighted feature fusion module are added to the original YOLOv4 model to enhance the network's feature extraction capabilities. The weighted feature fusion module introduces a learnable weight value w during the feature fusion process, and then multiplies the weight value w with the feature L to be fused to obtain the adjusted feature L', thereby dynamically adjusting the importance of different features. The expression is as follows:

[0015] L'=w*L

[0016] In the formula, L' represents the feature after weight adjustment, w represents the introduced weight value, and L represents the feature to be fused.

[0017] Furthermore, in step ②, a high-resolution detection layer is added to the original YOLOv4 model to improve the model's ability to detect small targets on the airport surface. For the model using a 608x320 input resolution, a 152x80 resolution detection feature layer is added to the original YOLOv4 algorithm. Finally, a total of 4 detection feature layers are used for detection, including 4 detection layers with resolutions of 152x80, 76x40, 38x20, and 19x10.

[0018] Furthermore, in step ③, K-means clustering is used to cluster the bounding boxes in the constructed airport scene dataset using IoU as the clustering metric to obtain anchor boxes that fit the airport scene dataset. The labeled dataset obtained in step ① is divided into training and test sets in a 3:1 ratio. Then, K-means clustering is applied to the constructed dataset, using IoU as the clustering metric to obtain the optimal detection anchor boxes. Simultaneously, since the IEN-YOLO detection model uses... There are 4 detection layers, so each layer is evenly distributed with 3 anchor boxes. The detection layers for 152x80 resolution are: [8, 11][12, 21][16, 9], the detection layers for 76x40 resolution are: [22, 34][44, 44][42, 20], the detection layers for 38x20 resolution are: [67, 25][96, 60][140, 54], and the detection layers for 19x10 resolution are: [287, 206][362, 171][574, 574].

[0019] Furthermore, in step ④, a small target data augmentation strategy is used in the model training to improve the network's detection performance of small targets on the airport scene. This strategy first uses manual cropping to crop sub-regions containing a large number of small targets in the dataset images to obtain sub-images. Then, four sub-images are randomly scaled, flipped, and their color and lighting are adjusted. After adjustment, the four sub-images are randomly stitched together to obtain more diverse training images. The weights of the IEN-YOLO model are initialized using the pre-trained EfficientNet model weights. The batch size for model training is 8, the initial learning rate is 0.001, and the learning rate decreases to 0.96 after each training round until the model converges, completing the network's learning.

[0020] Furthermore, in step ⑤, the trained IEN-YOLO detection model is used to calculate the detection and recognition results of the input image. Then, the Soft-NMS algorithm is used to post-process the detection boxes to obtain the final detection boxes. The Soft-NMS algorithm effectively handles the problem of high overlap of some detection boxes by reducing the detection box score. The score reduction method is as follows:

[0021]

[0022] In the formula, Si is the prediction score, M is the prediction box with the highest score, bi is the prediction box currently being judged, and σ is the weight parameter set. In this algorithm, σ is 0.5.

[0023] This invention offers several advantages: Addressing the shortcomings of traditional airport surface surveillance methods, such as poor performance in monitoring non-cooperative targets and high hardware deployment costs, this invention proposes a lightweight airport surface surveillance method based on YOLOv4. This method utilizes deep learning-based target detection technology to learn from a constructed airport surface dataset, thereby obtaining an accurate target detection model. Highly efficient airport surface surveillance can be achieved by installing surveillance cameras. Specifically, the use of the EfficientNet lightweight network with embedded ECA attention mechanism significantly reduces computational cost while achieving real-time detection speed. Weighted feature fusion and deconvolutional upsampling significantly improve the network's feature extraction capabilities. High-resolution detection layers and small target data augmentation strategies significantly enhance the network's detection performance for small targets on the airport surface. Furthermore, the Soff-NMS algorithm effectively handles the detection of highly overlapping small targets. Therefore, this invention improves airport surface surveillance efficiency and saves on hardware deployment costs, while effectively overcoming the challenge of small target detection on airport surfaces, achieving excellent detection performance and real-time detection speed. Attached Figure Description

[0024] Figure 1 This is a structural diagram of MBConv in step ② of the present invention;

[0025] Figure 2 This is a network structure diagram of the IEN-YOLO detection model in step ② of the present invention;

[0026] Figure 3 This is a structural diagram of the ECA attention mechanism in step ② of the present invention. Detailed Implementation

[0027] (Example 1)

[0028] This embodiment of a lightweight airport surface surveillance method based on YOLOv4 specifically includes the following steps:

[0029] Step 1: Collect surveillance video from airport surface monitoring cameras, and save each frame as an image by extracting keyframes from the surveillance video. Then, use the open-source software LabelImg to annotate the saved images. The annotation includes target location information and category information. The category information includes three categories: airplane, man, and car, thereby constructing an airport surface dataset.

[0030] Step 2: See Figure 2The YOLOv4 algorithm is used as the basic object detection model. Then, the following three improved modules are used to construct the IEN-YOLO detection model: (1) The lightweight EfficientNet network is used as the backbone feature extraction network, and the ECA attention mechanism is used to replace the SE attention mechanism in the backbone network to obtain the improved EfficientNet backbone network. The network parameters are shown in Appendix Table 1.

[0031] Table 1

[0032]

[0033] The original CSPDarknet53 backbone network of the YOLOv4 model was replaced with the lightweight EfficientNet network to reduce the computational parameters of the model. The EfficientNet network consists of 8 modules: the first module is a 3x3 convolutional module, and the remaining 7 modules are MBConv modules. See [link to documentation]. Figure 1 Each MBConv module is basically composed of: a depthwise separable convolutional layer, a switch activation function, a SE attention mechanism, and residual connections; see Figure 3 The ECA attention mechanism uses a 1D adaptive convolution operation of size K and a non-dimensionality reduction feature extraction operation to improve feature extraction capability and reduce the computational cost of the model. The ECA attention mechanism first performs global average pooling (Avgpool) on the input features F, and then uses a 1D convolution operation (Conv) of size K. k The attention weights are obtained, and then multiplied by the input feature F to obtain a new feature F'. The expression for the ECA attention mechanism is as follows:

[0034] F′=F*Sigmoid(Conv k (Avgpool(F)))

[0035] In the formula, F′ represents the final output feature, F represents the input feature, and Conv k Represents a 1D convolution operation of size K, and Avgpool represents the average pooling operation.

[0036] (2) To enhance the feature extraction capability of the YOLOv4 algorithm, a 2x deconvolution upsampling and a weighted feature fusion structure are added. The weighted feature fusion module introduces a learnable weight value w during the feature fusion process, and then multiplies the weight value w with the feature L to be fused to obtain the adjusted feature L', thereby dynamically adjusting the importance of different features. The expression is as follows:

[0037] L'=w*L

[0038] In the formula, L' represents the feature after weight adjustment, w represents the introduced weight value, and L represents the feature to be fused.

[0039] (3) Add a high-resolution detection layer to the YOLOv4 algorithm to improve the model’s ability to detect small targets on the airport surface. For the model using a 608x320 input resolution, add a 152x80 resolution detection feature layer on the basis of the original YOLOv4 algorithm. Finally, a total of 4 detection feature layers are used for detection, including 4 detection layers with resolutions of 152x80, 76x40, 38x20 and 19x10.

[0040] After adding the above three different improved modules to the YOLOv4 algorithm, the IEN-YOLO detection model is obtained, which can detect and identify airplanes, people, and vehicles on the airport surface.

[0041] Step 3: Using the constructed airport scene dataset and the IoU of the bounding boxes in the dataset as the clustering metric, calculate the pre-designed anchor box parameters required for the IEN-YOLO detection model. Specifically, use K-means clustering to cluster the bounding boxes in the constructed airport scene dataset using IoU as the clustering metric to obtain anchor boxes that fit the airport scene dataset. Divide the labeled dataset obtained in Step 1 into a training set and a test set in a 3:1 ratio. Then, the K-means clustering method was used on the constructed dataset, and the optimal detection anchor boxes were obtained with IoU as the clustering index. Since the IEN-YOLO detection model uses 4 detection layers, the anchor boxes were evenly distributed to each layer with 3 anchor boxes per layer. The 152x80 resolution detection layer is: [8, 11][12, 21][16, 9], the 76x40 resolution detection layer is: [22, 34][44, 44][42, 20], the 38x20 resolution detection layer is: [67, 25][96, 60][140, 54], and the 19x10 resolution detection layer is: [287, 206][362, 171][574, 574].

[0042] Step 4: Train the IEN-YOLO detection model using the constructed airport scene dataset and a small target data augmentation strategy. The optimal weight parameters are obtained through training, and then used to construct the final IEN-YOLO airport scene detection model. A small target data augmentation strategy is used to improve the network's detection performance for small targets on the airport scene. This strategy first uses manual cropping to crop sub-regions containing many small targets from the dataset images to obtain sub-images. Then, four sub-images are randomly scaled, flipped, and their color and lighting are adjusted. After adjustment, the four sub-images are randomly stitched together to obtain richer and more diverse training images. The weights of the IEN-YOLO model are initialized using pre-trained EfficientNet model weights. The batch size for model training is 8, and the initial learning rate is 0.001. After each training round, the learning rate decreases to 0.96 from the previous round until the model converges, completing the network's learning.

[0043] Step 5: Utilize the constructed IEN-YOLO airport surface detection model to detect and identify all targets in the input airport surface surveillance image, including the location information and category score information of each target. Finally, use the Soft-NMS algorithm to handle the overlap problem of the obtained detection boxes, thereby obtaining the final detection and recognition result: First, adjust the resolution of the input image to 608x320 to avoid feature distortion. Then, feed the image into the IEN-YOLO detection model for calculation. Finally, use four detection layers with different resolutions for detection, including 152x80, 76x40, 38x20, and 19x10 detection layers to handle target objects of different scales. Afterward, filter the category scores calculated by the model according to a set threshold. Use the Soft-NMS algorithm to handle the overlap problem of the detected boxes after filtering to obtain the final detection boxes. The Soft-NMS algorithm effectively handles the problem of high overlap of some detection boxes by reducing the detection box scores. The score reduction method is as follows:

[0044]

[0045] In the formula, Si is the predicted score, M is the predicted bounding box with the highest score, bi is the predicted bounding box currently being judged, and σ is the set weight parameter. In this algorithm, σ is taken as 0.5. Finally, the final detection result is output.

[0046] The aforementioned lightweight airport surface surveillance method based on YOLOv4 significantly reduces the number of computational parameters by using the EfficientNet lightweight network with embedded ECA attention mechanism on the basis of the YOLOv4 model. It effectively improves the feature extraction capability of the network by using deconvolution upsampling and weighted feature fusion structure. It effectively overcomes the difficulty of small target detection on the airport surface by using high-resolution detection layer and small target data augmentation strategy. It efficiently handles the problem of some targets with high overlap by using Soft-NMS algorithm. This invention can reduce the model's computational load to save the cost of deploying computing resources, while maintaining high detection speed and detection accuracy.

[0047] Obviously, the above embodiments are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, these obvious variations or modifications derived from the spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A lightweight airport surface surveillance method based on YOLOv4, characterized in that... Includes the following steps: Step 1: Collect surveillance video from airport surface surveillance cameras, and save each frame as an image by extracting keyframes from the surveillance video. Then, use the open-source software LabelImg to annotate the saved images. The annotation includes target location information and category information. The category information includes three categories: airplane, man, and car, thereby constructing an airport surface dataset. Step 2: Using the YOLOv4 algorithm as the basic object detection model, the following three improved modules are used to construct the IEN-YOLO detection model: (1) Using the lightweight EfficientNet network as the backbone feature extraction network, and replacing the SE attention mechanism with the ECA attention mechanism in the backbone network: The original CSPDarknet53 backbone of the YOLOv4 model is replaced with the lightweight EfficientNet network. The EfficientNet network consists of eight modules: the first is a 3x3 convolutional module, and the remaining seven are MBConv modules. Each MBConv module is basically composed of a depthwise separable convolutional layer, a switch activation function, an SE attention mechanism, and residual connections. The SE attention mechanism in the original EfficientNet network is replaced with the lighter ECA attention mechanism, which performs better for small object detection, resulting in an improved EfficientNet network. The ECA attention mechanism uses a 1D adaptive convolution operation of size K and a non-dimensionality reduction feature extraction operation to improve feature extraction capabilities and reduce the computational cost of the model. The ECA attention mechanism first performs global average pooling (Avgpool) on the input features F, and then uses a 1D convolution operation of size K (Conv53) to perform a convolution operation. k The attention weights are obtained, and then multiplied by the input feature F to obtain a new feature F'. The expression for the ECA attention mechanism is as follows: In the formula For the final output features, As input features, This represents a 1D convolution operation of size K. (1) Indicates average pooling operation; (2) Adds 2 times deconvolution upsampling and weighted feature fusion structure to YOLOv4 algorithm; (3) Adds a high-resolution detection layer to YOLOv4 algorithm; After adding the above three different improvement modules on the basis of YOLOv4 algorithm, the IEN-YOLO detection model is obtained, which realizes the detection and recognition of aircraft, people and vehicles on the airport surface; Step 3: Using the constructed airport scene dataset and the IoU of the labeled boxes in the dataset as the clustering metric, calculate the anchor box parameters that the IEN-YOLO detection model needs to design in advance; Step 4: Use the constructed airport surface dataset and a small target data augmentation strategy to train the IEN-YOLO detection model. Obtain the optimal weight parameters through training, and then use the optimal weight parameters to construct the final IEN-YOLO airport surface detection model. Step 5: Use the constructed IEN-YOLO airport surface detection model to detect and identify all targets in the input airport surface monitoring image, including the location information and category score information of each target. Finally, use the Soft-NMS algorithm to process the overlap problem of the obtained detection boxes to obtain the final detection and recognition results.

2. The lightweight airport surface surveillance method based on YOLOv4 according to claim 1, characterized in that: In step ②, a 2x deconvolution upsampling and weighted feature fusion module is added to the original YOLOv4 model to enhance the network's feature extraction capabilities; The weighted feature fusion module introduces a learnable weight value w during the feature fusion process, and then multiplies the weight value w with the feature L to be fused to obtain the adjusted feature L', thereby dynamically adjusting the importance of different features. The expression is as follows: L'=w*L In the formula L’ represents the feature after weight adjustment, w represents the introduced weight value, and L represents the feature to be fused.

3. The lightweight airport surface surveillance method based on YOLOv4 according to claim 2, characterized in that: In step ②, a high-resolution detection layer is added to the original YOLOv4 model to improve the model's ability to detect small targets on the airport surface. For the model using a 608x320 input resolution, a 152x80 resolution detection feature layer is added to the original YOLOv4 algorithm. Finally, a total of 4 detection feature layers are used for detection, including 4 detection layers with resolutions of 152x80, 76x40, 38x20, and 19x10.

4. The lightweight airport surface surveillance method based on YOLOv4 according to claim 3, characterized in that: In step ③, K-means clustering is used to cluster the bounding boxes in the constructed airport scene dataset using IoU as the clustering metric to obtain anchor boxes that fit the airport scene dataset. The labeled dataset obtained in step ① is divided into training and test sets in a 3:1 ratio. Then, K-means clustering is applied to the constructed dataset, using IoU as the clustering metric to obtain the optimal detection anchor boxes. Since the IEN-YOLO detection model uses 4 layers... The detection layers are divided into three anchor boxes per layer. The detection layers for 152x80 resolution are: [8,11][12,21][16,9], the detection layers for 76x40 resolution are: [22,34][44,44][42,20], the detection layers for 38x20 resolution are: [67,25][96,60][140,54], and the detection layers for 19x10 resolution are: [287,206][362,171][574,574].

5. A lightweight airport surface surveillance method based on YOLOv4 according to claim 4, characterized in that: In step ④, a small target data augmentation strategy is used in model training to improve the network's detection performance of small targets on the airport scene. This strategy first uses manual cropping to crop sub-regions containing a large number of small targets in the dataset images to obtain sub-images. Then, four sub-images are randomly scaled, flipped, and their color and lighting are adjusted. After adjustment, the four sub-images are randomly stitched together to obtain more diverse training images. The weights of the IEN-YOLO model are initialized using the pre-trained EfficientNet model weights. The batch size for model training is 8, the initial learning rate is 0.001, and the learning rate decreases to 0.96 after each training round until the model converges, completing the network's learning.

6. A lightweight airport surface surveillance method based on YOLOv4 according to claim 5, characterized in that: In step ⑤, the trained IEN-YOLO detection model is used to calculate the detection and recognition results of the input image. Then, the Soft-NMS algorithm is used to post-process the detection boxes to obtain the final detection boxes. The Soft-NMS algorithm effectively handles the problem of high overlap of some detection boxes by reducing the detection box score. The score reduction method is as follows: In the formula Si To predict scores, M The highest-scoring prediction box. bi This represents the currently being evaluated bounding box, and represents the set weight parameters in this algorithm. Take 0.5.

Images