Real-time target detection method and device based on panoramic video of airport remote tower

Abstract

The embodiment of the present invention discloses a real-time target detection method and device based on a panoramic video of a remote tower of an airport. The method includes: data set production, target detection algorithm design, model training and panoramic image target detection. Target detection includes: obtaining data to be detected, using a segmentation detection strategy to process the data to be detected, obtaining multiple sub-region small images, inputting the multiple sub-region small images into a target detection model, obtaining preliminary detection results under a large resolution of the remote tower, processing the preliminary detection results, and obtaining target detection results; the data to be detected includes a panoramic video of a remote tower of an airport. The method of the embodiment of the present invention can be implemented to detect and identify targets of different scales in a panoramic video of a remote tower, and the detection results can meet the accuracy of airport detection. At the same time, the method has excellent performance and is easy to implement in engineering.

Description

Technical Field

[0001] The present invention relates to the technical field of civil aviation airport scene target recognition, and in particular to a real-time target detection method and system based on panoramic video of an airport remote control tower. Background Art

[0002] With the continuous innovation and development of civil aviation technology, the emerging remote tower system will replace the traditional towers of many small airports. The remote tower can monitor and control unmanned airports and temporary landing and take-off points, change the traditional visual control method, use cameras instead of human eyes, and combine computer vision technology to analyze and identify the panoramic video of the airport scene.

[0003] Among the existing positioning technologies, surface surveillance radar, multi-lateration positioning (MLAT) and automatic dependent surveillance-broadcast (ADS-B) are currently the main means of surveillance at large airports. Surface surveillance radar uses expensive radar to monitor the activities of aircraft and vehicles on runways and aprons. The multi-lateration positioning system (MLAT) achieves accurate measurement and positioning of targets through airborne transponder signals received by multiple ground receiving stations. Automatic dependent surveillance-broadcast (ADS-B) obtains accurate positioning information of targets by receiving data messages from aircraft. It can be used for target monitoring in airways, terminal areas, and tower-controlled airspace, and can also serve as an important supplement to airport surface monitoring.

[0004] In recent years, computer vision technology has developed rapidly, and target positioning methods based on visible light video have been widely used in airport scene management systems, which is of great significance for improving positioning accuracy and enhancing the situational awareness of controllers.

[0005] However, the above-mentioned major airport methods all have certain defects:

[0006] (1) The surveillance mode of the surface surveillance radar is affected by obstacles such as terminal buildings and corridors. There are many blind spots in surveillance. The surveillance effect is not ideal under obstruction conditions, and the installation and maintenance costs are high.

[0007] (2) Both multi-lateration positioning (MLAT) and automatic dependent surveillance-broadcast (ADS-B) require wireless communication networks and transceivers installed on the monitored targets to achieve high-precision positioning and monitoring. For many non-cooperative targets that do not have transceivers installed, such as many vehicles and staff operating at airports, positioning and monitoring cannot be achieved.

[0008] (3) Traditional image target detection methods use manual features for feature extraction, which is not very robust and has poor detection effect on scenes with complex backgrounds;

[0009] (4) Compared with traditional methods, the target detection method based on deep learning has higher accuracy and efficiency. However, its accuracy and stability need to be further improved for the problem of cross-scale target detection under large field of view. Summary of the invention

[0010] In view of the deficiencies in the prior art, an object of the embodiments of the present invention is to provide a real-time target detection method and device based on panoramic video of an airport remote tower.

[0011] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a real-time target detection method based on panoramic video of an airport remote tower, comprising:

[0012] Data set preparation steps: obtaining airport tower data and public image data, processing the airport tower data and public image data to obtain training data;

[0013] Target detection algorithm design steps: Design a target detection network, the structure of which includes a backbone extraction network, a feature fusion network, and a detection head network connected in sequence, and add an SKNet attention mechanism to the early convolutional layer and the late convolutional layer of the backbone extraction network;

[0014] Model training step: using the training data to train the target detection network to obtain a target detection model;

[0015] Panoramic image target detection steps: obtain the data to be detected, use a segmentation detection strategy to process the data to be detected, obtain multiple sub-region small images, input the multiple sub-region small images into the target detection model, obtain preliminary detection results under the large resolution of the remote tower, process the preliminary detection results, and obtain target detection results; the data to be detected includes a panoramic video of the airport remote tower.

[0016] As a specific implementation of the present application, the step of preparing a data set specifically includes:

[0017] Acquire airport tower data through an airport camera, extract key frame images from the airport tower data, annotate the key frame images to obtain a label file for each key frame image, and convert the label file into a training data format to obtain the training data;

[0018] The training data is obtained by downloading the public data from the Internet through a search engine or by using a crawler, and the public data is cleaned and screened.

[0019] As a specific implementation of the present application, the model training step specifically includes:

[0020] Data enhancement step: performing data enhancement processing on the data set by using translation, rotation, splicing, tone transformation, size transformation and small object scale adaptation;

[0021] Model training steps:

[0022] Randomly initialize the parameters of each convolution kernel in the target detection network;

[0023] The frozen training method is adopted. First, the parameters of the backbone extraction network part of the target detection network are frozen, and the target detection network is trained for 500 rounds using the training data after data enhancement processing. Then, the parameters are unfrozen and the target detection network is trained for 500 rounds to obtain the target detection model.

[0024] As a specific implementation of the present application, the panoramic image target detection step specifically includes:

[0025] Data acquisition step: acquiring the data to be detected through the airport camera, processing the data to be detected, and obtaining a panoramic view of the airport remote control tower;

[0026] Segmentation step: cutting the panoramic large image into n different sub-region small images; wherein, when cutting the panoramic large image, an overlapping area is set between every two adjacent sub-region pieces;

[0027] Detection step: inputting a plurality of sub-region images into the target detection model to obtain a preliminary detection result at a large resolution of a remote tower;

[0028] Detection result processing step: extract and convert the preliminary detection results to obtain target detection results.

[0029] As a preferred implementation of the present application, after executing the target detection step, the target detection result includes target category information and target location information in the remote tower scene; the method further includes:

[0030] Visualization detection result step: the target position information is saved in the form of the center point of the target and the width and height of the target detection frame, and the detection frame is displayed in real time in the remote tower panoramic video.

[0031] In a second aspect, an embodiment of the present invention provides a real-time target detection device based on panoramic video of an airport remote tower, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are interconnected, wherein the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect above.

[0032] The method of the embodiment of the present invention can realize the detection and recognition of targets of different scales in the panoramic video of the remote tower, and the detection result can meet the accuracy of airport detection. At the same time, the method has excellent performance and is easy to implement in engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0034] Figure 1 It is a flow chart of a real-time target detection method based on panoramic video of an airport remote tower provided by an embodiment of the present invention;

[0035] Figure 2 It is a data production flow chart;

[0036] Figure 3 It is the target detection network structure diagram;

[0037] Figure 4 It is a schematic diagram of the data enhancement strategy for small target scale adaptation;

[0038] Figure 5 It is a flow chart of target detection combined with segmentation detection strategy;

[0039] Figure 6 It is a schematic diagram of overlapping area settings;

[0040] Figure 7 It is a structural diagram of a real-time target detection device based on panoramic video of an airport remote tower provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Please refer to Figure 1 The real-time target detection method based on the panoramic video of the airport remote tower provided by the embodiment of the present invention mainly includes:

[0043] S1, data set preparation: obtain airport tower data and public image data, process the airport tower data and public image data to obtain training data.

[0044] like Figure 2 As shown, the steps of data set preparation include data acquisition, screening, cleaning and labeling.

[0045] The data sources include the airport's remote tower data and public data. The airport's remote tower data is obtained through the airport's cameras, and the public data is obtained by downloading public data from the Internet and using crawlers.

[0046] For public data, since the data on the Internet include many categories, not just airplanes, cars, and trucks, the data will be processed first to extract the three categories of data required by the present invention, and then the data will be cleaned and screened to eliminate data with poor quality (such as many toy cars, toy airplanes, etc.), and only save clearly visible targets.

[0047] The tower data obtained by the airport camera is saved in the form of video. Therefore, it is necessary to extract key frames, obtain key frame images, and then annotate the key frames to obtain the label files of each image and convert them into training formats, which include information such as the target category and location in the image, and obtain a data format that can be directly used for model training. Among them, the remote tower data set obtained by the airport camera contains many distant scenes and small targets. Using it for training can improve the scale robustness of the model for targets in the remote tower panorama.

[0048] S2, object detection algorithm design: Design the object detection network.

[0049] like Figure 3 As shown, the structure of the target detection network designed in this embodiment includes three parts: a backbone extraction network, a feature fusion network and a detection head network.

[0050] The backbone feature extraction network is mainly built using Ghost convolution (a convolution method with half the number of parameters of ordinary convolution), and the three effective feature layers in the extraction process are input into the feature fusion network.

[0051] The feature fusion network gives different scale feature layers different weights and automatically learns the weights through the network, playing a role of feature integration.

[0052] The detection head network takes the integrated features as input, adjusts the number of channels through different numbers of 1×1 convolutions, and finally outputs the target detection results at three scales.

[0053] The results at the 80×80 scale are used to detect small objects in the image, and the results at the 20×20 scale are used to predict larger objects.

[0054] The optimization work of the network structure includes: adding SKNet attention mechanism in the early convolutional layer of the backbone extraction network, adding SENet attention mechanism in the later convolutional layer, using focal loss in the loss function (to solve the problem of sample imbalance), and setting a relatively large scale factor for the target detection box loss at a scale of 80.

[0055] S3, model training: using the training data to train the target detection network to obtain a target detection model.

[0056] When training a model, it is mainly divided into two parts: data enhancement and training.

[0057] 1. Data Augmentation

[0058] Using data enhancement strategies for training can improve the generalization performance of the model. In addition to using some commonly used methods such as translation, rotation, splicing, hue change, and size change to enhance the data set, the embodiment of the present invention uses a data enhancement method such as small target scale adaptation to improve scale robustness. Figure 4 As shown in the figure, each training data set will randomly select four pictures for scale scaling and randomly select a center point to splice the four pictures. This data enhancement method can improve the network's detection effect on small targets.

[0059] 2. Model Training

[0060] Use the pytorch framework to build a detection network, randomly initialize the parameters of each convolution kernel of the network, train the calibrated data set, and obtain a pre-trained model. According to the calibrated detection frame, calculate the three prior frame scales that need to be regressed in the feature layer at three resolutions.

[0061] Based on the pre-trained model, the frozen training method is used. First, the feature extraction parameters of the network are frozen for 500 rounds of training, and then the parameters are unlocked and trained for 500 rounds to obtain the final detection model. The hardware conditions for training are: two RTX6000 graphics cards. Software: Ubuntu 18.04 system, some parameter settings: batch size = 128, SGD gradient descent, initial learning rate 0.001. The network output results are network prediction results at three resolutions, and their dimensions are 20×20×24, 40×40×24, and 80×80×24 respectively. The subsequent real-time detection needs to process the detection results before they can be visualized in real-time video.

[0062] S4, panoramic image target detection: obtain the data to be detected, use a segmentation detection strategy to process the data to be detected, obtain multiple sub-region small images, input the multiple sub-region small images into the target detection model, obtain preliminary detection results under the large resolution of the remote tower, process the preliminary detection results, and obtain target detection results.

[0063] The panoramic image target detection part is mainly divided into the following sub-steps: data acquisition, panoramic image segmentation, target detection and detection result processing. For details, please refer to Figure 5 Flowchart shown.

[0064] The data acquisition step is specifically as follows: acquiring the data to be detected through the airport camera, processing the data to be detected, and obtaining a panoramic view of the airport remote control tower.

[0065] Since the resolution of the panoramic video of the airport remote tower is too large, the scale of the objects in the scene is too small. In order to improve the scale robustness of this method and improve the detection effect of small objects, the cutting detection strategy and data enhancement strategy are used. The cutting detection strategy cuts the panoramic image into n small images of different regions, inputs the small images into the model for detection, obtains the detection results and then returns the panoramic image. For details of the flow chart, see Figure 5 .

[0066] In order to avoid cutting off the target, an overlap is set between every two areas. Figure 6 shown.

[0067] Furthermore, using the target detection model to detect the data to be detected (real-time video), the detection results at three resolution scales will be obtained. The processing structure can be roughly divided into the following two steps:

[0068] Step 1: In order to extract the detection results with relatively ideal effects, a confidence threshold is set. Through this confidence threshold, the detection frames with low confidence scores in the results can be filtered out. Then an IOU threshold is set. Through this threshold, the overlapping frames of the same target with relatively high confidence scores can be filtered out, so that each target corresponds to a detection frame.

[0069] The second step is to convert the detection result data. First, the detection results at three resolutions are restored to the 640 resolution of the input network, and then the results at 640 resolution are restored to the original large image resolution. Since the previous cutting detection strategy sets the overlapping area, it is possible that one target has two detection frames. This problem can be solved by the non-maximum suppression method, that is, setting an IOU threshold, through which the overlapping frames of the same target with a relatively high confidence can be filtered, so that each target corresponds to a detection frame. Complete the detection.

[0070] S5, visualization of detection results: visualization of target detection results.

[0071] The detection results after the previous step contain the target category information and target location information in the remote tower scene. The category information is saved in the form of 0, 1, and 2, with airplanes as 0, cars as 1, and trucks as 2. The target location information is saved in the form of the center point of the target and the width and height of the target detection frame. This information can be used to display the detection frame in real time in the remote tower panoramic video. The real-time detection of the remote tower target is completed.

[0072] The method of the embodiment of the present invention can realize the detection and recognition of targets of different scales in the panoramic video of the remote tower, and the detection result can meet the accuracy of airport detection. At the same time, the method has excellent performance and is easy to implement in engineering.

[0073] Furthermore, compared with the prior art, the method has the following beneficial effects:

[0074] (1) Compared with the field surveillance radar, the present invention has low cost and does not require expensive sensors like radar;

[0075] (2) Compared with multi-lateration positioning (MLAT) and automatic dependent surveillance-broadcast (ADS-B) surveillance technologies, the present invention can detect and identify targets that are not equipped with receiving devices;

[0076] (3) Compared with the monitoring method based on traditional image processing, the present invention has high robustness and strong generalization ability;

[0077] (4) Compared with the existing deep learning detection method, the present invention has higher scale robustness, especially for the detection of small targets in airport scenes, and has better effects. It can meet both the accuracy and real-time requirements of airport target detection.

[0078] Based on the same inventive concept, an embodiment of the present invention provides a real-time target detection device based on panoramic video of an airport remote tower. Figure 7 As shown, the data processing end may include: one or more processors 101, one or more input devices 102, one or more output devices 103 and a memory 104, wherein the processors 101, the input devices 102, the output devices 103 and the memory 104 are interconnected via a bus 105. The memory 104 is used to store a computer program, wherein the computer program includes program instructions, and the processor 101 is configured to call the program instructions to execute the method of the embodiment of the real-time target detection method based on the panoramic video of the airport remote tower.

[0079] It should be understood that in the embodiment of the present invention, the processor 101 may be a central processing unit (CPU) and a graphics processing unit (GPU).

[0080] The input device 102 may include a camera array, a keyboard, a mouse, etc., and the output device 103 may include a display (LCD, etc.) and the like.

[0081] The memory 104 may include a read-only memory and a random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include a non-volatile random access memory. For example, the memory 104 may also store information on the device type.

[0082] In a specific implementation, the processor 101, input device 102, and output device 103 described in the embodiment of the present invention can execute the implementation method described in the embodiment of the real-time target detection method based on the panoramic video of the airport remote tower provided in the embodiment of the present invention, which will not be repeated here.

[0083] It should be noted that, for the specific workflow and related details of the real-time target detection device based on the panoramic video of the airport remote tower in the embodiment of the present invention, please refer to the aforementioned method embodiment part, which will not be repeated here.

[0084] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A real-time target detection method based on panoramic video of an airport remote tower, characterized in that: include: Data set preparation steps: obtaining airport tower data and public image data, processing the airport tower data and public image data to obtain training data; Target detection network design steps: Design a target detection network, the structure of which includes a backbone extraction network, a feature fusion network, and a detection head network connected in sequence, and add an SKNet attention mechanism to the early convolutional layer and the late convolutional layer of the backbone extraction network; Model training step: using the training data to train the target detection network to obtain a target detection model; Panoramic image target detection step: obtaining data to be detected, processing the data to be detected using a segmentation detection strategy to obtain a plurality of sub-region small images, inputting the plurality of sub-region small images into the target detection model, obtaining a preliminary detection result under a large resolution of a remote tower, and processing the preliminary detection result to obtain a target detection result; the data to be detected includes a panoramic video of a remote tower of an airport; The steps of panoramic image target detection specifically include: Data acquisition step: acquiring the data to be detected through the airport camera, processing the data to be detected, and obtaining a panoramic view of the airport remote control tower; Segmentation step: cutting the panoramic large image into n different sub-region small images; wherein, when cutting the panoramic large image, an overlapping area is set between every two adjacent sub-region pieces; Detection step: inputting a plurality of sub-region images into the target detection model to obtain a preliminary detection result at a large resolution of a remote tower; Detection result processing step: extracting and converting the preliminary detection results to obtain target detection results; The preliminary detection results include detection results at three resolution scales; the detection result processing steps specifically include: Step 1: Set a confidence threshold, and use the confidence threshold to filter the detection frames with lower confidence scores in the preliminary detection results; set an IOU threshold, and use the IOU threshold to filter the overlapping frames of the same target with higher confidence, so that each target corresponds to a detection frame; Step 2: Restore the detection results at the three resolutions and input them into the 640 resolution of the target detection network to obtain the first result. Restore the first result to the resolution of the panoramic image before cutting, and filter the overlapping frames of the same target through the non-maximum suppression method so that each target corresponds to a detection frame.

2. The real-time target detection method according to claim 1, characterized in that: The steps for making a dataset include: Acquire airport tower data through an airport camera, extract key frame images from the airport tower data, annotate the key frame images to obtain a label file for each key frame image, and convert the label file into a training data format to obtain the training data; The training data is obtained by downloading the public data from the Internet through a search engine or by using a crawler, and the public data is cleaned and screened.

3. The real-time target detection method according to claim 2, characterized in that: The model training steps specifically include: Data enhancement step: performing data enhancement processing on the data set by using translation, rotation, splicing, tone transformation, size transformation and small object scale adaptation; Model training steps: Randomly initialize the parameters of each convolution kernel in the target detection network; The frozen training method is adopted. First, the parameters of the backbone extraction network part of the target detection network are frozen, and the target detection network is trained for 500 rounds using the training data after data enhancement processing. Then, the parameters are unfrozen and the target detection network is trained for 500 rounds to obtain the target detection model.

4. The real-time target detection method according to claim 3, characterized in that: The data enhancement processing of the data set is performed by using a small target scale adaptation method as follows: Four images are randomly selected from the data set for scaling, and a center point is randomly selected to splice the four images.

5. The real-time target detection method according to any one of claims 1 to 4, characterized in that: After executing the target detection step, the target detection result includes target category information and target position information in the remote tower scene; The method further comprises: Visualization detection result step: the target position information is saved in the form of the center point of the target and the width and height of the target detection frame, and the detection frame is displayed in real time in the remote tower panoramic video.

6. A real-time target detection device based on panoramic video of an airport remote tower, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method as claimed in claim 5.

Images