A deep learning-based method for identifying moving targets at airports combined with ADS-B information

Abstract

The present invention discloses a method for identifying moving targets on airport scenes based on deep learning in conjunction with ADS‑B information. The method combines the idea of ​​motion and stillness judgment in change detection with feature extraction, classification and regression modules in target detection. At the same time, the background modeling method based on time histogram in the change detection algorithm is improved to obtain a more accurate background image. The existing ADS‑B technology of aircraft is used for auxiliary training to enhance feature extraction capabilities and improve detection results. The ideas and advantages of the comprehensive change detection algorithm and target detection algorithm are combined.

Description

Technical Field

[0001] The present invention relates to the field of airport target recognition, and in particular to a method for identifying airport scene moving targets based on deep learning in conjunction with ADS-B information. Background Art

[0002] In recent years, the number of civil aviation passengers has continued to increase worldwide, the airport structure has become increasingly complex, and the airport scene has become more crowded. It has become increasingly difficult for administrators to visually monitor ground moving targets in the field, and the safety risks of manual command and management have become more obvious. In order to meet the needs of automated scene monitoring, a monitoring system that covers the entire airport scene area through multiple cameras has been applied to modern airport scene monitoring. Compared with other sensors, computer vision technology has unique advantages because the way video transmits information is more intuitive and real, and the human eye can intuitively capture the key information displayed in the video. Therefore, airport intelligent applications based on computer vision technology have gradually become the mainstream of automated scene monitoring technology. The amount of video data obtained based on computer vision technology is very rich, and on this basis, intelligent applications such as illegal intrusion reminders and collision warnings can be developed. As the basis of many video-based intelligent applications, change detection and target detection algorithms have made great progress in recent years, but there are certain limitations in the application of airport scene monitoring.

[0003] The purpose of change detection algorithm is to present and mark foreground objects whose spatial position changes in image sequences or videos. Traditional change detection algorithms mainly use background subtraction techniques to estimate the initial clean background image using a small number of input video frames, and then perform pixel segmentation between the estimated background and the input video frames, and continuously update the background model. In the method ViBe, three background model update strategies are proposed: random background sample replacement to represent short-term and long-term history, memoryless update strategy and spatial diffusion strategy through background sample propagation. These strategies have been widely adopted by the latest state-of-the-art change detection techniques, but traditional methods are unsupervised. The performance of the algorithm depends on the quality of the established background model. In airport scenes, the background changes frequently due to factors such as lighting and shadows, which affects the detection results. In recent years, a large number of supervised change detection techniques based on convolutional neural networks (CNNs) have been proposed. The change detection algorithm based on convolutional neural network encodes the multi-scale features of the input image and then uses a transposed convolutional neural network to decode it, realizing the probability mapping of features to foreground pixels and determining whether each pixel belongs to the foreground or background one by one. Although the accuracy of the change detection algorithm based on convolutional neural network has been greatly improved compared with traditional methods and has achieved good accuracy in AGVS (airport scene monitoring data set), these methods are relatively complex and slow, making it difficult to play a role in real-time monitoring of airport scene videos.

[0004] As a classic task in computer vision, the object detection task has always been a research hotspot in this field. The purpose of object detection is to find objects with different geometric shapes and assign an accurate label to each detected object. This is a challenging task because objects can appear in natural images in any scale, shape, and position, and the appearance of objects of the same type may be very different. Traditional object detection algorithms usually use a sliding window method to extract features from each window through feature extraction algorithms such as SIFT and HOG, and then classify the extracted features using machine learning algorithms such as support vector machines to finally determine whether the window contains a certain type of object. This leads to the disadvantages of traditional object detection algorithms, such as large computational complexity, slow operation speed, and often multiple correct recognition results. Driven by the rapid development of deep learning, researchers have designed more effective object detection algorithms. Object detection algorithms can be divided into two-stage and one-stage methods. The two-stage object detection algorithm first generates candidate regions that may contain objects, and then further classifies and calibrates the candidate regions to obtain the final detection results. The one-stage method does not have the step of generating candidate regions, but directly generates results. Therefore, the former works slower but has higher detection accuracy. Compared with the change detection algorithm based on pixel-level segmentation, target detection only needs to find the area where the current target is located, which greatly reduces the computational cost. However, in the specific scenario of airport scene monitoring, most of the aircraft are often stationary on the apron, and only a few are moving. Among them, we are only interested in moving targets, and the target detection algorithm will detect all the targets in the picture, which will not only cause information redundancy, but also the display results of the algorithm are not intuitive enough.

[0005] Airport scenes are different from other scenes. Most civil aviation aircraft are equipped with the ADS-B system, which is a broadcast automatic dependent surveillance system. Aircraft equipped with this system can broadcast some of their own information through data links. This information includes the aircraft's location (four-dimensional coordinates, accuracy, dimension, altitude, time), the aircraft's flight speed and call sign (aircraft identity information), and some other information. At present, there are some works that can map the four-dimensional image information and camera position in ADS-B to two-dimensional image coordinates through mapping methods. The ADS-B signal is the unique information of the moving aircraft target. Therefore, with the help of ADS-B information, a new airport scene moving target recognition strategy can be proposed to meet the specificity and real-time requirements of airport scene monitoring, so as to realize the intelligent application of airports based on computer vision technology. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for identifying moving targets on airport surfaces based on deep learning in combination with ADS-B information.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] A method for identifying moving targets on airport surfaces based on deep learning combined with ADS-B information includes the following steps:

[0009] S1. Decode the ADS-B information and map the ADS-B data to a two-dimensional image using the three-dimensional data returned by the aircraft and the position of the camera;

[0010] S2. Label the scene surveillance video sequences in the airport scene surveillance dataset to obtain the real bounding box, and select the first ten video sequences as the training set and the last ten video sequences as the test set;

[0011] S3, constructing a moving target recognition network, and using the constructed moving target recognition network to output an estimated background image;

[0012] S4, using the estimated background image obtained in step S3 and the current frame image as a moving target recognition network to calculate the category of the target candidate frame and the boundary information of the target candidate frame;

[0013] S5, compare the target candidate box category and the boundary information of the target candidate box obtained in S4 with the real bounding box annotated in S2, and use the cross entropy loss function and smooth L1 The distance is used to calculate the classification loss function and the regression loss function, and the sum of the two loss functions is used as the loss function of the motion estimation network;

[0014] S6. Back-propagate the loss function of the motion target recognition network to obtain a trained motion estimation network, and use the trained motion estimation network to perform motion target recognition on the airport scene.

[0015] Furthermore, the moving target recognition network constructed in S3 includes a cascaded motion judgment module and a target detection module, wherein the input of the moving target recognition network is the current frame, the position information and label information of the current frame after ADS-B decoding, and the historical previous 30 frames of images of the current frame.

[0016] Furthermore, the method for calculating the estimated background image in S3 is:

[0017] The most recent historical frame image is input into the motion judgment module and continuously passed through multiple multi-scale feature receiving blocks. Each feature block captures the maximum response of the receptive fields of sizes 1×1, 3×3 and 5×5 to obtain the estimated background image.

[0018] Furthermore, the target detection module includes a backbone network, a feature pyramid network, and a regression and classification network that are cascaded in sequence, wherein the backbone network is a ResNet network, which is used to calculate the convolution features of the entire input image; the feature pyramid network enhances the target detection module through top-down paths and lateral connections; and the regression and classification networks are both multi-scale convolutional neural networks.

[0019] Furthermore, the classification loss function and regression loss function in S5 are respectively expressed as:

[0020]

[0021]

[0022]

[0023] Among them, t i ={t x ,t y ,t w ,th} represents the coordinates of the anchor point, which is the predicted offset in the training phase; is the ground truth label with the same dimension vector, and is the offset of the ground truth during the training phase; p i is the probability that the anchor point is the target; is the classification loss function; is the regression loss function.

[0024] The present invention has the following beneficial effects:

[0025] 1. This method solves the problem that the target detection algorithm cannot distinguish between moving and stationary targets, and is more suitable for airport scene monitoring scenarios.

[0026] 2. Using multi-scale receptive feature block cascade as the background modeling method can more accurately identify moving targets and obtain precise background images.

[0027] 3. Adopt aircraft-specific ADS-B technology to improve aircraft detection accuracy and provide airports with more effective scene surveillance solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the method for identifying moving targets on airport surfaces based on deep learning combined with ADS-B information of the present invention.

[0029] Figure 2 This is the network structure of the motion estimation module of the present invention.

[0030] Figure 3 This is the network structure of the target detection module of the present invention.

[0031] Figure 4 It is the comparison result between the output of the motion estimation module of the present invention and the estimated background obtained based on the time histogram method.

[0032] Figure 5 The comparison results of the moving target recognition algorithm implemented by the present invention and the RCNN target detection algorithm are given.

[0033] Figure 6 The mAP results of the moving target recognition algorithm and the RCNN target detection algorithm implemented by the present invention in some video sequences of the AGVS dataset. DETAILED DESCRIPTION

[0034] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0035] A method for identifying moving targets on airport surfaces based on deep learning combined with ADS-B information, such as Figure 1 As shown, the following steps are included:

[0036] S1. Decode the ADS-B information and map the ADS-B data to a two-dimensional image using the three-dimensional data returned by the aircraft and the position of the camera;

[0037] S2. Label the scene surveillance video sequences in the airport scene surveillance dataset to obtain the real bounding box, and select the first ten video sequences as the training set and the last ten video sequences as the test set;

[0038] For deep learning-based algorithms, data sets play a vital role. Therefore, we selected some airport scene surveillance video sequences from AGVS (Airport Scene Surveillance Dataset). In this embodiment, we manually labeled the aircraft targets in each frame of the picture using labelme software, and set the first ten video sequences as the training set, and the last ten video sequences as the test set.

[0039] S3, constructing a moving target recognition network, and using the constructed moving target recognition network to output an estimated background image;

[0040] According to the basic algorithm framework of the proposed motion judgment module and target detection module, we cascade the two to obtain the final moving target recognition network. The input of the network is the current frame, the position information and label information of the current frame after ADS-B decoding, and the historical 30 frames of images before the current frame. Among them, the historical 30 frames of images before the current frame are input into Figure 2 The output of the motion estimation network is the estimated background image. The output visualization result is compared with the temporal histogram-based result. Figure 4 As shown, both use 30 frames of historical images to obtain the background image of the current frame.

[0041] The background estimation method based on the time histogram is widely used in change detection algorithms. The method of implementing background estimation with the time histogram is as follows. First, at each pixel position, obtain the time histogram:

[0042]

[0043]

[0044] Among them, I(m,n,t) represents the pixel value at position (m,n) of the t-th frame image. From the pixel time histogram, the background pixel intensity at a specific position (m,n) can be obtained:

[0045]

[0046] Among them, argmax(.) is the maximum value of the pixel value l at the histogram point (m,n). Figure 4 The background estimation results based on the time histogram method are given. However, due to the slow speed of moving targets in airport scenes and the frequent motion stops, the method based on the time histogram cannot produce good results. Therefore, we propose a motion estimation network based on a convolutional neural network. The structure is as follows Figure 2 As shown in Figure 1. The input of the network is the most recent historical frame image, which is continuously passed through multiple multi-scale receptive feature blocks. Each feature block captures the maximum response of the receptive field of size 1×1, 3×3 and 5×5, and finally obtains the estimated background feature map. This enables us to obtain background statistical information while ensuring the adaptability of the network to different changing scenes. The visual comparison results of the proposed motion estimation network are shown in Figure 1. Figure 4 As can be seen from the figure, the background estimation method based on the time histogram will have ghosting due to the slow movement of the aircraft target, and the proposed motion estimation network obtains the background more accurately.

[0047] The object detection module consists of a backbone network, a feature pyramid network, and a regression and classification network, such as Figure 3As shown. We use the ResNet network as the backbone network, which is responsible for calculating the convolutional feature map of the entire input image and providing high-dimensional image features for the feature pyramid network. The Feature Pyramid Network (FPN) enhances the standard convolutional network through top-down paths and lateral connections, and can be used to detect objects of different scales. At each layer of the pyramid feature map, an input feature map of 256 channels is taken and 9 anchor points are set. After obtaining the feature map, the ADS-B signal is first used as a guide to select a probability region, such as Figure 3 As shown, the red area is the probability area, and the search is performed on the feature map in this area, and finally the target candidate box is generated by regression and classification.

[0048] S4, using the estimated background image obtained in step S3 and the current frame image as a moving target recognition network to calculate the category of the target candidate frame and the boundary information of the target candidate frame;

[0049] S5, compare the target candidate box category and the boundary information of the target candidate box obtained in S4 with the real bounding box annotated in S2, and use the cross entropy loss function and smooth L1 The distance is used to calculate the classification loss function and the regression loss function, and the sum of the two loss functions is used as the loss function of the motion estimation network;

[0050] The obtained background estimation image and the current frame image are used as the input of the backbone network and the feature pyramid network. The target detection network calculates the category of the candidate box and the bounding box information of the candidate box (the coordinates of the upper left vertex and the width and height of the box). These parameters are compared with the annotated real bounding box (ground truth), and the cross entropy loss function and smoothing are used. L1 Distance is used to calculate two loss functions, where the classification loss is recorded as The regression loss is denoted as

[0051]

[0052]

[0053]

[0054] where t i ={t x ,t y ,t w ,t h}, represents the coordinates of the anchor, and the predicted offset in the training phase. It is a vector with the same dimension, representing the anchor. During the training phase, it is the offset w relative to the ground truth. R stands for smoothness.L1 . p i is the probability that the anchor is the target, is the ground truth label. The network loss function is the sum of the classification loss and regression loss:

[0055]

[0056] S6. Back-propagate the loss function of the motion target recognition network to obtain a trained motion estimation network, and use the trained motion estimation network to perform motion target recognition on the airport scene.

[0057] In this example, the back propagation of the loss function also passes through the motion estimation module. I use an initial learning rate of 1×10 -5 The Adam optimizer is used for optimization, and the prediction threshold for the final detection is 0.5. Figure 5 Comparison of the results of the same frame of the experimental method with the target detection RCNN algorithm shows that in addition to having relatively better detection accuracy, the proposed algorithm can also accurately determine the movement and stillness of the target.

[0058] The number of historical frames that is most suitable for the airport scene is determined by analyzing the experimental results. We use the target detection evaluation index mAP to evaluate the accuracy of the detection results. Since this invention first proposed the concept of moving target recognition, there is no related algorithm for comparison, so the target detection algorithm RCNN is selected for comparison. The experimental results are as follows Figure 6 As shown, it can be seen that when the historical frames are selected as 30 frames, the average accuracy is the highest, and because the RCNN algorithm cannot distinguish between motion and stillness, the accuracy is poor.

[0059] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0063] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for identifying moving targets at airports based on deep learning combined with ADS-B information, characterized in that: The steps include: S1. Decode the ADS-B information and map the ADS-B data to a two-dimensional image using the three-dimensional data returned by the aircraft and the position of the camera; S2. Label the scene surveillance video sequences in the airport scene surveillance dataset to obtain the real bounding box, and select the first ten video sequences as the training set and the last ten video sequences as the test set; S3, constructing a moving target recognition network, and using the constructed moving target recognition network to output an estimated background image; S4, using the estimated background image obtained in step S3 and the current frame image as a moving target recognition network to calculate the category of the target candidate frame and the boundary information of the target candidate frame; S5, compare the target candidate box category and the boundary information of the target candidate box obtained in S4 with the real bounding box annotated in S2, and use the cross entropy loss function and smooth L1 The distance is used to calculate the classification loss function and the regression loss function, and the sum of the two loss functions is used as the loss function of the motion estimation network; S6. Back-propagate the loss function of the motion target recognition network to obtain a trained motion estimation network, and use the trained motion estimation network to perform motion target recognition on the airport scene.

2. The method for identifying moving targets at airports based on deep learning in combination with ADS-B information according to claim 1, characterized in that: The moving target recognition network constructed in S3 includes a cascaded motion judgment module and a target detection module, wherein the input of the moving target recognition network is the current frame, the position information and label information of the current frame after ADS-B decoding, and the historical previous 30 frames of images of the current frame.

3. The method for identifying moving targets at airports based on deep learning in combination with ADS-B information according to claim 2 is characterized in that: The method for calculating the estimated background image in S3 is: The most recent historical frame image is input into the motion judgment module and continuously passed through multiple multi-scale feature receiving blocks. Each feature block captures the maximum response of the receptive fields of sizes 1×1, 3×3 and 5×5 to obtain the estimated background image.

4. The method for identifying moving targets at airports based on deep learning in combination with ADS-B information according to claim 2, characterized in that: The target detection module includes a backbone network, a feature pyramid network, and a regression and classification network that are cascaded in sequence, wherein the backbone network is a ResNet network, which is used to calculate the convolution features of the entire input image; the feature pyramid network enhances the target detection module through top-down paths and lateral connections; and the regression and classification networks are both multi-scale convolutional neural networks.

5. The method for identifying moving targets at airports based on deep learning in combination with ADS-B information according to claim 2, characterized in that: The classification loss function and regression loss function in S5 are respectively expressed as: Among them, t i ={t x ,t y ,t w ,t h } represents the coordinates of the anchor point, which is the predicted offset during the training phase; is the ground truth label with the same dimension vector, and is the offset of the ground truth during the training phase; p i is the probability that the anchor point is the target; is the classification loss function; is the regression loss function.

Images