An end-to-end aircraft detection and tracking method based on deep learning

Abstract

This paper discloses an end-to-end deep learning-based aircraft detection and tracking method. It proposes a multi-scale adaptive joint framework based on deep learning to simultaneously complete target detection, appearance feature extraction, and position prediction. Three sub-branches are established to learn the appearance of aircraft, the location and size of aircraft, and the differences between the appearances of different aircraft. By learning both intra-class and inter-class appearances, the method not only enhances the ability to detect and distinguish aircraft in weak environments, but also effectively improves the accuracy and efficiency of multi-aircraft tracking.

Description

Technical Field

[0001] The present invention relates to the field of target recognition, and in particular to an end-to-end aircraft detection and tracking method based on deep learning. Background Art

[0002] With the rapid development of the national economy and aviation sector in recent years, more and more airports have sprung up, and their scale has grown year by year. The resulting increase in aviation business volume and total air transport volume has also led to increasingly complex airport surfaces, requiring more refined instructions from control centers. By manually monitoring aircraft on the airport surface and their corresponding motion states, aircraft routes can be predicted and guided, avoiding conflicts between different target aircraft while improving the operational efficiency of airport surface targets. Therefore, airport surface monitoring systems have become an indispensable auxiliary tool for airport control centers. With the rapid development of machine learning and deep neural networks, computer vision-based monitoring systems are gradually being implemented. Based on the collected real-time video data stream, aircraft data within the image is analyzed to provide the control center with accurate surface information, thereby enhancing airport safety management. This not only overcomes many shortcomings of manual visual monitoring, but also improves accuracy and enables real-time analysis of airport surface operations around the clock. The main functions of airport surface monitoring systems include multi-aircraft behavior recognition, path planning, and collision warning. A robust multi-target tracking algorithm is the cornerstone for implementing these higher-level applications.

[0003] The main task of multi-object tracking is to analyze the image information of a continuously input video stream, detect the objects in the image, and associate the same target object across frames to form a complete tracking trajectory. Currently, mainstream multi-object tracking frameworks are divided into three main tasks: detection, position prediction, and appearance feature extraction. The process involves predicting the position of the target in the current image frame through the position prediction task, extracting the appearance information of the target, and combining the position and appearance information to associate the target of interest in the current image frame detected by the detection task, thereby achieving multi-object tracking.

[0004] Unlike common multi-target tracking scenarios, the main difficulty in aircraft tracking lies in the high similarity in appearance between aircraft due to the relatively regular motion of aircraft. Therefore, using traditional appearance extraction methods will cause associations between different targets, increase the number of aircraft ID exchanges, cause trajectory interruptions, and affect tracking robustness. At the same time, the current mainstream detection tasks, appearance feature extraction tasks, and position prediction tasks all require separate training of corresponding network models, and the establishment of different databases. This not only increases the workload of early network model training, but also causes redundancy in the tracking framework. The addition of network models will result in poor real-time performance, making it impossible to implement normally. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides an end-to-end aircraft detection and tracking method based on deep learning.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] An end-to-end aircraft detection and tracking method based on deep learning includes the following steps:

[0008] S1. Mark the aircraft bounding box and correct the marked bounding box to obtain a corrected bounding box;

[0009] S2. Build a deep neural network model, input two adjacent image frames and the corrected bounding box into the deep neural network to extract features, and train the deep neural network;

[0010] S3. Use the trained deep neural network to perform target detection and tracking on the two input frames.

[0011] Furthermore, the method for correcting the marked bounding box in S1 is:

[0012] Define the bounding box (x, y, w, h) with the upper left coordinate and length and width. With the center as the origin, reduce the height of the bounding box by 20% to obtain the corrected bounding box (x, 1.2y, w, 0.8h).

[0013] Furthermore, the training process of the deep neural network in S2 is:

[0014] S21. Construct a feature pyramid to extract features from two adjacent frames of the input image sequence, use cross entropy loss to classify foreground and background, and use Smooth-L1 loss to perform bounding box regression to obtain feature maps of different scales;

[0015] S22. Construct an appearance feature map of the aircraft based on feature maps of different scales to complete the appearance feature extraction task;

[0016] S23. Determine the offset of the center point of the aircraft target through the convolution appearance feature map to perform position prediction.

[0017] Furthermore, the cross entropy loss function in S21 is expressed as:

[0018]

[0019] Among them, p i is the predicted classification probability of the i-th anchor point. When the IoU of the i-th anchor point exceeds the threshold, it is considered a positive sample. Otherwise it is a negative sample. t i The coordinates of the bounding box predicted for the anchor point, is the coordinate of the true width; is the Smooth-L1 function; N reg is the number of anchor positions.

[0020] Furthermore, the S22 specifically includes:

[0021] S221. Use the standard cross entropy loss function to train the appearance affinity branch, and use the cosine distance as the relative distance between samples, expressed as:

[0022]

[0023] Among them, f T is the selected anchor point sample, G + Indicates that the selected anchor point is a positive sample, Indicates that this is a negative sample;

[0024] S222. Use the discriminant method to train the positive and negative samples respectively to complete the position prediction. The logical loss is expressed as:

[0025]

[0026] in, is the appearance feature of the selected anchor point in frame t-1, g + Indicates the anchor points covered by the search area, which are positive samples, g - Aircraft samples that are not covered by the search area are called negative samples.

[0027] Furthermore, the S3 specifically includes:

[0028] S31. Input the new frame image into the trained network to obtain the detection coordinates D of the new frame image. i , detect appearance features δ(D i ) and tracking coordinates T j ;

[0029] S32, calculate detection coordinates D i and tracking coordinates T j The cross ratio is used to construct the spatial similarity matrix S between the tracking target and the detection target. iou , expressed as:

[0030]

[0031] Among them, Area(D i ) Detection target D i The coordinates of Area(T j ) represents the tracking target Tj coordinates of

[0032] S33, calculate the detection appearance feature δ(D i ) to construct the appearance similarity matrix S of the tracked target and the detected target cos , expressed as:

[0033]

[0034] Where δ(D i ) represents the detection target D i The appearance characteristics of δ(T j ) represents the tracking target T j appearance characteristics;

[0035] S34, setting the correlation matrix to determine the final score S of the detected target and the tracked target;

[0036] S35. Use the Hungarian algorithm to associate the tracking target with the detection target to obtain the tracking results of multiple aircraft.

[0037] Furthermore, in S34, the calculation formula of the final score S is:

[0038] S=α·S iou +α·S cos

[0039] Among them, α is the weight parameter.

[0040] The present invention has the following beneficial effects:

[0041] 1) Compared with previous detection-based tracking methods, the present invention adopts a deep network framework that integrates target detection and tracking. The information fusion is more complete, the model inference speed is significantly improved, and the tracking performance of the network is effectively improved.

[0042] 2) In view of the fact that there is a large amount of background information in the aircraft detection frame, a side attention mechanism is proposed to effectively improve the feature extraction capability of the network model.

[0043] 3) In view of the small-sized targets at the airport scene, a scale-adaptive method is proposed, which effectively solves the problem of small-sized targets being difficult to detect and improves the tracking performance of the network model, thereby effectively enhancing the airport scene surveillance capability and improving the management efficiency of aircraft at the airport scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a flow chart of the end-to-end aircraft detection and tracking method based on deep learning of the present invention.

[0045] Figure 2Schematic diagram of the edge attention mechanism, where (a) and (b) are the actual edge effects, and (c) is the edge parameter setting diagram.

[0046] Figure 3 Aircraft appearance shapes and anchor point proportions compiled for several sequences.

[0047] Figure 4 This is the overall network structure proposed in the embodiment of the present invention.

[0048] Figure 5 Schematic diagram of position prediction according to an embodiment of the present invention.

[0049] Figure 6 Schematic diagram of the deep neural network framework structure of an embodiment of the present invention.

[0050] Figure 7 Schematic diagram of aircraft detection and tracking according to an embodiment of the present invention DETAILED DESCRIPTION

[0051] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0052] An end-to-end aircraft detection and tracking method based on deep learning, such as Figure 1 As shown, the following steps are included:

[0053] S1. Mark the aircraft bounding box and correct the marked bounding box to obtain a corrected bounding box;

[0054] As a rigid body target, the aircraft will change its shape and size due to movement and force. Therefore, in the visual image, the aircraft has a relatively regular appearance and obvious geometric features, such as Figure 2(a) and Figure 2(b). For neural network models, shallow convolutional neural networks are primarily responsible for extracting detailed texture information about the target. As the number of layers increases, the semantic information of the extracted features becomes more abstract. In extracting the semantic model of an aircraft, the contour information of the aircraft's engine, tires, wings, tail, and fuselage plays a decisive role. However, through statistical analysis of the original detection frames of aircraft in different postures, we found that due to the aircraft's unique structure, the background area created by the tail accounts for 45% of the original detection frame's background area. As the aircraft's size decreases, the background area becomes larger, which is detrimental to model learning and convergence. Therefore, by limiting the original bounding box of the aircraft, we reduce the background information in most annotation frames to at least 80% of the original amount, which is more conducive to the analysis and learning of small target features by deep network models.

[0055] Based on this, we first need to correct the marked aircraft bounding box. Define the bounding box (x, y, w, h) with the upper left coordinates and length and width. With the center as the origin, reduce the height of the bounding box by 20% to obtain the corrected bounding box (x, 1.2y, w, 0.8h). Figure 2 (c)

[0056] S2. Build a deep neural network model, input two adjacent image frames, the aircraft ID, and the corrected bounding box into the deep neural network to extract features, and train the deep neural network;

[0057] The deep neural network framework built in this embodiment is as follows Figure 6 As shown, the input image is processed through the DarkNet53 network and the FPN network to extract deep learning features. The object detection task obtains the aircraft detection result for the current image frame, and then obtains the appearance features and search features of the detected aircraft in the current image frame.

[0058] like Figure 5 As shown, the position is predicted by convolving the appearance features of the aircraft in the previous frame with the search features of the aircraft in the current frame, and the intersection and union ratio of the predicted bounding box and the detected bounding box is calculated to obtain S iou .

[0059] Calculate the cosine distance between the appearance feature of the target tracked in the previous frame and the appearance feature of the target detected in the current frame to obtain S cos .

[0060] Among them, the detection task, appearance feature extraction task and location prediction task adopt an end-to-end learning method, such as Figure 6Figure 1 shows the network structure of the entire framework. The FPN is a three-layer structure, with each layer outputting a prediction head. Each prediction head is composed of a grid of the same size, each with three anchor points of varying proportions representing the depth features of the corresponding image. These features are used to calculate the confidence, bounding box, and appearance features of the object detected by that anchor point. Note that the search feature is a combination of appearance features from several networks, and the position is predicted using both the appearance and search features.

[0061] The present invention uses DarkNet53 as the backbone network. Since we need to merge the detection task, position prediction task and appearance feature extraction task into an end-to-end neural network model, we need to enhance the semantic features of the image. Feature extraction is performed on the input image by constructing a feature pyramid (FPN), wherein the FPN is composed of three layers of top-down downsampling and three layers of bottom-up upsampling. Through 32 times, 16 times, and 8 times the downsampling rate and upsampling rate, it is possible to fuse feature maps with strong low-resolution semantic information and feature maps with weak high-resolution semantic information but rich spatial information while increasing less computational complexity. Regarding the setting of anchor points, because the anchor points of the existing algorithm are not set for aircraft, this will cause the features contained in each anchor point box to have a certain amount of additional noise, making it impossible for the network to perform correct classification and correction, and also affecting the training of the appearance feature model to a certain extent. Therefore, the present invention combines the appearance aspect ratio of the aircraft at different angles and in different scenes to set a suitable anchor point aspect ratio suitable for the present invention, scores the features contained in each anchor point frame and adjusts the frame, and selects the frame most likely to contain the aircraft from different anchor points, thereby completing the target detection task.

[0062] The training process of the neural network constructed based on this embodiment includes the following steps:

[0063] S21. Construct a feature pyramid to extract features from two adjacent frames of the input image sequence, use cross entropy loss to classify foreground and background, and use Smooth-L1 loss to perform bounding box regression to obtain feature maps of different scales;

[0064] The input image is first subjected to feature extraction by DarkNet53 and FPN networks to form a multi-scale feature map. The ratios of the anchor points are {2:3, 2:5, 2:7}, as shown in the following example: Figure 3 As shown, we set IOU>0.5 as foreground and IOU<0.4 as background, use cross entropy loss to classify foreground and background, and use Smooth-L1 loss for bounding box regression:

[0065]

[0066] where pi is the predicted classification probability of the i-th anchor point. When the IoU of the i-th anchor point exceeds the threshold, it is considered a positive sample. Otherwise it is a negative sample. t i are the coordinates of the bounding box predicted by the anchor point, is the coordinate of the true width. is the Smooth-L1 function. N reg is the number of anchor positions, and N cls is the batch size.

[0067] S22. Construct an appearance feature map of the aircraft based on feature maps of different scales to complete the appearance feature extraction task;

[0068] In the appearance feature extraction task, an aircraft appearance feature map is constructed based on feature maps of different scales, so that each anchor point contains the appearance information of the identity it represents. By learning the cosine distance of different identity features, the appearance features of the same identity are brought closer together, while the appearance features of different identities are moved further away. Then, the convolution of the appearance features is used to determine the offset of the target center point for position prediction, such as Figure 5 As shown in Figure 2. Our template image features are the appearance features of the tracked target i in frame t-1, and the search image features are the identity features in the prediction head of frame t. The coordinates of the search image are centered on the coordinates of target i, and the length and width are proportionally enlarged to half the sum of the length and width of target i. The search image features are then mapped using the ratio of the length and width of the original image to the length and width of the prediction head. The center position offset of target i can be obtained by inversely mapping the position of the maximum corresponding point, where the length and width of target i in frame t are the same as those in frame t-1.

[0069] Regarding the appearance feature loss function, since the number of negative samples is significantly larger than the number of positive samples during training, we apply the standard cross-entropy loss function to train the appearance affinity branch, where the cosine distance is used as the relative distance between samples.

[0070]

[0071] where f T is selected as the anchor sample, G + Indicates that this anchor point is a positive sample, Indicates that this is a negative sample

[0072] S23. Determine the offset of the center point of the aircraft target through the convolution appearance feature map to perform position prediction.

[0073] Regarding the position prediction loss function, we adopt a discriminative method to train positive and negative sample pairs, and its logical loss is defined as follows:

[0074]

[0075] in is the appearance feature of the selected anchor point in frame t-1.

[0076] In order to jointly learn the appearance feature extraction task, the position prediction task, and the object detection task, two adjacent frames of images need to be input during training. Therefore, the total loss function of the model is as follows:

[0077]

[0078] The loss of the network is minimized by SGD with a momentum of 0.9 and a learning rate of 0.001.

[0079] S3. Use the trained deep neural network to perform target detection and tracking on the two input frames.

[0080] For a new frame image, the detection coordinate D is first obtained by the detection task i , detect appearance features δ(D i ) and tracking coordinates T j . Calculate the intersection and union ratio of coordinates and construct a spatial similarity matrix S between the tracked target and the detected target iou By calculating the cosine distance of the features, we can construct an appearance similarity matrix S between the tracked target and the detected target cos With a weight of 0.5, the final score is determined:

[0081] S=0.5·S iou +0.5·S cos

[0082] And use the Hungarian algorithm to associate the tracking target with the detection target to obtain the multi-aircraft tracking result. i , if the network's classification score for the box is greater than the threshold 0.5, the target is initialized as a new track. For the already associated target, the state and information of the target are updated, such as Figure 7 shown.

[0083] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

[0084] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. An end-to-end aircraft detection and tracking method based on deep learning, characterized in that: The steps include: S1. Mark the aircraft bounding box and correct the marked bounding box to obtain a corrected bounding box; S2. Build a deep neural network model, input two adjacent image frames and the corrected bounding box into the deep neural network to extract features, and train the deep neural network. The training process is as follows: S21. Construct a feature pyramid to extract features from two adjacent frames of the input image sequence, use cross entropy loss to classify foreground and background, and use Smooth-L1 loss to perform bounding box regression to obtain feature maps of different scales. The cross entropy loss function is expressed as: in, is the predicted classification probability of the i-th anchor point. When the IoU of the i-th anchor point exceeds the threshold, it is considered a positive sample. , otherwise it is a negative sample, ; The coordinates of the bounding box predicted for the anchor point, is the coordinate of the true width; is the Smooth-L1 function; is the number of anchor positions; S22. Construct an appearance feature map of the aircraft based on feature maps of different scales to complete the appearance feature extraction task, specifically including: S221. Use the standard cross entropy loss function to train the appearance affinity branch, and use the cosine distance as the relative distance between samples, expressed as: in, is the selected anchor point sample, Indicates that the selected anchor point is a positive sample, Indicates that this is a negative sample; S222. Use the discriminant method to train the positive and negative samples respectively to complete the position prediction. The logical loss is expressed as: in, is the appearance feature of the selected anchor point in frame t-1, Indicates the anchor points covered by the search area, which are positive samples. Aircraft samples that are not covered by the search area are called negative samples; S23, determining the offset of the center point of the aircraft target through the convolution appearance feature map to perform position prediction; S3. Use the trained deep neural network to perform target detection and tracking on the two input frames.

2. The end-to-end aircraft detection and tracking method based on deep learning according to claim 1, characterized in that: The method for correcting the marked bounding box in S1 is: Define the bounding box with the upper left coordinate and length and width ( x,y,w,h ), taking the center as the origin, reduce the bounding box height by 20% to obtain the corrected bounding box ( x,1.2y,w,0.8h ).

3. The end-to-end aircraft detection and tracking method based on deep learning according to claim 1, characterized in that: The S3 specifically includes: S31. Input the new frame image into the trained network to obtain the detection coordinates of the new frame image , detect appearance features and tracking coordinates ; S32, calculate detection coordinates and tracking coordinates The cross ratio is used to construct the spatial similarity matrix between the tracking target and the detection target. , expressed as: in, Detection target The coordinates of Indicates the tracking target coordinates of S33, calculate and detect appearance features The cosine distance is used to construct the appearance similarity matrix between the tracked target and the detected target. , expressed as: in Indicates the detection target appearance characteristics, Indicates the tracking target appearance characteristics; S34, setting the correlation matrix to determine the final score S of the detected target and the tracked target; S35. Use the Hungarian algorithm to associate the tracking target with the detection target to obtain the tracking results of multiple aircraft.

4. The end-to-end aircraft detection and tracking method based on deep learning according to claim 3, characterized in that: In S34, the calculation formula of the final score S is: in, is the weight parameter.

Images