An aircraft intelligent detection, tracking and anomaly analysis method

Abstract

The present invention discloses an aircraft intelligent detection, tracking and anomaly analysis method, which relates to the technical field of aircraft intelligent detection. It includes data collection and preprocessing: collecting airport video data in the takeoff and landing areas, and cleaning and normalizing it; sorting out each image frame and annotating the flight state; target tracking: detecting and tracking the aircraft target; behavior anomaly early warning: designing a behavior anomaly analysis algorithm to analyze the flight state of the aircraft target for detecting abnormal behavior in the flight state; behavior anomaly state monitoring and warning: displaying the target position information and whether there is behavior anomaly; construction of appearance anomaly detection algorithm: designing a self-supervised anomaly detection method based on image element decoupling, and promoting the model to better extract the features of the normal area and the abnormal area in the image by decoupling the normal image and the abnormal noise part; result display. The present invention conducts real-time monitoring, tracking and anomaly state analysis on the takeoff and landing process of the aircraft through intelligent means to ensure the safe flight of the aircraft. Aiming at the problem of detecting small-area anomalies on the aircraft appearance, a method for detecting aircraft anomalies based on image decoupling and generative adversarial is designed to improve the detection effect of the model on aircraft appearance anomalies.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft intelligent detection, and particularly to an aircraft intelligent detection, tracking and anomaly analysis method. Background Art

[0002] The takeoff and landing phases are one of the most critical and complex links in the flight process. Data shows that about 60% of air crash accidents occur during the takeoff and landing phases. This is mainly because the pilot's operation tasks are complex during the takeoff and landing phases, the ground airflow stability is relatively poor, and bird activities are also relatively frequent, resulting in the aircraft being prone to accidents. Abnormal accidents during the takeoff and landing phases require air traffic controllers and pilots to make preparations in advance, and to do a good job in emergency measures and rescue preparations in the air and on the ground. On the other hand, before takeoff and after landing, it is necessary to check whether there are cracks, impact damages and other areas on the aircraft hull. Therefore, detecting, tracking and monitoring abnormal flight states can provide necessary information for controllers and pilots, help air traffic control personnel to take appropriate disposal measures in time, and is of great significance for improving the automation level of air traffic control and reducing the personnel burden.

[0003] Traditional monitoring during the takeoff and landing phases mainly relies on the experience judgment of controllers, ground crew and pilots. With the continuous development of computer technology and data analysis methods, the use of technical means to monitor and analyze flight states has become increasingly emphasized. However, related methods (such as the document "An Aircraft Intelligent Tracking System and Method" (CN113691775A), etc.) usually only focus on the detection and tracking of aircraft. Currently, there is no algorithm that focuses on the automatic analysis of abnormal states of aircraft. Therefore, constructing aircraft detection, tracking and anomaly analysis to timely discover the abnormal states of aircraft is of great significance for reducing the visual burden of controllers, reminding controllers to make preparations in time, and improving the automation level of air traffic control. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the above-mentioned existing technologies and provide an aircraft intelligent detection, tracking and anomaly analysis method.

[0005] The technical solution provided by the present invention is as follows: An aircraft intelligent detection, tracking and anomaly analysis method, characterized in that it specifically includes the following steps:

[0006] Step 1, Data acquisition and preprocessing

[0007] Use multiple gun-type cameras to collect airport video data in the takeoff and landing areas, and clean and normalize the video data; sort out each image frame in the video data and label the flight states of aircraft targets;

[0008] Step 2, Target tracking

[0009] Detect and track aircraft targets;

[0010] Step 3, Abnormal behavior warning

[0011] Design an abnormal behavior analysis algorithm to analyze the flight state of aircraft targets, which is used to detect abnormal flight state behaviors caused by local ground airflow and pilot operations;

[0012] Step 4, Monitoring and warning of abnormal behavior status

[0013] Display the target position information and whether there is abnormal behavior;

[0014] Step 5, Construction of appearance abnormality detection algorithm

[0015] Design a self-supervised anomaly detection method based on image element decoupling, which can better extract the features of normal and abnormal regions in the image by decoupling normal images and abnormal noises; specifically including the following steps:

[0016] 5-1. Crop the image to obtain the aircraft target image;

[0017] Crop the image: For the image frames containing aircraft targets in the aircraft target detection and tracking dataset in step 1, crop the vertical box area marked by the position information (x1, y1, x2, y2) in step 1 to obtain an aircraft target slice with length and width dimensions of x2 - x1 and y2 - y1, and uniformly adjust the size to 64×64×3 to obtain the aircraft target image;

[0018] According to the above image cropping method, cut the aircraft target images from the detection and tracking training set, detection and tracking validation set, and detection and tracking test set images in the aircraft target detection and tracking dataset constructed in step 1 respectively to obtain the appearance anomaly detection training set, appearance anomaly detection validation set, and appearance anomaly detection test set of the appearance anomaly detection algorithm;

[0019] 5-2. For the appearance anomaly detection training set image A, randomly select one of the following three methods to obtain the input image X of the appearance anomaly detection algorithm;

[0020] The first method is to randomly generate a Gaussian noise mask M with a size of 64×64, a mean μ = 0.3, and a standard deviation σ = 0.2, and then copy it three times by channel and add it to the 64×64×3 appearance anomaly detection training set image A;

[0021] The second method is to randomly generate k masks M with a size of n×n and pixel value of 0 in a template with a size of 64×64 and all pixel values of 1, and then multiply the mask and the appearance anomaly detection training set image A pixel by pixel. Here, k = 5 and n = 8 are set;

[0022] The third method is to randomly select an image from the appearance anomaly detection training set and cut out j image patches of size n×n. After multiplying by the weight coefficient α, it is used as the mask M and randomly superimposed on the appearance anomaly detection training set image A. Here, j = 5, n = 8, and α takes a random value within 0.2 - 0.8;

[0023] 5-3. Construction of the Image Decoupling Network

[0024] The image decoupling network detects the abnormal area by decomposing and reconstructing the image; the input of this network is the input image X processed in step 5-2, and after passing through Enc, DecY, and DecM, the output is the reconstructed image Y of 64×64×3 out and the reconstructed mask noise M out ;

[0025] The image decoupling network is divided into an encoder Enc, decoders DecY and DecM. Among them, Enc encodes the input image X into a vector F of length 4096, and the input of DecY is the vector F, and the output is the reconstructed image Y out , and its training objective is to make Y decoded by DecY out as close as possible to the appearance anomaly detection training set image A in step 5-2; DecM decodes the vector F into the mask noise M added to the appearance anomaly detection training set image in step 5-2 out , and its training objective is to make the reconstructed mask noise M decoded by DecM out as close as possible to the mask M added in step 5-2;

[0026] 5-4. Optimization of the Reconstructed Image

[0027] Construct a generative adversarial network Dis to improve the reconstruction effect of the model on the appearance anomaly detection training set image A; the input D of the designed generative adversarial network Dis is the reconstructed image Y output by DecY in step 5-3 out , or the appearance anomaly detection training set image A corresponding to the reconstructed image Y out , and the output is the authenticity judgment of whether the input D of the generative adversarial network Dis is the appearance anomaly detection training set image A or the reconstructed image Y out ; among them, the reconstructed image Y out corresponds to the label 0, and the appearance anomaly detection training set image A corresponds to the label 1;

[0028] Randomly select a batch of appearance anomaly detection training set images, input them into the image decoupling network described in 5-3, and generate corresponding reconstructed images. Then shuffle the order of this batch of appearance anomaly detection training set images and the corresponding reconstructed images to form the input of the generative adversarial network, which is used as the input images of the generative adversarial network Dis. After passing through conv(3,2,32), conv(3,2,64), conv(3,2,128), conv(3,2,256), it is unfolded and then connected to a fully connected layer with a length of 2048, and finally connected to two real / fake nodes;

[0029] Use binary classification loss to optimize the authenticity discrimination of the generative adversarial network:

[0030]

[0031] where N is the batch size, that is, the number of images used in one training iteration; x i is the i-th image in the input D of the generative adversarial network, which can be an image from the appearance anomaly detection training set or an image reconstructed by DecY decoding; y i is the label of the source of x i For an image from the appearance anomaly detection training set, its value is 1, and for an image reconstructed by DecY decoding, its value is 0; D(x i ) is the output of the discriminator for the i-th image, that is, the degree to which the discriminator believes that this image is an image from the appearance anomaly detection training set;

[0032] 5-5. Optimization of decoupled reconstruction loss function

[0033] Design an extended Focal loss function L EF as follows:

[0034]

[0035] where is the true value of each pixel point; for the DecY branch, this true value is the appearance anomaly detection training set image A, and for the DecM branch, this true value is the mask M generated according to step 5-2; is the predicted value. For the DecY branch, this predicted value is the reconstruction output Y of DecY in 5-3 out , and for the DecM branch, this predicted value is the output reconstructed mask noise M of DecM in step 5-3 out ;

[0036] 5-6. EDAD model training

[0037] ① Select e aircraft target images A from the appearance anomaly detection training set constructed in 5-1 in, according to the 5-2 method, randomly select a noise addition method to generate the mask M in , and then randomly added to the aircraft target image A in , and obtain a corresponding batch of e input images X in ;

[0038] ② Take a batch of e input images X in Input Enc, DecY and DecM in the image decoupling network constructed by the method in step 5-3, and obtain e reconstructed images from DecY output Get e reconstruction mask noises from DecM output According to formula (5), we can calculate With A in , With M in The loss is then fed back to the optimized network DecY, DecM and Enc;

[0039] ③The reconstructed image output from step ② The original aircraft target image X selected in step ① in After obfuscation, input the generative adversarial network Dis constructed by the method in step 5-4, calculate the authenticity identification loss according to formula (2) and update the network weights of Enc and DecY;

[0040] ④ Repeat ① to ③ 500 times to finally obtain the trained EDAD model;

[0041] 5-7. Threshold calculation;

[0042] After completing the model training in 5-6, retain the Enc and DecY parts and embed them into the designed abnormal warning module as the application model of appearance abnormality detection; input the appearance abnormality detection verification set image Xv constructed in 5-1 into the above application model, obtain the reconstructed image Yv through DecY, and then calculate the average reconstruction error μ of the image according to formula (6): Y ;

[0043]

[0044] Repeat this for all validation set images and then calculate μ Y The mean and standard deviation σ Y , set θ = μ Y +σ Y As a threshold for appearance anomaly detection;

[0045] 5-8 Model Testing

[0046] For the images in the constructed appearance anomaly detection test set in 5-1, input them into the application model for appearance anomaly detection described in 5-7 to obtain the reconstructed images output by DecY, and then calculate the reconstruction error μ according to formula (6). Y ; If the reconstruction error μ Y is greater than θ, then in the display module, mark the corresponding position of the aircraft in the video collected in step 1 with a red frame and provide a warning of appearance anomaly;

[0047] Step 6, Result display

[0048] Stitch the collected images according to their positions and display them on the screen. Display the aircraft target tracking results according to step 2 and mark the aircraft targets with green frames; Run the behavior anomaly detection method constructed in steps 3 and 4 and the appearance anomaly detection method constructed in step 5 in the background. Mark the detected targets with anomalies with red frames and provide warnings.

[0049] Furthermore, in the data collection and preprocessing in step 1, organize each image frame in the video data, and use a vertical frame to mark the position (x1, y1, x2, y2) of the aircraft target for all images. Among them, x1 and y1 are the coordinate points of the upper left corner of the marked frame, and x2 and y2 are the coordinate points of the lower right corner of the marked frame; Obtain the aircraft target detection and tracking data set;

[0050] Frame by frame, mark the flight states of the aircraft targets in the video data in the aircraft target detection and tracking data set; For each image frame in the video data, if the flight state of an aircraft in a certain image frame is normal, mark it as 0, if there is a behavior anomaly, mark it as 1, and if there is an appearance anomaly, mark it as 2;

[0051] Divide the image frames in the aircraft target detection and tracking data set into a detection and tracking training set, a detection and tracking validation set, and a detection and tracking test set according to the quantity ratio of 7:1:2; Among them, the flight states of all aircraft targets in the detection and tracking training set and the detection and tracking validation set are normal, and all the video data containing aircraft targets with behavior anomalies and appearance anomalies are located in the detection and tracking test set; Among them, the frame-by-frame images, position information, and flight state information of the manually marked aircraft targets are stored in the data storage module.

[0052] Furthermore, in the target tracking in step 2, use the detection and tracking training set of the aircraft target detection and tracking data set marked in step 1 to train JDE. Its input is the image frames of the detection and tracking training set, and the output is the position (x1, y1, x2, y2) of each aircraft target in the image frame.

[0053] Further, in the abnormal behavior warning of step 3, the tracking result of the aircraft target in the video in the detection and tracking training set obtained in step 2 is used. Assuming that the aircraft appears in n image frames, and the image numbers of each frame are i = 1,..., n, the position of the aircraft target in the i-th frame image obtained in step 2 is Then the center point of the tracking box in each image frame is T i =(cx i , cy i ), where The flight trajectory of the aircraft target is expressed as T = [T1, T2,…, T n ; For the above aircraft target motion trajectory, the sliding window method is used to divide this section of the motion trajectory, the window size is z, and a total of n - z + 1 track segments are obtained; For the t-th track segment, its starting frame is the t-th frame image, and the ending frame is the (t + z)-th frame image. Then the stability degree s of the aircraft target within the [t, t + z] frames can be calculated according to the following formula:

[0054] s = var([T t+1 - T t , T t+2 - T t+1 ,..., T t+z - T t+z-1 )

[0055] var is the variance.

[0056] Further, in the abnormal behavior status monitoring and warning of step 4, its input is the image frame in the detection and tracking test set in step 1. The stability degree s is calculated using the method of step 3, and then compared with the threshold θ. The output is whether there is abnormal behavior;

[0057] The calculation method of the threshold θ is as follows: For all normal flights in the detection and tracking verification set, the target tracking algorithm described in step 2 is used to detect and track the aircraft target in the detection and tracking verification set in real time, obtain its flight trajectory T, then cut the track segment according to the sliding window method, calculate the stability degree s for each track segment, and then calculate the average value and the standard deviation σ of s, and obtain Take θ as the threshold for distinguishing abnormal behavior;

[0058] The test method is as follows: Calculate the stability degree s of the aircraft target according to step 3. If s > θ is detected in the flight trajectory of the test aircraft target, then the detection box of the aircraft target and this section of the trajectory are highlighted in red and a warning of abnormal status is provided.

[0059] The beneficial effects of the present invention are as follows: The present invention can achieve real-time tracking of airport aircraft targets and analysis of abnormal states. By deploying multiple cameras, a wider field of view can be obtained, which can assist or replace the visual observation of tower personnel on the airport status, automatically detect abnormal states and provide warnings, thus realizing automatic auxiliary monitoring of aircraft states.

[0060] Through the automatic tracking and abnormal analysis of aircraft targets by the algorithm of the present invention, the intelligent management of airport aircraft is realized, the burden of tower personnel is reduced, and the command efficiency is improved.

[0061] Through steps 1-6, it is possible to detect abnormal flight states and appearance abnormalities respectively from the aspects of statistics and machine learning by analyzing track data and image data, thereby automatically discovering unstable flight states caused by airflow such as ground wind shear or abnormalities caused by potential mechanical failures, bird strikes, mechanical damages, engine fires, etc., and providing warning information to air traffic controllers to improve the efficiency of handling special situations.

[0062] Through steps 3 and 4, a stability index is proposed to monitor the stability of the aircraft flight track. By calculating the variance of the position change amount of adjacent frames of the track coordinates, the stability of the track is measured, so as to monitor the problem of excessive flight track fluctuations caused by wind shear, etc. during takeoff and landing.

[0063] Through step 5, a method for detecting appearance abnormalities based on image decoupling is constructed. There are mainly two approaches to existing image abnormality detection schemes. One is the unsupervised method, where the input is a normal image or a normal image after mask overlay, and the output is a reconstructed normal image. The model is trained to fit the normal image. In the test stage, the model can accurately reconstruct normal images, but the reconstruction accuracy of abnormal parts in abnormal images is low, thus realizing the distinction between normal and abnormal images. This method does not require labeled data and can adapt to different abnormal targets / regions without knowing the abnormal targets / regions, but has a high false alarm rate. The other is the supervised method, where the input is an abnormal image and the output is the position or mask of the abnormal target / region. The model is trained to detect and segment the abnormal targets / regions in the abnormal image through image detection or segmentation methods. This method has strong accuracy and interpretability, but relies on labeled data and has limited generalization ability. To address the above problems, this paper proposes a method that combines the two ideas. By designing a multi-branch structure, a method of decoding normal images and abnormal noises in an image decoupling manner is proposed to enable the model to better understand and distinguish normal and abnormal information. The Focal loss function is improved to specifically optimize the pixel points with large reconstruction errors, thereby enhancing the fitting degree of the model to normal images. Finally, a model with a high fitting degree to normal samples and strong robustness is obtained, thus increasing the reconstruction error difference between abnormal samples and normal samples and improving the analysis effect of aircraft appearance abnormalities.

[0064] Through step 5-5, an optimized Focal loss function is proposed, and the Focal loss in image segmentation is extended to the image reconstruction task, so as to assign a greater weighted loss to the pixel points with high reconstruction error, prompting the model to focus on the pixel points with difficult reconstruction and perform targeted optimization, thereby improving the convergence speed of the model and reducing the reconstruction error of the model.

[0065] The present invention aims to perform real-time monitoring, tracking, and abnormal state analysis on the takeoff and landing processes of aircraft through intelligent means to ensure the safe flight of aircraft. Aiming at the problem of small-area abnormal detection on the appearance of aircraft, an aircraft abnormal detection method based on image decoupling and generative adversarial is designed to improve the detection effect of the model on aircraft appearance abnormalities. Description of the Drawings

[0066] Figure 1 is the overall framework diagram of the present invention;

[0067] Figure 2 is the schematic diagram of the method for adding a noise mask in step 5-2 of the present invention (taking k = 3, j = 3 as an example);

[0068] Figure 3 is the framework diagram of the image decoupling function part in step 5-3 of the present invention;

[0069] Figure 4 is the network framework diagram of the generative adversarial network Dis of the reconstruction image optimization module in step 5-4 of the present invention. Detailed Embodiments

[0070] The following will describe in detail the specific embodiments of the present invention with reference to the accompanying drawings:

[0071] As Figure 1 shown, an aircraft intelligent detection, tracking, and abnormal analysis method specifically includes the following steps:

[0072] Step 1, data collection and preprocessing

[0073] Multiple gun-type cameras are used to collect airport video data in the takeoff and landing areas, and the video data is cleaned and normalized. Images are collected once every 0.05 s, and the images in the obtained video data are resized to 224×224.

[0074] The image frames in the video data are sorted out, and the positions (x1, y1, x2, y2) of the aircraft targets are marked on all images using vertical frames, where x1 and y1 are the coordinate points of the upper left corner of the marked frame, and x2 and y2 are the coordinate points of the lower right corner of the marked frame; an aircraft target detection and tracking data set is obtained.

[0075] Frame by frame, label the flight states of aircraft targets in the video data of the aircraft target detection and tracking dataset. The flight states include three categories: normal, behavior anomaly, and appearance anomaly. Behavior anomaly means that the movement trajectory of the aircraft is abnormal within a certain period (usually 1 - 2 seconds); appearance anomaly means that the external shape of the aircraft changes due to engine fire, bird strike, lightning strike, or cracking of the aircraft. For each image frame in the video data, if the flight state of an aircraft in a certain image frame is normal, it is marked as 0; if there is a behavior anomaly, it is marked as 1; if there is an appearance anomaly, it is marked as 2.

[0076] For the abnormal recognition task of aircraft states, since the number of abnormal occurrences is very small, the present invention uses the method of anomaly detection for recognition. Specifically, the image frames in the aircraft target detection and tracking dataset are divided into a detection and tracking training set, a detection and tracking validation set, and a detection and tracking test set according to the quantity ratio of 7:1:2. Among them, the flight states of all aircraft targets in the detection and tracking training set and the detection and tracking validation set are normal, and all the video data containing aircraft targets with behavior anomalies and appearance anomalies are located in the detection and tracking test set. Among them, the frame-by-frame images, position information, and flight state information of the aircraft targets manually labeled are stored in the data storage module.

[0077] Step 2: Target tracking

[0078] The input of this step is the detection and tracking training set in the aircraft target detection and tracking dataset constructed in Step 1, and the output is the target position information. This method does not involve innovation in target tracking methods. Considering that the Joint Detection and Embedding (JDE, literature: Towards Real-Time Multi-Object Tracking) algorithm has reached the real-time level in the multi-object tracking task, and the tracking task in the scenario involved in the present invention is relatively simple, JDE is used as the tracking model for the target aircraft. Specifically, use the detection and tracking training set of the aircraft target detection and tracking dataset labeled in Step 1 to train JDE. Its input is the image frames of the detection and tracking training set, and the output is the position (x1, y1, x2, y2) of each aircraft target in the image frame, that is, the aircraft target is labeled in the form of a vertical box in each frame of the image, so as to realize the tracking of the aircraft target in the video and obtain the movement trajectory of the aircraft target.

[0079] Step 3: Behavior anomaly warning

[0080] Design a behavior anomaly analysis algorithm to analyze the flight state of the aircraft target, which is used to detect flight state behavior anomalies caused by local ground airflow, pilot operation, etc. Its input is the position (x1, y1, x2, y2) of each aircraft target in the image frame obtained in Step 2, and the output is the flight stability degree s.

[0081] Specifically, the tracking results of the aircraft target in the video can be obtained through Step 2. Assume that the aircraft appears in n image frames, and the image numbers of each frame are i = 1,..., n. The position of the aircraft target in the i-th frame image obtained through Step 2 is Then the center point of the tracking box in each image frame is T i =(cx i , cy i ), where The flight trajectory of the aircraft target is expressed as T = [T1, T2,…, T n .

[0082] For the above-mentioned aircraft target motion trajectory, use the sliding window method to divide this section of the motion trajectory. The window size is z, and a total of n - z + 1 track segments are obtained. For the t-th track segment, its starting frame is the t-th frame image, and the ending frame is the (t + z)-th frame image. Then the stability degree s of the aircraft target within the [t, t + z] frames can be calculated according to the following formula:

[0083] s = var([T t+1 - T t , T t+2 - T t+1 ,..., T t+z - T t+z-1 ) (1)

[0084] var is the variance. Here, z is taken as 20 frames. Since the images are collected once every 0.05 s, the duration of each track segment is 0.05 * z = 1 second. That is, calculate the stability degree within 1 second after the t-th frame.

[0085] Step 4. Monitoring and warning of abnormal behavior

[0086] Display the target position information and whether there is abnormal behavior. Its input is the image frames in the detection and tracking test set in Step 1. Use the method in Step 3 to calculate the stability degree s, and then compare it with the threshold θ. The output is whether there is abnormal behavior.

[0087] The calculation method of the threshold θ is as follows: For all flights (all normal flights) in the detection and tracking verification set, use the target tracking algorithm described in Step 2 to detect and track the aircraft target in the detection and tracking verification set in real time, obtain its flight trajectory T, then cut the track segments according to the sliding window method, calculate the stability degree s for each track segment, and then calculate the average value and standard deviation σ of s, and obtain Take θ as the threshold to distinguish abnormal behavior.

[0088] The test method is as follows: Calculate the stability degree s of the aircraft target according to Step 3. If s > θ is detected in the flight trajectory of the test aircraft target, the detection frame of the aircraft target and the section of the trajectory will be highlighted in red and a warning of abnormal status will be provided.

[0089] Step 5: Construction of Appearance Abnormality Detection Algorithm

[0090] Based on the ideas of unsupervised and supervised anomaly detection, the present invention designs a self-supervised anomaly detection method based on image element decoupling (ElementDecoupling Anomaly Detection, EDAD), which can better extract the features of normal and abnormal regions in the image by decoupling the normal image and abnormal noise parts. It includes the following steps:

[0091] 5-1. Crop the image to obtain the aircraft target image.

[0092] Crop the image: For the image frames containing aircraft targets in the aircraft target detection and tracking dataset in Step 1, crop the vertical frame region marked by the position information (x1, y1, x2, y2) in Step 1 to obtain an aircraft target slice with length and width dimensions of x2 - x1 and y2 - y1, and uniformly adjust the size to 64×64×3 to obtain the aircraft target image.

[0093] According to the above image cropping method, crop the aircraft target images from the detection and tracking training set, detection and tracking validation set, and detection and tracking test set in the aircraft target detection and tracking dataset constructed in Step 1 respectively to obtain the appearance anomaly detection training set, appearance anomaly detection validation set, and appearance anomaly detection test set of the appearance anomaly detection algorithm.

[0094] 5-2. Preprocess the appearance anomaly detection training set images obtained in 5-1 using self-supervised learning to enhance the algorithm's fitting ability for normal aircraft features.

[0095] Specifically, this algorithm uses three image preprocessing methods for self-supervised learning. For a randomly selected appearance anomaly detection training set image A, a mask M is randomly obtained in the following way and then superimposed on A to obtain the input image X of the appearance anomaly detection algorithm. The schematic diagrams of the second and third methods are as Figure 2 shown.

[0096] The first one is to randomly generate a Gaussian noise mask M with a size of 64×64, a mean μ = 0.3, and a standard deviation σ = 0.2, and add it to the 64×64×3 appearance anomaly detection training set image A after replicating it three times by channel.

[0097] The second method is to randomly generate k masks M with a size of n×n and pixel values of 0 in a template with all 64×64 pixel values being 1, and then multiply the masks pixel by pixel with the appearance anomaly detection training set image A. Here, k = 5 and n = 8 are set.

[0098] The third method is to randomly select an appearance anomaly detection training set image and cut out j image patches with a size of n×n, multiply them by the weight coefficient α, and then randomly superimpose them as the mask M onto the appearance anomaly detection training set image A. Here, j = 5, n = 8, and α takes a random value within 0.2 - 0.8.

[0099] By randomly adopting one of the above three methods for the appearance anomaly detection training set image, the input image X for the subsequent appearance anomaly detection algorithm after processing is obtained.

[0100] 5-3. Construction of Image Decoupling Network

[0101] The image decoupling network detects the abnormal area by decomposing and reconstructing the image. The input of this network is the input image X processed in step 5-2, and after passing through Enc, DecY, and DecM, the output is the reconstructed image Y of 64×64×3 out and the reconstructed mask noise M out .

[0102] Its network structure is as Figure 3 shown. The image decoupling network is divided into an encoder Enc, decoders DecY and DecM. Among them, Enc encodes the input image X into a vector F with a length of 4096, and the input of DecY is the vector F, and the output is the reconstructed image Y out , and its training objective is to make Y decoded by DecY out as close as possible to the appearance anomaly detection training set image A in step 5-2; DecM decodes the vector F into the mask noise M added to the appearance anomaly detection training set image in step 5-2 out , and its training objective is to make the reconstructed mask noise M decoded by DecM out as close as possible to the mask M added in step 5-2. Among them, the dashed arrow between DecY and DecM indicates weight sharing, and in actual code implementation, the same network layer can be used.

[0103] 5-4. Optimization of Reconstructed Image

[0104] A generative adversarial network Dis is constructed to improve the reconstruction effect of the model on the appearance anomaly detection training set image A. The input D of the designed generative adversarial network Dis is the reconstructed image Y output by DecY in step 5-3 out , or the reconstructed image Y outThe corresponding appearance anomaly detection training set image A, and the output is to determine whether the input D to the generative adversarial network Dis is the appearance anomaly detection training set image A or the reconstructed image Y out for authenticity. Among them, the reconstructed image Y out corresponds to the label 0, and the appearance anomaly detection training set image A corresponds to the label 1. Its structure is as Figure 4 shown. Randomly select a batch of appearance anomaly detection training set images, input them into the image decoupling network described in 5-3, and generate corresponding reconstructed images. Then shuffle the order of this batch of appearance anomaly detection training set images and the corresponding reconstructed images to form the input of the generative adversarial network, which is used as the input image of the generative adversarial network Dis. After passing through conv(3,2,32), conv(3,2,64), conv(3,2,128), conv(3,2,256), it is unfolded and then connected to a fully connected layer with a length of 2048, and finally connected to two true / false nodes.

[0105] Use the binary classification loss to optimize the authenticity discrimination of the generative adversarial network:

[0106]

[0107] Among them, N is the batch size, that is, the number of images used in one training iteration. x i is the i-th image in the input D of the generative adversarial network, which can be an image from the appearance anomaly detection training set or an image reconstructed by DecY decoding. y i is the label of the source of x i , and its value is 1 for images from the appearance anomaly detection training set and 0 for images reconstructed by DecY decoding. D(x i ) is the output of the discriminator for the i-th image, that is, the degree to which the discriminator believes that this image is from the appearance anomaly detection training set.

[0108] 5-5. Optimization of the decoupling reconstruction loss function

[0109] For the image decoupling network constructed in step 5-3, design a loss function to optimize the network parameters. A common loss function is the MSE loss function, which is calculated as follows:

[0110]

[0111] Among them, A i,j,k , Y i,j,k represent the coordinates of each pixel point in the appearance anomaly detection training set image A and the reconstructed image Y out . i, j = 1,..., 64, k = 1, 2, 3.

[0112] The MSE loss is difficult to optimize difficult pixel points such as the boundary region specifically. Therefore, the literature (Focal Loss for Dense Object Detection, ICCV 2017) proposed using Focal loss. This loss function is used in the image segmentation task, and its true label is a 0-1 mask. The formula for its loss is as follows:

[0113]

[0114] Among them, is the predicted value output by the DecM branch in the model for the appearance anomaly detection training set image A, that is, the reconstructed mask noise M out in the pixel points, i, j, k are the coordinate indices of the pixel points in M out in, represents the probability that the coordinates of each pixel point in the reconstructed mask noise M out match the true label mask M. If the true label of a certain pixel point is 1, that is, the true label mask M i,j,k = 1, then If the true label is 0, then α is a hyperparameter for balancing the class imbalance. For pixel points with a true label of 1, it is set to α = 0.25, otherwise it is set to α = 0.75. γ is a hyperparameter that needs to be manually set to adjust the degree of attention of the model to pixel points with large reconstruction errors, and is set to 2.

[0115] Since Focal loss is designed for the image segmentation task, in the image segmentation task, the output is a 0-1 mask image for marking the position of the segmented target. And the labels predicted by this method are normal images and noises, the reconstructed image Y out and the reconstructed mask noise M out , Focal loss cannot be applied to this method to measure the loss of network prediction. Therefore, Focal loss is improved so that in addition to predicting the 0-1 mask scenario, it can adapt to more extensive problems such as image reconstruction. Specifically, the designed extended Focal loss function L EF is as follows:

[0116]

[0117] Among them is the true value of each pixel point; for the DecY branch, this true value is the appearance anomaly detection training set image A, and for the DecM branch, this true value is the mask M generated according to step 5-2. is the predicted value. For the DecY branch, this predicted value is the reconstructed output Y of DecY in 5-3 outFor the DecM branch, the predicted value is the output reconstruction mask noise M of DecM in step 5-3 out .

[0118] That is, use the predicted value of each pixel and the true value to measure the difficulty of predicting this pixel. If the difference is larger, it indicates that the prediction accuracy is lower, and the learning of this pixel is more difficult. At this time, this pixel can obtain higher attention through the above formula.

[0119] 5-6. EDAD model training

[0120] Set the learning rate to 0.0002, the number of training times to 500, and the batch size to e = 64. Train the EDAD model in the following order.

[0121] ① Take e aircraft target images A from the appearance anomaly detection training set constructed in 5-1 in , randomly select a noise addition method according to the method in 5-2 to generate a mask M in , and then randomly add it to the aircraft target image A in to obtain a corresponding batch of e input images X in ;

[0122] ② Input a batch of e input images X in into Enc, DecY, and DecM in the image decoupling network constructed by the method in step 5-3. Obtain e reconstructed images from the output of DecY Obtain e reconstructed mask noises from the output of DecM Calculate respectively according to formula (5) and A in , and M in losses, and then backpropagate to optimize the networks DecY, DecM, and Enc.

[0123] ③ Confuse the reconstructed images output in step ② with the original aircraft target images X selected in step ① in and input them into the discriminator Dis of the generative adversarial network constructed by the method in step 5-4. Calculate the real and fake discrimination loss according to formula (2) and update the network weights of Enc and DecY.

[0124] ④ Repeat ① to ③ 500 times to finally obtain the trained EDAD model.

[0125] 5-7. Threshold calculation.

[0126] This paper uses the reconstruction error of DecY for the appearance anomaly detection validation set image (denoted as Xv) to calculate the threshold of the degree of abnormality. Since DecY is trained using the training set images in steps 5-2 to 5-6, its function is to reconstruct the input image without adding noise. Therefore, if a test image contains an abnormal area, its reconstruction loss will be larger than that of a normal image without abnormalities. Therefore, the reconstruction loss can be calculated as a measure of the degree of abnormality.

[0127] Specifically, after completing the model training in 5-6, retain the Enc and DecY parts and embed them into the designed abnormal warning module as the application model of appearance abnormality detection. Input the appearance abnormality detection verification set image Xv constructed in 5-1 into the above application model, obtain the reconstructed image Yv through DecY, and then calculate the average reconstruction error μ of the image according to formula (6): Y .

[0128]

[0129] Repeat this for all validation set images and then calculate μ Y The mean and standard deviation σ Y , set θ = μ Y +σ Y As the threshold for appearance anomaly detection.

[0130] 5-8 Model Testing

[0131] For the appearance anomaly detection test set image constructed in 5-1, input it into the application model of appearance anomaly detection described in 5-7 to obtain the reconstructed image output by DecY, and then calculate the reconstruction error μ according to formula (6): Y If the reconstruction error μ Y If the value is greater than θ, the aircraft is marked with a red frame at the corresponding position in the video collected in step 1 in the display module, and a warning of abnormal appearance is provided.

[0132] Step 6: Results display

[0133] The images collected by each gun-type camera are stitched according to their positions and displayed on the screen. The aircraft target tracking results are displayed according to step 2, and the aircraft target is marked with a green frame. The behavior anomaly detection method constructed in steps 3 and 4 and the appearance anomaly detection method constructed in step 5 are run in the background. The targets detected to be abnormal are marked with a red frame and a warning is provided.

[0134] It should be understood that the parts not elaborated in this specification all belong to the prior art. The above embodiments merely describe the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An aircraft intelligent detection, tracking and anomaly analysis method, characterized in that, Specifically, it includes the following steps: Step 1: Data collection and preprocessing Use multiple gun-shaped cameras to collect airport video data in the takeoff and landing areas, and clean and normalize the video data; sort out each image frame in the video data and label the flight states of aircraft targets; Step 2: Target tracking Detect and track aircraft targets; Step 3: Abnormal behavior warning Design an abnormal behavior analysis algorithm to analyze the flight states of aircraft targets, which is used to detect abnormal flight state behaviors caused by local ground airflow and pilot operations; Step 4: Monitoring and warning of abnormal behavior states Display the target position information and whether there is abnormal behavior; Step 5: Construction of appearance anomaly detection algorithm Design a self-supervised anomaly detection method based on image element decoupling, which can better extract the features of normal and abnormal regions in the image by decoupling normal images and abnormal noises; Specifically, it includes the following steps: 5-1. Crop the image to obtain an aircraft target image; Crop the image: For the image frames containing aircraft targets in the aircraft target detection and tracking dataset in Step 1, crop the vertical box area marked by the position information (x1, y1, x2, y2) in Step 1 to obtain an aircraft target slice with length and width dimensions of x2 - x1 and y2 - y1, and uniformly adjust the size to 64×64×3 to obtain an aircraft target image; According to the above image cropping method, cut the aircraft target images from the image frames of the detection and tracking training set, detection and tracking validation set, and detection and tracking test set in the aircraft target detection and tracking dataset constructed in Step 1 to obtain the appearance anomaly detection training set, appearance anomaly detection validation set, and appearance anomaly detection test set of the appearance anomaly detection algorithm; 5-2. For the appearance anomaly detection training set image A, randomly use one of the following three methods to obtain the input image X of the appearance anomaly detection algorithm; The first method is to randomly generate a Gaussian noise mask M with a size of 64×64, a mean μ = 0.3, and a standard deviation σ = 0.2, and add it to the 64×64×3 appearance anomaly detection training set image A after replicating it three times by channel; The second method is to randomly generate k masks M with a size of n×n and pixel value of 0 in a template with all pixel values of 1 at 64×64 pixels, and then multiply the mask and the appearance anomaly detection training set image A pixel by pixel. Here, k = 5 and n = 8 are set; The third method is to randomly select an appearance anomaly detection training set image and cut j image blocks with a size of n×n, multiply them by the weight coefficient α, and then randomly superimpose them on the appearance anomaly detection training set image A as the mask M. Here, j = 5, n = 8, and α takes a random value within 0.2 - 0.8; 5-3. Construction of image decoupling network The image decoupling network realizes the detection of abnormal regions by decomposing and reconstructing images; the input of this network is the input image X processed in step 5-2, and after passing through Enc, DecY, and DecM, the output is the reconstructed image Y of 64×64×3 out and the reconstructed mask noise M out ; The image decoupling network is divided into an encoder Enc, a decoder DecY, and a decoder DecM. Among them, Enc encodes the input image X into a vector F with a length of 4096, and the input of DecY is the vector F, and the output is the reconstructed image Y out , and its training objective is to make Y decoded by DecY out as close as possible to the appearance anomaly detection training set image A in step 5-2; DecM decodes the vector F into the masked noise M added to the appearance anomaly detection training set image in step 5-2 out , and its training objective is to make the reconstructed masked noise M decoded by DecM out as close as possible to the mask M added in step 5-2; 5-4. Reconstruction image optimization Construct the generative adversarial network Dis to improve the reconstruction effect of the model on the images A in the appearance anomaly detection training set; the input D of the designed generative adversarial network Dis is the reconstructed image Y output by DecY in step 5-3 out , or the image A in the appearance anomaly detection training set corresponding to the reconstructed image Y out , and the output is the authenticity judgment of whether the input D of the generative adversarial network Dis is the image A in the appearance anomaly detection training set or the reconstructed image Y out ; among them, the reconstructed image Y out corresponds to the label 0, and the image A in the appearance anomaly detection training set corresponds to the label 1; Randomly select a batch of appearance anomaly detection training set images, input them into the image decoupling network described in 5-3, and generate corresponding reconstructed images. Then shuffle the order of this batch of appearance anomaly detection training set images and the corresponding reconstructed images to form the input of the generative adversarial network, which is used as the input images of the generative adversarial network Dis. After passing through conv(3,2,32), conv(3,2,64), conv(3,2,128), conv(3,2,256), it is unfolded and then connected to a fully connected layer with a length of 2048, and finally connected to two real / fake nodes; Optimize the real / fake discrimination of the generative adversarial network using binary classification loss: Among them, N is the batch size, that is, the number of images used in one training iteration; x i is the i-th image in the input D of the generative adversarial network, which can be an image from the appearance anomaly detection training set or an image reconstructed by DecY decoding; y i is the label of the source of x i , with a value of 1 for images from the appearance anomaly detection training set and a value of 0 for images reconstructed by DecY decoding; D(x i ) is the output of the discriminator for the i-th image, that is, the degree to which the discriminator believes that the image is from the appearance anomaly detection training set; 5-5. Optimization of the decoupling reconstruction loss function Design extended Focal loss function L EF As follows: wherein is the true value of each pixel point; for the DecY branch, this true value is the appearance anomaly detection training set image A, and for the DecM branch, this true value is the mask M generated according to step 5-2; is the predicted value. For the DecY branch, this predicted value is the reconstructed output Y of DecY in 5-3 out and for the DecM branch, this predicted value is the output reconstructed mask noise M of DecM in step 5-3 out ; 5-6. Training of the EDAD model ① Take e aircraft target images A from the constructed appearance anomaly detection training set in 5-1 in , randomly select a noise addition method according to the method in 5-2 to generate a mask M in , and then randomly add it to the aircraft target image A in , obtaining a corresponding batch of e input images X in ; ② Input a batch of e input images X in into Enc, DecY, and DecM in the image decoupling network constructed by the method in step 5-3, and obtain e reconstructed images from the output of DecY Obtain e reconstructed mask noises from the output of DecM Calculate respectively according to formula (5) and A in , and M in losses, and then backpropagate to optimize the networks DecY, DecM, and Enc; ③ Input the reconstructed image output in step ② and the original aircraft target image X selected in step ① in into the generative adversarial network Dis constructed by the method in step 5-4 after confusion, calculate the true and false discrimination loss according to formula (2), and update the network weights of Enc and DecY; ④ Repeat ①~③ 500 times to finally obtain the trained EDAD model; 5-7. Threshold calculation; After completing the model training at 5-6, retain the Enc and DecY parts and embed them into the designed anomaly warning module as the application model for appearance anomaly detection. Input the appearance anomaly detection validation set image Xv constructed at 5-1 into the above application model, obtain the reconstructed image Yv through DecY, and then calculate the average reconstruction error μ of this image according to formula (6). Y ; Repeat this operation for all validation set images, and then calculate the mean value of μ Y and the standard deviation σ Y , and set θ = μ Y + σ Y as the threshold for appearance anomaly detection; 5-8. Model testing For the images in the constructed appearance anomaly detection test set in 5-1, input them into the application model for appearance anomaly detection described in 5-7 to obtain the reconstructed images output by DecY, and then calculate the reconstruction error μ according to formula (6). Y ; If the reconstruction error μ Y is greater than θ, then in the display module, mark the corresponding position of the aircraft in the video collected in step 1 with a red box and provide a warning of appearance anomaly. Step 6. Result display Stitch the collected images according to their positions and display them on the screen. According to Step 2, display the aircraft target tracking results and mark the aircraft target with a green box; Run the behavior anomaly detection method constructed in Step 3 and the appearance anomaly detection method constructed in Step 5 in the background. Mark the detected abnormal targets with a red box and provide a warning.

2. The aircraft intelligent detection, tracking and anomaly analysis method according to claim 1, wherein In the data collection and preprocessing of Step 1, organize each image frame in the video data, and use a vertical box to label the position (x1, y1, x2, y2) of the aircraft target for all images. Among them, x1 and y1 are the coordinates of the upper left corner of the annotation box, and x2 and y2 are the coordinates of the lower right corner of the annotation box; Obtain the aircraft target detection and tracking data set; Frame by frame, label the flight state of the aircraft target in the aircraft target detection and tracking data set for the video data; For each image frame in the video data, if the flight state of a certain aircraft in a certain image frame is normal, mark it as 0, if there is a behavior anomaly, mark it as 1, and if there is an appearance anomaly, mark it as 2; Divide the image frames in the aircraft target detection and tracking data set into a detection and tracking training set, a detection and tracking validation set, and a detection and tracking test set according to the quantity ratio of 7:1:2; Among them, the flight states of all aircraft targets in the detection and tracking training set and the detection and tracking validation set are normal, and all the video data containing aircraft targets with behavior anomalies and appearance anomalies are located in the detection and tracking test set; Among them, the frame-by-frame images, position information, and flight state information of the aircraft targets manually annotated are stored in the data storage module.

3. An aircraft intelligent detection, tracking and anomaly analysis method according to claim 1, characterized in that, In the target tracking of Step 2, use the detection and tracking training set of the aircraft target detection and tracking data set annotated in Step 1 to train JDE. Its input is the image frames of the detection and tracking training set, and the output is the position (x1, y1, x2, y2) of each aircraft target in the image frame.

4. A method for intelligent detection, tracking and anomaly analysis of an aircraft, according to claim 1, wherein In the abnormal behavior warning of step 3, the tracking result of the aircraft target in the video is obtained through step 2. Assuming that the aircraft appears in n image frames, and the image numbers of each frame are i = 1,..., n, the position of the aircraft target in the i-th frame image obtained through step 2 is Then the center point of the tracking box in each image frame is T i =(cx i , cy i ), where The flight trajectory of the aircraft target is expressed as T = [T1, T2,…, T n ; For the above aircraft target motion trajectory, use the sliding window method to divide this section of the motion trajectory, the window size is z, and a total of n - z + 1 track segments are obtained; For the t-th track segment, its starting frame is the t-th frame image, and the ending frame is the (t + z)-th frame image. Then the stability degree s of the aircraft target within the [t, t + z] frames can be calculated according to the following formula: s = var([T t+1 -T t , T t+2 -T t+1 ,..., T t+z -T t+z-1 ) var is the variance.

5. A method for intelligent detection, tracking and abnormal analysis of an aircraft, as claimed in claim 1, wherein In the behavior anomaly status monitoring and warning of Step 4, its input is the image frames in the detection and tracking test set in Step 1. Use the method in Step 3 to calculate the stability degree s, and then compare it with the threshold θ. The output is whether there is a behavior anomaly; The calculation method of the threshold θ is as follows: For all normal flights in the detection and tracking validation set, use the target tracking algorithm described in step 2 to detect and track the aircraft targets in the detection and tracking validation set in real time, obtain their flight trajectories T, then segment the flight tracks in the way of a sliding window, calculate the stability degree s for each track segment, and then calculate the average value of s and the standard deviation σ to obtain Use θ as the threshold to distinguish abnormal behaviors; The testing method is as follows: Calculate the stability degree s of this aircraft target according to Step 3; If s > θ is detected in the flight trajectory of the test aircraft target, the detection box of the aircraft target and this section of the trajectory will be highlighted in red and a warning of abnormal status will be provided.

Images