Air traffic supervision anomaly detection method, device and system based on optical remote sensing image and medium

Abstract

The invention provides an air traffic supervision anomaly detection method, device and system based on an optical remote sensing image, and a medium, and relates to the technical field of remote sensing image application and target detection. The method comprises the steps of obtaining an optical remote sensing image of a target area and imaging time and an imaging range of the optical remote sensing image; obtaining target ADS-B data from pre-stored ADS-B data based on the imaging time and the imaging range; detecting a moving target in the optical remote sensing image to obtain position information of the moving target; and matching the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image applications and target detection, and particularly to an abnormal detection method, device, system and medium for air traffic supervision based on optical remote sensing images. Background Art

[0002] With the rapid development of the aviation industry, aircraft are increasingly widely used in daily life. Automatic Dependent Surveillance-Broadcast (ADS-B), as an advanced navigation technology, has attracted increasing attention. Based on the global positioning system, ADS-B uses air-to-ground and air-to-air data links to achieve a new type of air traffic supervision and information transmission method. This system can automatically (for example, once per second) obtain parameters from relevant on-board equipment without manual operation or interrogation, and broadcast information such as the position, altitude, speed, heading, and identification number of the aircraft to other aircraft or ground stations. This method provides an effective means for air traffic controllers to monitor the state of aircraft, and becomes an effective supplement and alternative to the traditional air traffic control radar surveillance method.

[0003] ADS-B technology can not only enhance the efficiency of air traffic management, but also improve flight safety. For example, in complex flight environments or low visibility conditions, the real-time data provided by ADS-B is particularly critical for flight safety.

[0004] However, since ADS-B data is broadcast in plain text and lacks basic security mechanisms, the supervision of aircraft navigation through ADS-B is easily interfered with. Therefore, it is particularly important to study effective ADS-B data anomaly detection schemes. Summary of the Invention

[0005] In view of this, the present invention provides an abnormal detection method, device, system and medium for air traffic supervision based on optical remote sensing images, which are used to at least partially solve the above technical problems.

[0006] The first aspect of the embodiments of the present invention provides an abnormal detection method for air traffic supervision based on optical remote sensing images, including: obtaining an optical remote sensing image of a target area, as well as the imaging time and imaging range of the optical remote sensing image; obtaining target ADS-B data from the pre-stored ADS-B data based on the imaging time and imaging range; detecting moving targets in the optical remote sensing image to obtain the position information of the moving targets; and matching the moving targets with the target ADS-B data based on the position information of the moving targets to obtain an anomaly detection result.

[0007] According to an embodiment of the present invention, obtaining target ADS-B data from pre-stored ADS-B data based on imaging time and imaging range includes: obtaining initial ADS-B data associated with a moving target in an optical remote sensing image from the pre-stored ADS-B data based on imaging time and imaging range; and determining, among the initial ADS-B data, the initial ADS-B data for which the time difference between the transmission time of the ADS-B data and the recording time of the last recorded ADS-B data of the moving target satisfies a preset condition as the target ADS-B data.

[0008] According to an embodiment of the present invention, in the case where the optical remote sensing image is a multi-spectral remote sensing image, detecting a moving target in the optical remote sensing image to obtain the position information of the moving target includes: performing image denoising, normalization, and data augmentation on the multi-spectral remote sensing image to obtain a first image; inputting the first image into a pre-trained deep convolutional neural network model to extract spatial features of different scales in the first image and output spatial feature maps of different scales; and inputting the spatial feature maps of different scales into a region proposal network to identify candidate regions containing the moving target and adjusting the candidate regions to obtain the position information of the moving target.

[0009] According to an embodiment of the present invention, inputting the spatial feature maps of different scales into a region proposal network to identify candidate regions containing the moving target includes: for each scale of spatial feature map, sliding a sliding window over the spatial feature map to obtain a plurality of detection frames; performing foreground and background classification and bounding box regression on each detection frame, and determining at least one target detection frame from the plurality of detection frames as the initial candidate region corresponding to the spatial feature map of this scale; and fusing the initial candidate regions of different scales to obtain a candidate region containing the moving target, where non-maximum suppression is used during the fusion process to remove multiple detection frames corresponding to the same moving target.

[0010] According to an embodiment of the present invention, in the case where the optical remote sensing image is a video satellite image, detecting a moving target in the optical remote sensing image to obtain the position information of the moving target includes: performing denoising and enhancement processing on the video satellite image to obtain a second image; inputting the second image into a siamese network model to extract the spatial and temporal features of the second image, obtaining a spatio-temporal feature map, and based on the spatio-temporal feature map, comparing the feature representations of the moving target in consecutive image frames to obtain the position information of the moving target.

[0011] According to an embodiment of the present invention, the moving target is matched with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result, including: when there is no target in the target ADS-B data that matches the moving target, it is determined that the moving target does not use the ADS-B system in a standardized manner; or: when the target ADS-B data matches the moving target, the moving speed and moving azimuth angle of the moving target are determined based on the optical remote sensing image; the moving speed and moving azimuth angle are matched with the target ADS-B data, and when the deviation is greater than the threshold, it is determined that there is an anomaly in the ADS-B data of the moving target.

[0012] According to an embodiment of the present invention, determining the moving speed and moving azimuth angle of the moving target based on the optical remote sensing image includes: determining the moving speed and moving azimuth angle of the moving target based on the pixel positions of the moving target in the optical remote sensing images of different bands and the spatial resolutions corresponding to different bands.

[0013] A second aspect of the embodiments of the present invention provides an anomaly detection device for air traffic supervision based on optical remote sensing images, including: a first acquisition module for acquiring the optical remote sensing image of the target area, as well as the imaging time and imaging range of the optical remote sensing image; a second acquisition module for acquiring target ADS-B data from the pre-stored ADS-B data based on the imaging time and imaging range; a target detection module for detecting the moving target in the optical remote sensing image to obtain the position information of the moving target; a matching module for matching the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result.

[0014] A third aspect of the embodiments of the present invention provides an airspace supervision system, including: a remote sensing satellite constellation configured to acquire the optical remote sensing image of the target area, as well as the imaging time and imaging range of the optical remote sensing image; an aviation safety warning platform configured to acquire target ADS-B data from the pre-stored ADS-B data based on the imaging time and imaging range; detect the moving target in the optical remote sensing image to obtain the position information of the moving target; match the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result, and visually display the anomaly detection result.

[0015] A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, which is arranged on the on-board edge and stores executable instructions. When the instructions are executed by a processor, the processor is caused to implement the following method: acquire the optical remote sensing images of the moving target in the target area in different bands; determine the moving speed and moving azimuth angle of the moving target based on the pixel positions of the moving target in the optical remote sensing images of different bands and the spatial resolutions corresponding to different bands.

[0016] A fifth aspect of the present invention provides an electronic device, including: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above method.

[0017] The method, device, system and medium for detecting anomalies in air traffic supervision based on optical remote sensing images according to the embodiments of the present invention have at least the following beneficial effects:

[0018] A remote sensing satellite is used to perform high-resolution imaging on a specific airspace to obtain an optical remote sensing image, and object detection is performed to identify moving objects (aircraft) in the image. According to information such as the imaging time and imaging range of the image, the ADS-B data is matched to obtain the target ADS-B data, and then based on the position information of the moving object, the detection result of air traffic supervision anomalies can be obtained, thereby realizing the monitoring of the usage of the ADS-B system, which is of great significance for improving the coverage ability and response efficiency of the global aviation monitoring system.

[0019] Parameters such as the navigation speed and direction in the image are calculated and cross-validated with the ADS-B data. Through the cross-validation of the ADS-B data, problems such as whether the aircraft uses the ADS-B device irregularly, reports incorrect data or has a large accuracy deviation can be identified. This can not only verify the usage of the ADS-B device of the aircraft, but also evaluate the accuracy and reliability of the ADS-B data. This will help to promote the standardized use of ADS-B data, improve the quality and efficiency of aviation safety monitoring, and at the same time provide more accurate data support for relevant aviation management departments to ensure the synchronous improvement of aviation safety and efficiency.

[0020] For different types of remote sensing satellites, different methods can be used to detect aircraft targets to improve the detection accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Through the following description of the embodiments of the present invention with reference to the drawings, the above and other objects, features and advantages of the present invention will become clearer. In the drawings:

[0022] Figure 1 Schematically shows a flowchart of a method for detecting anomalies in air traffic supervision based on optical remote sensing images according to an embodiment of the present invention;

[0023] Figure 2 Schematically shows a logic diagram of detecting anomalies in air traffic supervision based on optical remote sensing images according to an embodiment of the present invention;

[0024] Figure 3 Schematically shows a map of the pixel positions of aircraft in different bands of a hyperspectral image according to an embodiment of the present invention;

[0025] Figure 4 Schematically shows a structural block diagram of an air traffic supervision anomaly detection device based on an optical remote sensing image according to an embodiment of the present invention;

[0026] Figure 5 Schematically shows a structural block diagram of an airspace supervision system according to an embodiment of the present invention;

[0027] Figure 6 Schematically shows a block diagram of an electronic device suitable for implementing an anomaly detection method according to an embodiment of the present invention. Detailed implementation manners

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0029] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, having only B, having only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0032] Currently, there are already some methods for detecting abnormal ADS-B data. For example, the ADS-B anomaly detection method based on self-supervised learning uses a conditional diffusion model to achieve the reconstruction of messages, and finally detects anomalies through the reconstruction error. The multi-source ADS-B data anomaly time-scale detection method can achieve the detection of the anomaly time-scale of real-time dynamic target trajectories in a large-scale airspace.

[0033] Although the above methods can perform anomaly detection through direct analysis of ADS-B data, all of the above methods must rely on the already obtained ADS-B data. If the ADS-B system of an aircraft fails or is deliberately turned off, the monitoring ability of this system will be affected. In addition, although the ADS-B signal coverage is extensive, in some remote areas or blank areas between ground base stations, the reliability and integrity of the signal may decline. Therefore, for ensuring flight safety and supervision globally, other technologies and methods, such as radar monitoring and satellite remote sensing images, still need to be combined to supplement and improve the deficiencies of the ADS-B system. This multi-source data fusion method can not only enhance the robustness of the monitoring network but also continue to provide critical flight tracking information when the ADS-B system is unavailable.

[0034] Remote sensing satellite images can provide extensive visual information of the flight area. By combining ADS-B data with remote sensing images, the position of an aircraft can be accurately identified and tracked when the ADS-B signal is insufficient or visual confirmation is required. In addition, this method can also enhance the reliability and effectiveness of flight monitoring in specific application scenarios, such as border monitoring, environmental supervision, or search and rescue operations. Some existing methods of combining ADS-B data with remote sensing images are mainly used for quickly retrieving and positioning abnormal aviation targets in a large area by using low-earth orbit satellites, but this method relies on on-board ADS-B data and the application scenarios are severely limited. There are also some methods that project the sampled ADS-B data into image information and achieve spoofing detection of ADS-B route information by detecting the structural similarity between images. However, this method only discriminates based on ADS-B data and does not make full use of the image information provided by remote sensing images.

[0035] In view of this, an embodiment of the present invention provides an anomaly detection method for air traffic supervision based on optical remote sensing images. This method combines the matching verification of optical remote sensing images and ADS-B data. First, a remote sensing satellite is used to image the area, and an advanced target detection algorithm is adopted to identify aircraft targets in the image. According to the imaging time of the satellite image and the position information of the aircraft in the image, the matching of the aircraft in the image and the ADS-B data is achieved. Then, the motion parameters of the aircraft are estimated through the remote sensing image and compared with the ADS-B data to realize the inspection of the usage situation of the aircraft's ADS-B system.

[0036] Figure 1 Schematically shows a flowchart of an anomaly detection method for air traffic supervision based on optical remote sensing images according to an embodiment of the present invention.

[0037] As Figure 1As shown, the method for abnormal detection of air traffic supervision based on optical remote sensing images in this embodiment may include operation S110 to operation S120.

[0038] In operation S110, obtain the optical remote sensing image of the target area, as well as the imaging time and imaging range of the optical remote sensing image.

[0039] In operation S120, obtain the target ADS-B data from the pre-stored ADS-B data based on the imaging time and imaging range.

[0040] In operation S130, detect moving targets in the optical remote sensing image to obtain the position information of the moving targets.

[0041] In operation S140, match the moving targets with the target ADS-B data based on the position information of the moving targets to obtain the abnormal detection result.

[0042] According to an embodiment of the present invention, the target area may be some remote areas, blank areas between ground base stations, or areas of interest. For example, eligible optical remote sensing images can be collected according to the area of interest and time range, and information such as the scene number (sceneid), task number (jobtaskid), satellite number (satelliteid), imaging start time (scenestarttime), imaging end time (sceneendtime), cloud cover (cloudcover), and imaging range (spatialdata) of each optical remote sensing image can be extracted.

[0043] According to an embodiment of the present invention, the imaging time and imaging range of each obtained optical remote sensing image can be used to screen the pre-stored ADS-B data. The ADS-B data that meets the requirements returns information such as the identification code, longitude and latitude, flight speed, azimuth, altitude, the time of sending ADS-B, and the time of the last record of the aircraft's ADS-B. The application programming interface (Application Programming Interface, API) of OpenSky can be called to screen the ADS-B data in the database. OpenSky is an open aviation data platform operated by OpenSky Network. OpenSky has collected more than 30 trillion messages, and these data come from more than 6,000 sensors around the world. This makes OpenSky the largest aviation traffic surveillance data repository of its kind, and researchers and developers can access these data in various ways, including API and Impala database.

[0044] In some embodiments, obtaining target ADS-B data from pre-stored ADS-B data based on the imaging time and imaging range may include:

[0045] Obtaining initial ADS-B data associated with moving targets in the optical remote sensing image from the pre-stored ADS-B data based on the imaging time and imaging range.

[0046] Determining, from the initial ADS-B data, the initial ADS-B data for which the time difference between the transmission time of the ADS-B data and the recording time of the last recorded ADS-B data of the moving target satisfies a preset condition as the target ADS-B data.

[0047] According to an embodiment of the present invention, due to the delay in the transmission of ADS-B data, it is necessary to further screen the matched data. For example, retaining the data with a time difference of one second between the time of sending ADS-B and the time of the last recorded ADS-B of the aircraft. By this method, the reliability of the matched data can be maximized.

[0048] For the detection of moving targets (aircraft), according to the different types of remote sensing satellites used, it can be divided into: target detection based on multi-spectral remote sensing images and target detection based on video satellite images.

[0049] In some embodiments, when the optical remote sensing image is a multi-spectral remote sensing image, detecting moving targets in the optical remote sensing image to obtain the position information of the moving targets includes:

[0050] Performing image denoising, normalization, and data enhancement on the multi-spectral remote sensing image to obtain a first image.

[0051] Inputting the first image into a pre-trained deep convolutional neural network model to extract spatial features of different scales in the first image and outputting spatial feature maps of different scales.

[0052] Inputting the spatial feature maps of different scales into a region proposal network to identify candidate regions containing moving targets and adjusting the candidate regions to obtain the position information of the moving targets.

[0053] Further, inputting the spatial feature maps of different scales into a region proposal network to identify candidate regions containing moving targets includes:

[0054] For each scale of spatial feature map, sliding a sliding window over the spatial feature map to obtain a plurality of detection frames.

[0055] Performing foreground and background classification and bounding box regression on each detection frame, and determining at least one target detection frame from the plurality of detection frames as the initial candidate region corresponding to the spatial feature map of this scale.

[0056] Fuse the initial candidate regions of different scales to obtain candidate regions containing moving objects. During the fusion process, non-maximum suppression is used to remove multiple detection boxes corresponding to the same moving object.

[0057] According to the embodiments of the present invention, traditional object detection methods are limited by algorithm complexity and environmental changes, and often cannot meet the requirements of high accuracy and real-time performance. Therefore, using deep learning technology for automatic detection of aircraft targets can not only improve the detection accuracy, but also effectively adapt to the complex and changeable backgrounds and environmental conditions of multi-spectral images.

[0058] The overall architecture of aircraft target detection based on deep learning includes several key steps: data preprocessing, feature extraction, target detection, and post-processing.

[0059] In the data preprocessing stage, image denoising, normalization, and data augmentation techniques are used to improve the quality of the input image and enhance the robustness of the model under different lighting and background conditions. The feature extraction process uses pre-trained deep convolutional neural networks, such as VGG and ResNet, which can effectively extract rich spatial features from remote sensing images.

[0060] In the target detection stage, the Region Proposal Network (RPN) and anchor box technology are used to automatically identify potential aircraft target regions in the image, and the position of the aircraft is accurately located through bounding box regression technology. In addition, to improve the detection accuracy of the system for aircraft targets, a multi-scale feature fusion strategy is introduced. By fusing network features at different levels, the detection ability for small or blurred targets is enhanced. At the same time, the detection threshold is dynamically adjusted. Based on the size and shape features of the target, the signal-to-noise ratio in the detection process is optimized to further improve the detection accuracy.

[0061] In the target detection task, for the recognition and localization of aircraft targets, a series of advanced technical means can be adopted, including RPN, anchor box technology, bounding box regression, multi-scale feature fusion strategy, and dynamic adjustment of the detection threshold, etc.

[0062] RPN is a key component in the target detection model, responsible for generating candidate regions that may contain targets. It slides a window on the feature map and generates multiple anchor boxes (detection boxes), classifies each anchor box as foreground / background, and performs bounding box regression, thereby screening out high-quality candidate regions. RPN can efficiently generate candidate regions, reduce the computational amount in the subsequent processing stage, and improve the efficiency and accuracy of target detection.

[0063] An anchor box is a rectangle with predefined fixed scales and aspect ratios on the feature map by the RPN, used to cover possible target regions. The anchor box technology provides a benchmark for the RPN to generate candidate regions, enabling the model to more comprehensively cover potential targets in the image.

[0064] Bounding box regression is an important step in object detection, used to precisely adjust the candidate regions generated by the RPN to more accurately locate the position of the target object. By learning the offset between the candidate region and the true bounding box of the target object through a regression model, the candidate region is fine-tuned.

[0065] In object detection, small or blurred targets are often difficult to be accurately detected because their feature representations in the image are weak. By fusing network features at different levels (e.g., shallow feature maps contain more detailed information, and deep feature maps contain more semantic information), the detection ability of the model for small or blurred targets is improved. Structures such as the Feature Pyramid Network (FPN) can be adopted to fuse feature maps at different levels to form multi-scale feature representations.

[0066] The purpose of dynamically adjusting the detection threshold is to optimize the signal-to-noise ratio in the detection process and improve the detection accuracy. The method of dynamically adjusting the detection threshold can include dynamically adjusting the detection threshold according to features such as the size and shape of the target. For example, for small targets, the detection threshold can be lowered to increase the recall rate; for large targets, the detection threshold can be raised to reduce false detections. The method of dynamically adjusting the detection threshold can also include an adaptive threshold, which enables the detection threshold to be adaptively adjusted through model learning or rule setting to meet the detection requirements in different scenarios.

[0067] In practical applications, combining technologies such as the RPN, anchor box technology, bounding box regression, multi-scale feature fusion strategy, and dynamically adjusting the detection threshold can build an efficient and accurate object detection model. This model can automatically identify potential aircraft target regions in the image and accurately locate the position of the aircraft through the bounding box regression technology. At the same time, the multi-scale feature fusion strategy and the method of dynamically adjusting the detection threshold further improve the detection ability of the model for small or blurred targets and the overall detection accuracy.

[0068] In the post-processing stage, the Non-Maximum Suppression (NMS) technology is used to process multiple overlapping detection results to ensure that each target can be correctly and separately identified, thereby improving the performance of the overall system. In addition, new deep learning optimization algorithms are adopted, such as Adam and SGDR. These algorithms not only accelerate the model training process but also improve the convergence speed of the learning process, enabling a high accuracy to be achieved in a relatively short time.

[0069] In some other embodiments, when the optical remote sensing image is a video satellite image, detecting a moving target in the optical remote sensing image to obtain the position information of the moving target, including:

[0070] Performing denoising and enhancement processing on the video satellite image to obtain a second image.

[0071] Inputting the second image into a siamese network model to extract the spatial and temporal features of the second image, obtaining a spatio-temporal feature map, and based on the spatio-temporal feature map, comparing the feature representations of the moving target in consecutive image frames to obtain the position information of the moving target.

[0072] According to the embodiments of the present invention, the object detection based on video satellite images mainly utilizes the temporal and spatial information of the video sequence, combined with a siamese network, to achieve high-precision detection of aircraft targets. The application of the siamese network in moving object detection is mainly reflected in its advantages in similarity measurement. The siamese network consists of two or more sub-networks sharing parameters, and is usually used to compare and match the similarity of input image pairs. In the aircraft target detection of video satellite images, the siamese network can identify and track aircraft targets by comparing the similarity of consecutive frames or feature maps. This method generally includes steps such as image preprocessing, feature extraction, object detection and tracking.

[0073] In the image preprocessing stage, it is first necessary to perform denoising and enhancement processing on the video satellite image to improve the image quality and reduce the impact of environmental noise on the detection accuracy. Since satellite images are often limited by imaging conditions and interference during transmission, denoising and enhancement processing can significantly improve the accuracy of subsequent detections.

[0074] In the feature extraction stage, the siamese network extracts the feature representations of the input image pair through sub-networks sharing parameters. These sub-networks are usually convolutional neural networks, which can effectively capture the spatial and temporal features in the image. By comparing the similarity of consecutive frames or feature maps, the siamese network can detect the motion trajectory and change pattern of the target. Specifically, during the training process, the siamese network optimizes the network parameters by calculating the similarity loss of the image pair, enabling it to distinguish similar and different target features.

[0075] In the object detection and tracking stage, the siamese network uses the extracted features for object matching and similarity measurement. By comparing the feature representations of the target in consecutive frames, the siamese network can accurately identify and track the position and morphological changes of the moving target. This process generally involves steps such as generating candidate regions, calculating similarity scores, and selecting the best matching region. The siamese network performs well in dealing with complex backgrounds and diverse targets, and can effectively handle problems such as target occlusion and appearance changes.

[0076] In addition, in order to further improve the accuracy and robustness of moving target detection, technologies such as multi-scale feature fusion and attention mechanism are introduced. Multi-scale feature fusion enhances the network's target recognition ability by combining feature representations at different levels. The attention mechanism, on the other hand, improves the detection performance by dynamically adjusting feature weights, highlighting important features, and suppressing interference information.

[0077] Figure 2 Schematically shows a logic diagram of air traffic supervision anomaly detection based on optical remote sensing images according to an embodiment of the present invention.

[0078] As Figure 2 shown, in some embodiments, matching the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result may include:

[0079] In the case where there is no target in the target ADS-B data that matches the moving target, it is determined that the moving target does not use the ADS-B system regularly.

[0080] In the case where the target ADS-B data matches the moving target, determine the moving speed and moving azimuth angle of the moving target based on the optical remote sensing image; match the moving speed and moving azimuth angle with the target ADS-B data, and in the case where the deviation is greater than the threshold, determine that there is an anomaly in the ADS-B data of the moving target.

[0081] In some embodiments, determining the moving speed and moving azimuth angle of the moving target based on the optical remote sensing image includes:

[0082] Determine the moving speed and moving azimuth angle of the moving target based on the pixel position of the moving target in the optical remote sensing images of different bands and the corresponding spatial resolution of different bands.

[0083] According to an embodiment of the present invention, since a multi-spectral satellite has different imaging times for different bands during imaging, this results in different pixel positions of a moving target in the images of different bands.

[0084] Figure 3 Schematically shows a diagram of the pixel positions of an airplane in the images of different bands of a hyperspectral image according to an embodiment of the present invention.

[0085] As Figure 3 shown, the moving speed of the target can be obtained by determining the change amount of the target pixel and the moving time in the images of different bands.

[0086] Assume that the pixel displacement of the same moving target in the remote sensing image between two different bands is , which are the horizontal direction and the vertical direction respectively (the positive x direction is to the right horizontally, and the positive y direction is downward vertically), that is, if at time The pixel coordinates are , and the time The pixel coordinates are , then , so the approximate speed of the moving target can be obtained, that is:

[0087]

[0088] where r is the spatial resolution of the corresponding band. For the multispectral band of the high-resolution multi-mode satellite, r = 2m. The moving azimuth angle of the moving target can also be calculated simultaneously through pixel displacement. The azimuth angle is defined as 0 degrees in the due north direction and increases sequentially to 360 degrees in the clockwise direction. It is stipulated that the direction is uncertain at , and the azimuth angle is 90 degrees ( ) or 270 degrees ( ) at , and the azimuth angle is 180 degrees ( ) or 0 degrees ( ) at (in degrees), and the azimuth angle of the moving target

[0089]

[0090] is determined as follows: (in degrees).

[0091] To reduce the uncertainty of the estimation results, in the case of multiple bands, the estimated values of multiple pairs of bands can be selected and averaged. In this way, the difference method can be used to calculate the estimated values of the moving target speed / direction and reduce the error. Let (m, n) in the subscript represent the change value from band m to band n, then the average speed can be expressed as:

[0092]

[0093] Through the above calculations, the moving speed and moving azimuth degree of the moving target in the image can be obtained. However, due to the high-speed movement of the satellite itself, the speed estimated from the multispectral image is not the actual speed of the target movement, but the combined speed of the target and the satellite. After canceling the speed of the satellite, the true speed of the aircraft can be obtained. Similarly, the azimuth angle calculated according to the above formula is not the true azimuth angle. By analyzing the shape of the target in the image, the true angle information can be extracted according to the characteristics of the aircraft target in the spectral image.

[0094] Then, the position of the aircraft in the image and the calculated speed and azimuth angle parameters are verified with the matched ADS-B data. If the deviation is greater than 20%, it means that the accuracy of the aircraft ADS-B system is relatively low.

[0095] Regarding the problem of monitoring the irregular use of the ADS-B system by aircraft, the method of the embodiment of the present invention fuses remote sensing images and the ADS-B data of aircraft, extracts information from optical remote sensing images by using deep learning methods and image processing means, and performs matching verification on a large amount of ADS-B data, effectively solving the problem of supervising the irregular use of the aircraft ADS-B system and the estimation of the flight state of the aircraft in the case of lack of ADS-B data.

[0096] Based on the above-mentioned method for detecting anomalies in air traffic supervision based on optical remote sensing images, an embodiment of the present invention also provides a device for detecting anomalies in air traffic supervision based on optical remote sensing images.

[0097] Figure 4 The structural block diagram of the device for detecting anomalies in air traffic supervision based on optical remote sensing images according to an embodiment of the present invention is schematically shown.

[0098] As Figure 4 shown, the device 400 for detecting anomalies in air traffic supervision based on optical remote sensing images of this embodiment may include a first acquisition module 410, a second acquisition module 420, a target detection module 430, and a matching module 440.

[0099] The first acquisition module 410 is configured to acquire an optical remote sensing image of a target area, as well as the imaging time and imaging range of the optical remote sensing image.

[0100] The second acquisition module 420 is configured to acquire target ADS-B data from the pre-stored ADS-B data based on the imaging time and imaging range.

[0101] The target detection module 430 is configured to detect moving targets in the optical remote sensing image to obtain the position information of the moving targets;

[0102] The matching module 440 is configured to match the moving targets with the target ADS-B data based on the position information of the moving targets to obtain an anomaly detection result.

[0103] Any number of modules, sub-modules, units, and sub-units according to embodiments of the present invention, or at least part of the functions of any number thereof, may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention may be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), programmable logic array (PLA), system on chip, system on substrate, system on package, application specific integrated circuit (ASIC), or may be implemented by any other reasonable manner of integrating or packaging circuits, in hardware or firmware, or may be implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0104] For example, any number of the first acquisition module 410, the second acquisition module 420, the target detection module 430, and the matching module 440 may be combined and implemented in one module / unit / sub-unit, or any one of the module / unit / sub-unit may be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units may be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to embodiments of the present invention, at least one of the first acquisition module 410, the second acquisition module 420, the target detection module 430, and the matching module 440 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), programmable logic array (PLA), system on chip, system on substrate, system on package, application specific integrated circuit (ASIC), or may be implemented by any other reasonable manner of integrating or packaging circuits, etc., in hardware or firmware, or may be implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the first acquisition module 410, the second acquisition module 420, the target detection module 430, and the matching module 440 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0105] It should be noted that the device part in the embodiments of the present invention corresponds to the method part in the embodiments of the present invention, and the specific implementation details are also the same, and will not be elaborated here.

[0106] Figure 5Schematically shows a structural block diagram of an airspace supervision system according to an embodiment of the present invention.

[0107] As Figure 5 shown, the airspace supervision system 500 of this embodiment may include a remote sensing satellite constellation 510 and an aviation safety warning platform 520 for sending modules.

[0108] The remote sensing satellite constellation 510 is configured to acquire an optical remote sensing image of a target area, as well as the imaging time and imaging range of the optical remote sensing image.

[0109] The aviation safety warning platform 520 is configured to obtain target ADS-B data from pre-stored ADS-B data based on the imaging time and imaging range; detect moving targets in the optical remote sensing image to obtain the position information of the moving targets; match the moving targets with the target ADS-B data based on the position information of the moving targets to obtain an anomaly detection result, and visually display the anomaly detection result.

[0110] It should be noted that the system part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention, and their specific implementation details are also the same, which will not be elaborated here.

[0111] Figure 6 Schematically shows a block diagram of an electronic device suitable for implementing the anomaly detection method according to an embodiment of the present invention.

[0112] As Figure 6 shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as, an application specific integrated circuit (ASIC)), etc. The processor 601 may also include on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0113] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present invention by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method flow according to the embodiments of the present invention by executing the programs stored in the one or more memories.

[0114] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, and the input / output (I / O) interface 605 is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read from it can be installed into the storage part 608 as needed.

[0115] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist alone without being assembled into the device / apparatus / system.

[0116] In one embodiment, the computer-readable storage medium is disposed at the on-board edge, and executable instructions are stored thereon. When the instructions are executed by a processor, the processor is caused to implement the following method: obtaining optical remote sensing images of a moving target in a target area in different bands; determining the moving speed and moving azimuth angle of the moving target based on the pixel positions of the moving target in the optical remote sensing images in different bands and the spatial resolutions corresponding to different bands. That is, the on-board edge computing storage medium solidifies a multi-spectral motion solution algorithm, supports real-time processing of images on the satellite orbit and extraction of aircraft motion parameters. For specific details, please refer to the foregoing method part and will not be elaborated here.

[0117] The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0118] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the above-described ROM 602 and / or RAM 603 and / or one or more memories other than ROM602 and RAM 603.

[0119] An embodiment of the present invention further includes a computer program product, which includes a computer program that contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to enable the computer system to implement the processing method provided by the embodiment of the present invention.

[0120] When the computer program is executed by the processor 601, it executes the above functions defined in the system / apparatus of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0121] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 609, and / or installed from the removable medium 611. The program code included in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0122] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it executes the above functions defined in the system of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0123] According to embodiments of the present invention, program code for executing the computer programs provided by the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming languages include, but are not limited to, such as Java, C++, Python, the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly recited in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recited in the various embodiments and / or claims of the present invention can be combined and combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0126] The embodiments of the present invention have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. An abnormal detection method for air traffic supervision based on optical remote sensing images, characterized in that, Including: Obtaining an optical remote sensing image of a target area, as well as the imaging time and imaging range of the optical remote sensing image; Obtaining target ADS-B data from pre-stored ADS-B data based on the imaging time and the imaging range; Detecting moving targets in the optical remote sensing image to obtain the position information of the moving targets; Matching the moving targets with the target ADS-B data based on the position information of the moving targets to obtain an anomaly detection result.

2. The method according to claim 1, characterized in that The obtaining of the target ADS-B data from the pre-stored ADS-B data based on the imaging time and the imaging range includes: Obtaining initial ADS-B data associated with the moving targets in the optical remote sensing image from the pre-stored ADS-B data based on the imaging time and the imaging range; Determining the initial ADS-B data with a time difference between the sending time of the ADS-B data and the recording time of the last recorded ADS-B data of the moving targets meeting a preset condition in the initial ADS-B data as the target ADS-B data.

3. The method according to claim 1, wherein When the optical remote sensing image is a multi-spectral remote sensing image, the detecting of the moving targets in the optical remote sensing image to obtain the position information of the moving targets includes: Performing image denoising, standardization, and data enhancement on the multi-spectral remote sensing image to obtain a first image; Inputting the first image into a pre-trained deep convolutional neural network model to extract spatial features of different scales in the first image and output spatial feature maps of different scales; Inputting the spatial feature maps of different scales into a region proposal network to identify candidate regions containing the moving targets and adjusting the candidate regions to obtain the position information of the moving targets.

4. The method according to claim 3, wherein The inputting of the spatial feature maps of different scales into the region proposal network to identify candidate regions containing the moving targets includes: For each scale of spatial feature map, sliding a sliding window over the spatial feature map to obtain a plurality of detection frames; Performing foreground and background classification and bounding box regression on each detection frame, and determining at least one target detection frame from the plurality of detection frames as the initial candidate region corresponding to the spatial feature map of this scale; Fusing the initial candidate regions of different scales to obtain a candidate region containing the moving targets, where non-maximum suppression is used during the fusion process to remove multiple detection frames corresponding to the same moving target.

5. The method according to claim 1, wherein When the optical remote sensing image is a video satellite image, the detecting of the moving targets in the optical remote sensing image to obtain the position information of the moving targets includes: Performing denoising and enhancement processing on the video satellite image to obtain a second image; Inputting the second image into a Siamese network model to extract the spatial and temporal features of the second image, obtaining a spatio-temporal feature map, and based on the spatio-temporal feature map, comparing the feature representations of the moving targets in consecutive image frames to obtain the position information of the moving targets.

6. The method according to claim 1, wherein The matching of the moving targets with the target ADS-B data based on the position information of the moving targets to obtain an anomaly detection result includes: In the case that there is no target matching the moving target in the target ADS-B data, it is determined that the moving target does not use the ADS-B system in a standardized manner; Or: In the case that the target ADS-B data matches the moving target, the moving speed and moving azimuth angle of the moving target are determined based on the optical remote sensing image; The moving speed and moving azimuth angle are matched with the target ADS-B data. In the case that the deviation is greater than the threshold, it is determined that there is an abnormality in the ADS-B data of the moving target.

7. The method according to claim 6, characterized in that, The determining of the moving speed and moving azimuth angle of the moving target based on the optical remote sensing image includes: Based on the pixel positions of the moving target in the optical remote sensing images of different bands and the spatial resolutions corresponding to different bands, the moving speed and moving azimuth angle of the moving target are determined.

8. An abnormal detection device for air traffic supervision based on optical remote sensing images, characterized in that, Including: A first acquisition module, configured to acquire an optical remote sensing image of a target area, and the imaging time and imaging range of the optical remote sensing image; A second acquisition module, configured to acquire target ADS-B data from the pre-stored ADS-B data based on the imaging time and the imaging range; A target detection module, configured to detect a moving target in the optical remote sensing image to obtain the position information of the moving target; A matching module, configured to match the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result.

9. An airspace supervision system, characterized in that, Including: A remote sensing satellite constellation, configured to acquire an optical remote sensing image of a target area, and the imaging time and imaging range of the optical remote sensing image; An aviation safety warning platform, configured to acquire target ADS-B data from the pre-stored ADS-B data based on the imaging time and the imaging range; Detect a moving target in the optical remote sensing image to obtain the position information of the moving target; match the moving target with the target ADS-B data based on the position information of the moving target to obtain an anomaly detection result, and visually display the anomaly detection result.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is set on the on-board edge, and an executable instruction is stored thereon. When the instruction is executed by a processor, the processor implements the following method: Acquire optical remote sensing images of a moving target in a target area in different bands; Based on the pixel positions of the moving target in the optical remote sensing images of different bands and the spatial resolutions corresponding to different bands, determine the moving speed and moving azimuth angle of the moving target.

Images