A method for monitoring moving targets on the airport surface based on satellite positioning and image data

Abstract

This invention discloses a method for monitoring moving targets on airport surfaces based on "satellite positioning + image data," applied to the field of target surveillance. It addresses the low detection accuracy problem of existing target detection algorithms in airport surface target detection. This invention preprocesses satellite positioning data and image data, utilizes a neural network to align, interact with, and fuse features from different sources, and performs target detection based on the fused features. Satellite positioning data provides precise target location information, while image data provides rich visual context. By combining satellite positioning data and image data, the complementary advantages of both can be fully utilized, improving the accuracy and robustness of target detection. This multimodal fusion method not only effectively copes with the complex environment of airport surfaces but also maintains good performance under various lighting and weather conditions. Furthermore, the method of this invention can also improve the detection capability of occluded targets.

Description

Technical Field

[0001] This invention belongs to the field of target surveillance, and specifically relates to a technology for surveillance of targets on the airport surface. Background Technology

[0002] With the rapid development of the air transport industry, airports, as important nodes in the air transport network, are receiving increasing attention for their operational efficiency and safety. The movement, location, and status of all objects on the airport surface—including aircraft, vehicles, and personnel—are crucial for ensuring normal flight takeoffs and landings, improving airport operational efficiency, and safeguarding ground safety.

[0003] Traditional surveillance systems primarily rely on image data. In recent years, driven by deep learning, image target detection algorithms have developed rapidly. Algorithms such as YOLOv8, DINO, and RT-DETR have achieved good performance on benchmark datasets like VOC and COCO. However, these methods still face some challenges when applied to target detection in airport scenes. Dynamically changing environment: Illumination changes; lighting conditions in airport scenes vary drastically with time and weather, from dawn to dusk, from sunny to rainy days, all of which affect image quality and consequently the performance of target detection algorithms. Weather effects; adverse weather conditions such as rain, snow, and fog reduce image clarity and increase noise, making targets difficult to identify correctly. Target diversity: Targets of varying sizes; airport scenes exhibit a wide range of target sizes, from small baggage carts and staff to large passenger aircraft, requiring algorithms to effectively handle targets of different scales. Changing poses; aircraft, vehicles, and pedestrians constantly change poses during movement, placing higher demands on target detection. Partial occlusion: In busy airports, targets are often partially occluded by other objects, requiring algorithms with strong occlusion handling capabilities.

[0004] The main problems with existing target detection algorithms for airport scenes are as follows:

[0005] 1) Poor robustness: Airports are typical outdoor scenes with complex and variable lighting and weather conditions. Existing target detection methods perform relatively poorly under low light and low visibility conditions, and often experience missed detections and false detections under low light, rainy, and foggy conditions. Therefore, it is necessary to improve the robustness of the algorithm under complex backgrounds and changing lighting conditions, and design a target detection algorithm that performs well under different environments and conditions.

[0006] 2) Insufficient accuracy: Existing methods based on a single data source (such as images or satellite positioning alone) are insufficient to provide high-precision target location and status information, especially in high-density target environments.

[0007] 3) Multimodal data alignment: Satellite positioning information comes from ADS-B transponders, and image data comes from surveillance cameras. Since they come from different sensors and have different information types, how to align the two types of data in a unified coordinate system and assist in target detection is an urgent problem to be solved. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a method for monitoring moving targets on the airport surface based on "satellite positioning + image data". When the image data quality is poor, the method utilizes the location information provided by the satellite as prior knowledge to guide the image processing algorithm, which can effectively improve the detection accuracy.

[0009] The technical solution adopted in this invention is: a method for monitoring moving targets on the airport surface based on "satellite positioning + image data", comprising:

[0010] S1. Obtain the satellite positioning information of the aircraft; and obtain airport surface image data through surveillance cameras deployed on the airport surface;

[0011] S2. Convert the geographic coordinates in the satellite positioning information of the aircraft into pixel coordinates in the image data;

[0012] S3. Calculate the signal strength of the spacecraft's satellite positioning information;

[0013] S4. ResNet50 is used as the backbone network to extract features from the input image data, obtain the mapping features at each stage, and then the features are merged through convolution and upsampling operations to obtain the feature layer {C1,C2,C3,C4,C5}.

[0014] S5. After obtaining the feature layer {C1,C2,C3,C4,C5}, input {C3,C4,C5} into the first Focus module. The output of the first Focus module is passed through a set of depthwise parallel convolutions to obtain {P41}.

[0015] {P41} is downsampled and concatenated with {C5} to obtain {P51};

[0016] {C2} first passes through the SPDConv module to obtain features rich in small target information. After downsampling, it is fused with the upsampled features of {P41} and {C3} to obtain {P31}.

[0017] The Integrate module is used to integrate the features of {P31} to obtain {P32};

[0018] {P32, P41, P51} are obtained by capturing cross-scale information through the second Focus module using deep parallel convolution;

[0019] {P42} is upsampled and fused with {P32} to obtain {P33}, and {P42} is downsampled and fused with {P51} to obtain {P52};

[0020] Finally, the features {P33, P42, P52} are encoded and input into the Decoder module;

[0021] S6. Adjust the initialization of Tokens in the Decoder module according to the signal strength calculated in step S3;

[0022] S7. The Decoder module, after adjustment in step S6, outputs the class probability and bounding box coordinates of each target.

[0023] The beneficial effects of this invention are as follows: The image data acquired by the surveillance camera in the method of this invention provides detailed information such as the appearance, shape, and texture of the target. Combined with the more accurate location information of the target provided by satellite positioning data, the position and attitude of the target can be further refined, thereby improving the accuracy of detection. This invention adopts a multimodal fusion method to enhance the algorithm's adaptability to environmental changes. Even if a single data source fails, such as when the image data quality deteriorates under severe weather conditions, the system can still maintain high detection performance by relying on another data source. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention;

[0025] Figure 2 This is a diagram of the multi-scale feature fusion structure of the present invention;

[0026] Figure 3 This is a structural diagram of the Decoder of the present invention;

[0027] Figure 4 The results of the detection of targets on the airport surface. Detailed Implementation

[0028] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0029] In recent years, with the development and maturation of satellite positioning technology and the continuous expansion of its application scope, satellite positioning technology can provide new solutions for monitoring moving targets at airports. By integrating the precise location information provided by satellite positioning with target features derived from image data analysis, it is possible not only to effectively improve the accuracy of target identification and tracking continuity, but also to enhance the robustness and intelligence of the entire surveillance system.

[0030] Against this backdrop, this invention proposes a target detection algorithm based on satellite positioning data and image data, aiming to improve detection accuracy and robustness. This method preprocesses satellite positioning data and image data, utilizes a neural network to perform feature alignment, interaction, and fusion of data from different sources, and performs target detection based on the fused features. Satellite positioning data provides precise target location information, while image data provides rich visual context. By combining satellite positioning data and image data, the complementary advantages of both can be fully utilized to improve the accuracy and robustness of target detection. This multimodal fusion method can not only effectively cope with the complex environment of airport scenes but also maintain good performance under various lighting and weather conditions. Furthermore, this method can also improve the detection capability of occluded targets.

[0031] Compared to image data, satellite positioning information, such as ADS-B transponder data, provides aircraft coordinates. Even in low-light conditions like nighttime or foggy weather, ADS-B information can provide relatively accurate aircraft location information. This invention proposes an airport target detection algorithm that combines satellite positioning information and image information, aiming to improve the accuracy of detecting moving targets on the airport surface, enhance the robustness of the detection algorithm, and achieve better detection results. The specific technical solution includes the following steps, such as... Figure 1 As shown.

[0032] Step 1: Acquisition and Analysis of Satellite Positioning and Image Information

[0033] In this invention, the aircraft's satellite positioning information is obtained through ADS-B information. By capturing ADS-B signals using an ADS-B receiver, the aircraft's position information can be decoded. ADS-B data is typically provided in JSON or CSV format and includes information such as the aircraft's latitude, longitude, altitude, speed, and heading. Image data is acquired through surveillance cameras deployed at the airport.

[0034] Step 2: Conversion between geographic coordinates and image coordinates

[0035] The images captured by the camera are based on the pixel coordinate system (x,y), while the location information in the ADS-B data is based on the WGS84 coordinate system (latitude and longitude). Therefore, it is necessary to convert the geographic coordinates in the ADS-B data into pixel coordinates on the image in order to match the aircraft with its corresponding ADS-B information.

[0036] Camera calibration: Intrinsic parameter calibration refers to calibrating the camera's intrinsic parameters using a checkerboard or other calibration objects to obtain parameters such as focal length, principal point coordinates, and distortion coefficients. Extrinsic parameter calibration refers to determining the camera's absolute position and attitude (rotation and translation matrix) relative to the airport scene, which is achieved by placing calibration points at known locations and performing calibration.

[0037] Establish coordinate mapping relationship: Use perspective projection model to project points in the three-dimensional world coordinate system onto the two-dimensional image plane. Assume that the points in the world coordinate system are (X,Y,Z) and the points in the image coordinate system are (u,v). The projection formula is shown in equation (1), where K is the intrinsic parameter matrix of the camera and [R|t] is the extrinsic parameter matrix, which includes the rotation matrix R and the translation vector t.

[0038]

[0039] Step 3: ADS-B Information Strength Estimation

[0040] Estimating the strength of ADS-B information is crucial for airport surface moving target surveillance systems, as the strength of the ADS-B signal directly reflects the distance between the aircraft and the camera. Higher signal strength generally indicates a closer aircraft to the camera; conversely, lower signal strength suggests a farther aircraft. This information is invaluable for dynamically adjusting target detection strategies. By estimating signal strength, the bounding box of a local area can be dynamically initialized and adjusted. For aircraft with high signal strength (i.e., closer aircraft), a larger bounding box can be used for accurate tracking; while for aircraft with low signal strength (i.e., farther aircraft), a smaller bounding box can be used.

[0041] The aircraft's coordinates are still obtained from the ADS-B data carried by the aircraft, and its position and altitude are recorded as (X). aircraft ,Y aircraft H aircraft ); the precise location and height (X) of the camera on the airport scene. camera ,Y camera H camera ), and use the Euclidean distance formula (2) to calculate the approximate distance d between the aircraft and the camera.

[0042]

[0043] The attenuation of radio signals in free space is estimated using formula (3) of the Free Space Path Loss (FSPL) model, where d is the distance (in meters) calculated by formula (2), f is the signal frequency (in Hertz), the frequency of ADS-B signals is usually 1090 MHz, and c is the speed of light, approximately 3*10 8 meters per second:

[0044] FSPL(dB) = 20log 10 (d)+20log 10 (f)+20log 10 (4Π / c) (3)

[0045] After obtaining the signal attenuation strength, the received ADS-B signal strength can be calculated. In formula (4), Received Power is the received signal strength, Transmit Power is the transmission power, ADS-B transmission power is usually 250 mW (about 24 dBm), and FSPL (dB) is the attenuation strength.

[0046] Received Power(dBm)=Transmit Power(dBm)-FSPL(dB)(4)

[0047] Step 4: Image Data Preprocessing

[0048] Effective preprocessing is crucial before feeding image data into the backbone of the object detection model. Preprocessing not only improves image quality but also enhances the model's robustness and detection accuracy. Lighting conditions at airports can vary due to time and weather, leading to image quality degradation. For images with low illumination, an adaptive histogram equalization method is used, performing histogram equalization locally to avoid noise amplification issues associated with global equalization. In severe weather conditions such as fog, rain, or snow, images may be affected by fog, resulting in blurring and loss of detail. Dehazing can significantly improve image clarity and visibility. Therefore, for this portion of the data, a dark channel prior method is used, based on an atmospheric scattering model, to recover a clear image by estimating transmittance and atmospheric light intensity.

[0049] Step 5: Multi-scale feature selection, fusion, and encoding

[0050] ResNet50 is used as the backbone network to extract features, obtaining the mapped features at each stage. Then, convolution and upsampling operations are used to merge the features to obtain {C1, C2, C3, C4, C5}. In object detection tasks, the detection of small objects is often more challenging than that of large objects. Due to the wide monitoring range of airports, pedestrians and vehicles are relatively small compared to aircraft, and distant aircraft also appear small in the image. Therefore, there are a large number of small moving targets in airport scenes. The representation of small targets on feature maps is often not obvious enough, especially on high-level feature maps. The features extracted by deep networks may be too abstract, leading to the loss of key details of small targets. Even the {P3} feature layer struggles to detect small targets. Therefore, we considered adding a {P2} feature layer to improve small target detection. However, directly adding the {P2} detection layer would lead to excessive computation and more time-consuming processing. To address this, we designed a new feature selection and fusion structure.

[0051] After obtaining the feature layer {C1,C2,C3,C4,C5}, {C3,C4,C5} is first processed through the Focus module (i.e., ... Figure 3 The feature aggregation module (in the code) captures higher-level, multi-scale information through a set of deep parallel convolutions to obtain {P41}. {P41} is downsampled and concatenated with {C5} to obtain {P51}. {C2} first passes through the SPDConv module to obtain features rich in small target information, then is downsampled and fused with the features from {P41} and the feature layer from {C3} to obtain {P31}. Then, the feature layer from {P31} is integrated through the Integrate module to obtain {P32}. {P32, P41, and P51} are then captured through the Focus module using deep parallel convolutions to obtain {P42}. {P42} is upsampled and fused with {P32} to obtain {P33}, then downsampled and fused with {P51} to obtain {P52}. Finally, the features of {P33, P42, and P52} are encoded and input into the Decoder module.

[0052] Step Six: ADS-B Signal Adjustment Token Initialization

[0053] Token initialization is a crucial step in the decoder. In traditional decoder frameworks, to ensure fair and comprehensive detection of every region, tokens are typically evenly distributed across the entire image. By incorporating information from the ADS-B signal, we can initialize tokens more intelligently, thereby improving detection accuracy and efficiency. Specifically, during token initialization, we can utilize the aircraft position information provided by the ADS-B signal to adjust the initialization of some tokens, concentrating them in the areas where the ADS-B signal appears, while maintaining the even distribution of other tokens.

[0054] Token initialization is adjusted based on signal strength: The influence range of each aircraft is determined based on the ADS-B signal strength. A subset of tokens is selected, and their initialization positions are adjusted according to the influence range of each aircraft. Aircraft with higher signal strength should have a larger influence range, while those with lower signal strength should have a smaller influence range.

[0055] Initialize Points: Image size H*W, total number of Tokens N, first generate uniformly distributed points. init (5):

[0056] points init ={(x j ,y j ,)~Uniform(0,W)*Uniform(0,H)|j=1,2,…N} (5)

[0057] Determine the influence range of the ADS-B signal: Assume there are M ADS-B signals in the image region, and the coordinates of each ADS-B signal after coordinate transformation are (u... i ,v i ), intensity is s i i = 1, 2, ..., M (signal strength is calculated using formula (4)). Determine the influence range Δ of each ADS-B signal. i The range of influence can be adjusted according to the signal strength, as shown in formula (6), where Δ and β are adjustment parameters, and α controls the signal strength s. i Impact range Δ i The contribution ratio is α, where a larger α value means a stronger signal corresponds to a wider range of influence; β is an offset that ensures that even the weakest signal has a certain minimum range of influence, which helps maintain a certain monitoring capability even when the signal completely fails; in this embodiment, α = 1 and β = 120.

[0058] Δ i =α·s i +β (6)

[0059] Adjusting Point Distribution: For the i-th ADS-B signal, select points within its influence range for distribution adjustment. After coordinate transformation, the coordinates of the i-th ADS-B signal on the image are (u... i ,v i The affected area is (u i -Δ i ,u i +Δ i )×(v i -Δ i ,v i +Δ i Within the influence range, a two-dimensional Gaussian distribution is used to redistribute and match the positions of these points, as shown in formula (7), where σ is the standard deviation of the Gaussian distribution, which determines the width of the distribution and the influence range Δ. i Proportional, as shown in formula (8), where k is a scaling factor less than 1:

[0060] x j ~N(ui,σ 2 );y j ~N(v i ,σ 2 (7)

[0061] σ=k·Δ i (8)

[0062] In this embodiment, when k is between 0.25 and 0.5, the final recognition accuracy can reach above 0.66. In practical applications, the value of k can also be adjusted according to actual needs.

[0063] Merge Points: Merge the adjusted Points with the unaffected Points to obtain the final initialized points. final As shown in formula (9).

[0064] points final ={(x j ,y j (9) |j=1,2,…,N}

[0065] Step 7: The adjusted Points are fed into the decoder and subsequent processing.

[0066] Since self-attention mechanisms themselves lack information about the order of elements, positional encoding is needed to preserve the order information of elements in the sequence. Positional encoding is a fixed function added to each input embedding vector, typically a combination of sine and cosine functions, which provides the model with absolute and relative positional information for each location. Before the adjusted Points are fed into the Decoder, such as... Figure 3 As shown, these Points need to be given corresponding position codes.

[0067] The decoder consists of multiple identical stacked layers, each containing a self-attention mechanism, a cross-attention mechanism, and a feedforward network. The decoder's final output is passed through a linear layer and mapped to the output space of the object detection task. The final output includes the class probability and bounding box coordinates for each object. See [link to documentation]. Figure 4 . Figure 4 The numbers in the Chinese box indicate the following: The first number, 0 or 1, represents the category, with 0 representing the aircraft category and 1 representing the vehicle category. The decimal number after the number represents the confidence level.

[0068] Traditional target detection methods rely primarily on image data, which is easily affected by environmental factors such as changes in lighting, occlusion, and noise. This invention introduces satellite positioning data, enabling the use of satellite-provided location information as prior knowledge to guide image processing algorithms even when image data quality is poor. Satellite positioning data provides the approximate location of the target, allowing the image processing algorithm to focus on these areas, reducing false positives and false negatives, thereby improving detection accuracy.

[0069] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method for monitoring moving targets on the airport surface based on "satellite positioning + image data", characterized in that, include: S1. Obtain the satellite positioning information of the aircraft; and obtain airport surface image data through surveillance cameras deployed on the airport surface; S2. Convert the geographic coordinates in the satellite positioning information of the aircraft into pixel coordinates in the image data; S3. Calculate the signal strength of the spacecraft's satellite positioning information; S4. ResNet50 is used as the backbone network to extract features from the input image data, obtain the mapping features at each stage, and then the features are merged through convolution and upsampling operations to obtain the feature layer {C1,C2,C3,C4,C5}. S5. After obtaining the feature layer {C1,C2,C3,C4,C5}, input {C3,C4,C5} into the first Focus module. The output of the first Focus module is passed through a set of depthwise parallel convolutions to obtain {P41}. {P41} is downsampled and concatenated with {C5} to obtain {P51}; {C2} first passes through the SPDConv module to obtain features rich in small target information. After downsampling, it is fused with the upsampled features of {P41} and {C3} to obtain {P31}. The Integrate module is used to integrate the features of {P31} to obtain {P32}; {P32, P41, P51} are input into the second Focus module. The output of the second Focus module is captured by deep parallel convolution to obtain cross-scale information, resulting in {P42}. {P42} is upsampled and fused with {P32} to obtain {P33}, and {P42} is downsampled and fused with {P51} to obtain {P52}; Finally, the features {P33, P42, P52} are encoded and input into the Decoder module; S6. Initialize Tokens based on the signal strength calculated in step S3; S7. The Decoder module encodes the input features {P33, P42, P52} and outputs the class probability of each target. The Decoder module also outputs the corresponding bounding box coordinates based on the Tokens initialized in step S6.

2. The airport surface activity target surveillance method based on "satellite positioning + image data" according to claim 1, characterized in that, The specific implementation process of step S3 is as follows: S31. Calculate the distance between the aircraft and the camera based on Euclidean distance; S32. Based on the distance calculated in step S31, estimate the attenuation of the radio signal in free space using the free space path loss model; S33. Based on the attenuation of the radio signal in free space estimated in step S32, calculate the signal strength of the spacecraft's satellite positioning information.

3. The airport surface activity target surveillance method based on "satellite positioning + image data" according to claim 2, characterized in that, The implementation process of step S6 is as follows: S61. Determine the range of influence of each radio signal on the image data based on the signal strength; S62. Based on the pixel coordinates of the radio signal in the image data and its influence range, the area affected by the radio signal is obtained; S63. Select the Points located within the affected area and adjust their distribution to complete the initialization of the Tokens.

4. The airport surface activity target surveillance method based on "satellite positioning + image data" according to claim 3, characterized in that, Step S63 specifically involves redistributing the positions of these Points within the influence area using a two-dimensional Gaussian distribution.

5. The airport surface activity target surveillance method based on "satellite positioning + image data" according to claim 4, characterized in that, The standard deviation σ of a Gaussian distribution is calculated as follows: σ=k·Δ i Where k represents the scaling factor, Δ i Indicates the scope of influence.

6. The airport surface activity target surveillance method based on "satellite positioning + image data" according to claim 5, characterized in that, After merging the adjusted Points with the unaffected Points, add the corresponding position code, and then input them into the Decoder module.

Images