Airport scene moving target monitoring method based on satellite positioning and image data

Abstract

The invention discloses an airport scene moving target monitoring method based on satellite positioning and image data, which is applied to the field of target monitoring and aims to solve the problem of low detection accuracy of an existing target detection algorithm in airport scene target detection. According to the method, satellite positioning data and image data are preprocessed, feature alignment, interaction and fusion are performed on data of different sources by using a neural network, and target detection is performed on fused features; the satellite positioning data provides accurate positioning information of a target, and the image data provides rich visual context; by combining satellite positioning data and image data, the complementary advantages of the satellite positioning data and the image data can be fully utilized, and the precision and robustness of target detection are improved; the multi-modal fusion method provided by the invention not only can effectively cope with a complex environment in an airport scene, but also can keep good performance under various illumination and weather conditions. In addition, the method provided by the invention can also improve the detection capability of the occluded target.

Description

Technical Field

[0001] The invention belongs to the field of target monitoring, and in particular relates to a technology for monitoring targets on airport field activities. Background Art

[0002] With the rapid development of the aviation industry, airports, as important nodes in the aviation transportation network, are receiving more and more attention for their operational efficiency and safety. The moving targets at the airport, including aircraft, vehicles, and personnel, their movement trajectories, locations, and status are crucial to ensuring the normal takeoff and landing of flights, improving airport operational efficiency, and ensuring ground safety.

[0003] Traditional surveillance systems mainly rely on image data. In recent years, driven by deep learning, image target detection algorithms have developed rapidly. For example, target detection algorithms such as yolov8, DINO, and RT-DETR have achieved good performance on benchmark datasets such as VOC and COCO. However, these methods still face some challenges when applied to target detection in airport scenes. Dynamically changing environment: lighting changes. The lighting conditions in airport scenes will change dramatically with time and weather, from early morning to evening, from sunny to rainy days, which will affect the quality of the image and thus the performance of the target detection algorithm; weather influence. Adverse weather conditions such as rain, snow, and fog will reduce image clarity and increase noise, making it difficult for targets to be correctly identified. Target diversity: targets of different scales. The target sizes in airport scenes vary greatly, from small baggage handling vehicles, staff to large passenger aircraft. The algorithm needs to be able to effectively handle targets of different scales; posture changes. The postures of aircraft, vehicles, and pedestrians change continuously during movement, which puts higher requirements on target detection; partial occlusion: In busy airports, targets are often partially occluded by other objects, which requires the algorithm to have strong occlusion processing capabilities.

[0004] The main problems of existing target detection algorithms for airport scenes are as follows:

[0005] 1) Poor robustness: Airports are typical outdoor scenes with complex and changeable lighting and weather conditions. Existing target detection methods perform relatively poorly in low light and low visibility. In low light, rainy days, foggy days, etc., the algorithm often misses or misdetects objects. Therefore, it is necessary to improve the robustness of the algorithm under complex backgrounds and changing lighting conditions, and design a target detection algorithm that performs well in different environments and conditions.

[0006] 2) Insufficient accuracy: Existing methods based on a single data source (such as those based only on images or only on satellite positioning) find it difficult to provide high-precision target location and status information, especially in high-density target environments.

[0007] 3) Multimodal data alignment: Satellite positioning information comes from the ADS-B transponder, and image data comes from surveillance cameras. The two come from different sensors and have different information types. How to align the two types of data in a unified coordinate system and assist in target detection is an urgent problem to be solved. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes a method for monitoring active targets on airport scene based on "satellite positioning + image data". When the image data quality is poor, the position information provided by the satellite is used as prior knowledge to guide the image processing algorithm, which can effectively improve the detection accuracy.

[0009] The technical solution adopted by the present invention is: a method for monitoring targets on the airport surface based on "satellite positioning + image data", comprising:

[0010] S1. Obtain satellite positioning information of the aircraft; and obtain airport scene image data through surveillance cameras arranged on the airport scene;

[0011] S2, converting the geographic coordinates in the satellite positioning information of the aircraft into pixel coordinates in the image data;

[0012] S3, calculating the signal strength of the satellite positioning information of the aircraft;

[0013] S4, using ResNet50 as the backbone network to extract the features of the input image data, obtain the mapping features of each stage, and then merge the features through convolution and upsampling operations to obtain the feature layer {C1, C2, C3, C4, C5};

[0014] S5. After obtaining the feature layer {C1, C2, C3, C4, C5}, {C3, C4, C5} is input into the first Focus module. The output of the first Focus module is passed through a set of deep parallel convolutions to obtain {P41}.

[0015] {P41} is downsampled and concatenated with {C5} to obtain {P51};

[0016] {C2} first passes through the SPDConv module to obtain features rich in small target information, which are then downsampled and fused with the upsampled features of {P41} and {C3} to obtain {P31};

[0017] Perform feature integration on {P31} through the Integrate module to obtain {P32};

[0018] {P32, P41, P51} are obtained by capturing cross-scale information using deep parallel convolution in the second Focus module {P42};

[0019] {P42} is upsampled and fused with {P32} to obtain {P33}, and {P42} is downsampled and fused with {P51} to obtain {P52};

[0020] Finally, {P33, P42, P52} features are encoded and input into the Decoder module;

[0021] S6. Adjust the Tokens initialization in the Decoder module according to the signal strength calculated in step S3;

[0022] S7. The decoder module adjusted in step S6 outputs the category probability and bounding box coordinates of each target.

[0023] Beneficial effects of the present invention: The image data acquired by the monitoring camera in the method of the present invention provides detailed information such as the appearance, shape, texture, etc. of the target. Combined with the relatively accurate location information of the target provided by the satellite positioning data, the location and posture of the target can be further refined, thereby improving the accuracy of detection; the present invention adopts a multimodal fusion method to enhance the algorithm's adaptability to environmental changes. Even in the event of failure of a single data source, such as the quality of image data deteriorates in bad weather, the system can still rely on another data source to maintain a high detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of the present invention;

[0025] Figure 2 It is a multi-scale feature fusion structure diagram of the present invention;

[0026] Figure 3 It is a structure diagram of the Decoder of the present invention;

[0027] Figure 4 This is the detection result of the airport surface moving target. DETAILED DESCRIPTION

[0028] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0029] In recent years, with the development and maturity of satellite positioning technology and the continuous expansion of its application scope, satellite positioning technology can provide new solutions for monitoring active targets on airport surfaces. By integrating the precise location information provided by satellite positioning with the target features obtained by image data analysis, it can not only effectively improve the recognition accuracy and tracking continuity of active targets, but also enhance the robustness and intelligence level of the entire monitoring system.

[0030] In this context, the present invention proposes a target detection algorithm based on satellite positioning data and image data, aiming to improve detection accuracy and robustness. The method preprocesses the satellite positioning data and image data, uses a neural network to align, interact and fuse the data from different sources, and performs target detection on the fused features. Satellite positioning data provides precise positioning information of the target, while image data provides rich visual context. By combining satellite positioning data and image data, the complementary advantages of the two can be fully utilized to improve the accuracy and robustness of target detection. This multimodal fusion method can not only effectively cope with the complex environment in the airport scene, but also maintain good performance under various lighting and weather conditions. In addition, the method can also improve the detection capability of occluded targets.

[0031] Compared with image data, satellite positioning information, such as ADS-B transponders, provides the coordinate information of the aircraft. Even in low-light nights or in rainy and foggy days with low visibility, ADS-B information can provide relatively accurate aircraft location information. The present invention proposes an airport target detection algorithm that combines satellite positioning information and image information, aiming to improve the accuracy of airport scene activity target detection, enhance the robustness of the detection algorithm, and obtain better detection results. The specific technical solution includes the following steps, such as Figure 1 shown.

[0032] Step 1: Acquisition and analysis of satellite positioning information and image information

[0033] In the present invention, the satellite positioning information of the aircraft is obtained through ADS-B information. By using an ADS-B receiver to capture the ADS-B signal, the position information of the aircraft can be decoded from it. ADS-B data is usually provided in JSON or CSV format, including the latitude and longitude, altitude, speed, heading and other information of the aircraft. Image data is obtained through surveillance cameras deployed on the airport scene.

[0034] Step 2: Conversion between geographic coordinates and image coordinates

[0035] The images captured by the camera are based on the pixel coordinate system (x, y), while the location information in the ADS-B data is based on the WGS84 coordinate system (latitude and longitude). Therefore, the geographic coordinates in the ADS-B data need to be converted into pixel coordinates on the image to match the aircraft with its corresponding ADS-B information.

[0036] Camera calibration: Intrinsic calibration refers to the use of a chessboard or other calibration objects to calibrate the camera's internal parameters, obtaining the camera's focal length, principal point coordinates, distortion coefficient and other parameters. External calibration refers to determining the camera's absolute position and attitude (rotation and translation matrix) relative to the airport surface, which is achieved by placing calibration points at known positions and performing calibration.

[0037] Establish coordinate mapping relationship: Use perspective projection model to project points in the 3D world coordinate system onto the 2D image plane. Assuming that the point in the world coordinate system is (X, Y, Z) and the point in the image coordinate system is (u, v), the projection formula is shown in equation (1), where K is the intrinsic parameter matrix of the camera, and [R|t] is the extrinsic parameter matrix, which includes the rotation matrix R and the translation vector t.

[0038]

[0039] Step 3: ADS-B information strength estimation

[0040] Estimating the strength of ADS-B information is very important for airport surface active target monitoring systems. The strength of the ADS-B signal can directly reflect the distance between the aircraft and the camera. The higher the signal strength, the closer the aircraft is to the camera; conversely, the lower the signal strength, the aircraft may be farther away from the camera. This information is very useful for dynamically adjusting the target detection strategy. By estimating the signal strength, the bounding box of the local range can be dynamically initialized and adjusted. For aircraft with high signal strength (i.e., aircraft at a closer distance), a larger bounding box can be used for accurate tracking; while for aircraft with low signal strength (i.e., aircraft at a farther distance), a smaller bounding box can be used.

[0041] The coordinates of the aircraft are still obtained from the ADS-B data carried by the aircraft, and the position and altitude are recorded (X aircraft ,Y aircraft ,H aircraft ); the exact position and height of the camera on the airport surface (X camera ,Y camera ,H camera ), and use the Euclidean distance formula (2) to calculate the approximate distance d between the aircraft and the camera.

[0042]

[0043] The attenuation of the radio signal in free space is estimated by the free space path loss model (FSPL) formula (3), where d is the distance (unit: meter) calculated by formula (2), f is the signal frequency (unit: Hz), the frequency of the ADS-B signal is usually 1090 MHz, and c is the speed of light, which is approximately 3*10 8 Meters / second:

[0044] FSPL(dB)=20log 10 (d)+20log 10 (f)+20log 10 (4Π / c) (3)

[0045] After obtaining the signal attenuation strength, the received ADS-B signal strength can be calculated. In formula (4), Received Power is the received signal strength, Transmit Power is the transmit power, the ADS-B transmit power is usually 250 mW (about 24 dBm), and FSPL (dB) is the attenuation strength.

[0046] Received Power(dBm)=Transmit Power(dBm)-FSPL(dB)(4)

[0047] Step 4: Image data preprocessing

[0048] Before feeding the image data into the backbone network of the target detection model, it is crucial to perform effective preprocessing. The preprocessing step can not only improve the quality of the image, but also enhance the robustness and detection accuracy of the model. The lighting conditions of the airport scene may vary due to factors such as time and weather, resulting in a decrease in image quality. For images with too low illumination, an adaptive histogram equalization method is used to perform histogram equalization in local areas to avoid the noise amplification problem caused by global equalization. Under severe weather conditions such as haze, rain and snow, the image may be affected by fog, resulting in image blur and loss of details. Dehazing can significantly improve the clarity and visibility of the image. Therefore, for this part of the data, the dark channel prior method is used to restore a clear image by estimating the transmittance and atmospheric light intensity based on the atmospheric scattering model.

[0049] Step 5: Multi-scale feature selection, fusion and encoding

[0050] ResNet50 is used as the backbone network to extract features, obtain mapping features at each stage, and then {C1, C2, C3, C4, C5} are obtained by merging features through convolution and upsampling operations. In the target detection task, the detection of small targets is often more challenging than that of large targets. Due to the wide monitoring range of the airport, pedestrians and vehicles are relatively small in size compared to aircraft, and distant aircraft also appear small in the image, so there are a large number of small active targets in the airport scene. The representation of small targets on the feature map is often not obvious enough, especially on high-level feature maps, the features extracted by the deep network may be too abstract, resulting in the loss of key details of small targets. Even the {P3} feature layer is slightly difficult to detect small targets on this, so we consider adding the {P2} feature layer to improve small target detection, but directly adding the {P2} detection layer will lead to problems such as excessive calculation and more time-consuming processing. For this reason, we designed a new feature selection and fusion structure.

[0051] After obtaining the feature layer {C1, C2, C3, C4, C5}, {C3, C4, C5} first passes through the Focus module (i.e. Figure 3 The feature aggregation module in the image processing module captures higher-level information across multiple scales through a set of deep parallel convolutions to obtain {P41}. {P41} is downsampled and concatenated with {C5} to obtain {P51}. {C2} first passes through the SPDConv module to obtain features rich in small target information, which are then downsampled and fused with the features adopted from {P41} and the {C3} feature layer to obtain {P31}. The {P31} feature layer is then integrated through the Integrate module to obtain {P32}. {P32, P41, and P51} are then captured through the Focus module using deep parallel convolutions to obtain {P42}. {P42} is upsampled and fused with {P32} to obtain {P33}, which is then downsampled and fused with {P51} to obtain {P52}. Finally, {P33, P42, and P52} features are encoded and input into the Decoder module.

[0052] Step 6: Initialize ADS-B signal adjustment tokens

[0053] Token initialization in the decoder is a key step. In the traditional decoder framework, in order to detect every area fairly without missing any, tokens are usually evenly distributed throughout the image. By combining the information of the ADS-B signal, we can initialize tokens more intelligently, thereby improving the accuracy and efficiency of detection. Specifically, during the initialization stage of tokens, we can use the aircraft position information provided by the ADS-B signal to adjust the initialization of some tokens so that they are concentrated in the area where the ADS-B signal appears, while other tokens remain evenly distributed.

[0054] Adjust token initialization based on signal strength: Determine the impact range of each aircraft based on the ADS-B signal strength, select a portion of tokens, and adjust the initialization position of these tokens based on the impact range of each aircraft. Aircraft with higher signal strength should have a larger impact range, and aircraft with lower signal strength should have a smaller impact range.

[0055] Initialize Points: Image size H*W, total number of Tokens N, first generate evenly distributed points init (5):

[0056] points init ={(x j ,y j ,)~Uniform(0,W)*Uniform(0,H)|j=1,2,…N} (5)

[0057] Determine the impact range of the ADS-B signal: Assume that there are M ADS-B signals in the image area, and the coordinates of each ADS-B signal on the image after coordinate transformation are (u i ,v i ), the intensity is s i ,i=1,2,…M(Signal strength is calculated by formula (4)) Determine the influence range of each ADS-B signal Δ i The influence range can be adjusted according to the signal strength, as shown in formula (6), where Δ and β are adjustment parameters, and α controls the signal strength s i Impact range Δ i The contribution ratio of α is 1, and β is 120. A larger α value means that a stronger signal will correspond to a wider impact range. β is an offset that ensures that even the weakest signal has a certain minimum impact range, which is necessary to maintain a certain monitoring capability when the signal fails completely.

[0058] Δ i =α·s i +β (6)

[0059] Adjust Points distribution: For the i-th ADS-B signal, select the points within its influence range for distribution adjustment. After the i-th ADS-B signal is transformed, the coordinates on the image are (u i ,v i ), the affected area is (u i -Δ i ,u i +Δ i )×(v i -Δ i ,v i +Δ i ), and use a two-dimensional Gaussian distribution to redistribute the positions of these points within the influence range, as shown in formula (7), where σ is the standard deviation of the Gaussian distribution, which determines the width of the distribution and is related to the influence range Δ i proportional to, as shown in formula (8), where k is a proportional factor less than 1:

[0060] x j ~Ν(ui,σ 2 );y j ~Ν(v i ,σ 2 ) (7)

[0061] σ=k·Δ i (8)

[0062] In this embodiment, when the value of k is between 0.25 and 0.5, the final recognition accuracy can reach above 0.66. In practical applications, the value of k can also be adjusted according to actual needs.

[0063] Merge Points: Merge the adjusted Points with the unaffected Points to get the final initialization points final , as shown in formula (9).

[0064] points final ={(x j ,y j )|j=1,2,…,N} (9)

[0065] Step 7: Adjusted Points enter the decoder and subsequent processing

[0066] Since the self-attention mechanism itself has no information about the order of elements, position encoding is needed to preserve the order information of elements in the sequence. Position encoding is a fixed function added to each input embedding vector, usually a combination of sine and cosine wave functions, which can provide the model with absolute and relative position information of each position. Before sending the adjusted Points to the Decoder, such as Figure 3 As shown, corresponding position codes need to be added to these Points.

[0067] The decoder is composed of multiple identical layers stacked together, each of which contains a self-attention mechanism, a cross-attention mechanism, and a feedforward network. The final output of the decoder is mapped to the output space of the target detection task through a linear layer. The final output includes the category probability and bounding box coordinates of each target, see Figure 4 . Figure 4 The meaning of the numbers in the box is as follows: the number 0 or 1 in front represents the category, 0 is the aircraft category, 1 is the vehicle category, and the decimal behind it represents the confidence level.

[0068] Traditional target detection methods mainly rely on image data and are easily affected by environmental factors such as illumination changes, occlusion, noise, etc. By introducing satellite positioning data, the present invention can use the location information provided by the satellite as prior knowledge to guide the image processing algorithm when the image data quality is poor. Satellite positioning data provides the approximate location of the target, so that the image processing algorithm can focus on these areas, reduce false detection and missed detection, and thus improve the accuracy of detection.

[0069] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for monitoring targets on airport surface activity based on "satellite positioning + image data", characterized in that: include: S1. Obtain satellite positioning information of the aircraft; and obtain airport scene image data through surveillance cameras arranged on the airport scene; S2, converting the geographic coordinates in the satellite positioning information of the aircraft into pixel coordinates in the image data; S3, calculating the signal strength of the satellite positioning information of the aircraft; S4, using ResNet50 as the backbone network to extract the features of the input image data, obtain the mapping features of each stage, and then merge the features through convolution and upsampling operations to obtain the feature layer {C1, C2, C3, C4, C5}; S5. After obtaining the feature layer {C1, C2, C3, C4, C5}, {C3, C4, C5} is input into the first Focus module. The output of the first Focus module is passed through a set of deep parallel convolutions to obtain {P41}. {P41} is downsampled and concatenated with {C5} to obtain {P51}; {C2} first passes through the SPDConv module to obtain features rich in small target information, which are then downsampled and fused with the upsampled features of {P41} and {C3} to obtain {P31}; Perform feature integration on {P31} through the Integrate module to obtain {P32}; {P32, P41, P51} are input into the second Focus module. The output of the second Focus module is subjected to deep parallel convolution to capture cross-scale information and obtain {P42}; {P42} is upsampled and fused with {P32} to obtain {P33}, and {P42} is downsampled and fused with {P51} to obtain {P52}; Finally, {P33, P42, P52} features are encoded and input into the Decoder module; S6. Initialize Tokens according to the signal strength calculated in step S3; S7. The Decoder module outputs the category probability of each target based on the input {P33, P42, P52} feature encoding, and the Decoder module outputs the corresponding bounding box coordinates based on the Tokens initialized in step S6.

2. The method for monitoring airport surface moving targets based on "satellite positioning + image data" according to claim 1 is characterized in that: The specific implementation process of step S3 is: S31, calculating the distance between the aircraft and the camera according to the Euclidean distance; S32, based on the distance calculated in step S31, estimating the attenuation of the radio signal in free space by using a free space path loss model; S33. Based on the attenuation of the radio signal in free space estimated in step S32, calculate the signal strength of the satellite positioning information of the aircraft.

3. The method for monitoring airport surface moving targets based on "satellite positioning + image data" according to claim 2 is characterized in that: The implementation process of step S6 is: S61, determining the influence range of each radio signal on the image data according to the signal strength; S62, obtaining an influence area of ​​the radio signal according to the pixel coordinates of the radio signal in the image data and its influence range; S63. Select Points within the affected area for distribution adjustment, thereby completing the initialization of Tokens.

4. The method for monitoring airport surface moving targets based on "satellite positioning + image data" according to claim 3 is characterized in that: Step S63 specifically includes: using a two-dimensional Gaussian distribution to redistribute and match the positions of these Points within the influence range.

5. The method for monitoring airport surface moving targets based on "satellite positioning + image data" according to claim 4 is characterized in that: The calculation formula for the standard deviation σ of the Gaussian distribution is: σ=k·Δ i Where k is the scaling factor, Δ i Indicates the scope of influence.

6. The method for monitoring airport surface moving targets based on "satellite positioning + image data" according to claim 5 is characterized in that: After merging the adjusted Points with the unaffected Points, add the corresponding position code and then input them into the Decoder module.

Images