An Airport Target Detection and Tracking System and Method Based on High-Definition Array Cameras

Abstract

This invention discloses an airport target detection and tracking system and method based on a high-definition array camera. The system includes obtaining the local ID and bounding box of each target using real-time target detection and tracking algorithms from both the panoramic view of the array camera and the sub-camera views; recording the timestamp and target latitude and longitude information (i.e., the spatiotemporal information of the target) of the corresponding frame; using an ADS-B system to obtain complete information about the aircraft target at that latitude and longitude, achieving matching between the local ID and the global ID; comparing the target ID with the previous frame's aircraft target ID to identify missing aircraft targets; determining whether the target's disappearance is abnormal based on the disappearance area and ADS-B system information; when a target disappears abnormally, querying the detection results of the sub-cameras, transforming the bounding box in the sub-camera image coordinate system to the bounding box in the panoramic image coordinate system of the array camera, performing matching and restoration, and redrawing and outputting the restored detection box. This invention improves the accuracy and stability of real-time airport target detection and tracking through multi-camera collaborative work and multi-information fusion.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to an airport target detection and tracking system and method based on a high-definition array camera. Background Technology

[0002] With the rapid development of the aviation industry, airport safety management and operational efficiency face increasing challenges. Traditional airport monitoring systems mainly rely on video streams from single or a small number of cameras. These systems have significant limitations in areas such as large-scale monitoring, small target detection, and multi-target tracking. For example, a single camera has limited coverage, making it difficult to achieve comprehensive monitoring of the entire airport; in long-distance monitoring, targets such as aircraft appear as small objects in images, and traditional target detection algorithms are prone to missed or false detections; airport environments contain numerous moving targets, and accurately tracking multiple targets in complex backgrounds while maintaining consistent target IDs is an extremely challenging task; furthermore, targets may suddenly disappear due to occlusion, signal loss, or other reasons, and quickly and accurately recovering the tracking of these targets is crucial for improving system robustness. Summary of the Invention

[0003] To address the aforementioned issues, this invention discloses an airport target detection and tracking system and method based on a high-definition array camera. Through multi-camera collaborative operation, target detection and tracking algorithm optimization, geographic coordinate transformation and information fusion, and anomaly handling mechanisms, it achieves efficient and stable detection and tracking of targets within the airport.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] An airport target detection and tracking system based on a high-definition array camera includes a high-definition array camera image acquisition system, an ADS-B system (Automatic Dependent Surveillance-Broadcast), a target detection and tracking system, a coordinate matching system with the camera, and a discrimination and correction system.

[0006] A design method for an airport target detection and tracking system based on a high-definition array camera includes the following steps:

[0007] (1) The target detection and tracking system applicable to the panoramic image of the array camera and the target detection and tracking system applicable to the sub-camera image operate simultaneously. After the panoramic image frame of the array camera passes through the target detection system, target tracking is further performed to obtain the ID and bounding box of each target in the image.

[0008] (2) The calibration box is converted into geographic coordinates, i.e. latitude and longitude, through the conversion module, and the complete information of the aircraft target located at that latitude and longitude is obtained through the ADS-B system;

[0009] (3) By comparing with the ID of the aircraft target in the previous frame, the ID of the disappeared target is obtained, and the target is judged to be abnormally disappeared based on the image area and latitude and longitude.

[0010] (4) When the target disappears abnormally, the detection box of the target needs to be restored. The detection calibration box of the sub-camera at the same time as the target is found, and the calibration box in the sub-camera image coordinate system is transformed into the calibration box in the array camera panoramic image coordinate system.

[0011] (5) The calibration boxes obtained by transformation in the sub-camera are matched with the calibration boxes obtained by panoramic image detection of the array camera through a matching algorithm. The unmatched calibration boxes are designated as detection boxes that need to be restored, and the restored detection boxes are output.

[0012] Furthermore, step (2) of the ADS-B system information acquisition and database design includes:

[0013] The ADS-B system (Automatic Dependent Surveillance-Broadcast) can acquire real-time spatiotemporal information of the aircraft (longitude, latitude, speed, and timestamp), other possible supplementary information (conflict warning information, pilot input information, track angle, route inflection points, etc.), and aircraft identification and category information. In addition, it may include other supplementary information such as heading, airspeed, wind speed, wind direction, and outside temperature.

[0014] Furthermore, some of the information obtained by the ADS-B system in step (2) needs to be stored in a database for later use. Meanwhile, to meet the requirements of the target tracking system design methodology, the key fields involved in the airport target tracking database include:

[0015] Global ID: Used to uniquely identify an aircraft target throughout the entire airport; typically, it uses the aircraft's identification information.

[0016] Latitude and longitude: Geographic coordinates of the aircraft target, obtained from the ADS-B system;

[0017] Local ID: Generated by the target detection and tracking system, it is the ID of the target in the image;

[0018] Pixel coordinates: The pixel coordinates of the aircraft target in the image, determined by the bounding box. The bounding box contains the pixel coordinates of the upper left and lower right corners of the aircraft target and is obtained by the target detection and tracking system.

[0019] Whether it is in a monitorable area: This is used to determine whether the aircraft target is in the monitorable area of ​​the array camera. It is determined by whether the aircraft's latitude and longitude are within the latitude and longitude range corresponding to the monitorable area.

[0020] Timestamp: Used to identify the time when information belongs.

[0021] The ADS-B system stores the global ID, latitude and longitude information, whether it is in a monitorable area, and timestamp fields in the database. Subsequently, by matching the global ID and local ID, the local ID and pixel coordinate fields are filled in.

[0022] Furthermore, the matching process between the global ID and the local ID includes:

[0023] The local ID and the bounding box are linked through an image coordinate to geographic coordinate conversion module. The bounding box is converted into the corresponding latitude and longitude region. The database is queried to obtain the ADS-B information and global ID of the aircraft target located in the region. Through these steps, the local ID and the global ID are bound together.

[0024] Furthermore, the determination process for the abnormal disappearance of the target in step (3) includes:

[0025] By comparing the global IDs of all targets obtained in the current frame with the global IDs of all targets obtained in the previous frame, the targets that disappeared in the current frame can be identified.

[0026] There are two ways to determine whether a target has disappeared abnormally. First, the surveillance camera footage at the airport is relatively open with little obstruction. The aircraft target can only disappear in the boundary area of ​​the image or be obstructed. By marking the boundary area and the normally obstructed area, a normal disappearance area can be identified, while the disappearance in other areas is abnormal.

[0027] On the other hand, the information obtained by ADS-B can directly indicate the location of the aircraft and determine whether the aircraft is within the monitoring range. If it is no longer within the monitoring range, it is considered to have disappeared normally; otherwise, it is considered to have disappeared abnormally.

[0028] The above two methods are used to determine whether the target has disappeared abnormally. If either method indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance is the target considered to have disappeared normally.

[0029] Furthermore, the abnormal disappearance target bounding box recovery process in step (4) includes:

[0030] To calculate the correspondence between the sub-camera view and the panoramic view of the array camera, the sub-camera and the array camera first need to be calibrated to obtain their internal and external parameters.

[0031] The purpose of camera calibration is to determine the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. The intrinsic parameter matrix K includes the focal length f. x and f y and the principal point coordinates cx and c y The format is as follows: The extrinsic parameter matrix [R|t] includes the rotation matrix R and the translation vector t, in the form of [R|t] = [R t]. These parameters can be solved using a calibration board and calibration algorithm, providing a foundation for subsequent image processing.

[0032] Feature points are extracted from the sub-camera view and the panoramic view of the array camera, and then matched. The purpose of feature point detection is to find points with uniqueness and stability in an image. Common feature point detection methods include Harris corner detection, SIFT, SURF, and ORB. The descriptor of a feature point is usually a high-dimensional vector used to describe the local information around the feature point.

[0033] Feature point matching finds the most similar point pairs by comparing the descriptors of feature points in different images. Common matching methods include nearest neighbor matching and ratio testing. Assume feature point p... i and q i In the images from the sub-camera and the panoramic camera, their feature descriptors are respectively: and di q The matching process can then be represented as: Where m represents a matching point pair. The above formula means that for each feature point p in the sub-camera image... i Find the feature point q in the panoramic image of the array camera that is most similar to its descriptor. i Specifically, the distance between each feature point descriptor (such as Euclidean distance or Hamming distance) is calculated, and the point pair with the smallest distance is selected as the matching point.

[0034] The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes. Assume there are n pairs of matching feature points ((x...)) in the sub-camera image. i ,y i ),(x′ i ,y′ i )), where (x i ,y i ) and (x′ i ,y′ i Let x' and y' represent the coordinates of points in the sub-camera and panoramic camera, respectively. The homography matrix H can be solved by the following equation: (x') i y′ i 1)=H(x i y i 1) Specifically, a system of linear equations can be constructed: By solving the above system of linear equations, the elements (h) of the homography matrix (H) can be obtained. 11,h 12 ,…,h 32 ), the last element h 33 It is usually set to 1.

[0035] For each pixel coordinate (x′, y′) in the array camera image, use the homography matrix H of each sub-camera. i Perform an inverse transformation to find the corresponding sub-camera coordinates (x) i ,y i The inverse transform formula is: The final coordinates are: Check the coordinates (x) after the inverse transformation i ,y i Whether it is within the effective range of the sub-camera (i.e., whether it is within the image boundary of the sub-camera). If it is within the effective range, then the pixel coordinate corresponds to sub-camera i.

[0036] With the homography matrix H, perspective transformation can be performed on the sub-camera images, mapping them onto the panoramic image of the array camera. The formula for perspective transformation is: [x′ y′ w]=H[xy 1], where (x,y) are the coordinates of a point in the sub-camera image, (x′,y′) are the coordinates of the transformed point in the panoramic image of the array camera, and w is the normalization factor. The final coordinates are:

[0037] Using the above expression, a set of bounding boxes in the sub-camera image coordinates can be converted into bounding boxes in the panoramic image coordinates of the array camera. These converted bounding boxes are then matched with all the bounding boxes obtained by the array camera panoramic image target detection algorithm using the Hungarian algorithm.

[0038] Before matching, it is necessary to determine the metric for evaluating the similarity between two bounding boxes. DIoU adds the distance between the center points of the bounding boxes to IoU, thus measuring the spatial proximity of the two boxes while considering overlapping areas. Its formula is defined as follows:

[0039] Wherein, IoU is the intersection-union ratio of two bounding boxes, calculated using the following formula:

[0040] Area intersection This represents the area of ​​the intersection of two detection boxes, while Area... union This represents the area of ​​their union. d is the Euclidean distance between the center points of the two boxes, and c is the minimum diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle containing the two boxes that can be formed.

[0041] After calculating the DIoU between all bounding boxes, the Hungarian algorithm is applied for matching. The Hungarian algorithm is a classic method for solving assignment problems, used to find the globally optimal or near-optimal matching scheme while ensuring that each pair of objects matches at most once. In this scenario, the specific implementation steps of the Hungarian algorithm include:

[0042] Step 1: Construct the cost matrix: Using DIoU as the matching cost, construct an n×m cost matrix (n is the number of calibration boxes after the sub-camera transformation, and m is the number of calibration boxes obtained by the object detection algorithm). Each element in the matrix represents the DIoU value of the corresponding box pair.

[0043] Step 2: Finding an augmenting path: By continuously selecting unmatched points and trying to find the matching point with the lowest cost, an augmenting path is formed until all points are matched or no better match can be found.

[0044] Step 3: Update matching status and cost: Adjust the matching status based on the augmenting path, and may rematch some already matched points to reduce the total cost.

[0045] Step 4: Repeat steps 2 and 3 until no new augmentation path can be found.

[0046] Step 5: The matching result is the one-to-one correspondence between the converted bounding boxes and the bounding boxes obtained by the object detection algorithm. Boxes that fail to match using the Hungarian algorithm in the converted bounding boxes are those that need to be restored. The coordinate boxes and their corresponding local IDs are then redrawn on the camera array image and output.

[0047] The beneficial effects of this invention are as follows:

[0048] This invention discloses a design method for an airport target detection and tracking system based on a high-definition array camera. By combining ADS-B system information and manual calibration information to determine the target detection effect, and by coordinating the panoramic view of the array camera and the view of the sub-cameras, the accuracy and stability of airport target detection and tracking are improved. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the airport target tracking system and method based on a high-definition array camera provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram illustrating how the target detection box is recovered using the detection results of a sub-camera when the target disappears abnormally, as provided in an embodiment of the present invention.

[0051] Figure 3This is a schematic diagram illustrating the process of restoring the panoramic camera image based on the sub-camera detection results provided in an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] like Figure 1 As shown, the airport target tracking system and method based on a high-definition array camera of the present invention includes the following steps:

[0054] S1. The target detection system for array camera footage and the target detection and tracking system for sub-camera footage operate simultaneously.

[0055] A target detection and tracking system suitable for panoramic images from array cameras and a target detection system suitable for sub-camera images are run simultaneously. These two algorithms are obtained by capturing enough panoramic images from array cameras and sub-camera images, labeling aircraft targets, and then training the target detection network.

[0056] The panoramic frame images acquired by the array camera are processed by the target detection algorithm to obtain the bounding boxes of the aircraft target. The bounding boxes contain the pixel coordinates of the upper left and lower right corners of the aircraft target, specifying the position of the target in the image. The target tracking algorithm assigns an ID to each bounding box. The same ID is considered to be the same target. This ID is also called the local ID.

[0057] The frame images acquired by the sub-camera can be used by the target detection system to obtain the bounding boxes of the aircraft target. The bounding boxes contain the pixel coordinates of the upper left and lower right corners of the aircraft target, specifying the position of the target in the image. These bounding boxes, timestamps, and the number of rows and columns of the sub-camera in the camera array are stored together for subsequent recovery of the detection boxes of abnormally disappeared targets.

[0058] S2. The calibration box is converted into geographic coordinates, i.e., latitude and longitude, through the conversion module, and the complete information of the aircraft target located at that latitude and longitude is obtained through the ADS-B system;

[0059] The ADS-B system (Automatic Dependent Surveillance-Broadcast) can acquire real-time spatiotemporal information of the aircraft (longitude, latitude, speed, and timestamp), other possible supplementary information (conflict warning information, pilot input information, track angle, route inflection points, etc.), and aircraft identification and category information. In addition, it may include other supplementary information such as heading, airspeed, wind speed, wind direction, and outside temperature.

[0060] Furthermore, some of the information acquired by the ADS-B system needs to be stored in a database for later use. Additionally, to meet the requirements of the target tracking system design methodology, the key fields involved in the airport target tracking database include:

[0061] Global ID: Used to uniquely identify an aircraft target throughout the entire airport; typically, it uses the aircraft's identification information.

[0062] Latitude and longitude: Geographic coordinates of the aircraft target, obtained from the ADS-B system;

[0063] Local ID: Generated by the target detection and tracking system, it is the ID of the target in the image;

[0064] Pixel coordinates: The pixel coordinates of the aircraft target in the image, determined by the bounding box. The bounding box contains the pixel coordinates of the upper left and lower right corners of the aircraft target and is obtained by the target detection and tracking system.

[0065] Whether it is in a monitorable area: This is used to determine whether the aircraft target is in the monitorable area of ​​the array camera. It is determined by whether the aircraft's latitude and longitude are within the latitude and longitude range corresponding to the monitorable area.

[0066] Timestamp: Used to identify the time when information belongs.

[0067] The ADS-B system stores the global ID, latitude and longitude information, whether it is in a monitorable area, and timestamp fields in the database. Subsequently, by matching the global ID and local ID, the local ID and pixel coordinate fields are filled in.

[0068] The target detection system obtains local IDs and bounding boxes. Through an image coordinate to geographic coordinate conversion module, the bounding boxes are converted into corresponding latitude and longitude regions. The database is queried to obtain the ADS-B information and global ID of aircraft targets located in that region. Through these steps, the local IDs and global IDs are bound together.

[0069] S3. By comparing with the ID of the aircraft target in the previous frame, obtain the ID of the disappeared target, and determine whether the target disappeared abnormally based on two methods: image region and latitude and longitude.

[0070] By comparing the global IDs of all targets obtained in the current frame with the global IDs of all targets obtained in the previous frame, the targets that disappeared in the current frame can be identified.

[0071] There are two ways to determine whether a target has disappeared abnormally. First, the surveillance camera footage at the airport is relatively open with little obstruction. The aircraft target can only disappear in the boundary area of ​​the image or be obstructed. By marking the boundary area and the normally obstructed area, a normal disappearance area can be identified, while the disappearance in other areas is abnormal.

[0072] On the other hand, the information obtained by ADS-B can directly indicate the location of the aircraft and determine whether the aircraft is within the monitoring range. If it is no longer within the monitoring range, it is considered to have disappeared normally; otherwise, it is considered to have disappeared abnormally.

[0073] The above two methods are used to determine whether the target has disappeared abnormally. If either method indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance is the target considered to have disappeared normally.

[0074] S4, such as Figure 2 As shown, when a target disappears abnormally, its detection bounding box needs to be restored. First, it is necessary to determine which sub-camera corresponds to the disappeared target. Then, the detection results (i.e., the calibration boxes) of the sub-cameras that were present at the same time as the target are retrieved. The calibration boxes in the sub-camera image coordinate system are transformed into calibration boxes in the panoramic image coordinate system of the array camera. The transformed calibration boxes obtained in the sub-camera are then matched with the calibration boxes detected in the panoramic frame image of the array camera using a matching algorithm. The transformed calibration boxes that do not match are the detection boxes that need to be restored. Once the detection boxes are restored, they can be output.

[0075] To calculate the correspondence between the sub-camera view and the panoramic view of the array camera, the sub-camera and the array camera first need to be calibrated to obtain their internal and external parameters.

[0076] The purpose of camera calibration is to determine the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. The intrinsic parameter matrix K includes the focal length f. x and f y and the principal point coordinates c x and c y The format is as follows: The extrinsic parameter matrix [R|t] includes the rotation matrix R and the translation vector t, in the form of [R|t] = [R t]. These parameters can be solved using a calibration board and calibration algorithm, providing a foundation for subsequent image processing.

[0077] Feature points are extracted from the sub-camera view and the panoramic view of the array camera, and then matched. The purpose of feature point detection is to find points with uniqueness and stability in an image. Common feature point detection methods include Harris corner detection, SIFT, SURF, and ORB. The descriptor of a feature point is usually a high-dimensional vector used to describe the local information around the feature point.

[0078] Feature point matching finds the most similar point pairs by comparing the descriptors of feature points in different images. Common matching methods include nearest neighbor matching and ratio testing. Assume feature point p... i and q i In the images from the sub-camera and the array camera, their feature descriptors are respectively: and di q The matching process can then be represented as: Where m represents a pair of matching points. The above formula means that for each feature point p in the sub-camera... i Find the feature point q in the panoramic image of the array camera that is most similar to its descriptor. i Specifically, the distance between each feature point descriptor (such as Euclidean distance or Hamming distance) is calculated, and the point pair with the smallest distance is selected as the matching point.

[0079] The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes. Assume there are n pairs of matching feature points ((x...)) in the sub-camera frame. i ,y i ),(x′ i ,y′ i )), where (x i ,y i ) and (x′ i ,y′ i (x′) represent the point coordinates in the sub-camera view and the panoramic view of the array camera, respectively. The homography matrix H can be solved by the following equation: (x′) i y′ i 1)=H(x i y i 1) Specifically, a system of linear equations can be constructed: By solving the above system of linear equations, the elements (h) of the homography matrix H can be obtained. 11 ,h 12 ,…,h 32 ), the last element h 33 It is usually set to 1.

[0080] For each pixel coordinate (x′, y′) in the panoramic view of the array camera, use the homography matrix H of each sub-camera. iPerform an inverse transform to find the corresponding sub-camera frame coordinates (x... i ,y i The inverse transform formula is: The final coordinates are: Check the coordinates (x) after the inverse transformation i ,y i Is the pixel coordinate within the valid range of the sub-camera (i.e., within the image boundary of the sub-camera)? If it is within the valid range, then the pixel coordinate corresponds to sub-camera i.

[0081] With the homography matrix H, perspective transformation can be performed on the sub-camera's image, mapping it to the panoramic image of the array camera. The formula for perspective transformation is: [x′ y′ w]=H[xy 1], where (x,y) are the coordinates of a point in the sub-camera's image, (x′,y′) are the coordinates of the transformed point in the array camera's panoramic image, and w is the normalization factor. The final coordinates are:

[0082] Using the above expression, a set of bounding boxes in the sub-camera image coordinates can be converted into bounding boxes in the panoramic image coordinates of the array camera. These converted bounding boxes are then matched with all the bounding boxes obtained by the array camera panoramic image target detection algorithm using the Hungarian algorithm.

[0083] Before matching, it is necessary to determine the metric for evaluating the similarity between two bounding boxes. DIoU adds the distance between the center points of the bounding boxes to IoU, thus measuring the spatial proximity of the two boxes while considering overlapping areas. Its formula is defined as follows: Wherein, IoU is the intersection-union ratio of two bounding boxes, calculated using the following formula:

[0084] Area intersection This represents the area of ​​the intersection of two detection boxes, while Area... union This represents the area of ​​their union. d is the Euclidean distance between the center points of the two boxes, and c is the minimum diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle containing the two boxes that can be formed.

[0085] After calculating the DIoU between all bounding boxes, the Hungarian algorithm is applied for matching. The Hungarian algorithm is a classic method for solving assignment problems, used to find the globally optimal or near-optimal matching scheme while ensuring that each pair of objects matches at most once. In this scenario, the specific implementation steps of the Hungarian algorithm include:

[0086] Step 1: Construct the cost matrix: Using DIoU as the matching cost, construct an n×m cost matrix (n is the number of calibration boxes after the sub-camera transformation, and m is the number of calibration boxes obtained by the object detection algorithm). Each element in the matrix represents the DIoU value of the corresponding box pair.

[0087] Step 2: Finding an augmenting path: By continuously selecting unmatched points and trying to find the matching point with the lowest cost, an augmenting path is formed until all points are matched or no better match can be found.

[0088] Step 3: Update matching status and cost: Adjust the matching status based on the augmenting path, and may rematch some already matched points to reduce the total cost.

[0089] Step 4: Repeat steps 2 and 3 until no new augmentation path can be found.

[0090] Step 5: The matching result obtained is the one-to-one correspondence between the converted bounding boxes and the bounding boxes obtained by the object detection algorithm. The bounding boxes that fail to match using the Hungarian algorithm in the converted bounding boxes are the boxes that need to be restored. The coordinate boxes and their corresponding local IDs are then redrawn on the panoramic image of the array camera and output; for example... Figure 3 As shown.

[0091] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for airport target detection and tracking based on a high-definition array camera, characterized in that: The system includes an airport target detection and tracking system based on a high-definition array camera. The system includes a high-definition array camera image acquisition system, an ADS-B system, a target detection and tracking system, a pixel coordinate matching system with the camera, and a discrimination and correction system. The method includes the following steps: (1) The target detection and tracking system applicable to the panoramic image of the array camera and the target detection and tracking system applicable to the sub-camera image operate simultaneously. After the panoramic image frame of the array camera passes through the target detection system, target tracking is further performed to obtain the ID and bounding box of each target in the image. (2) The calibration box is converted into geographic coordinates, i.e. latitude and longitude, through the conversion module, and the complete information of the aircraft target located at that latitude and longitude is obtained through the ADS-B system; Some of the information acquired by the ADS-B system needs to be stored in a database for later use. Additionally, to meet the requirements of the target tracking system design methodology, the key fields involved in the airport target tracking database include: Global ID: Used to uniquely identify an aircraft target throughout the entire airport; typically, it uses the aircraft's identification information. Latitude and longitude: Geographic coordinates of the aircraft target, obtained from the ADS-B system; Local ID: Generated by the target detection and tracking system, it is the ID of the target in the image; Pixel coordinates: The pixel coordinates of the aircraft target in the image, determined by the bounding box, which includes the pixel coordinates of the upper left and lower right corners of the aircraft target, and is obtained by the target detection and tracking system; Whether it is in a monitorable area: This is used to determine whether the aircraft target is in the monitorable area of ​​the array camera. It is determined by whether the aircraft's latitude and longitude are within the latitude and longitude range corresponding to the monitorable area. Timestamp: Used to identify the time when information was acquired; The ADS-B system stores the global ID, latitude and longitude information, whether it is in a monitorable area, and timestamp fields in the database. Subsequently, by matching the global ID and local ID, the local ID and pixel coordinate fields are filled in. The matching process between the global ID and the local ID includes: The local ID and the bounding box are linked through an image coordinate to geographic coordinate conversion module. The bounding box is converted into the corresponding latitude and longitude region. The database is queried to obtain the ADS-B information and global ID of the aircraft target located in the region. Through these steps, the local ID and the global ID are bound together. (3) By comparing with the ID of the aircraft target in the previous frame, the ID of the disappeared target is obtained, and the target is judged to be abnormally disappeared based on the image area and latitude and longitude. The process for determining the abnormal disappearance of a target includes: By comparing the global IDs of all targets obtained in the current frame with the global IDs of all targets obtained in the previous frame, the targets that disappeared in the current frame can be identified. Determine whether a target has disappeared abnormally using two methods: image region and latitude / longitude coordinates. Image area: Aircraft targets can only disappear at the edge of the image or in an obscured area; disappearance in other areas is considered abnormal. Latitude and longitude: Use ADS-B information to determine whether the aircraft is still within the monitoring range. If it is not within the range, it is considered to have disappeared normally; otherwise, it is considered to have disappeared abnormally. If either of the two methods indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance is the target considered to have disappeared normally. (4) When the target disappears abnormally, the detection box of the target needs to be restored. The detection calibration box of the sub-camera at the same time as the target is found, and the calibration box in the sub-camera image coordinate system is transformed into the calibration box in the array camera panoramic image coordinate system. (5) The calibration boxes obtained by transformation in the sub-camera are matched with the calibration boxes obtained by panoramic image detection of the array camera through a matching algorithm. The unmatched calibration boxes are designated as detection boxes that need to be restored, and the restored detection boxes are output.

2. The airport target detection and tracking method according to claim 1, characterized in that: Step (2) of the ADS-B system information acquisition and database design includes: The ADS-B system acquires aircraft identification information, spatiotemporal information, aircraft model, and flight number information in real time.

3. The airport target detection and tracking method according to claim 1, characterized in that: Step (4) of the abnormal disappearance target bounding box recovery process includes: To calculate the correspondence between the sub-camera view and the panoramic view of the array camera, the sub-camera and the array camera first need to be calibrated to obtain their internal and external parameters. The purpose of camera calibration is to determine the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t]; the intrinsic parameter matrix K includes the focal length f. x and f y and the principal point coordinates c x and c y The format is as follows: The extrinsic parameter matrix [R|t] includes the rotation matrix R and the translation vector t, in the form of [R|t] = [R t]. These parameters are solved using a calibration board and calibration algorithm, providing a foundation for subsequent image processing. Feature points are extracted from the sub-camera view and the panoramic view of the array camera, and then matched. Feature point matching finds the most similar point pairs by comparing the descriptors of feature points in different images; the matching methods are nearest neighbor matching and ratio testing. Assume feature point p i and q i In the images from the sub-camera and the array camera, their feature descriptors are respectively: and The matching process can be represented as follows: Where m represents a matching point pair; the above formula indicates that for each feature point p in the sub-camera frame... i Find the feature point q in the panoramic image of the array camera that is most similar to its descriptor. i Specifically, the distance between each feature point descriptor is calculated, and the point pair with the smallest distance is selected as the matching point. The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes; assuming there are n pairs of matching feature points ((x...) in the sub-camera frame... i ,y i ),(x′ i ,y′ i )), where (x i ,y i ) and (x′ i ,y′ i (x′) represent the coordinates of points in the sub-camera view and the panoramic view of the array camera, respectively. The homography matrix H is solved by the following equation: (x′) i y′ i 1)=H(x i y i 1) Specifically, construct a system of linear equations: By solving the above system of linear equations, the elements (h) of the homography matrix H are obtained. 11 ,h 12 ,…,h 32 ), the last element h 33 It is usually set to 1; For each pixel coordinate (x′, y′) in the panoramic view of the array camera, use the homography matrix H of each sub-camera view. i Perform an inverse transformation to find the corresponding sub-camera coordinates (x... i ,y i The inverse transform formula is: Check the coordinates (x) after the inverse transformation i ,y i Whether it is within the effective range of the sub-camera; if it is within the effective range, then the pixel coordinates correspond to the sub-camera; With the homography matrix H, the perspective transformation of the sub-camera image is performed and mapped onto the panoramic image of the array camera. The formula for perspective transformation is: [x′ y′ w]=H[xy 1], where (x,y) are the coordinates of the point in the sub-camera image, (x′,y′) are the coordinates of the point in the panoramic image of the array camera after transformation, and w is the normalization factor. The above expression converts a set of bounding boxes in the image coordinates of the sub-camera into bounding boxes in the panoramic image coordinates of the array camera. These converted bounding boxes are then matched with all the bounding boxes obtained by the panoramic target detection algorithm of the array camera using the Hungarian algorithm. Before performing matching, it is necessary to determine the metric for evaluating the similarity between two bounding boxes: DIoU adds the distance between the center points of the bounding boxes to IoU, thus measuring the spatial proximity of the two boxes while considering overlapping areas; its formula is defined as follows: Wherein, IoU is the intersection-union ratio of the two bounding boxes, calculated using the following formula: Area intersection This represents the area of ​​the intersection of two detection boxes, while Area... union Then it represents the area of ​​their union; d is the Euclidean distance between the center points of the two boxes; c is the minimum diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle containing the two boxes that can be formed. After completing the DIoU calculation between all calibration boxes, the Hungarian algorithm is applied for matching; The specific implementation steps of the Hungarian algorithm include: Step 1: Construct the cost matrix: Using DIoU as the matching cost, construct an n×m cost matrix, where n is the number of calibration boxes after the sub-camera transformation, m is the number of calibration boxes obtained by the object detection algorithm, and each element in the matrix represents the DIoU value of the corresponding box pair; Step 2: Finding an augmenting path: By continuously selecting unmatched points and trying to find matching points with the lowest cost, an augmenting path is formed until all points are matched or no better match can be found. Step 3: Update matching status and cost: Adjust the matching status according to the augmenting path and rematch some of the already matched points to reduce the total cost; Step 4: Repeat steps 2 and 3 until no new augmenting path can be found; Step 5: The matching result obtained is the one-to-one correspondence between the converted calibration box and the calibration box obtained by the target detection algorithm; the boxes that cannot be matched by the Hungarian algorithm in the converted calibration box are the boxes that need to be restored. The coordinate boxes and their corresponding local IDs are redrawn on the panoramic image of the array camera and output.

Images