Airport target detection tracking system and method based on high-definition array camera

Abstract

The invention discloses an airport target detection tracking system and method based on a high-definition array camera, and the method comprises the steps: obtaining a local ID and a calibration frame of each target through a target real-time detection tracking algorithm of a panoramic image of the array camera and a target real-time detection tracking algorithm of an image of a sub-camera; recording a timestamp of a frame corresponding to the calibration frame and target longitude and latitude information (namely time-space information of the target), acquiring complete information of the aircraft target under the longitude and latitude by using an ADS-B system, and realizing matching of a local ID and a global ID; comparing with a previous frame of aircraft target ID to obtain a disappeared aircraft target, and judging whether the target disappears abnormally or not according to an aircraft target disappearing area and ADS-B system information; when the target abnormities disappear, the detection result of the sub-camera is inquired, the calibration frame under the image coordinate system of the sub-camera is converted into the calibration frame under the panoramic image coordinate system of the array camera, matching recovery is carried out, and the recovered detection frame is redrawn and output. According to the invention, through cooperative work of multiple cameras and multi-information fusion, the accuracy and stability of real-time detection and tracking of the airport target are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to an airport target detection and tracking system and method based on a high-definition array camera. Background Art

[0002] With the rapid development of the aviation industry, airport safety management and operational efficiency are facing increasing challenges. Traditional airport monitoring systems mainly rely on video streams from a single camera or a small number of cameras. This system has obvious limitations in large-scale monitoring, small target detection, and multi-target tracking. For example, a single camera has limited coverage, making it difficult to achieve comprehensive monitoring of the entire airport; in long-distance monitoring, targets such as aircraft appear as small objects in the image, and traditional target detection algorithms are prone to missed detection or false detection; there are a large number of moving targets in the airport environment, and how to accurately track multiple targets in a complex background and maintain the consistency of target IDs is a very challenging task; in addition, targets may suddenly disappear due to occlusion, signal loss, etc. How to quickly and accurately restore the tracking of these targets is the key to improving the robustness of the system. Summary of the invention

[0003] To solve the above problems, the present invention discloses an airport target detection and tracking system and method based on high-definition array cameras, which realizes efficient and stable detection and tracking of targets in the airport through the collaborative work of multiple cameras, optimization of target detection and tracking algorithms, geographic coordinate conversion and information fusion, and abnormal situation processing mechanism.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] The invention discloses an airport target detection and tracking system based on a high-definition array camera, comprising a high-definition array camera image acquisition system, an ADS-B system (Automatic Dependent Surveillance-Broadcast), a target detection and tracking system, a coordinate and camera matching system, and a discrimination and correction system.

[0006] A design method for an airport target detection and tracking system based on a high-definition array camera includes the following steps:

[0007] (1) The target detection and tracking system for the panoramic image of the array camera and the target detection and tracking system for the sub-camera image are run simultaneously. The panoramic image frame image of the array camera is further tracked after passing through the target detection system, so as to obtain the ID and calibration frame of each target in the image;

[0008] (2) The calibration frame is converted into geographic coordinates, i.e., longitude and latitude, through a conversion module, and the complete information of the aircraft target located at the longitude and latitude is obtained through the ADS-B system;

[0009] (3) By comparing the ID of the aircraft target in the previous frame, the ID of the disappeared target is obtained, and the target is judged to be abnormally disappeared based on the image area and longitude and latitude;

[0010] (4) When the target disappears abnormally, the detection frame of the target needs to be restored, and the detection calibration frame of the sub-camera at the same time as the target is found, and the calibration frame in the sub-camera image coordinate system is transformed into the calibration frame in the array camera panoramic image coordinate system;

[0011] (5) The calibration frame transformed in the sub-camera is matched with the calibration frame detected by the panoramic image of the array camera through a matching algorithm, and the unmatched calibration frame is defined as the detection frame that needs to be restored, and the restored detection frame is output.

[0012] Furthermore, the ADS-B system information acquisition and database design in step (2) includes:

[0013] The ADS-B system (Automatic Dependent Surveillance-Broadcast System) can obtain the aircraft's time and space information (longitude, latitude, speed and timestamp), other possible additional information (conflict warning information, pilot input information, track angle, route turning point, etc.) and aircraft identification information and category information in real time. In addition, it may also include some other additional information, such as heading, airspeed, wind speed, wind direction and aircraft external temperature;

[0014] Furthermore, part of the information obtained by the ADS-B system in step (2) needs to be stored in a database for subsequent use. At the same time, in order to meet the needs of the target tracking system design method, the key fields involved in the airport target tracking database include:

[0015] Global ID: used to uniquely identify an aircraft target in the entire airport, usually using the aircraft's identification information;

[0016] Latitude and longitude: geographical coordinates of the aircraft target, obtained by the ADS-B system;

[0017] Local ID: generated by the target detection and tracking system, which is the ID of the target in the picture;

[0018] Pixel coordinates: The pixel coordinates of the aircraft target in the image are determined by the calibration frame. The calibration frame contains the upper left corner pixel coordinates and the lower right corner pixel coordinates of the aircraft target, which are obtained by the target detection and tracking system;

[0019] Whether it is in the monitorable area flag: It is used to determine whether the aircraft target is in the monitorable area of ​​the array camera, and is determined by whether the longitude and latitude of the aircraft are in the longitude and latitude range corresponding to the monitorable area;

[0020] Timestamp: used to identify the time at which information is attributed.

[0021] The ADS-B system stores the global ID, longitude and latitude information, whether it is in a monitorable area flag and timestamp field in the database, and then completes the filling of the local ID and pixel coordinate fields by matching the global ID and the local ID.

[0022] Furthermore, the matching process of the global ID and the local ID includes:

[0023] The local ID and the calibration frame are converted into the corresponding latitude and longitude area through an image coordinate and geographic coordinate conversion module, and the database is queried to obtain the ADS-B information and global ID of the aircraft target located in the area. Through these steps, the local ID and the global ID are bound;

[0024] Furthermore, the determination process of the target abnormal disappearance in step (3) includes:

[0025] By comparing the global IDs of all targets acquired in the current frame image with the global IDs of all targets acquired in the previous frame image, the targets that disappeared in the current frame are acquired;

[0026] There are two ways to determine whether the target disappears abnormally. On the one hand, the surveillance camera screen at the airport is relatively open and less blocked. The aircraft target can only disappear in the boundary area of ​​the screen or be blocked. By calibrating the boundary area and the normal blocked area, a normal disappearance area can be determined, and those in other areas are abnormal disappearances.

[0027] On the other hand, the information obtained by ADS-B can directly indicate the location of the aircraft and determine whether the aircraft is within the monitoring range. If it is no longer within the monitoring range, it is considered to have disappeared normally, otherwise it is considered to have disappeared abnormally;

[0028] The above two methods are used to determine whether the target disappears abnormally. If one of the two methods indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance, the target is considered to have disappeared normally.

[0029] Furthermore, the abnormally disappeared target calibration frame recovery process in step (4) includes:

[0030] To calculate the correspondence between the sub-camera image and the panoramic image of the array camera, first, the sub-camera and the array camera need to be calibrated to obtain their internal and external parameters.

[0031] The purpose of camera calibration is to determine the camera’s intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. The intrinsic parameter matrix K includes the focal length f x and f y And the principal point coordinates cx and c y , of the following form: The external parameter matrix [R|t] includes the rotation matrix R and the translation vector t, and is in the following form: [R|t] = [R t]. These parameters can be solved through the calibration board and calibration algorithm, providing a basis for subsequent image processing.

[0032] Feature points are extracted from the sub-camera image and the array camera panoramic image respectively, and then matched. The purpose of feature point detection is to find unique and stable points in the image. Common feature point detection methods include Harris corner detection, SIFT, SURF, and ORB. The descriptor of a feature point is usually a high-dimensional vector that describes the local information around the feature point.

[0033] Feature point matching is to find the most similar point pair by comparing the descriptors of feature points in different images. Common matching methods include nearest neighbor matching and ratio test. Assume that feature point p i and q i In the images of the sub-camera and the panoramic camera, their feature descriptors are and q , then the matching process can be expressed as: Where m represents a matching point pair. The above formula indicates that for each feature point p in the sub-camera image i , find the feature point q in the panoramic image of the array camera that is most similar to its descriptor i Specifically, the distance (such as Euclidean distance or Hamming distance) between each feature point descriptor is calculated, and the point pair with the smallest distance is selected as the matching point.

[0034] The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes. Assume that there are n pairs of matching feature points in the sub-camera image ((x i ,y i ),(x′ i ,y′ i )), where (x i ,y i ) and (x′ i ,y′ i ) represent the point coordinates in the sub-camera and panoramic camera respectively, and the homography matrix H can be solved by the following equation: (x′ i y′ i 1) = H(x i y i 1); Specifically, a linear system of equations can be constructed: By solving the above linear equations, we can obtain the elements (h 11,h 12 ,…,h 32 ), the last element h 33 Typically set to 1.

[0035] For each pixel coordinate (x′, y′) in the array camera image, use the homography matrix H of each sub-camera i Perform inverse transformation to find the corresponding sub-camera coordinates (x i ,y i ). The inverse transformation formula is: The final coordinates are: Check the inverse transformed coordinates (x i ,y i ) is within the valid range of the sub-camera (i.e., within the image boundary of the sub-camera). If it is within the valid range, the pixel coordinates correspond to sub-camera i.

[0036] With the homography matrix H, the sub-camera image can be transformed into the panoramic image of the array camera. The formula for perspective transformation is: [x′ y′ w] = H[xy 1], where (x, y) is the coordinate of the point in the sub-camera image, (x′, y′) is the coordinate of the point in the panoramic image of the array camera after transformation, and w is the normalization factor. The final coordinates are:

[0037] Through the above expressions, a set of calibration frames in the sub-camera image coordinates can be converted into calibration frames in the array camera panoramic image coordinates, and this set of converted calibration frames is matched with all calibration frames obtained by the array camera panoramic image object detection algorithm through the Hungarian algorithm;

[0038] Before matching, it is necessary to determine the metric for evaluating the similarity of two calibrated boxes. DIoU adds the distance between the center points of the bounding boxes to IoU, thereby measuring the spatial proximity of the two boxes while considering the overlapping area. Its formula is defined as follows:

[0039] Among them, IoU is the intersection over union ratio of two bounding boxes, and the calculation formula is:

[0040] Area intersection Represents the area of ​​the intersection of two detection frames, and Area union Then represents the area of ​​their union. d is the Euclidean distance between the center points of the two boxes, c is the minimum diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle that can be formed by the two boxes and contains the two boxes;

[0041] After the DIoU calculations between all the calibrated frames are completed, the Hungarian algorithm is applied for matching. The Hungarian algorithm is a classic method for solving the assignment problem, which is used to find the global optimal or approximately optimal matching solution while ensuring that each pair of objects can only be matched once at most. In this scenario, the specific implementation steps of the Hungarian algorithm include:

[0042] Step 1: Construct a cost matrix: Use DIoU as the matching cost and construct an n×m cost matrix (n is the number of calibration boxes after sub-camera conversion, and m is the number of calibration boxes obtained by the target detection algorithm). Each element in the matrix represents the DIoU value of the corresponding box pair.

[0043] Step 2: Find an augmenting path: By continuously selecting unmatched points and trying to find matching points with the lowest cost, an augmenting path is formed until all points are matched or no better match can be found.

[0044] Step 3: Update matching status and cost: Adjust the matching status according to the augmented path, and possibly re-match some matched points to reduce the total cost.

[0045] Step 4: Repeat steps 2 and 3 until no new augmenting path is found.

[0046] Step 5: The matching result is the one-to-one correspondence between the converted calibration frame and the calibration frame obtained by the target detection algorithm. The frame that cannot be matched by the Hungarian algorithm in the converted calibration frame is the frame that needs to be restored. The coordinate frame and its corresponding local ID are redrawn on the camera array image and output.

[0047] The beneficial effects of the present invention are:

[0048] The present invention discloses a design method for an airport target detection and tracking system based on a high-definition array camera, which improves the accuracy and stability of airport target detection and tracking by combining ADS-B system information and manual calibration information to determine the target detection effect and the coordinated work of the array camera panoramic picture and sub-camera picture target detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic flow chart of an airport target tracking system and method based on a high-definition array camera provided in an embodiment of the present invention.

[0050] Figure 2 A schematic diagram of restoring a target detection frame using a sub-camera detection result when a target abnormally disappears provided by an embodiment of the present invention.

[0051] Figure 3A schematic diagram of a restoration process of sub-camera detection results corresponding to panoramic camera images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0053] like Figure 1 As shown, the airport target tracking system and method based on a high-definition array camera described in the present invention includes the following steps:

[0054] S1. The target detection system applicable to the array camera image and the target detection and tracking system applicable to the sub-camera image run simultaneously.

[0055] A target detection and tracking system for the panoramic image of the array camera and a target detection system for the sub-camera image are run simultaneously. These two algorithms respectively capture enough images of the panoramic image of the array camera and the sub-camera image, and mark the aircraft targets respectively, and then train the target detection network to obtain two sets of target detection algorithms;

[0056] The panoramic frame image acquired by the array camera can be used to obtain the calibration frame of the aircraft target through the target detection algorithm. The calibration frame contains the upper left corner pixel coordinates and the lower right corner pixel coordinates of the aircraft target, specifies the position of the target in the image, and uses the target tracking algorithm to assign an id to each calibration frame. The same id is considered to be the same target, and this id is also called a local id.

[0057] The frame image acquired by the sub-camera can be used to obtain the calibration frame of the aircraft target through the target detection system. The calibration frame contains the upper left corner pixel coordinates and the lower right corner pixel coordinates of the aircraft target, specifying the position of the target in the image. These calibration frames, timestamps and the row and column numbers of the sub-camera in the camera array are stored together for the subsequent recovery of the detection frame of the abnormally disappeared target.

[0058] S2, converting the calibration frame into geographic coordinates, i.e., longitude and latitude, through a conversion module, and obtaining complete information of the aircraft target located at the longitude and latitude through the ADS-B system;

[0059] The ADS-B system (Automatic Dependent Surveillance-Broadcast System) can obtain the aircraft's time and space information (longitude, latitude, speed and timestamp), other possible additional information (conflict warning information, pilot input information, track angle, route turning point, etc.) and aircraft identification information and category information in real time. In addition, it may also include some other additional information, such as heading, airspeed, wind speed, wind direction and aircraft external temperature;

[0060] Furthermore, some of the information obtained by the ADS-B system needs to be stored in the database for subsequent application. At the same time, in order to meet the needs of the target tracking system design method, the key fields involved in the airport target tracking database include:

[0061] Global ID: used to uniquely identify an aircraft target in the entire airport, usually using the aircraft's identification information;

[0062] Latitude and longitude: geographical coordinates of the aircraft target, obtained by the ADS-B system;

[0063] Local ID: generated by the target detection and tracking system, which is the ID of the target in the picture;

[0064] Pixel coordinates: The pixel coordinates of the aircraft target in the image are determined by the calibration frame. The calibration frame contains the upper left corner pixel coordinates and the lower right corner pixel coordinates of the aircraft target, which are obtained by the target detection and tracking system;

[0065] Whether it is in the monitorable area flag: It is used to determine whether the aircraft target is in the monitorable area of ​​the array camera, and is determined by whether the longitude and latitude of the aircraft are in the longitude and latitude range corresponding to the monitorable area;

[0066] Timestamp: used to identify the time at which information is attributed.

[0067] The ADS-B system stores the global ID, longitude and latitude information, whether it is in a monitorable area flag and timestamp field in the database, and then completes the filling of the local ID and pixel coordinate fields by matching the global ID and the local ID.

[0068] The local ID and calibration frame obtained by the target detection system are converted into the corresponding longitude and latitude area through an image coordinate and geographic coordinate conversion module, and the database is queried to obtain the ADS-B information and global ID of the aircraft target located in the area. Through these steps, the local ID and the global ID are bound.

[0069] S3, by comparing with the ID of the aircraft target in the previous frame, the ID of the disappeared target is obtained, and whether the target disappears abnormally is determined based on the image area and the longitude and latitude;

[0070] By comparing the global IDs of all targets acquired in the current frame image with the global IDs of all targets acquired in the previous frame image, the targets that disappeared in the current frame are acquired;

[0071] There are two ways to determine whether the target disappears abnormally. On the one hand, the surveillance camera screen at the airport is relatively open and less blocked. The aircraft target can only disappear in the boundary area of ​​the screen or be blocked. By calibrating the boundary area and the normal blocked area, a normal disappearance area can be determined, and those in other areas are abnormal disappearances.

[0072] On the other hand, the information obtained by ADS-B can directly indicate the location of the aircraft and determine whether the aircraft is within the monitoring range. If it is no longer within the monitoring range, it is considered to have disappeared normally, otherwise it is considered to have disappeared abnormally;

[0073] The above two methods are used to determine whether the target disappears abnormally. If one of the two methods indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance, the target is considered to have disappeared normally.

[0074] S4, such as Figure 2 As shown, when the target disappears abnormally, the detection frame of the target needs to be restored. First, it is necessary to obtain which sub-camera corresponds to the disappeared target, query the detection result of the sub-camera at the same time as the target, that is, the calibration frame, and transform the calibration frame in the sub-camera image coordinate system into the calibration frame in the array camera panoramic picture image coordinate system; match the calibration frame transformed in the sub-camera with the calibration frame detected in the array camera panoramic frame image through a matching algorithm, and the unmatched transformed calibration frame is the detection frame that needs to be restored, and the detection frame can be output after it is restored;

[0075] To calculate the correspondence between the sub-camera image and the panoramic image of the array camera, first, the sub-camera and the array camera need to be calibrated to obtain their internal and external parameters.

[0076] The purpose of camera calibration is to determine the camera’s intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. The intrinsic parameter matrix K includes the focal length f x and f y And the principal point coordinates c x and c y , of the following form: The external parameter matrix [R|t] includes the rotation matrix R and the translation vector t, and is in the following form: [R|t] = [R t]. These parameters can be solved through the calibration board and calibration algorithm, providing a basis for subsequent image processing.

[0077] Feature points are extracted from the sub-camera image and the array camera panoramic image respectively, and then matched. The purpose of feature point detection is to find unique and stable points in the image. Common feature point detection methods include Harris corner detection, SIFT, SURF, and ORB. The descriptor of a feature point is usually a high-dimensional vector that describes the local information around the feature point.

[0078] Feature point matching is to find the most similar point pair by comparing the descriptors of feature points in different images. Common matching methods include nearest neighbor matching and ratio test. Assume that feature point p i and q i In the images of the sub-camera and the array camera, their feature descriptors are and q , then the matching process can be expressed as: Where m represents a matching point pair. The above formula indicates that for each feature point p in the sub-camera i , find the feature point q in the panoramic image of the array camera that is most similar to its descriptor i Specifically, the distance (such as Euclidean distance or Hamming distance) between each feature point descriptor is calculated, and the point pair with the smallest distance is selected as the matching point.

[0079] The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes. Assume that there are n pairs of matching feature points in the sub-camera image ((x i ,y i ),(x′ i ,y′ i )), where (x i ,y i ) and (x′ i ,y′ i ) represent the point coordinates in the sub-camera image and the array camera panoramic image respectively. The homography matrix H can be solved by the following equation: (x′ i y′ i 1) = H(x i y i 1); Specifically, a linear system of equations can be constructed: By solving the above linear equations, we can obtain the elements of the homography matrix H (h 11 ,h 12 ,…,h 32 ), the last element h 33 Typically set to 1.

[0080] For each pixel coordinate (x′, y′) in the panoramic image of the array camera, use the homography matrix H of each sub-camera iPerform inverse transformation to find the corresponding sub-camera screen coordinates (x i ,y i ). The inverse transformation formula is: The final coordinates are: Check the inverse transformed coordinates (x i ,y i ) is within the valid range of the sub-camera (i.e., within the image boundary of the sub-camera). If it is within the valid range, the pixel coordinates correspond to sub-camera i.

[0081] With the homography matrix H, the sub-camera image can also be perspective transformed to map it to the array camera panoramic image. The formula for perspective transformation is: [x′ y′ w] = H[xy 1], where (x, y) is the point coordinate in the sub-camera image, (x′, y′) is the point coordinate in the array camera panoramic image after transformation, and w is the normalization factor. The final coordinates are:

[0082] Through the above expressions, a set of calibration frames in the sub-camera image coordinates can be converted into calibration frames in the array camera panoramic image coordinates, and this set of converted calibration frames is matched with all calibration frames obtained by the array camera panoramic image object detection algorithm through the Hungarian algorithm;

[0083] Before matching, it is necessary to determine the metric for evaluating the similarity of two calibrated boxes. DIoU adds the distance between the center points of the bounding boxes to IoU, thereby measuring the spatial proximity of the two boxes while considering the overlapping area. Its formula is defined as follows: Among them, IoU is the intersection over union ratio of two bounding boxes, and the calculation formula is:

[0084] Area intersection Represents the area of ​​the intersection of two detection frames, and Area union Then represents the area of ​​their union. d is the Euclidean distance between the center points of the two boxes, c is the minimum diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle that can be formed by the two boxes and contains the two boxes;

[0085] After the DIoU calculations between all the calibrated frames are completed, the Hungarian algorithm is applied for matching. The Hungarian algorithm is a classic method for solving the assignment problem, which is used to find the global optimal or approximately optimal matching solution while ensuring that each pair of objects can only be matched once at most. In this scenario, the specific implementation steps of the Hungarian algorithm include:

[0086] Step 1: Construct a cost matrix: Use DIoU as the matching cost and construct an n×m cost matrix (n is the number of calibration boxes after sub-camera conversion, and m is the number of calibration boxes obtained by the target detection algorithm). Each element in the matrix represents the DIoU value of the corresponding box pair.

[0087] Step 2: Find an augmenting path: By continuously selecting unmatched points and trying to find matching points with the lowest cost, an augmenting path is formed until all points are matched or no better match can be found.

[0088] Step 3: Update matching status and cost: Adjust the matching status according to the augmented path, and possibly re-match some matched points to reduce the total cost.

[0089] Step 4: Repeat steps 2 and 3 until no new augmenting path is found.

[0090] Step 5: The matching result is the one-to-one correspondence between the converted calibration frame and the calibration frame obtained by the target detection algorithm. The frame that cannot be matched by the Hungarian algorithm in the converted calibration frame is the frame that needs to be restored. The coordinate frame and its corresponding local ID are redrawn on the panoramic image of the array camera and output; Figure 3 shown.

[0091] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. An airport target detection and tracking system based on a high-definition array camera, characterized in that: It includes a high-definition array camera image acquisition system, an ADS-B system, a target detection and tracking system, a pixel coordinate and camera matching system, and a discrimination and correction system.

2. A method for airport target detection and tracking using the airport target detection and tracking system based on a high-definition array camera as described in claim 1, characterized in that: The following steps are involved: (1) The target detection and tracking system for the panoramic image of the array camera and the target detection and tracking system for the sub-camera image are run simultaneously. The panoramic image frame image of the array camera is further tracked after passing through the target detection system, so as to obtain the ID and calibration frame of each target in the image; (2) The calibration frame is converted into geographic coordinates, i.e., longitude and latitude, through a conversion module, and the complete information of the aircraft target located at the longitude and latitude is obtained through the ADS-B system; (3) By comparing the ID of the aircraft target in the previous frame, the ID of the disappeared target is obtained, and the target is judged to be abnormally disappeared based on the image area and longitude and latitude; (4) When the target disappears abnormally, the detection frame of the target needs to be restored, and the detection calibration frame of the sub-camera at the same time as the target is found, and the calibration frame in the sub-camera image coordinate system is transformed into the calibration frame in the array camera panoramic image coordinate system; (5) The calibration frame transformed in the sub-camera is matched with the calibration frame detected by the panoramic image of the array camera through a matching algorithm, and the unmatched calibration frame is defined as the detection frame that needs to be restored, and the restored detection frame is output.

3. The method for detecting and tracking an airport target according to claim 2, characterized in that: The ADS-B system information acquisition and database design described in step (2) include: The ADS-B system obtains the aircraft's identification information, time and space information, aircraft model, and flight number information in real time.

4. The method for detecting and tracking an airport target according to claim 2, characterized in that: Step (2) Part of the information obtained by the ADS-B system needs to be stored in the database for subsequent use. At the same time, in order to meet the needs of the target tracking system design method, the key fields involved in the airport target tracking database include: Global ID: used to uniquely identify an aircraft target in the entire airport, usually using the aircraft's identification information; Latitude and longitude: geographical coordinates of the aircraft target, obtained by the ADS-B system; Local ID: generated by the target detection and tracking system, which is the ID of the target in the picture; Pixel coordinates: The pixel coordinates of the aircraft target in the image are determined by the calibration frame. The calibration frame contains the upper left corner pixel coordinates and the lower right corner pixel coordinates of the aircraft target, which are obtained by the target detection and tracking system; Whether it is in the monitorable area flag: It is used to determine whether the aircraft target is in the monitorable area of ​​the array camera, and is determined by whether the longitude and latitude of the aircraft are in the longitude and latitude range corresponding to the monitorable area; Timestamp: used to identify the time at which information is attributed; The ADS-B system stores the global ID, longitude and latitude information, whether it is in a monitorable area flag and timestamp field in the database, and then completes the filling of the local ID and pixel coordinate fields by matching the global ID and the local ID.

5. The method for detecting and tracking an airport target according to claim 4, characterized in that: The matching process of the global ID and the local ID includes: The local ID and the calibration frame are converted into the corresponding longitude and latitude area through an image coordinate and geographic coordinate conversion module, and the database is queried to obtain the ADS-B information and global ID of the aircraft target located in the area. Through these steps, the local ID and the global ID are bound.

6. The method for detecting and tracking an airport target according to claim 2, characterized in that: The process of determining whether the target disappears abnormally in step (3) includes: By comparing the global IDs of all targets acquired in the current frame image with the global IDs of all targets acquired in the previous frame image, the targets that disappeared in the current frame are acquired; Determine whether the target disappears abnormally by using image area and longitude and latitude: Image area: Aircraft targets can only disappear at the image boundary or in the blocked area. Disappearance in other areas is considered abnormal. Latitude and longitude: Use ADS-B information to determine whether the aircraft is still within the monitoring range. If it is not within the range, it is considered to have disappeared normally; otherwise, it is considered to have disappeared abnormally. If one of the two methods indicates abnormal disappearance, the target is considered to have disappeared abnormally. Only when both methods indicate normal disappearance, the target is considered to have disappeared normally.

7. The method for detecting and tracking an airport target according to claim 2, characterized in that: The process of restoring the abnormally disappeared target calibration frame in step (4) includes: To calculate the correspondence between the sub-camera image and the panoramic image of the array camera, first, the sub-camera and the array camera need to be calibrated to obtain their internal and external parameters; The purpose of camera calibration is to determine the camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t]; the intrinsic parameter matrix K includes the focal length f x and f y And the principal point coordinates c x and c y , of the following form: The external parameter matrix [R|t] includes the rotation matrix R and the translation vector t, and is in the following form: [R|t] = [R t]. Through the calibration board and calibration algorithm, these parameters are solved to provide a basis for subsequent image processing. Extract feature points from the sub-camera image and the array camera panoramic image respectively, and perform matching; Feature point matching is to find the most similar point pairs by comparing the descriptors of feature points in different images; the matching methods are nearest neighbor matching and ratio test: Assume that the feature point p i and q i In the images of the sub-camera and the array camera, their feature descriptors are and The matching process is expressed as: Where m represents a matching point pair; the above formula indicates that for each feature point p in the sub-camera image, i , find the feature point q in the panoramic image of the array camera that is most similar to its descriptor i ; Specifically, the distance between each feature point descriptor is calculated, and the point pair with the smallest distance is selected as the matching point; The homography matrix H is a 3x3 matrix used to describe the projection transformation relationship between two planes. Assume that there are n pairs of matching feature points in the sub-camera image ((x i ,y i ),(x' i ,y' i )), where (x i ,y i ) and (x' i ,y' i ) represent the point coordinates in the sub-camera image and the array camera panoramic image respectively, and the homography matrix H is solved by the following equation: (x' i y' i 1) = H(x i y i 1); Specifically, construct a linear system of equations: By solving the above linear equations, we can obtain the elements of the homography matrix H (h 11 ,h 12 ,…,h 32 ), the last element h 33 Usually set to 1; For each pixel coordinate (x', y') in the panoramic image of the array camera, use the homography matrix H of each sub-camera image i Perform inverse transformation to find the corresponding sub-camera coordinates (x i ,y i ); the inverse transformation formula is: The final coordinates are: Check the inverse transformed coordinates (x i ,y i ) is within the valid range of the sub-camera; if it is within the valid range, the pixel coordinate corresponds to the sub-camera i; After obtaining the homography matrix H, the sub-camera image is transformed into the panoramic image of the array camera; the perspective transformation formula is: [x' y' w] = H[xy 1], where (x, y) is the point coordinate in the sub-camera image, (x', y') is the point coordinate in the panoramic image of the array camera after transformation, and w is the normalization factor; the final coordinates are: Through the above expressions, a set of calibration frames in the image coordinates of the sub-camera screen is converted into calibration frames in the image coordinates of the panoramic screen of the array camera, and this set of converted calibration frames is matched with all calibration frames obtained by the panoramic screen object detection algorithm of the array camera through the Hungarian algorithm; Before matching, it is necessary to determine the indicator for evaluating the similarity of two calibration boxes: DIoU adds the distance between the center points of the bounding boxes to the IoU, thereby measuring the spatial proximity of the two boxes while considering the overlapping area; its formula is defined as follows: Among them, IoU is the intersection over union ratio of two bounding boxes, and the calculation formula is: Area intersection Represents the area of ​​the intersection of two detection frames, and Area union Then represents the area of ​​their union; d is the Euclidean distance between the center points of the two boxes, and c is the minimum value of the diagonal length of the two boxes, that is, the diagonal length of the smallest rectangle that can be formed by the two boxes and contains the two boxes; After completing the DIoU calculation between all calibrated frames, the Hungarian algorithm is applied for matching; The specific implementation steps of the Hungarian algorithm include: Step 1: Construct a cost matrix: Use DIoU as the matching cost and construct an n×m cost matrix, where n is the number of calibration boxes after sub-camera conversion, and m is the number of calibration boxes obtained by the target detection algorithm. Each element in the matrix represents the DIoU value of the corresponding box pair. Step 2: Find an augmenting path: By continuously selecting unmatched points and trying to find matching points with the lowest cost, an augmenting path is constructed until all points are matched or no better match can be found; Step 3: Update matching status and cost: adjust the matching status according to the augmented path, and re-match some matched points to reduce the total cost; Step 4: Repeat steps 2 and 3 until no new augmenting path is found; Step 5: The matching result is the one-to-one correspondence between the converted calibration frame and the calibration frame obtained by the target detection algorithm; the frame in the converted calibration frame that cannot be matched by the Hungarian algorithm is the frame that needs to be restored, and the coordinate frame and its corresponding local ID are redrawn on the panoramic screen of the array camera for output.

Images