Cross-satellite airport aircraft target tracking method combined with road constraint

Abstract

The invention discloses a cross-satellite airport aircraft target tracking method in combination with road constraints, and the method comprises the following steps: S101, carrying out the aircraft target detection of an airport image collected by a satellite constellation through a target detector, and obtaining the detection frame data of an aircraft target in a scene, calculating the time difference of data acquired by adjacent satellites in the constellation; step S102, establishing a trajectory prediction model; s101, using the detection frame data obtained in the step S101, and respectively adopting a re-identification network and a multi-mode trajectory prediction network to carry out appearance and motion information extraction on an aircraft target in an airport area; and S103, realizing cross-frame target matching by using an optimized matching algorithm, performing rationality correction on a track obtained by matching in combination with airport road constraints, and drawing a final aircraft target motion track. According to the method, tracking errors can be reduced, and the precision and stability of cross-satellite aircraft target tracking are improved.

Description

Technical Field

[0001] The present invention relates to the field of satellite monitoring and target tracking, and in particular to a method for realizing aircraft target tracking on a data set of cross-satellite imaging and introducing airport road constraints to optimize the tracking path, so as to improve the accuracy and stability of cross-satellite aircraft target tracking. Background Art

[0002] With the large-scale deployment of low-orbit commercial Internet satellites and the increase in the number of remote sensing satellites, the collaborative work of multiple satellites can achieve high-frequency revisits to ground areas. In the field of satellite monitoring and target tracking, the existing technology mainly relies on the data processing of a single imaging platform and performs target tracking based on traditional Kalman filtering. However, it faces problems such as insufficient tracking path accuracy, difficulty in cross-satellite data integration, limited data collection time, and inability to provide long-term continuous airport monitoring capabilities. In particular, under satellite imaging conditions at different times and different perspectives, the robustness of target tracking is poor. The existing method does not fully utilize the road information in the airport scene for prediction, which makes the accuracy of cross-satellite target matching limited, making cross-satellite target tracking difficult to achieve. Since the cross-satellite target tracking method in the prior art has the technical problems of large tracking errors, insufficient tracking accuracy and stability, the present invention provides a solution to the above technical problems. Summary of the invention

[0003] In view of the problems existing in the prior art, the purpose of the present invention is to provide a cross-satellite airport aircraft target tracking method combined with road constraints, so as to reduce tracking errors and improve the accuracy and stability of cross-satellite aircraft target tracking.

[0004] To achieve the above object, the present invention provides a method for tracking an aircraft target across satellite airports combined with road constraints, the method comprising the following steps: Step S101: using a target detector to detect aircraft targets on the airport image collected by the satellite constellation, obtaining detection frame data of the aircraft targets in the scene, and calculating the time difference of the data collected by adjacent satellites in the constellation; Step S102: establishing a trajectory prediction model; using the detection frame data obtained in step S101, respectively using a re-identification network and a multi-modal trajectory prediction network to extract appearance and motion information of aircraft targets in the airport area; Step S103: Use the optimized matching algorithm to achieve cross-frame target matching, and combine the airport road constraints to make reasonable corrections to the matched trajectory to draw the final aircraft target motion trajectory.

[0005] Further, step S101 specifically includes: Step S201: Use the object detector to identify the aircraft in the field of view, generate a rotated bounding box and crop the image data, combine the channel and spatial attention modules, and strengthen the features extracted by the backbone network; Step S202: Calculate and provide the imaging time difference of adjacent satellites in the satellite constellation according to the imaging time information contained in the image.

[0006] Furthermore, the target detector is a YOLOv7 target detection network.

[0007] Furthermore, step S102 further includes collecting ADS-B data of aircraft at the airport, and cleaning the data, ignoring stationary aircraft tracks.

[0008] Furthermore, step S102 further includes: Step S301: collecting aircraft historical motion trajectory data through ADS-B and preprocessing the data, including extracting aircraft target trajectories from satellite images, and performing data cleaning, outlier removal and normalization processing; Step S302: Use a LSTM-based multimodal trajectory prediction network to predict future trajectories.

[0009] Furthermore, in step S302, a multimodal prediction method is used to generate multiple possible trajectories, and Gaussian noise is added during the training process to simulate possible motion changes of the aircraft and improve the adaptability of the model to uncertain motion patterns.

[0010] Furthermore, step S102 further includes: Step S303: Determine the number of prediction steps of the trajectory prediction model according to the calculated time difference, give a candidate range in combination with the trajectory prediction end point, perform cropping and obtain the cropped target image; Step S304: Using a lightweight re-identification network to extract the appearance features of the target.

[0011] Furthermore, the lightweight re-ID network consists of four convolutional blocks and a downsampling module, which takes the cropped target image as input and outputs a 128×1 feature vector.

[0012] Furthermore, the airport road node graph is used to correct the trajectory, matching the start and end points with the nearest neighbor road nodes, and then forming a tracking trajectory that conforms to the roads within the airport.

[0013] The beneficial effects of the present invention are as follows: The present invention proposes a cross-satellite airport aircraft target tracking method combined with road constraints. The multi-mode trajectory prediction model is trained by stacking multiple layers of long short-term memory (LSTM) networks, and combined with an optimized cross-frame matching algorithm, the trajectory is corrected using airport road constraints, effectively improving the airport aircraft target tracking accuracy on the cross-satellite data set. Road constraints include not only traditional airport road node information, but also aircraft motion trajectory constraints. Since the aircraft's motion trajectory in the airport is naturally restricted by roads, the trajectory itself contains road information. This method can not only effectively reduce tracking errors, but also provide new technical means and application prospects for the field of satellite remote sensing data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of a method for tracking an aircraft target across satellite airports combined with road constraints according to the present invention; Figure 2 It is a diagram of the multi-mode trajectory network structure; Figure 3 It is a schematic diagram of trajectory prediction screening; Figure 4 This is the target tracking effect diagram. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0017] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0018] The following combination Figure 1-Figure 4 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0019] The present invention proposes a novel aircraft target tracking method, which comprehensively utilizes trajectory prediction, time difference calculation and optimized matching strategy to improve tracking accuracy and robustness, and can achieve accurate and stable tracking of aircraft targets on a cross-satellite imaging data set.

[0020] The core of the tracking method of the present invention is to use historical aircraft motion data containing trajectory information for deep learning modeling, build a trajectory prediction model through a multi-layer stacked LSTM network to predict the possible position of the aircraft at a future moment, and combine it with an optimized matching algorithm to perform cross-frame target tracking; finally, the tracking trajectory is corrected through airport road constraints to ensure the rationality of the trajectory and improve the accuracy of tracking. Figure 1 As shown, the method comprises the following steps: Step S101: Use the YOLOv7 target detector to detect aircraft targets on the airport image collected by the satellite constellation, obtain the detection frame information of the aircraft target in the scene, and calculate the time difference between the data collected by adjacent satellites in the constellation.

[0021] Step S102: Collect the ADS-B (Automatic Dependent Surveillance-Broadcast) data of the airport aircraft, clean the data, and ignore the static aircraft trajectory. Train the multi-mode trajectory prediction network based on the historical aircraft motion trajectory data, learn the motion pattern of the aircraft targets in the airport, and establish a trajectory prediction model. Using the detection box data obtained in step S101, use the re-identification network and the multi-mode trajectory prediction network to extract the appearance and motion information of the aircraft targets in the airport area.

[0022] Step S103: Use the optimized matching algorithm to achieve cross-frame target matching, and combine the airport road constraints to make reasonable corrections to the matched trajectory, draw the final aircraft target motion trajectory, and ensure that the trajectory accurately reflects the dynamic motion characteristics of the aircraft target and complies with the airport road constraints.

[0023] The following are the detailed steps of the relevant modules: Regarding step S101, the detailed steps of the method provided in this solution are as follows (S201-S203): Step S201: Use the low-orbit satellite constellation to collect satellite remote sensing image data of the airport area and perform preprocessing (including orthorectification, registration, etc.) (This part corresponds to Figure 1 The YOLOv7 target detector is used to identify the aircraft in the field of view, generate a rotation bounding box and crop the image data. The channel and spatial attention modules are combined to strengthen the features extracted by the backbone network, so that the detection network focuses on the key parts of the feature map containing the target, and improves the recognition ability of weak targets.

[0024] Step S202: Calculate and give the imaging time difference of adjacent satellites in the satellite constellation according to the imaging time information contained in the image, introduce a time variable for the trajectory prediction network, and improve the accuracy of predicting the moving trajectory of the target during the target tracking process.

[0025] Regarding step S102, the detailed steps of the method provided by this solution are as follows (S301-S304): Step S301: Collect the historical motion trajectory data of the aircraft through ADS-B (Automatic Dependent Surveillance-Broadcast) and pre-process the data, including extracting the aircraft target trajectory from the satellite image, and performing data cleaning, outlier removal and normalization to ensure data consistency. Road constraints include trajectory constraints and road network constraints. The historical motion trajectory data of the aircraft includes the static information of the airport runway and the dynamic information of the aircraft movement, which can provide more effective information for target tracking. The dynamic information includes movement speed and direction, etc.

[0026] Step S302: A multimodal trajectory prediction network based on LSTM is used, such as Figure 2As shown, the network structure is optimized. The present invention adopts an encoding and decoding structure, in which the encoder maps the historical observation trajectory into trajectory features (saved in LSTM hidden state), and the decoder uses the existing historical trajectory features (output of the encoder) and random variables as input to realize multi-mode trajectory prediction. Multi-layer LSTM is used in the encoder to realize the fitting of complex motion trajectories of aircraft in airport scenarios. At the end of the decoder, multiple rounds of linear mapping are used to obtain multiple possible motion trajectories while extrapolating the positions of subsequent multiple moments to adapt to time series trajectory data and improve long-time span prediction capabilities. In order to improve the prediction accuracy, interpolation technology is used to generate finer-grained intermediate positions to make the trajectory data smoother and continuous. The historical motion pattern is captured by the cyclic unit to predict the future trajectory. A multi-modal prediction method is used to generate multiple possible trajectories, and Gaussian noise is added during the training process to simulate the possible motion changes of the aircraft to improve the adaptability of the model to uncertain motion patterns. During the training process, the combined prediction end point and the loss of the entire trajectory are used for supervision to improve the accuracy of the predicted trajectory end point.

[0027] The composite loss function supervises the trajectory prediction network by combining the absolute distance of the final prediction point and the matching distance of the entire trajectory. Due to the long time interval of satellite constellation imaging data, multiple rounds of position prediction are required to estimate the target position. Therefore, more emphasis is placed on the accuracy of the final prediction point. Defined as: ; in and are the ground truth values ​​of the samples and the predicted final positions. is the number of predicted trajectories, that is, the number of endpoints of all trajectories. The average absolute distance loss of the endpoints of multiple predicted trajectories is obtained by calculating the absolute distance between the predicted position and the true position.

[0028] Overall trajectory loss Combining L1 (mean absolute error) and L2 (mean squared error) losses to explain the error at each time step during trajectory prediction: ; In the formula is the number of time steps of the input trajectory, is the number of time steps for predicting the trajectory, and are the weights of L1 and L2 losses.

[0029] Adding weights to the comprehensive loss function To the end point loss, to emphasize its importance: ; This loss is used to supervise the trajectory prediction network. As a measurement indicator, when the value is large, the network prediction result is inaccurate. During the training process, the parameters are gradually adjusted to reduce the loss. Finally, the network can make the prediction result closer to the true value.

[0030] Step S303: According to the calculated time difference, determine the prediction steps of the trajectory prediction model, so that the prediction step length is close to the time span of adjacent frames, while avoiding errors caused by excessive extrapolation. Input the position information of the aircraft target in the previous frames into the trajectory prediction model to generate multiple possible candidate sets of future aircraft target positions. Combined with the trajectory prediction end point, a candidate range is given, which is determined by the time difference of adjacent data. Figure 3 As shown, the extrapolated trajectory of the aircraft target at the previous moment is obtained, and the candidate range is obtained at the end position of the trajectory, and the detection targets outside the range are excluded to reduce the calculation consumption in the subsequent matching process.

[0031] Step S304: A lightweight re-identification network is used to extract the appearance features of the target. This network is mainly used for pedestrian re-identification and is used to calculate the appearance similarity of targets in different images. In the present invention, the network is retrained using aircraft data to obtain a re-identification network suitable for aircraft targets. The network consists of four convolution blocks and a downsampling module, takes the cropped target image as input, and outputs a 128×1 feature vector. The feature vector contains the specific appearance features of the aircraft target and is robust to changes in illumination, perspective, and scale.

[0032] Regarding step S103, the detailed steps of the method provided by this solution are as follows (S401-S403): In cross-frame matching, considering the low temporal resolution of satellite constellation data, moving targets will have a large range of movement, resulting in a large difference between moving targets and stationary targets. Therefore, different matching schemes need to be adopted for different targets.

[0033] Step S401: The displacement of static targets between adjacent frames is small, and their position changes are mainly due to angle or geo-registration errors. The intersection over union (IoU) is used to calculate the overlap of detection boxes in adjacent frames, and the candidate set of static targets is screened out. Preliminary matching is performed on targets with IoU higher than the set threshold, and the Hungarian algorithm is used to optimize the combination of all candidate matching pairs. In view of the airport queuing phenomenon, IoU matching may mistakenly identify different targets as the same target. At this time, the target appearance features extracted by the re-identification network are used to calculate the cosine similarity of the targets in adjacent frames: ; in, and Represent the appearance feature vectors of the target respectively. If the cosine similarity is lower than the set threshold, the matching pair is considered inconsistent in appearance, the matching pair will be eliminated, and the unmatched target will be moved to the moving target matching stage.

[0034] Step S402: Generate multiple possible motion trajectories based on the LSTM trajectory prediction model. Calculate the Euclidean distance between the predicted trajectory and the detection target in the current frame: ; in, is the detected position of the target in the current frame, The target predicted position generated by the trajectory prediction network. If the distance exceeds the set threshold, the target is considered to be far away from the predicted trajectory and the matching pair is excluded. For the target pairs that pass the position screening, the cosine distance of the appearance features is further calculated: ; For multiple possible matching results, the Hungarian algorithm is used to generate the optimal matching combination according to the cosine distance to complete the matching of moving targets.

[0035] Step S403: After the matching is completed, the tracking trajectory of each target is updated in combination with the matching results. Since the confidence of the detection box in the satellite remote sensing video is usually higher than the confidence of the prediction box, a weighted strategy is introduced when updating the target position: ; in, and Respectively represent the coordinate positions of the detection box and the prediction box, and Represent the position weights of the detection frame and the prediction frame respectively. Increasing the weight of the detection frame improves the accuracy of the position update. Then, the trajectory is corrected using the airport road node graph, and the starting point and the end point are matched with the nearest neighbor road node. Then, the Dijkstra algorithm is used to calculate the shortest path between the two points in the airport road node graph to form a tracking trajectory that conforms to the roads within the airport, avoiding the target trajectory from appearing in abnormal areas (such as non-taxiways). The road node graph contains the static information of the road, and the obtained tracking trajectory is corrected by hard constraints to make it conform to the actual movement of the aircraft.

[0036] Through the above steps, the present invention realizes high-precision aircraft target tracking on the satellite constellation imaging data set, and optimizes the tracking path by introducing airport road constraints, effectively improving the tracking accuracy and trajectory drawing accuracy when the time difference between adjacent data is large.

[0037] like Figure 4The figure shows the target tracking effect after satellite image processing. The present invention can realize multi-target tracking of airport scenes from a satellite perspective. The figure shows the tracking frame of the target's position in the current frame, as well as the position marks and trajectory of the moving target in the past frames. The color of the moving target's trajectory line and the position points in the past frames are the same as its tracking frame. It can be seen that the target in the figure will have long-distance nonlinear motion in adjacent frames. The method of the present invention can correctly match it, and correct the trajectory in combination with the road network to obtain a historical motion trajectory that conforms to the road.

[0038] The technical advantages of the present invention are: 1. Combine trajectory prediction to realize a two-stage tracking network (a two-stage tracking network refers to the two stages of detection and tracking, which is a tracking network based on detection (TBD) to improve the performance of continuous tracking of targets and reduce the tracking error caused by time intervals.

[0039] 2. An improved multi-level matching algorithm is proposed (S401 and S402 correspond to the specific work of multi-level matching), which can effectively reduce the impact of target displacement caused by the large time difference between adjacent image frames and optimize the multi-target tracking effect.

[0040] 3. By introducing the road node graph, we can avoid the situation where the target's tracking trajectory is linear in long time difference data and does not conform to the airport runway, thereby effectively improving the accuracy of airport dynamic activity monitoring.

[0041] The present invention introduces airport road constraints on the cross-satellite imaging data set and uses the airport road information in different satellite imaging data to optimize the aircraft target tracking path. Compared with the traditional method, the present invention can more accurately limit the trajectory range of the aircraft target, make the target tracking path more reasonable, and effectively improve the target matching accuracy and trajectory continuity in complex environments.

[0042] Airport road information not only provides clear spatial constraints, but also naturally contains trajectory information, which enables this method to achieve high-precision target tracking in satellite image data with low temporal resolution.

[0043] The present invention constructs an LSTM trajectory prediction model through deep learning and combines airport road constraint information to construct a multimodal trajectory prediction method. Compared with traditional Kalman filtering, this method has higher stability and adaptability in long-term prediction.

[0044] Trajectory prediction combined with road constraints can effectively reduce prediction errors and improve the rationality of tracking trajectories when aircraft targets enter complex airport environments (such as taxiway intersections or take-off and landing path changes).

[0045] In the target matching process, the present invention adopts a double-layer optimization strategy for stationary targets and moving targets. For stationary targets, IoU matching is combined with appearance feature similarity for matching to reduce mismatches caused by aircraft stops or short movements.

[0046] For moving targets, a matching mechanism based on LSTM trajectory prediction is adopted and combined with airport road constraints for correction to improve the robustness of matching and avoid target matching loss due to long time intervals.

[0047] Relying on the collaborative imaging capability of the low-orbit satellite constellation, this method can carry out long-term (the imaging monitoring capability of the low-orbit satellite constellation cannot reach the high frequency of video data, and the present invention is more for the purpose of achieving long-term regional monitoring and making up for the short duration of satellite staring video data) and multi-angle imaging monitoring of the airport area. Trajectory prediction and road constraint optimization matching are used between multiple imaging data to effectively reduce the loss of targets due to the long time span between frames.

[0048] This method is suitable for large-scale airport target monitoring and can provide accurate dynamic tracking data support for airport activity analysis, flight trajectory monitoring, and airspace safety management.

[0049] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0050] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0051] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. A method for tracking aircraft targets across satellite airports combined with road constraints, characterized in that: The method comprises the following steps: Step S101: using a target detector to detect aircraft targets on the airport image collected by the satellite constellation, obtaining detection frame data of the aircraft targets in the scene, and calculating the time difference of the data collected by adjacent satellites in the constellation; Step S102: establishing a trajectory prediction model; using the detection box data obtained in step S101, respectively using a re-identification network and a multi-modal trajectory prediction network to extract the appearance and motion information of the aircraft target in the airport area; Step S103: Use the optimized matching algorithm to achieve cross-frame target matching, and combine the airport road constraints to make reasonable corrections to the matched trajectory to draw the final aircraft target motion trajectory.

2. The method for tracking cross-satellite airport aircraft targets in combination with road constraints according to claim 1, characterized in that: Step S101 specifically includes: Step S201: Use the object detector to identify the aircraft in the field of view, generate a rotated bounding box and crop the image data, combine the channel and spatial attention modules, and strengthen the features extracted by the backbone network.

3. The method for tracking aircraft targets across satellite airports combined with road constraints according to claim 1, characterized in that: Step S101 also includes: Step S202: calculating and giving the imaging time difference of adjacent satellites in the satellite constellation according to the imaging time information contained in the image.

4. The method for tracking aircraft targets across satellite airports combined with road constraints according to claim 2, characterized in that: The target detector is a YOLOv7 target detection network.

5. The method for tracking aircraft targets across satellite airports combined with road constraints according to claim 1, characterized in that: Step S102 further includes collecting ADS-B data of aircraft at the airport, cleaning the data, and ignoring stationary aircraft trajectories.

6. The method for tracking aircraft targets across satellite airports combined with road constraints according to claim 1, characterized in that: Step S102 further includes: Step S301: collecting aircraft historical motion trajectory data through ADS-B and preprocessing the data, including extracting aircraft target trajectories from satellite images, and performing data cleaning, outlier removal and normalization processing; Step S302: Use a LSTM-based multimodal trajectory prediction network to predict future trajectories.

7. The method for tracking aircraft targets across satellite airports combined with road constraints according to claim 5, characterized in that: In step S302, a multimodal prediction method is used to generate multiple trajectories, and Gaussian noise is added during the training process to simulate the movement changes of the aircraft and improve the adaptability of the model to uncertain movement patterns.

8. The method for tracking aircraft targets across satellite airports in combination with road constraints according to claim 5, characterized in that: Step S102 further includes: Step S303: Determine the number of prediction steps of the trajectory prediction model according to the calculated time difference, give a candidate range in combination with the trajectory prediction end point, perform cropping and obtain the cropped target image; Step S304: Using a lightweight re-identification network to extract the appearance features of the target.

9. The method for tracking aircraft targets across satellite airports in combination with road constraints according to claim 7, characterized in that: The lightweight re-ID network consists of four convolutional blocks and downsampling modules, takes the cropped target image as input, and outputs a 128×1 feature vector.

10. The method for tracking aircraft targets across satellite airports in combination with road constraints according to claim 9, characterized in that: The airport road node graph is used to correct the trajectory, matching the start and end points with the nearest neighbor road nodes, and then forming a tracking trajectory that conforms to the roads within the airport.

Images