Satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression

Abstract

The invention relates to the technical field of satellite remote sensing images, in particular to a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression, which comprises the following steps of: performing adaptive Kalman filtering data association on a plurality of target results detected by each frame of image in a satellite video to form a plurality of motion trajectories; adopting Kalman filtering to predict the plurality of motion tracks; and smoothing the motion track by adopting a Gaussian process regression model to reduce the error of the motion track so as to realize the satellite video track prediction method based on Kalman filtering and Gaussian process regression. According to the method, the ID conversion frequency can be effectively reduced, the IDF1 is improved, and the larger the value of the IDF1 represents the lower the ID conversion frequency and the higher the precision. According to the method, indexes of different methods are obtained through testing under five satellite videos in a viso data set after a plurality of tracking methods are fused.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite remote sensing images, and in particular relates to a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression. Background Art

[0002] Multi-object tracking in satellite video aims to simultaneously estimate the positions and motion trajectories of multiple tracked objects in a satellite video sequence, frame by frame. In recent years, satellite video tracking technology has been urgently needed in various fields and has gradually become a hot topic in remote sensing image processing.

[0003] Due to the characteristics of satellite video, existing tracking methods for satellite video still have some shortcomings, making it impossible to obtain the positions of multiple tracked objects frame by frame, resulting in fragmented trajectories. In terms of data association, existing multi-target tracking methods often use the appearance and motion features of the target for association. However, due to the resolution of satellite video, it is difficult to extract appearance features that can be used for data association. Some methods only use the target's motion features for data association, namely using Kalman filtering to extract the target's motion characteristics and then using the Hungarian algorithm for matching. However, due to the complex background in satellite video, the variance of the observation noise is often unknown, which can have a certain impact on the performance of the Kalman filter and ultimately the effectiveness of data association.

[0004] While linear interpolation is effective and convenient in many situations, it also has some limitations. First, it assumes that data changes linearly. For some nonlinear data, linear interpolation can lead to inaccurate estimates. Furthermore, in the presence of large data gaps or extreme values, linear interpolation can introduce significant errors. Single-object tracking is an important problem in computer vision with broad application prospects, but the module is very time-consuming to use, making it difficult to meet the requirements of some real-time engineering tasks. During the Kalman filter update process, new observations are typically used to update the estimate of the system state. This involves calculating the difference between the observed and predicted values ​​and using this difference to correct the state estimate. The update phase also involves updating the uncertainty of the state estimate by updating the error covariance matrix. However, when completing missing trajectories due to missed detections, there is no new observation data to update the system state, which can lead to some errors. Summary of the Invention

[0005] In view of this, the present invention aims to provide a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression to solve the problem of target trajectory missing in the prior art.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows: A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression includes the following steps: Perform data association of multiple targets detected in each frame of satellite video using adaptive Kalman filtering to form multiple motion trajectories. A Kalman filter is used to predict multiple motion trajectories; a Gaussian process regression model is used to smooth the motion trajectories, reduce the errors of the motion trajectories, and realize a satellite video trajectory prediction method based on Kalman filter and Gaussian process regression.

[0007] Furthermore, the data association includes: initializing the detection frame of the first frame image to an existing motion trajectory, adaptive Kalman filtering predicting the motion trajectory of the target in the second frame image through the existing motion trajectory, calculating the IOU of the predicted motion trajectory of the target in the second frame image and the second frame detection frame to obtain the similarity, fusing the predicted motion trajectory with high similarity with the existing motion trajectory to form a new motion trajectory, and so on, to achieve trajectory prediction.

[0008] Furthermore, it also includes: dividing the detection frame of each frame image into a high-score detection frame and a low-score detection frame; first associating the motion trajectory with the high-score detection frame, and then associating the low-score detection frame with a motion trajectory with low similarity, and distinguishing the background and the real target through the similarity of the motion trajectory.

[0009] Furthermore, the high-score detection frame and the low-score detection frame are determined based on confidence.

[0010] Furthermore, after forming multiple motion trajectories, the multiple targets are matched to the multiple motion trajectories using the Hungarian algorithm.

[0011] Furthermore, in the process of smoothing the motion trajectory using the Gaussian process regression model, the Gaussian process regression model of the trajectory is expressed as:

[0012] in, is the state vector of the predicted target, is the state vector of the target in the previous frame, p is the position coordinate and size of the target in the t frame, and ε is the state vector of the target in the previous frame (0, ) distributed Gaussian noise, f is a function of target position and time.

[0013] Compared with the prior art, the present invention can achieve the following beneficial effects: (1) The present invention achieves a lower number of ID (target identity) conversions during satellite video tracking than other technologies: Because satellite video tracking is prone to broken tracks, ID failure and high conversion times occur, i.e., the same ID tracking box flashes continuously and the same target has multiple different IDs. The present invention can effectively reduce the number of ID conversions and improve IDF1. The present invention integrates multiple tracking methods and then tests them on five satellite videos in the VISO dataset to obtain indicators for different methods.

[0014] (2) The present invention has higher tracking accuracy than other technologies in the satellite video tracking process: the tracking accuracy (MOTA) is reflected in the accuracy of determining the number of tracked objects and the relevant attributes of the tracked objects, and is used to count the error accumulation in tracking. MOTA is calculated by GT (true trajectory), FP (false positive), and FN (false negative). The higher the MOTA, the better the tracking effect; the lower the MOTP, the better the tracking effect; the lower the FP and FN, the better the tracking effect; after incorporating many representative tracking models, the present invention has higher tracking accuracy (MOTA) and tracking precision (MOTP).

[0015] (3) The present invention can form a more complete trajectory during the satellite video tracking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings: Figure 1 This is a comparison chart of the trajectories formed when performing multi-target tracking under satellite video using the satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression described in the embodiment of the present invention and FairMOT, TGraM, and OC-SORT.

[0017] Figure 2 This is a detailed diagram comparing the trajectories formed by the satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression described in the embodiment of the present invention and OC-SORT when performing multi-target tracking under satellite video. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention. Similar elements in different embodiments use associated similar element numbers. In the following embodiments, many detailed descriptions are intended to enable the present invention to be better understood. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, or methods. In some cases, some operations related to the present invention are not shown or described in the specification. This is to avoid the core part of the present invention being overwhelmed by too much description. For those skilled in the art, it is not necessary to describe these related operations in detail. They can fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0019] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other to form various implementation methods. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various orders in the description and the drawings are only for the purpose of clearly describing a certain embodiment and are not intended to be a required order, unless otherwise specified that a certain order must be followed.

[0020] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0021] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0022] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0023] A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression includes the following steps: Step 1: Obtain the detection bounding boxes for each frame in the satellite video, where each frame contains multiple targets. Classify the detection bounding boxes into high-scoring and low-scoring ones based on their confidence. Initialize the detection bounding boxes of the first frame to the existing motion trajectory.

[0024] When the detection frame in the second frame is a high-scoring one, the adaptive Kalman filter predicts the target's trajectory in the second frame using the existing trajectory. The IOU (Intersection-over-Union) between the predicted trajectory and the high-scoring detection frame in the second frame is calculated to determine the similarity. When the IOU is greater than or equal to 0.5, the similarity is set to high; when the IOU is less than 0.5, the similarity is set to low. The predicted trajectory with high similarity is fused with the existing trajectory to form a new trajectory, and so on, to achieve preliminary trajectory prediction. The Hungarian algorithm is used to match multiple targets to multiple trajectories.

[0025] When the detection frame of the second frame image is a low-score detection frame, the adaptive Kalman filter predicts the motion trajectory of the target in the second frame image through the existing motion trajectory, calculates the IOU of the predicted motion trajectory of the target in the second frame image and the low-score detection frame of the second frame to obtain the similarity, and merges the predicted motion trajectory with high similarity with the existing motion trajectory to form a new motion trajectory. The predicted motion trajectory with low similarity is marked for deletion, distinguishing false alarms from targets, so as to successfully track some targets with scale changes, occlusions and motion blur and filter out false alarms.

[0026] In the adaptive Kalman filter algorithm, the state vector of each target is selected as an octet:

[0027] Where x represents the horizontal coordinate of the target, and y represents the vertical coordinate of the target. The following four parameters represent the rate of change of the horizontal coordinate, the rate of change of the vertical coordinate, the rate of change of the aspect ratio of the target prediction box, and the rate of change of the height of the target prediction box. In the prediction step, the state transition matrix is ​​used to predict the new state of the target. The specific prediction formula is as follows:

[0028]

[0029] in is the predicted state vector, is the state vector of the previous frame, is the state transition matrix, is the predicted state covariance matrix, is the state covariance matrix of the previous frame, Is the process noise covariance matrix. In the update step of the adaptive Kalman filter algorithm, the predicted state vector is compared with the observation vector and the predicted state vector is updated. and the predicted state covariance matrix , forming a preliminary trajectory. The specific update formula is as follows:

[0030]

[0031]

[0032] where K is the Kalman gain matrix, is the observation matrix, is the observation noise covariance matrix, is the observation vector, is the identity matrix, is the updated target state vector, is the updated forecast state covariance matrix, the observation noise covariance matrix is considered a known constant. In practical applications, the observation noise covariance matrix may change over time or be uncertain due to other factors. To solve this problem, an adaptive method is used to estimate the observation noise variance. A commonly used adaptive method is to use the minimum variance unbiased estimate:

[0033]

[0034] where e is the estimated error vector.

[0035] Step 2: Adaptive Kalman filtering can obtain longer trajectories and reduce the number of ID conversions to a certain extent, but the trajectory will still be broken after the target is blocked for a long time. In this regard, Kalman filtering is used to predict the missing trajectory: the state vector of each target is selected as a four-tuple: , the trajectory prediction formula for the Kalman filter algorithm is: .

[0036] Due to the absence of the target in the image, there is no observation value to update the target's motion state, which brings a certain deviation to the final trajectory. Therefore, the Gaussian process regression model is used to smooth the final trajectory.

[0037] The Gaussian process regression model of the trajectory is expressed as:

[0038] in, is the state vector of the predicted target, is the state vector of the target in the previous frame, and p is the position coordinate and size of the target in the t frame, that is, ,ε is subject to (0, ) distributed Gaussian noise, f is a function of target position and time, assuming f obeys a Gaussian process:

[0039] Where k is the Gaussian process kernel, is a hyperparameter that controls the smoothness of the trajectory, 、 are the independent variables before smoothing and the newly given independent variables respectively. According to the properties of the Gaussian process, a new frame set is given , the smoothed position It is given by the following formula:

[0040] Among them, P is the position of the target in each frame before smoothing, F and is the set of frames before and after smoothing, is the variance of Gaussian noise, and I is the identity matrix.

[0041] Implement a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression.

[0042] This method reduces the number of ID (target identity) transitions during satellite video tracking compared to other technologies. Tracking with satellite video often results in broken tracks, leading to ID failure and high ID transitions. This means the same ID tracking box constantly flashes and the same target has multiple different IDs. This method effectively reduces ID transitions and improves IDF1 (an evaluation metric for multi-target tracking). By integrating multiple tracking methods with this method, we tested the performance of each method using five satellite videos from the VISO dataset. Specific performance indicators are shown in Table 1. A higher IDF1 indicates better tracking performance, while a lower IDs (target identity transitions) indicates better tracking performance.

[0043] The calculation formulas for MOTA and MOTP are as follows:

[0044]

[0045] GT, FP, and FN represent the true trajectory, false positive, and false negative indicators, respectively. t Indicates the number of successful matches in t frame, d t The experimental results in Table 1 show that the present invention has higher tracking accuracy (MOTA) and tracking precision (MOTP) after incorporating many representative tracking models.

[0046] During the experiment, by comparing the trajectories formed by the present invention when tracking multiple targets under satellite video with the trajectories formed by other methods (FairMOT, TGraM, OC-SORT) when tracking multiple targets under satellite video, it can be found that the present invention can obtain more complete trajectories. The trajectory diagram formed is as follows: Figure 1 The specific details are as shown in Figure 2 shown.

[0047] Table 1. Tracking metrics under the same test set

[0048] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0049] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression, characterized in that: The method comprises the following steps: Perform adaptive Kalman filtering on the data association of multiple target results detected in each frame of satellite video to form multiple motion trajectories; Use Kalman filtering to predict multiple motion trajectories; A Gaussian process regression model is used to smooth the motion trajectory, reduce the error of the motion trajectory, and implement a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression.

2. The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression according to claim 1 is characterized in that: The data association includes: initializing the detection frame of the first frame image to an existing motion trajectory, adaptive Kalman filtering predicting the motion trajectory of the target in the second frame image through the existing motion trajectory, calculating the IOU of the predicted motion trajectory of the target in the second frame image and the second frame detection frame to obtain similarity, fusing the predicted motion trajectory with high similarity with the existing motion trajectory to form a new motion trajectory, and so on to achieve trajectory prediction.

3. The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression according to claim 2 is characterized in that: Also includes: The detection frame of each frame image is divided into high-score detection frame and low-score detection frame; First, the motion trajectory is associated with the high-score detection frame, and then the low-score detection frame is associated with the motion trajectory with low similarity, and the background and the real target are distinguished by the similarity of the motion trajectory.

4. The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression according to claim 3 is characterized in that: The high-score detection frame and the low-score detection frame are determined based on confidence.

5. The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression according to claim 1 is characterized in that: After forming multiple motion trajectories, multiple targets are matched into the multiple motion trajectories using the Hungarian algorithm.

6. The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression according to claim 1 is characterized in that: In the process of smoothing the motion trajectory using the Gaussian process regression model, the Gaussian process regression model of the trajectory is expressed as: in, is the state vector of the predicted target, is the state vector of the target in the previous frame, p is the position coordinate and size of the target in the t frame, and ε is the state vector of the target in the previous frame (0, ) distributed Gaussian noise, f is a function of target position and time.

Images