Satellite video track prediction method based on kalman filtering and gaussian process regression

Abstract

This invention relates to the field of satellite remote sensing image technology, and particularly to a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression. The method includes the following steps: adaptive Kalman filtering is used to correlate multiple target results detected in each frame of satellite video to form multiple motion trajectories; Kalman filtering is used to predict the multiple motion trajectories; and a Gaussian process regression model is used to smooth the motion trajectories, reducing the error and realizing the satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression. This invention can effectively reduce the number of ID transformations and improve IDF1. A higher IDF1 value indicates a lower number of ID transformations and higher accuracy. This invention integrates multiple tracking methods and tests them on five satellite videos in the Visio dataset to obtain the metrics of different methods.

Description

Technical Field

[0001] This invention belongs to the field of satellite remote sensing image technology, and in particular relates to a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression. Background Technology

[0002] Multi-object tracking technology in satellite video aims to simultaneously estimate the positions and acquire the motion trajectories of multiple objects frame by frame in a satellite video sequence. In recent years, tracking technology based on satellite video has been urgently needed in many fields and has gradually become one of the hot topics in the field of remote sensing image processing.

[0003] Due to the characteristics of satellite video, existing tracking methods still have some shortcomings, such as the inability to acquire the positions of multiple tracked objects frame by frame, resulting in fragmented trajectories. Regarding data association, existing multi-target tracking methods mostly use the appearance and motion features of targets for association. However, in satellite video, due to its low resolution, it is difficult to extract usable appearance features for data association. Some methods only use the motion features of targets for data association, i.e., using Kalman filtering to extract the motion characteristics of targets and then using the Hungarian algorithm for matching. However, because of the complex background in satellite video, the variance of observation noise is often unknown, which can affect the performance of Kalman filtering and ultimately the effectiveness of data association.

[0004] While linear interpolation is effective and convenient in many cases, it also has some limitations. First, linear interpolation assumes that data changes linearly; for some non-linear data, it may lead to inaccurate estimates. Furthermore, linear interpolation can introduce significant errors in cases of large data intervals or extreme values. Single-object tracking is an important problem in computer vision with broad application prospects, but the modules are very time-consuming in practice, making it difficult to meet the needs of some real-time engineering tasks. In the Kalman filter update process, new observation data is typically used to update the estimate of the system state. This involves calculating the difference between the observed and predicted values ​​and using this difference to correct the state estimate. The update phase also involves updating the uncertainty of the state estimate by updating the error covariance matrix. However, when completing missing trajectories due to missed detections, no new observation data is available to update the system state, thus introducing some errors. Summary of the Invention

[0005] In view of this, the present invention aims to provide a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression, to solve the problem of missing target trajectories in the prior art.

[0006] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0007] A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression includes the following steps:

[0008] Adaptive Kalman filtering is used to correlate multiple target results detected in each frame of satellite video to form multiple motion trajectories;

[0009] Kalman filtering is used to predict multiple motion trajectories; a Gaussian process regression model is used to smooth the motion trajectories, reducing the error of the motion trajectories, thus realizing a satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression.

[0010] Furthermore, the data association includes: initializing the detection box of the first frame image as an existing motion trajectory; using adaptive Kalman filtering to predict the motion trajectory of the target in the second frame image through the existing motion trajectory; calculating the similarity between the predicted motion trajectory of the target in the second frame image and the detection box of the second frame image; fusing the predicted motion trajectory with high similarity with the existing motion trajectory to form a new motion trajectory; and so on, to achieve trajectory prediction.

[0011] Furthermore, it also includes: dividing the detection boxes of each frame of the image into high-scoring detection boxes and low-scoring detection boxes; first associating the motion trajectory with the high-scoring detection boxes, and then associating the low-scoring detection boxes with motion trajectories with low similarity, so as to distinguish the background and the real target by the similarity of the motion trajectories.

[0012] Furthermore, the high-scoring detection boxes and low-scoring detection boxes are determined based on confidence levels.

[0013] Furthermore, after forming multiple motion trajectories, the Hungarian algorithm is used to match multiple targets to multiple motion trajectories.

[0014] Furthermore, in the process of smoothing the motion trajectory using a Gaussian process regression model, the Gaussian process regression model of the trajectory is expressed as:

[0015]

[0016] in, The state vector of the predicted target. Let p be the state vector of the target in the previous frame, p be the position coordinates and size of the target in frame t, and ε be the value that follows the property (0, 1). The target is a Gaussian noise distributed as f, where f is a function of the target's position and time.

[0017] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0018] (1) This invention achieves a lower number of ID (target identification) conversions during satellite video tracking compared to other technologies: Satellite video tracking is prone to broken trajectories, leading to ID failures and high conversion rates, i.e., the same ID tracking frame constantly flashes and the same target has multiple different IDs. This invention effectively reduces the number of ID conversions and improves IDF1. This invention combines multiple tracking methods and tests them on five satellite videos within the Visio dataset to obtain the metrics for different methods.

[0019] (2) The present invention achieves higher tracking accuracy than other technologies in satellite video tracking: Tracking accuracy (MOTA) is reflected in the accuracy of determining the number of tracked objects and the relevant attributes of the tracked objects, and is used to statistically analyze the accumulation of errors during tracking. MOTA is calculated using GT (True Track), FP (False Positive), and FN (False Negative). The higher the MOTA, the better the tracking effect; the lower the MOTP, the better the tracking effect; the lower the FP and FN, the better the tracking effect; the present invention, after incorporating many representative tracking models, has higher tracking accuracy (MOTA) and tracking precision (MOTP).

[0020] (3) The present invention can form a more complete trajectory during satellite video tracking. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0022] Figure 1 The satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression described in the embodiments of the present invention is compared with the trajectory generated by FairMOT, TGraM, and OC-SORT when performing multi-target tracking in satellite video.

[0023] Figure 2 A detailed comparison of the satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression described in the embodiments of the present invention with the trajectory formed by OC-SORT when performing multi-target tracking on satellite video. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0026] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression includes the following steps:

[0030] Step 1: Obtain the detection bounding boxes for each frame of the satellite video, where each frame contains multiple targets; classify the detection bounding boxes into high-scoring and low-scoring boxes based on confidence level. Initialize the detection bounding boxes of the first frame image with existing motion trajectories.

[0031] When the detection bounding box in the second frame is a high-resolution detection bounding box, the adaptive Kalman filter predicts the motion trajectory of the target in the second frame using the existing motion trajectory. The similarity is calculated by the Intersection over Union (IOU) between the predicted motion trajectory of the target in the second frame and the high-resolution detection bounding box. A high similarity is assigned when the IOU is greater than or equal to 0.5, and a low similarity is assigned when the IOU is less than 0.5. The predicted motion trajectory with high similarity is fused with the existing motion trajectory to form a new motion trajectory, and this process is repeated to achieve initial trajectory prediction. The Hungarian algorithm is used to match multiple targets to multiple motion trajectories.

[0032] When the detection box in the second frame image is a low-resolution detection box, the adaptive Kalman filter predicts the motion trajectory of the target in the second frame image based on the existing motion trajectory. It calculates the similarity between the predicted motion trajectory of the target in the second frame image and the low-resolution detection box in the second frame image. The predicted motion trajectory with high similarity is fused with the existing motion trajectory to form a new motion trajectory. The predicted motion trajectory with low similarity is marked for deletion. This distinguishes between false alarms and targets, so as to successfully track targets with scale changes, occlusion, and motion blur and filter out false alarms.

[0033] In the adaptive Kalman filter algorithm, the state vector of each target is selected as an octet:

[0034]

[0035] Where x represents the target's horizontal coordinate and y represents the target's vertical coordinate. `h` represents the aspect ratio of the target detection box, and `h` represents the height of the target detection box. The last four parameters of the octet represent the rate of change of the horizontal coordinate, the rate of change of the vertical coordinate, the rate of change of the aspect ratio of the target prediction box, and the rate of change of the height of the target prediction box. In the prediction step, the state transition matrix is ​​used to predict the new state of the target. The specific prediction formula is as follows:

[0036]

[0037]

[0038] in It is the predicted state vector. It is the state vector of the previous frame. It is the state transition matrix. It is the predicted state covariance matrix. It is the state covariance matrix of the previous frame. This is the process noise covariance matrix. In the update step of the adaptive Kalman filter algorithm, the predicted state vector is compared with the observed vector, and the predicted state vector is updated accordingly. and the predicted state covariance matrix This forms a preliminary trajectory. The specific update formula is as follows:

[0039]

[0040]

[0041]

[0042] Where K is the Kalman gain matrix. It is the observation matrix. It is the observation noise covariance matrix. It is the observation vector. It is an identity matrix. It is the updated target state vector. It is the updated predicted state covariance matrix and the observation noise covariance matrix. It is considered a known constant. In practical applications, the observation noise covariance matrix may change over time or be uncertain due to other factors. To address this issue, an adaptive method is used to estimate the observation noise variance. A commonly used adaptive method is to use the minimum variance unbiased estimate:

[0043]

[0044]

[0045] Where e is the estimation error vector.

[0046] Step 2: Adaptive Kalman filtering can achieve longer trajectories and reduce the number of ID transitions to some extent, but trajectory breaks still occur after prolonged target occlusion. To address this, Kalman filtering is used to predict the missing trajectories: the state vector of each target is selected as a four-tuple: The trajectory prediction formula for the Kalman filter algorithm is as follows: .

[0047] Because the target is missing in the image, there are no observations to update the target's motion state, which introduces a certain deviation into the final trajectory. Therefore, a Gaussian process regression model is used to smooth the final trajectory.

[0048] The Gaussian process regression model of the trajectory is expressed as:

[0049]

[0050] in, The state vector of the predicted target. Let p be the state vector of the target in the previous frame, and p be the position coordinates and size of the target in frame t. ε is subject to (0, The noise is distributed as Gaussian noise, where f is a function of the target position and time, and f is assumed to follow a Gaussian process:

[0051]

[0052] Where k is the kernel of the Gaussian process, The hyperparameters for controlling trajectory smoothness, , These are the independent variables before smoothing and the newly given independent variables, respectively. Based on the properties of Gaussian processes, given a new frame set... Smoothed position Given by the following formula:

[0053]

[0054] Where P is the position of the target in each frame before smoothing, and F and It is the frame set before and after smoothing. Let V be the variance of the Gaussian noise, and I be the identity matrix.

[0055] A satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression is implemented.

[0056] This invention achieves a lower number of ID (target identity) transformations during satellite video tracking compared to other technologies. Satellite video tracking is prone to broken trajectories, leading to ID failures and high transformation counts, such as the same ID tracking frame constantly flashing and multiple different IDs for the same target. This invention effectively reduces the number of ID transformations and improves IDF1 (multi-target tracking performance index). By fusing multiple tracking methods using this invention, the metrics for different methods were obtained through testing on five satellite videos using the Visio dataset. Specific metrics are shown in Table 1, where a higher IDF1 indicates better tracking performance, and lower IDs (target identity transformation counts) also indicate better tracking performance.

[0057] The formulas for calculating MOTA and MOTP are shown below.

[0058]

[0059]

[0060] GT, FP, and FN represent the true trajectory, false positive, and false negative indicators, respectively. t d represents the number of successful matches in frame t. t This represents the distance between the target and the Kalman filter prediction result. The experimental results in Table 1 show that this invention achieves higher tracking accuracy (MOTA) and tracking precision (MOTP) after incorporating many representative tracking models.

[0061] During the experiment, by comparing the trajectory generated by this invention for multi-target tracking under satellite video with the trajectories generated by other methods (FairMOT, TGraM, OC-SORT) for multi-target tracking under satellite video, it can be found that this invention can obtain a more complete trajectory, as shown in the trajectory diagram below. Figure 1 As shown, specific details are as follows: Figure 2 As shown.

[0062] Table 1. Tracking metrics under the same test set

[0063]

[0064] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0065] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A satellite video track prediction method based on Kalman filtering and Gaussian process regression, characterized in that, The method comprises the following steps: Adaptive Kalman filtering data association is performed on the multiple target results detected in each frame of image in the satellite video to form multiple motion trajectories; in the process of adaptive Kalman filtering, an adaptive method is used to estimate the observation noise variance; Kallman filter is used to predict multiple motion trajectories: the state vector of each target is selected as a four-tuple: The trajectory prediction formula for the Kallman filter algorithm is: where x represents the horizontal coordinate of the target, y represents the vertical coordinate of the target, the aspect ratio of the target detection box, h represents the height of the target detection box, is the state transition matrix, is the predicted state vector of the target, is the state vector of the target in the previous frame; Gaussian process regression model is used to smooth the motion trajectories, reduce the error of the motion trajectories, and realize the satellite video trajectory prediction method based on Kalman filtering and Gaussian process regression.

2. The satellite video track prediction method based on Kalman filtering and Gaussian process regression according to claim 1, characterized in that, The data association comprises the following steps: the detection frame of the first frame of image is initialized as an existing motion trajectory, the adaptive Kalman filtering is used to predict the motion trajectory of the target in the second frame of image through the existing motion trajectory, the IOU of the predicted motion trajectory of the target in the second frame of image and the second frame detection frame is calculated to obtain the similarity, the predicted motion trajectory with high similarity is fused with the existing motion trajectory to form a new motion trajectory, and the trajectory prediction is realized through the similar method.

3. The satellite video track prediction method based on Kalman filtering and Gaussian process regression according to claim 2, characterized in that, Further comprising: The detection frame of each frame of image is divided into high-score detection frames and low-score detection frames; The motion trajectory is associated with the high-score detection frame first, and then the low-score detection frame is associated with the motion trajectory with low similarity, and the similarity of the motion trajectory is used to distinguish the background and the real target.

4. The satellite video track prediction method based on Kalman filtering and Gaussian process regression according to claim 3, characterized in that, The high-score detection frame and the low-score detection frame are determined according to the confidence.

5. The satellite video track prediction method based on Kalman filtering and Gaussian process regression according to claim 1, characterized in that, After the multiple motion trajectories are formed, the multiple targets are matched into the multiple motion trajectories through the Hungarian algorithm.

6. The satellite video track prediction method based on Kalman filtering and Gaussian process regression according to claim 1, characterized in that, In the process of smoothing the motion trajectories by using the Gaussian process regression model, the Gaussian process regression model of the trajectory is expressed as: where p is the position coordinate and size of the target in frame t, ε is a Gaussian noise obeying (0, ) distribution, and f is a function of the target position and time.

Images