Satellite video multi-target tracking method and system based on space-time state propagation and progressive trajectory association

Abstract

The invention relates to the technical field of remote sensing earth observation, in particular to a satellite video multi-target tracking method and system based on space-time state propagation and progressive trajectory association, and the method comprises the steps: obtaining a target detection result and a state transition result of adjacent frames in a video sequence frame image through a target tracking model; the target detection result comprises a target detection frame position representation and a target category representation, the state transition result comprises a coordinate transition result and a foreground probability, calculating a spatial position matching degree by using an intersection-to-union ratio, and calculating an adjacent frame target appearance feature similarity by using a cosine distance; and combining the spatial position matching degree and the appearance feature similarity to calculate a target space-time matching affinity, associating each target tracking frame in each video sequence frame based on the target space-time matching affinity, and adding the tracking frames to the corresponding target trajectory fragments to generate each target tracking trajectory. The core intelligence information can be accurately and efficiently extracted from the satellite video data, and the utilization efficiency of the satellite video data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing earth observation, and in particular to a satellite video multi-target tracking method and system based on spatiotemporal state propagation and progressive trajectory association. Background Art

[0002] With the advancement of remote sensing Earth observation technology, more and more new satellite sensors have been introduced, and the application areas of remote sensing data are continuously expanding. Currently, static remote sensing imagery can no longer fully meet the dynamic Earth observation needs of related industries. Video satellites "stare" at target areas based on real-time observation missions or pre-stored observation plans, recording changes within the field of view as video, thereby providing information on the target's state in both time and space. The massive amount of Earth observation data acquired by video satellites not only provides static images of the sensor's field of view but also increases the temporal resolution of remote sensing data imaging to the second level. This has become a critical data source for providing real-time services in intelligent systems for constellations and a major direction for the future development of remote sensing dynamic monitoring.

[0003] Satellite video multi-object tracking (MOT) is the process of processing and analyzing satellite video (sequences) and leveraging the information captured by sensors to locate targets. Key technologies include obtaining high-precision target detection results, predicting the inter-frame propagation of target states, and data association and trajectory management for identical targets. Multi-object tracking, a fundamental aspect of satellite video interpretation, effectively captures information such as an object's speed, acceleration, direction, and posture, providing a safe, efficient, and large-scale monitoring method. Therefore, satellite video multi-object tracking technology can play a vital role in national economic and defense development, including emergency response, traffic control, battlefield monitoring, and weapon guidance.

[0004] Early multi-object tracking algorithms in satellite videos mostly used shallow semantic features. These methods had poor robustness and generalization capabilities and could only be applied to simple scenes. The successful application of convolutional neural networks (CNNs) in multi-object tracking in natural scene videos, as well as the rapid development of tracking by detection (TBD) and joint detection and tracking (JDT) methods in natural scenes, have provided new ideas for multi-object tracking in satellite videos. Commonly tracked objects in satellite videos, such as aircraft, ships, and vehicles, are typically significantly smaller than those in natural scene videos or drone videos. Some objects occupy only a few pixels, and the semantic features extracted from objects in satellite videos are very limited. Although the improvement in spatial resolution of satellite videos enables tracking of smaller ground targets, it also leads to more interference in the feature extraction process. Similar background interference, partial occlusion, and excessively fast movement speeds can lead to problems such as missed detections, model drift (the tracker loses the target or can only track a small part of the target), and frequent switching and disconnection of track segments. Moreover, the resolution of most natural scene multi-target tracking datasets and drone MOT datasets is approximately 1080P (1920×1080 pixels) or even smaller. In contrast, satellite videos cover a wider ground area with a field of view of almost 4K (4096×3072 pixels). Existing satellite video MOT algorithms lack practical verification on very large image sizes, hindering their application in real-world scenarios. Summary of the Invention

[0005] To this end, the present invention provides a satellite video multi-target tracking method and system based on spatiotemporal state propagation and progressive trajectory association, which solves the application limitations of existing satellite video multi-target tracking caused by limited spatial feature representation, frequent switching and disconnection of trajectory segments, and lack of ultra-wide-angle video verification.

[0006] According to the design scheme provided by the present invention, on the one hand, a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association is provided, comprising:

[0007] Obtain the video sequence frames that require target tracking;

[0008] Utilize a target tracking model to obtain target detection results and state transition results for adjacent frames in a video sequence frame image. The target detection results include: target detection frame position representation and target category representation. The state transition results include: coordinate transition and foreground probability. Based on the target detection frame position representation and coordinate transition, an intersection-over-union (IoU) spatial matching is obtained. Based on the target category representation and foreground probability, appearance features of adjacent frames are extracted and similarity of appearance features of adjacent frames is calculated. Coordinate transition is used to represent the change in spatial position of the same target in adjacent frames. Foreground probability represents the category prediction result of the target in the next frame. The target tracking model is obtained by training a neural network using image data samples.

[0009] The target spatiotemporal matching affinity is calculated by combining the intersection-over-union spatial matching result and the appearance feature similarity. Based on the target spatiotemporal matching affinity, each target tracking frame in each video sequence frame is obtained and added to the corresponding target trajectory segment to generate each target tracking track.

[0010] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, the target tracking model is obtained by using static image data as training data, constructing enhanced image data for each static image through data enhancement, and training a neural network using training sample pairs consisting of the static image data and the enhanced image data;

[0011] Alternatively, dynamic satellite video sequence image data is used as training data, adjacent frames are randomly sampled at a specified interval, and the neural network is trained using training sample pairs consisting of the randomly sampled adjacent frames.

[0012] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, the target tracking model includes: a feature extraction network that combines spatial resolution and semantic information to perform multi-scale feature extraction on the image, a detection branch that obtains target category representation and detection frame position representation based on the extracted image features, and an association branch that associates the spatial positions of the same target in related frames based on the extracted image features and obtains coordinate migration and foreground probability. The detection branch includes an RPN network and a regional target recognition network. The association branch takes the center point of each detection frame in the previous frame as a reference and expands outward a specified number of times along the length and width directions on the current frame image and performs similarity association on the same target features between adjacent frames based on the target context search area and channel attention mechanism.

[0013] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, using image data samples to train a neural network, it includes:

[0014] Constructing a network training loss function, wherein the network training loss function includes RPN network training loss, regional target recognition network training loss, associated branch coordinate migration loss and appearance feature similarity loss;

[0015] The neural network is trained using image data samples and based on the network training loss function.

[0016] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, based on target category representation and foreground probability, the appearance features of adjacent frames are extracted and the similarity of the appearance features of adjacent frames is calculated, which includes:

[0017] Taking the center point of the target detection frame of the previous frame as the reference, expand the current frame image outward by λ times in width and length respectively to obtain the current frame target context search area, where λ is greater than 1;

[0018] Extract each target area in the previous frame according to the specified pixel size, and extract the context search target area corresponding to λ times the pixel size in the target context search area in the current frame;

[0019] Convolutional coding is used to calculate the similarity feature map between each target area in the previous frame and the context search target area in the current frame;

[0020] For the cross-frame state propagation features represented by foreground probabilities, a shared network consisting of average pooling and maximum pooling is used to encode them respectively. The encoded vectors are element-wise added and sigmoid normalized to obtain the channel attention response map. The attention response map is element-wise multiplied with the cross-frame state propagation features, and the multiplication result is again feature-added with the cross-frame state propagation features. A fully connected layer is used to map the feature addition result into an appearance feature vector.

[0021] Cosine similarity is used to measure the similarity of appearance feature vectors between frames.

[0022] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, the target spatiotemporal matching affinity is calculated by combining the intersection-over-union spatial matching result and the appearance feature similarity, including:

[0023] Generate a target position affinity matrix based on the intersection-over-union ratio between the spatial positions of the target detection and the predicted bounding box in the current frame, where the predicted bounding box is obtained by migrating the coordinates of the target motion state;

[0024] The cosine distance is used to measure the similarity between the target appearance features and the target apparent state transition features, and the target appearance affinity matrix is generated;

[0025] The target spatiotemporal matching affinity discriminant matrix is constructed based on the target position affinity matrix and the target appearance affinity matrix.

[0026] As a satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association of the present invention, further, each target tracking frame in each video sequence frame is obtained based on the target spatiotemporal matching affinity, including:

[0027] If the target spatiotemporal affinity is greater than the threshold, the corresponding target detection result is deleted, the corresponding state transition result is retained, and the detection confidence of the corresponding state transition result and the detection confidence of the remaining target detection results are calculated;

[0028] If the detection confidence of the remaining target detection results is greater than the specified detection threshold, a new track is added; otherwise, the corresponding target detection results are deleted.

[0029] If the detection confidence of the corresponding state transition result is greater than the specified detection threshold, the target trajectory length is increased by 1. Otherwise, the target trajectory is temporarily dormant and an interval frame T is set. Every T / 3 interval frame, the specified detection threshold is reduced by 0.1. It is then determined whether the detection confidence of the state transition result within the corresponding interval frame is greater than the reduced specified detection threshold. If so, the dormant target trajectory is activated and the target trajectory length is increased by 1. Otherwise, the corresponding target trajectory tracking is terminated.

[0030] On the other hand, the present invention also provides a satellite video multi-target tracking system based on spatiotemporal state propagation and progressive trajectory association, comprising: a data acquisition module, a target tracking module and a trajectory generation module, wherein:

[0031] A data acquisition module is used to acquire video sequence frames that require target tracking;

[0032] A target tracking module is configured to use a target tracking model to obtain target detection results and state transition results for adjacent frames in a video sequence frame image. The target detection results include: a target detection frame position representation and a target category representation; the state transition results include: a coordinate transition and a foreground probability; and to obtain an intersection-over-union (IoU) spatial matching based on the target detection frame position representation and the coordinate transition. Appearance features of adjacent frames are extracted based on the target category representation and the foreground probability, and the similarity of appearance features of adjacent frames is calculated. Coordinate transition is used to represent the change in the spatial position of the same target in adjacent frames, and the foreground probability represents the category prediction result of the target in the next frame. The target tracking model is obtained by training a neural network using image data samples.

[0033] The trajectory generation module is used to calculate the target spatiotemporal matching affinity by combining the intersection-over-union spatial matching results and the appearance feature similarity. Based on the target spatiotemporal matching affinity, the tracking frame of each target in each video sequence frame is obtained and the tracking frame is added to the corresponding target trajectory segment to generate the tracking trajectory of each target.

[0034] Beneficial effects of the present invention:

[0035] The present invention uses a target tracking model to realize the recognition and positioning of each target category to be tracked in the scene, combines the spatiotemporal information transmission target inter-frame filling, obtains the target position and appearance migration results, improves the target trajectory integrity, and generates unique appearance features for each target through the channel attention mechanism, which can resist the mismatch problem caused by background interference; fully considers the inter-frame consistency, jointly uses the target appearance and motion information, and adopts different association thresholds for different trajectory segments to effectively integrate the detection box into the existing target trajectory, thereby improving the integrity of the trajectory segment and the target tracking accuracy, and has good application prospects in the field of remote sensing earth observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a multi-target tracking process of satellite video based on spatiotemporal state propagation and progressive trajectory association in an embodiment;

[0037] Figure 2 This is a schematic diagram of the satellite video multi-target tracking algorithm framework in the embodiment;

[0038] Figure 3 This is a schematic diagram of the spatiotemporal state propagation principle between frames in the embodiment;

[0039] Figure 4 This is a schematic diagram of the target appearance modeling process based on the channel attention mechanism in the embodiment;

[0040] Figure 5 This is a schematic diagram of the trajectory segment association strategy process in the embodiment;

[0041] Figure 6 Schematic diagram of the trajectory visualization results of the selected video in the dataset using the comparison method in the embodiment. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below with reference to the accompanying drawings and technical solutions.

[0043] Multi-object tracking (MOT) aims to identify multiple objects of interest in a given video while keeping the identification (ID) of each object consistent, so as to achieve data association of the same object in different frames. Data association aims to match the detection boxes of the same target in the video sequence images and form a continuous trajectory. Existing multi-object tracking methods in natural scenes have failed to fully consider the impact of factors such as limited spatial feature representation, frequent switching and disconnection of trajectory segments, and lack of ultra-wide viewing angle video verification on multi-object tracking. Compared with natural scene videos, the targets to be tracked in satellite videos are mostly rigid objects. Except for the large deformation of trains when making sharp turns, the deformation amplitude of targets such as airplanes and ships is much smaller than that of targets to be tracked in natural scene videos (mainly pedestrians). In other words, the appearance and posture of the same target between two adjacent frames have a high similarity. In addition, the trajectories of these rigid targets are approximately straight lines in a short period of time, and the moving distance of the same target between two adjacent frames is spatially limited. Based on the above analysis of the characteristics of satellite video targets, an embodiment of the present invention provides a satellite video multi-object tracking method based on spatiotemporal state propagation and progressive trajectory association, such as Figure 1 As shown, it specifically includes the following contents:

[0044] S101: Obtain video sequence frames requiring target tracking.

[0045] S102. Utilize a target tracking model to obtain target detection results and state migration results of adjacent frames in a video sequence frame image, wherein the target detection results include: target detection frame position representation and target category representation, and the state migration results include: coordinate migration and foreground probability, and obtain intersection-over-union spatial matching based on the target detection frame position representation and coordinate migration, extract appearance features of adjacent frames based on the target category representation and foreground probability, and calculate the similarity of appearance features of adjacent frames, wherein coordinate migration is used to represent the change in spatial position of the same target in adjacent frames, and foreground probability represents the category prediction result of the target in the next frame, and the target tracking model is obtained by training a neural network using image data samples.

[0046] Among them, the target tracking model can be obtained by using static image data as training data, constructing enhanced image data for each static image through data enhancement, and training the neural network using training sample pairs consisting of static image data and enhanced image data; or by using dynamic satellite video sequence image data as training data, randomly extracting adjacent frames at a specified interval, and training the neural network using training sample pairs consisting of randomly extracted adjacent frames.

[0047] S103. Calculate the target spatiotemporal matching affinity by combining the IoU spatial matching result and the appearance feature similarity. Obtain each target tracking frame in each video sequence frame based on the target spatiotemporal matching affinity, and add the tracking frame to the corresponding target trajectory segment to generate each target tracking trajectory.

[0048] like Figure 2 The Propagating Spatio-temporal State and Progressively Associating Trajectory for Satellite Video Multiple Object Tracking (PS2PAT-MOT) architecture shown in Figure 2 takes a pair of images as input during training. The branch that processes the original image is the original branch (denoted as ori), and the data-augmented image is fed into the augmented branch (denoted as aug). Available training data includes static target detection image data and dynamic satellite video sequence image data. When using static target detection images as training data, data augmentation can be used to construct training sample pairs for each image. When using dynamic satellite video sequence images as training data, two adjacent frames can be randomly sampled at a certain interval as training sample pairs. The final optimization results of the detection branch include category representation and position representation.

[0049] Specifically, the target tracking model can be designed to include: a feature extraction network that combines spatial resolution and semantic information to extract multi-scale features of the image, a detection branch that obtains target category representation and detection box position representation based on the extracted image features, and an association branch that associates the spatial positions of the same target in related frames based on the extracted image features and obtains coordinate migration and foreground probability. The detection branch includes an RPN network and a regional target recognition network. The association branch takes the center point of each detection box in the previous frame as the reference and expands outward a specified number of times in the length and width directions of the current frame image and performs similarity association on the same target features between adjacent frames based on the target context search area and channel attention mechanism.

[0050] The feature extraction network can use DLA34 for multi-scale feature extraction, combining high spatial resolution and strong semantic information. The extracted features are then sent to the detection branch and the association branch respectively to achieve target recognition and similar target association capabilities. The detection branch can be composed of an RPN network to generate a large number of candidate boxes and pass L rpn Optimize, and a Region-based target recognition network, through L odOptimize. The explanation of these two loss functions can be found in FasterR-CNN. The final optimization results of the detection branch include category representation and position representation

[0051] In the association branch, the spatio-temporal State Propagation (STSP) module can be used to associate the spatial positions of the same target in adjacent frames. Specifically, the candidate boxes generated by RPN are sampled to obtain the RoI (region of interest) with the highest score corresponding to each real target. For each RoI in the ori branch image block, its coordinates are mapped to the aug image block based on the center point and expanded outward by λ times (λ>1) in the length and width directions. The features of the RoI in the ori image block are cross-correlated with the context search area in the aug image block in a depth-wise convolution manner. The state propagation process is performed through the loss function L sp Optimize coordinate migration and prospect probability

[0052] During target motion, occlusion, shadows, and similar background interference may lead to incomplete visual appearance information, which poses a huge challenge to spatial position prediction and matching feature extraction, leading to data association failure. This is also the root cause of model drift. Therefore, in order to cope with the interference of multiple similar backgrounds in the tracking scene, the classification representation is obtained. and prospect probability Based on this, in this embodiment, the SPST module uses an appearance modeling method based on the channel attention mechanism to generate appearance features. and The cosine distance is used to measure the similarity between the two vectors, which is expressed as:

[0053]

[0054] The similarity is measured through the loss function L appa To optimize:

[0055]

[0056] The final satellite video multi-target tracking framework PS2PAT-MOT optimization loss function can be expressed as:

[0057] L=L rpn +L od +L sp +L appa (3)

[0058] During tracking, the PS2PAT-MOT framework uses the results of the previous frame as prior tracking information for the current frame. After extracting shared features, the detection branch detects the current frame to obtain multi-category targets. The association branch uses the center point of each tracking box in the previous frame as the reference point and expands the current frame image by a factor of λ in both length and width, encoding similarities through depthwise convolution. The resulting cross-correlation feature map propagates the location and probability of the target.

[0059] At the same time, the position representation in the detection branch and coordinate migration in related branches Spatial matching is performed through Intersection over Union (IoU). Category representation in the detection branch and the foreground probability in the relevant branch First, the appearance feature (AF) is extracted through the attention mechanism, and then the similarity is calculated using the cosine vector. After the progressively associating trajectory (PAT) strategy, the affinity metric formed by the IoU matching results and the similarity is used to generate tracking boxes for each target in the current frame. These tracking boxes are then added to the existing track segments to generate the tracking results.

[0060] Among them, the training of the neural network using image data samples can be designed to include:

[0061] Constructing a network training loss function, wherein the network training loss function includes RPN network training loss, regional target recognition network training loss, associated branch coordinate migration loss and appearance feature similarity loss;

[0062] The neural network is trained using image data samples and based on the network training loss function.

[0063] Among them, based on the target category representation and foreground probability, the appearance features of adjacent frames are extracted and the similarity of the appearance features of adjacent frames is calculated, which can be designed to include:

[0064] Taking the center point of the target detection frame of the previous frame as the reference, expand the current frame image outward by λ times in width and length respectively to obtain the current frame target context search area, where λ is greater than 1;

[0065] Extract each target area in the previous frame according to the specified pixel size, and extract the context search target area corresponding to λ times the pixel size in the target context search area in the current frame;

[0066] Convolutional coding is used to calculate the similarity feature map between each target area in the previous frame and the context search target area in the current frame;

[0067] For the cross-frame state propagation features represented by foreground probabilities, a shared network consisting of average pooling and maximum pooling is used to encode them respectively. The encoded vectors are element-wise added and sigmoid normalized to obtain the channel attention response map. The attention response map is element-wise multiplied with the cross-frame state propagation features, and the multiplication result is again feature-added with the cross-frame state propagation features. A fully connected layer is used to map the feature addition result into an appearance feature vector.

[0068] Cosine similarity is used to measure the similarity of appearance feature vectors between frames.

[0069] In satellite videos, most targets to be tracked are rigid objects, and the same target between two adjacent frames has a strong correlation in appearance and position. In addition, the trajectories of these rigid targets are approximately straight lines in a short period of time, and the moving distance of the same target between two adjacent frames is spatially limited. Based on the above characteristics, in this embodiment, the spatio-temporal state propagation module (STSP) between frames, such as Figure 3 As shown in the figure, the motion features and appearance features of the target are transferred between two adjacent frames. The module includes a contextual search region expansion mechanism (CSREM) and an appearance modeling method based on the channel attention mechanism.

[0070] Figure 3 In the context search area expansion mechanism shown in FIG, since the satellite video sequence images have the same size, the center point of the target tracking result of the previous frame is used as the reference, and the width and height are expanded outward by λ (λ>1) times respectively to obtain the context search area of the target in the current frame, as shown in FIG. Figure 3 The yellow dotted box is shown. For each target in the previous frame, RoI Align is used to divide it into 15*15 pixels. The context search area in the current frame is divided into 15λ*15λ pixels in the same way. Depth-wise convolution encoding can be used to calculate the similarity between the image area where the target tracking result of the previous frame is located and the context search area containing the target in the current frame, thereby predicting the position of the target in the current frame. The encoding process can be expressed as:

[0071] F θ (c,p)=F c *F p (4)

[0072] In formula (4), * is the cross-correlation operation represented by Depth-wise convolution, F θ (c, p) is the similarity feature map obtained through the cross-correlation operation. The window corresponding to the feature point with the highest responsiveness is the target's position in the current frame, and the responsiveness is the predicted foreground target probability score. The spatial position and probability score obtained through depth-wise convolutional coding above realize the transfer of the target's motion state between frames.

[0073] Occlusion, shadows, and similar background interference during target motion can lead to incomplete visual appearance information, or even the entire target disappearing from the scene. Incomplete or even missing visual appearance features pose significant challenges to spatial position prediction and matching information extraction, leading to data association failures and the fundamental cause of model drift. Therefore, in addition to predicting the target's spatial position, we also predict and extract its appearance features, combining its motion and appearance states to achieve high-precision trajectory matching.

[0074] like Figure 4 As shown, cross-frame state propagation features Where C represents the channel. Average pooling and maximum pooling can be used to count the feature information on the channel (expressed as a 1×C vector), and encoded separately through a shared network consisting of a one-dimensional convolution and a fully connected layer. The encoded vectors are added element-wise to make up for the defects of a single statistic, and the channel attention response map (expressed as a 1×C vector) is obtained through the Sigmoid normalization operation. Its purpose is to enhance the feature representation on the high-order semantic channel that is conducive to learning discriminative matching vectors. The channel attention response map is then compared with Perform element-wise multiplication to enhance the response of target information and filter out background noise. In addition, in order to avoid accidental loss of information, the original feature representation The robust appearance features are then added to the features enhanced by channel attention. Finally, a fully connected layer is used to map the appearance features into a C-dimensional ID vector, which is used to represent the discriminative appearance features of the target, thereby enabling the target's appearance state to be transferred between frames.

[0075] Among them, the target spatiotemporal matching affinity is calculated by combining the intersection-over-union spatial matching result and the appearance feature similarity, which can be designed to include:

[0076] Generate a target position affinity matrix based on the intersection-over-union ratio between the spatial positions of the target detection and the predicted bounding box in the current frame, where the predicted bounding box is obtained by migrating the coordinates of the target motion state;

[0077] The cosine distance is used to measure the similarity between the target appearance features and the target apparent state transition features, and the target appearance affinity matrix is generated;

[0078] The target spatiotemporal matching affinity discriminant matrix is constructed based on the target position affinity matrix and the target appearance affinity matrix.

[0079] The standard overall trajectory matching method relies on the relevant threshold α and the interval frame number T to manage the activation, dormancy, extension and termination of the trajectory. Specifically, when the score of the trajectory is still lower than the threshold α after T frames, the trajectory will be terminated. However, in satellite videos, factors such as shadows and occlusions may lead to reduced detection and related confidence. This situation may lead to trajectory shortening when relying on a single threshold and a fixed-length interval frame number. In this embodiment, a progressively associating trajectory (PAT) strategy with inter-frame consistency is considered. First, the affinity calculation is performed on the measurement values representing the same motion and appearance attributes. Then, the calculation results are gradually associated with the existing trajectory to generate trajectory segments.

[0080] The target detection result and target motion state migration result represent the detected and predicted spatial positions of the target in the current frame, respectively. For these two results, in this embodiment, an affinity calculation can be performed using an IoU-based spatiotemporal matching method. By measuring the IoU between two sets of bounding boxes, a position affinity matrix is generated, which can be expressed as:

[0081]

[0082] Among them, S D and S P They represent the area of the detection box and the area of the prediction box respectively.

[0083] For the calculation of apparent affinity, the cosine distance shown in formula (6) can be used to measure the similarity between the apparent features and the target apparent state transition results, which can be expressed as:

[0084]

[0085] Based on the position affinity matrix and the apparent affinity matrix, the discriminant matrix can be constructed:

[0086]

[0087] Among them, the detection result and prediction result corresponding to the maximum value of each row or column in the discriminant matrix are the matching results.

[0088] Specifically, each target tracking frame in each video sequence frame is obtained based on the target spatiotemporal matching affinity, which can be designed to include:

[0089] If the target spatiotemporal affinity is greater than the threshold, the corresponding target detection result is deleted, the corresponding state transition result is retained, and the detection confidence of the corresponding state transition result and the detection confidence of the remaining target detection results are calculated;

[0090] If the detection confidence of the remaining target detection results is greater than the specified detection threshold, a new track is added; otherwise, the corresponding target detection results are deleted.

[0091] If the detection confidence of the corresponding state transition result is greater than the specified detection threshold, the target trajectory length is increased by 1. Otherwise, the target trajectory is temporarily dormant and an interval frame T is set. Every T / 3 interval frame, the specified detection threshold is reduced by 0.1. It is then determined whether the detection confidence of the state transition result within the corresponding interval frame is greater than the reduced specified detection threshold. If so, the dormant target trajectory is activated and the target trajectory length is increased by 1. Otherwise, the corresponding target trajectory tracking is terminated.

[0092] In actual tracking scenarios, due to shadows, occlusions, and insufficient model feature extraction capabilities, problems such as reduced detection and prediction confidence are likely to occur. In this case, using a unified threshold and a fixed-length interval frame for discrimination will reduce the length of the trajectory. To address the above issues, this embodiment adopts a trajectory segmentation association strategy. The specific process is as follows: Figure 5 As shown, after obtaining the target detection result set D={D1,D2,D3,...,D S} and state transition result set C={C1,C2,C3,...,C T After that, the process of trajectory management can be summarized as follows:

[0093] i. Calculate the affinity between the target detection result and the state transition result, and remove Target detection results.

[0094] For the remaining detection results, if the detection confidence is greater than the given detection threshold β, consider adding a new track ID. Combining detection confidence and detection location, detection confidence can be directly obtained from this representation;

[0095] ii. For the state prediction result, if its prediction confidence is greater than the given prediction threshold α, tracking is considered to continue and the track length is increased by 1. Otherwise, the track is considered to be temporarily dormant and the track ID is retained;

[0096] iii. For a trajectory in a dormant state, set the interval frame T. Every T / 3 frames, the prediction threshold is reduced by 0.1. If the prediction confidence in the corresponding frame is greater than the given prediction threshold, the trajectory is activated from the dormant state and the trajectory length is increased by 1. Otherwise, the trajectory is considered terminated.

[0097] This solution can fully consider the motion characteristics of rigid targets in satellite videos. That is, except for trains that undergo large deformation when making sharp turns, the deformation amplitude of targets such as airplanes and ships is much smaller than that of targets to be tracked in natural scene videos (mainly pedestrians). In other words, the appearance and posture of the same target between two adjacent frames are highly similar. In addition, the trajectories of these rigid targets are approximately straight lines in a short period of time, and the moving distance of the same target between two adjacent frames is spatially limited. Based on the above analysis of the target characteristics of satellite videos, by combining the spatiotemporal information of two adjacent frames, the cross-frame changes in target motion and appearance are modeled, and the spatiotemporal state is propagated, thereby improving the integrity of the trajectory.

[0098] Furthermore, based on the above method, an embodiment of the present invention also provides a satellite video multi-target tracking system based on spatiotemporal state propagation and progressive trajectory association, comprising: a data acquisition module, a target tracking module and a trajectory generation module, wherein:

[0099] A data acquisition module is used to acquire video sequence frames that require target tracking;

[0100] A target tracking module is configured to use a target tracking model to obtain target detection results and state transition results for adjacent frames in a video sequence frame image. The target detection results include: a target detection frame position representation and a target category representation; the state transition results include: a coordinate transition and a foreground probability; and to obtain an intersection-over-union (IoU) spatial matching based on the target detection frame position representation and the coordinate transition. Appearance features of adjacent frames are extracted based on the target category representation and the foreground probability, and the similarity of appearance features of adjacent frames is calculated. Coordinate transition is used to represent the change in the spatial position of the same target in adjacent frames, and the foreground probability represents the category prediction result of the target in the next frame. The target tracking model is obtained by training a neural network using image data samples.

[0101] The trajectory generation module is used to calculate the target spatiotemporal matching affinity by combining the intersection-over-union spatial matching results and the appearance feature similarity. Based on the target spatiotemporal matching affinity, the tracking frame of each target in each video sequence frame is obtained and the tracking frame is added to the corresponding target trajectory segment to generate the tracking trajectory of each target.

[0102] To verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0103] The PS2PAT-MOT framework designed in this paper is compared with existing multi-target tracking methods on the public satellite video target tracking dataset AIR-MOT. The performance is evaluated using metrics such as MOTA, IDF1, IDs, FN, FP, MT, ML, IDP, and IDR. The quantitative results are shown in Table 1. ↑ indicates that the higher the metric, the better the performance; ↓ indicates that the lower the metric, the better the performance.

[0104] Table 1 Comparison results of different multi-target tracking methods on the AIR-MOT dataset The highest index is shown in bold black

[0105]

[0106]

[0107] As shown in Table 1, the PS2PAT-MOT framework achieved the highest scores among the nine compared methods in all six metrics (MOTA, IDF1, FN, MT, ML, and IDR). Notably, in terms of the core metric MOTA, the PS2PAT-MOT framework outperformed the second-ranked CenterTrack by 3.7%. In terms of the IDF1 metric, which is positively correlated with MOTA, the PS2PAT-MOT framework outperformed the second-ranked OC-SORT by 2.2%. Overall, general-purpose video trackers performed better in terms of MOTA and IDF1. Although CFTracker and DSFNet-SORT, two methods specifically designed for satellite video processing, failed to achieve satisfactory results in terms of MOTA and IDF1, they performed well in maintaining trajectory consistency (IDs and IDP metrics) and reducing false positives (FP metric). This demonstrates their targeted considerations for object recognition in complex scenes and the problem of trajectory fragmentation in satellite videos. The PS2PAT-MOT framework in this case fully considers the motion characteristics of rigid targets in satellite videos, improves recognition accuracy by transferring the target's inter-frame state, and adopts a progressive matching strategy to enhance trajectory integrity, thereby achieving robust tracking performance on the AIR-MOT dataset.

[0108] Figure 6Track visualization results for selected videos from the AIR-MOT dataset are shown for StrongSORT, SiamMOT, OC-SORT, CenterTrack, and the proposed method, PS2PAT-MOT, arranged from low to high in terms of MOTA value. Key areas are highlighted with yellow frames and magnified. StrongSORT's missed detections of the cluster of ships in the upper left corner of video 016 and the single aircraft in the upper left corner of video 046 result in low MOTA, IDs, and FP values. This demonstrates the importance of a high-performance detector for achieving satisfactory tracking results, further validating the complete detection theory. Although CenterTrack achieved the second-highest MOTA value, it performed the worst in maintaining track consistency. This phenomenon is also confirmed by the track fragmentation observed in the lower right corner of video 016, where the ship's track is fragmented. The proposed PS2PAT-MOT framework accurately identifies small targets in crowded scenes while simultaneously resisting interference from similar backgrounds, maintaining a complete track. As a result, it achieves the most stable tracking performance.

[0109] The above experimental data shows that this solution combines the spatiotemporal information of two adjacent frames to model the cross-frame changes in target motion and appearance. By propagating the spatiotemporal state, it can effectively improve the integrity of the trajectory and the accuracy of multi-target tracking.

[0110] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0112] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0113] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0114] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association, characterized in that: Include: Obtain the video sequence frames that require target tracking; Utilize a target tracking model to obtain target detection results and state transition results for adjacent frames in a video sequence frame image. The target detection results include: target detection frame position representation and target category representation. The state transition results include: coordinate transition and foreground probability. Based on the target detection frame position representation and coordinate transition, an intersection-over-union (IoU) spatial matching is obtained. Based on the target category representation and foreground probability, appearance features of adjacent frames are extracted and similarity of appearance features of adjacent frames is calculated. Coordinate transition is used to represent the change in spatial position of the same target in adjacent frames. Foreground probability represents the category prediction result of the target in the next frame. The target tracking model is obtained by training a neural network using image data samples. The target spatiotemporal matching affinity is calculated by combining the intersection-over-union spatial matching result and the appearance feature similarity. Based on the target spatiotemporal matching affinity, each target tracking frame in each video sequence frame is obtained and added to the corresponding target trajectory segment to generate each target tracking track.

2. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 1, characterized in that: The target tracking model is obtained by using static image data as training data, constructing enhanced image data for each static image through data enhancement, and training a neural network using training samples consisting of static image data and enhanced image data; Alternatively, dynamic satellite video sequence image data is used as training data, adjacent frames are randomly sampled at a specified interval, and the neural network is trained using training sample pairs consisting of the randomly sampled adjacent frames.

3. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 1 or 2, characterized in that: The target tracking model includes: a feature extraction network that extracts multi-scale features from image images in combination with spatial resolution and semantic information, a detection branch that obtains target category representation and detection frame position representation based on the extracted image features, and an association branch that associates the spatial positions of the same target in related frames based on the extracted image features and obtains coordinate migration and foreground probability. The detection branch includes an RPN network and a regional target recognition network. The association branch uses the center point of each detection frame in the previous frame as a reference and expands outward a specified number of times along the length and width directions on the current frame image and performs similarity association on the same target features between adjacent frames based on the target context search area and channel attention mechanism.

4. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 3, characterized in that: Use image data samples to train neural networks, including: Constructing a network training loss function, wherein the network training loss function includes RPN network training loss, regional target recognition network training loss, associated branch coordinate migration loss and appearance feature similarity loss; The neural network is trained using image data samples and based on the network training loss function.

5. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 1, characterized in that: Extract the appearance features of adjacent frames based on the target category representation and foreground probability and calculate the similarity of the appearance features of adjacent frames, including: Taking the center point of the target detection frame of the previous frame as the reference, expand the current frame image outward by λ times in width and length respectively to obtain the current frame target context search area, where λ is greater than 1; Extract each target area in the previous frame according to the specified pixel size, and extract the context search target area corresponding to λ times the pixel size in the target context search area in the current frame; Convolutional coding is used to calculate the similarity feature map between each target area in the previous frame and the context search target area in the current frame; For the cross-frame state propagation features represented by foreground probabilities, a shared network consisting of average pooling and maximum pooling is used to encode them respectively. The encoded vectors are element-wise added and sigmoid normalized to obtain the channel attention response map. The attention response map is element-wise multiplied with the cross-frame state propagation features, and the multiplication result is again feature-added with the cross-frame state propagation features. A fully connected layer is used to map the feature addition result into an appearance feature vector. Cosine similarity is used to measure the similarity of appearance feature vectors between frames.

6. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 1, characterized in that: The target spatiotemporal matching affinity is calculated by combining the intersection-over-union spatial matching results and the appearance feature similarity, including: Generate a target position affinity matrix based on the intersection-over-union ratio between the spatial positions of the target detection and the predicted bounding box in the current frame, where the predicted bounding box is obtained by migrating the coordinates of the target motion state; The cosine distance is used to measure the similarity between the target appearance features and the target apparent state transition features, and the target appearance affinity matrix is generated; The target spatiotemporal matching affinity discriminant matrix is constructed based on the target position affinity matrix and the target appearance affinity matrix.

7. The satellite video multi-target tracking method based on spatiotemporal state propagation and progressive trajectory association according to claim 1 or 6, characterized in that: Obtain each target tracking frame in each video sequence frame based on the target spatiotemporal matching affinity, including: If the target spatiotemporal affinity is greater than the threshold, the corresponding target detection result is deleted, the corresponding state transition result is retained, and the detection confidence of the corresponding state transition result and the detection confidence of the remaining target detection results are calculated; If the detection confidence of the remaining target detection results is greater than the specified detection threshold, a new track is added; otherwise, the corresponding target detection results are deleted. If the detection confidence of the corresponding state transition result is greater than the specified detection threshold, the target trajectory length is increased by 1. Otherwise, the target trajectory is temporarily dormant and an interval frame T is set. Every T / 3 interval frame, the specified detection threshold is reduced by 0.

1. It is then determined whether the detection confidence of the state transition result within the corresponding interval frame is greater than the reduced specified detection threshold. If so, the dormant target trajectory is activated and the target trajectory length is increased by 1. Otherwise, the corresponding target trajectory tracking is terminated.

8. A satellite video multi-target tracking system based on spatiotemporal state propagation and progressive trajectory association, characterized in that: Contains: data acquisition module, target tracking module and trajectory generation module, among which, A data acquisition module is used to acquire video sequence frames that require target tracking; A target tracking module is configured to use a target tracking model to obtain target detection results and state transition results for adjacent frames in a video sequence frame image. The target detection results include: a target detection frame position representation and a target category representation; the state transition results include: a coordinate transition and a foreground probability; and to obtain an intersection-over-union (IoU) spatial matching based on the target detection frame position representation and the coordinate transition. Appearance features of adjacent frames are extracted based on the target category representation and the foreground probability, and the similarity of appearance features of adjacent frames is calculated. Coordinate transition is used to represent the change in the spatial position of the same target in adjacent frames, and the foreground probability represents the category prediction result of the target in the next frame. The target tracking model is obtained by training a neural network using image data samples. The trajectory generation module is used to calculate the target spatiotemporal matching affinity by combining the intersection-over-union spatial matching results and the appearance feature similarity. Based on the target spatiotemporal matching affinity, each target tracking frame in each video sequence frame is obtained and the tracking frame is added to the corresponding target trajectory segment to generate each target tracking trajectory.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images