Aerial video image target tracking method and system based on broken trajectory repair

Abstract

The invention provides an aerial video image target tracking method and system based on broken trajectory repair, and the method comprises the steps: inputting an aerial video image into a sliding window slicing module frame by frame, and segmenting each frame of image into a plurality of image slices P1-Pn; respectively inputting the original images P0 and P1-Pn of each frame into a trained joint detection-tracking model, wherein the model comprises a target detection network and a Re-ID network integrated with a broken trajectory prediction and restoration algorithm; the target detection network carries out target detection on P0-Pn; and the Re-ID network performs target tracking according to the target detection result of each target of each frame: initializing the detection frame coordinate of each target of the first frame into a target trajectory, matching the detection frame coordinate of each target of the current frame with the existing target trajectory by combining the ID identity vector from the second frame, and if the matching succeeds, performing target tracking. And updating the detection frame coordinate of each target of the current frame to the target track, and if the matching is not successful, creating a new target track. According to the invention, target tracking can be carried out on aerial video images.

Description

Technical Field

[0001] This invention relates to the field of video image target tracking technology, and in particular to a method and system for aerial video image target tracking based on broken trajectory repair. Background Technology

[0002] Aerial video images captured by drones or aircraft are crucial tools for real-time monitoring and dynamic analysis. They not only provide target location information but also capture dynamic changes in target time and space. In-depth analysis of this information enables precise tracking, behavior recognition, and trajectory analysis. Currently, multi-target tracking technology based on traditional ground images is relatively mature, capable of accurately identifying multiple different targets and continuously and stably tracking them. However, the application of traditional target tracking algorithms to aerial video images still faces numerous challenges due to the high mobility and diverse perspectives of the images. How to achieve efficient target identification and accurate tracking in aerial video images is a hot research topic in the field of drone and aerial photography applications.

[0003] To track targets in an image, it's necessary to extract the target's location information and associate the same target across consecutive frames to construct the trajectory of each target. Common target tracking methods include modeling the target's appearance to match the most similar features across different frames for target localization, using prediction algorithms to search for targets near predicted values, and using deep learning-based target detection algorithms to obtain bounding boxes for matching. However, compared to ground images, aircraft and drones can capture target data from multiple angles and viewpoints, including overhead, oblique, and vertical perspectives. This change in perspective leads to significant deformation of the target, which traditional appearance-based target tracking algorithms cannot adapt to, easily resulting in target loss. Furthermore, when drones and aircraft fly at high speeds, the spatial position of targets in the captured images can change abruptly, making traditional deep learning algorithms prone to target loss and incorrect target identification in such situations. Additionally, due to the influence of flight altitude, targets captured in aerial video images are often small in size, while the image size is large, easily leading to missed detections by the detection model, thus affecting target tracking. These problems indicate that existing target tracking methods are not suitable for target tracking in aerial video images, making it difficult to achieve subsequent target behavior recognition and trajectory analysis. Summary of the Invention

[0004] To address the technical problems existing in the prior art, the present invention provides a method and system for target tracking in aerial video images based on broken trajectory repair, the technical solution of which is as follows:

[0005] On the one hand, a target tracking method based on broken trajectory repair for aerial video images is provided, the method comprising:

[0006] S1. Collect aerial video images to be tracked using drones or airplanes;

[0007] S2. The aerial video images are input into the sliding window slicing module frame by frame. The sliding window slicing module divides each input frame into multiple small-resolution image slices P1 to P2. n ;

[0008] S3, combine the original image P0 of each frame with the images P1 to P2. n Input the trained joint detection-tracking model into the target. The joint detection-tracking model adopts the paradigm of detection before tracking, which detects the target first and then tracks it. It includes a target detection network and a Re-ID network that integrates a broken trajectory prediction and repair algorithm.

[0009] The target detection network, for P0 to P n A total of n+1 images are used for object detection, resulting in n+1 detection results. P1 to P... n The detection bounding box is restored from the coordinates on the image slice to the coordinates on the original image, and then for all P0 to P1... n After performing non-maximum suppression post-processing on the detection boxes, the same target is merged, and the final target detection result is output, including target category information, detection box coordinates, and ID identity vector.

[0010] The Re-ID network performs target tracking based on the target detection results of each target in each frame. The target tracking includes:

[0011] The detection box coordinates of each target in the first frame are initialized as the target trajectory. Starting from the second frame, the detection box coordinates of each target in the current frame are matched with the existing target trajectory in combination with the ID identity vector. If the match is successful, the detection box coordinates of each target in the current frame are updated to the target trajectory. If the match is unsuccessful, a new target trajectory is created.

[0012] The matching process includes using a broken trajectory prediction and repair module to predict the trajectory information of the target in the current frame in multiple subsequent time periods, forming multiple different predicted trajectory information, and associating and matching the correct predicted trajectory when a sudden change in spatial position occurs in a subsequent frame, using it as complete trajectory information to complete the target's actual discrete trajectory to form a continuous motion trajectory.

[0013] Optionally, the target detection network uses YOLOv11 as the backbone detector and calculates the regression error of the target detection box through smoothed L1 loss optimization, the formula of which is:

[0014]

[0015] Among them, b i This represents the actual coordinates of the i-th frame. The bounding box coordinates are represented by the smoothed L1 loss, which penalizes the regression error and avoids the impact of large errors.

[0016] The classification loss uses cross-entropy loss to optimize class prediction, and its formula is:

[0017]

[0018] Where, p i This represents the true category label of the i-th target. It is the predicted class probability, and the classification loss improves the model's classification accuracy by minimizing the difference between the predicted class and the true class;

[0019] Meanwhile, in addition to the backbone detector, the model also includes an ID embedding branch. Its core objective is to generate features that can distinguish different targets, ensuring consistency between target detection and the Re-ID network tracking. The ID embedding branch is a convolutional layer with 128 kernels, used to extract unique identity embedding features from each location. These identity embedding features not only provide a distinctive representation for each target, but also help the model better capture the target's identity information. The ID embedding branch maps the target's feature map into a 128-dimensional identity vector through the convolutional layer, serving as the target's identity features. These identity vectors are optimized together with the target's location and category information during training, ensuring that the model can generate accurate and highly discriminative target features while locating the target.

[0020] During training, to enhance the feature consistency of the same target across different frames while maximizing the feature differences between different targets, an ID consistency loss is introduced, the formula of which is as follows:

[0021]

[0022] Where N is the total number of detection boxes, K is the total number of IDs, and L i (k) is the labeled ID number. The loss function, which extracts target identity features for the network, further improves the stability and distinguishability of target identity by minimizing the distance between features of the same target and increasing the distance between different targets.

[0023] Finally, the model will merge and output the target category information and target detection box output by the backbone detector, as well as the identity vector output by the ID embedding branch.

[0024] Optionally, the processing procedure of the breakage trajectory prediction and repair module includes:

[0025] First, the existing target trajectory is embedded in high dimension. A linear projection layer maps the spatial location information into a feature vector with semantic representation capabilities, and position encoding is added to preserve the temporal structure, resulting in the embedded trajectory feature z. t ;

[0026] Next, the embedded trajectory feature z t In the temporal modeling module that integrates motion perception, by introducing the target's velocity and acceleration, the model can identify motion trends and change patterns while performing time modeling, thus improving its ability to perceive fast or irregular motion. The calculation formula is as follows:

[0027]

[0028] in, Enc(v) is the trajectory feature sequence after fusing motion information. t ), Enc(a t This indicates that the velocity and acceleration vectors are embedded and transformed separately through a linear layer or a small neural network;

[0029] Next, the aforementioned Input to a state-space model consisting of N stacked elements:

[0030] At each time step t, the gated representation g of the input features is normalized and computed. t Next, the hidden state is updated using a state space unit, and the calculation formula is as follows:

[0031]

[0032] Among them, h t The obtained temporal features; W g A and B are both learnable weight matrices, and W... g Let b be the gate matrix. g h is the bias vector. t h t-1 Let A represent the hidden states at the current time step and the previous time step, respectively; let B represent the state transition matrix; and let B represent the input mapping matrix.

[0033] Finally, regarding the time series feature h t Decode the target and output its possible location in the next time step.

[0034] Optionally, the matching process specifically includes:

[0035] For the t-th iFor each target in the frame, the traditional SORT algorithm is first used to process t. i Frame target detection box coordinates and t i-1 The frame ID motion trajectory information is initially matched to obtain the matched trajectory information T1 to T2. n And unmatched trajectory information L1~L m ;

[0036] Next, for each unmatched trajectory L j The data is input as time-series data into the broken trajectory prediction and repair module to predict the coordinates of the next time step, resulting in multiple coordinate points P. 11 ~P 1s Then put P 11 ~P 1s spliced ​​to trajectory L respectively j In the middle, s predicted trajectories L are formed. 11 ~L 1s Then, these trajectories are re-input into the broken trajectory prediction and repair module for prediction, and this process is repeated to obtain predicted trajectories PL1 to PL2 that connect the prediction results of multiple future time steps. d ;

[0037] Next, the t-th i The coordinates of the unmatched target detection boxes in the frame and the predicted trajectories PL1 to PL2 d Secondary matching is performed based on IoU. The predicted positions of the predicted trajectories up to the matching point in the previous multiple time steps are embedded into the actual trajectories as actual positions, and the actual trajectories are updated to complete the broken trajectories caused by the sudden change in the target position.

[0038] Finally, for target points that still fail to match after the second matching, a new motion trajectory is created and merged with the trajectory from the first and second matching, becoming the current trajectory t. i The trajectory information of the frame.

[0039] Optionally, the training process of the joint detection-tracking model includes:

[0040] Aerial video images are acquired, each frame is extracted to form a video frame sequence, the video frame sequence is preprocessed and target is labeled, and the resulting labeled file and the video frame sequence constitute the original dataset.

[0041] The video frame sequences in the original dataset are input into a multi-view feature generation and fusion network. The multi-view feature generation and fusion network first uses a target segmentation network to automatically extract and segment multiple target images and a background image from the image. Then, for each target image, a diffusion generation model generates images from different viewpoints from a single viewpoint. Next, a feature fusion network extracts and fuses target features from different viewpoints. Finally, the fused target features and the background features extracted directly from the background image through the DLA34 network are concatenated to obtain the final fused features. The final fused features are then input into the joint detection-tracking model for feature learning training.

[0042] Optionally, the processing steps of the diffusion generation model include:

[0043] First, Gaussian noise is gradually added to the input single-view target image within a preset time step to form a noisy image;

[0044] Next, progressive reverse denoising is performed on the noisy image to gradually reconstruct a clear image consistent with the input viewpoint from a pure noise state;

[0045] Next, the reconstructed image is used as input, combined with the target viewpoint information described by the preset text prompts, to form a multi-view representation of the target object in the latent space. By repeatedly performing an iterative process of adding noise and reversing noise under different viewpoint conditions, a viewpoint transformation image H with consistent style and geometric rationality corresponding to the input viewpoint is progressively generated. i1 ~H im At the same time, the target image from the original viewpoint is defined as H. i0 H i0 ~H im There are a total of (m+1) target images.

[0046] Optionally, the processing procedure of the feature fusion network includes:

[0047] First, H i0 ~H im The target feature F is extracted using a feature extraction network. i0 ~F im ;

[0048] Then, based on the multi-head self-attention mechanism, F is calculated using multiple independent self-attention heads. i0 ~F im The similarity between each pair of viewpoint features is calculated, and a fusion weight is learned to capture the global dependencies between viewpoints. The complementarity and consistency of features between viewpoints are modeled, where the target feature F of the i-th target under the j-th viewpoint is... ij First calculate the query key value

[0049]

[0050] in, Both are learnable self-attention matrices, where h represents the h-th attention head;

[0051] Then, the attention weights are calculated, and the formula is as follows:

[0052]

[0053] in, For queries and keys from another perspective, d represents the attention weight of the h-th head between viewpoints j and l. k For feature dimensions;

[0054] Will The weighted values ​​are used as the output of the head, and the outputs of all heads are concatenated to obtain the dynamically weighted fusion feature F. attn The calculation formula is as follows:

[0055]

[0056] in, Let W be the output of viewpoint j at the h-th head, N be the number of attention heads, and W be the output of viewpoint j at the h-th head. O It is a learnable linear transformation matrix;

[0057] Next, the F i,attn The spatial transformation matrix T is learned by inputting it into a lightweight convolutional network, and F is transformed through T. i,attn Perform spatial alignment to obtain the aligned feature F' i,attn The calculation formula is as follows:

[0058] F′ i,attn =Γ(F i,attn ,T)

[0059] Where Γ represents a space transformation operation;

[0060] Next, the F' i,attn The DLA34 network is used to extract and fuse feature maps of different scales, ultimately obtaining the multi-scale semantic fusion feature F. i .

[0061] On the other hand, a target tracking system based on broken trajectory repair for aerial video images is provided, the system comprising:

[0062] The acquisition module is used to acquire aerial video images to be tracked via drones or airplanes;

[0063] The segmentation module is used to input the aerial video images into the sliding window slicing module frame by frame. The sliding window slicing module divides each input frame into multiple small-resolution image slices P1 to P2. n ;

[0064] The detection and tracking module is used to detect and track the original image P0 of each frame and the images P1 to P2. n Input the trained joint detection-tracking model into the target. The joint detection-tracking model adopts the paradigm of detection before tracking, which detects the target first and then tracks it. It includes a target detection network and a Re-ID network that integrates a broken trajectory prediction and repair algorithm.

[0065] The target detection network, for P0 to P n A total of n+1 images are used for object detection, resulting in n+1 detection results. P1 to P... n The detection bounding box is restored from the coordinates on the image slice to the coordinates on the original image, and then for all P0 to P1... n After performing non-maximum suppression post-processing on the detection boxes, the same target is merged, and the final target detection result is output, including target category information, detection box coordinates, and ID identity vector.

[0066] The Re-ID network performs target tracking based on the target detection results of each target in each frame. The target tracking includes:

[0067] The detection box coordinates of each target in the first frame are initialized as the target trajectory. Starting from the second frame, the detection box coordinates of each target in the current frame are matched with the existing target trajectory in combination with the ID identity vector. If the match is successful, the detection box coordinates of each target in the current frame are updated to the target trajectory. If the match is unsuccessful, a new target trajectory is created.

[0068] The matching process includes using a broken trajectory prediction and repair module to predict the trajectory information of the target in the current frame in multiple subsequent time periods, forming multiple different predicted trajectory information, and associating and matching the correct predicted trajectory when a sudden change in spatial position occurs in a subsequent frame, using it as complete trajectory information to complete the target's actual discrete trajectory to form a continuous motion trajectory.

[0069] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described aerial video image target tracking method based on broken trajectory repair.

[0070] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described aerial video image target tracking method based on broken trajectory repair.

[0071] The beneficial effects of the technical solution provided by this invention include at least the following:

[0072] 1) This invention designs a broken trajectory repair algorithm, which combines a broken trajectory prediction and repair module with multi-time step prediction and IoU matching to predict and match subsequent trajectories, allowing the model to adapt to sudden changes in the target spatial position and improve tracking accuracy.

[0073] 2) By designing a sliding window slicing method, this invention can quickly and effectively apply a general joint detection-tracking algorithm to small target detection, reducing the training cost of the model and improving the detection accuracy of the model.

[0074] 3) Compared with traditional target tracking methods, the multi-view feature generation and fusion network of the present invention can learn target features from multiple perspectives through a single perspective, thereby maintaining stable tracking of the same target when the perspective changes. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a flowchart of a target tracking method for aerial video images based on broken trajectory repair provided by an embodiment of the present invention;

[0077] Figure 2 This is a general block diagram of an aerial video image target tracking method based on broken trajectory repair provided by an embodiment of the present invention;

[0078] Figure 3 This is a flowchart of the sliding window slicing module processing procedure provided in an embodiment of the present invention;

[0079] Figure 4 This is a block diagram of the target detection network structure provided in an embodiment of the present invention;

[0080] Figure 5 This is a flowchart of the processing procedure of the fracture trajectory prediction and repair module provided in this embodiment of the invention;

[0081] Figure 6 This is a flowchart of the matching process provided in an embodiment of the present invention;

[0082] Figure 7 This is a schematic diagram of the fracture trajectory prediction and repair provided in an embodiment of the present invention;

[0083] Figure 8 This is a flowchart of the multi-view feature generation and fusion network processing procedure provided in the embodiments of the present invention;

[0084] Figure 9 This is a flowchart of the processing procedure of the diffusion generation model provided in the embodiments of the present invention;

[0085] Figure 10 This is a flowchart of the processing procedure of the feature fusion network provided in the embodiments of the present invention;

[0086] Figure 11 This is a block diagram of an aerial video image target tracking system based on broken trajectory repair provided by an embodiment of the present invention;

[0087] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0088] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0089] This invention provides a method for target tracking in aerial video images based on broken trajectory repair. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps:

[0090] S1. Collect aerial video images to be tracked using drones or airplanes;

[0091] S2. The aerial video images are input into the sliding window slicing module frame by frame. The sliding window slicing module divides each input frame into multiple small-resolution image slices P1 to P2. n ;

[0092] Since the target size for aerial video image acquisition is often small, this invention embodiment designs a sliding window slicing module for small targets, such as... Figure 3 As shown, the sliding window slicing module will slide the window according to the set image size and overlap, dividing the original video frame image into several small image slices P1 to P2. n And sort the slices according to the sliding window order.

[0093] Furthermore, to avoid missed or false detections of large-resolution targets due to small segmentation sizes during sliding window slicing, the original-size image is retained after segmentation. This introduces a global-scale supplement on top of the conventional small-block segmentation, ensuring that the subsequent target detection network still has sufficient discriminative information when processing large-scale targets, thereby improving overall detection accuracy and robustness. Based on this, the original-size image is defined as P0, and P0~P n There are a total of (n+1) images.

[0094] S3, combine the original image P0 of each frame with the images P1 to P2. n Input the trained joint detection-tracking model into the target. The joint detection-tracking model adopts the paradigm of detection before tracking, which detects the target first and then tracks it. It includes a target detection network and a Re-ID network that integrates a broken trajectory prediction and repair algorithm.

[0095] The target detection network, for P0 to P n A total of n+1 images are used for object detection, resulting in n+1 detection results. P1 to P... n The detection bounding box is restored from the coordinates on the image slice to the coordinates on the original image, and then for all P0 to P1... n After performing non-maximum suppression post-processing on the detection boxes, the same target is merged, and the final target detection result is output, including target category information, detection box coordinates, and ID identity vector.

[0096] The Re-ID network performs target tracking based on the target detection results of each target in each frame. (The Re-ID network combines the ID identity vector to match the target detection results with existing target trajectories, thereby keeping the same target ID unchanged and achieving stable tracking. Re-ID networks typically employ the SORT tracking algorithm, which combines Kalman filtering and IoU, and uses the target's historical frame motion trajectory to associate and match the target, achieving stable tracking. However, this method is not suitable for scenarios where the target position in the image changes abruptly due to the high speed of the aircraft or drone, resulting in discontinuous and broken trajectories. Therefore, for broken trajectories caused by abrupt changes in target position, this embodiment of the invention designs a Re-ID that integrates a broken trajectory prediction and repair algorithm. An additional broken trajectory prediction and repair module is trained to predict trajectory points after multiple time steps based on the existing target trajectory, solving the problem of target loss caused by position jumps where the existing target trajectory cannot match the target coordinates in the current frame.) The target tracking includes:

[0097] The detection box coordinates of each target in the first frame are initialized as the target trajectory. Starting from the second frame, the detection box coordinates of each target in the current frame are matched with the existing target trajectory in combination with the ID identity vector. If the match is successful, the detection box coordinates of each target in the current frame are updated to the target trajectory. If the match is unsuccessful, a new target trajectory is created.

[0098] The matching process includes using a broken trajectory prediction and repair module to predict the trajectory information of the target in the current frame in multiple subsequent time periods, forming multiple different predicted trajectory information, and associating and matching the correct predicted trajectory when a sudden change in spatial position occurs in a subsequent frame, using it as complete trajectory information to complete the target's actual discrete trajectory to form a continuous motion trajectory.

[0099] Optionally, such as Figure 4 As shown, the target detection network uses YOLOv11 as the backbone detector and calculates the regression error of the target detection box through smoothed L1 loss optimization. The formula is as follows:

[0100]

[0101] Among them, b i This represents the actual coordinates of the i-th frame. The bounding box coordinates are represented by the smoothed L1 loss, which penalizes the regression error and avoids the impact of large errors.

[0102] The classification loss uses cross-entropy loss to optimize class prediction, and its formula is:

[0103]

[0104] Where, p i This represents the true category label of the i-th target. It is the predicted class probability, and the classification loss improves the model's classification accuracy by minimizing the difference between the predicted class and the true class;

[0105] Meanwhile, in addition to the backbone detector, the model also includes an ID embedding branch. Its core objective is to generate features that can distinguish different targets, ensuring consistency between target detection and the Re-ID network tracking. The ID embedding branch is a convolutional layer with 128 kernels, used to extract unique identity embedding features from each location. These identity embedding features not only provide a distinctive representation for each target, but also help the model better capture the target's identity information. The ID embedding branch maps the target's feature map into a 128-dimensional identity vector through the convolutional layer, serving as the target's identity features. These identity vectors are optimized together with the target's location and category information during training, ensuring that the model can generate accurate and highly discriminative target features while locating the target.

[0106] During training, to enhance the feature consistency of the same target across different frames while maximizing the feature differences between different targets (ideally, the features of the same target should remain consistent across different frames, while the features of different targets should have sufficient differences to ensure that targets can be effectively distinguished), an ID consistency loss is introduced, the formula of which is as follows:

[0107]

[0108] Where N is the total number of detection boxes, K is the total number of IDs, and L i (k) is the labeled ID number. The loss function, which extracts target identity features for the network, further improves the stability and distinguishability of target identity by minimizing the distance between features of the same target and increasing the distance between different targets.

[0109] Finally, the model will merge and output the target category information and target detection box output by the backbone detector, as well as the identity vector output by the ID embedding branch.

[0110] Optionally, such as Figure 5 As shown, the processing procedure of the broken trajectory prediction and repair module (in target tracking, the trajectory coordinate sequence generated by the ID information corresponding to each target consists of the position coordinates of the target in consecutive frames, recording its motion path and dynamic features in the image space; therefore, in this embodiment of the invention, these trajectory coordinate sequences can be regarded as time series data, and a state space-based temporal prediction model is designed for trajectory modeling and prediction) includes:

[0111] First, the existing target trajectory is embedded in high dimension. A linear projection layer maps the spatial location information into a feature vector with semantic representation capabilities, and position encoding is added to preserve the temporal structure, resulting in the embedded trajectory feature z. t ;

[0112] Next, the embedded trajectory feature z t In the temporal modeling module that integrates motion perception, by introducing the target's velocity and acceleration, the model can identify motion trends and change patterns while performing time modeling, thus improving its ability to perceive fast or irregular motion. The calculation formula is as follows:

[0113]

[0114] in, Enc(v) is the trajectory feature sequence after fusing motion information. t ), Enc(a tThis indicates that the velocity and acceleration vectors are embedded and transformed separately through a linear layer or a small neural network;

[0115] Next, the aforementioned Input to a state-space model consisting of N stacked elements:

[0116] At each time step t, the gated representation g of the input features is normalized and computed. t Next, the hidden state is updated using a state space unit, and the calculation formula is as follows:

[0117]

[0118] Among them, h t The obtained temporal features; W g A and B are both learnable weight matrices, and W... g Let b be the gate matrix. g h is the bias vector. t h t-1 Let A represent the hidden states at the current time step and the previous time step, respectively; let B represent the state transition matrix; and let B represent the input mapping matrix.

[0119] Finally, regarding the time series feature h t Decode the target and output its possible location in the next time step.

[0120] Optionally, such as Figure 6 As shown, the matching process specifically includes:

[0121] For the t-th i For each target in the frame, the traditional SORT algorithm is first used to process t. i Frame target detection box coordinates and t i-1 The frame ID motion trajectory information is initially matched to obtain the matched trajectory information T1 to T2. n And unmatched trajectory information L1~L m ;

[0122] Next, for each unmatched trajectory L j The data is input as time-series data into the broken trajectory prediction and repair module to predict the coordinates of the next time step, resulting in multiple coordinate points P. 11 ~P 1s Then put P 11 ~P 1s spliced ​​to trajectory L respectively j In the middle, s predicted trajectories L are formed. 11 ~L 1sThen, these trajectories are re-input into the broken trajectory prediction and repair module for prediction, and this process is repeated to obtain predicted trajectories PL1 to PL2 that connect the prediction results of multiple future time steps. d ,like Figure 7 As shown (for the fragmented trajectory prediction and repair of the aerial video image to be tracked) Figure 7 (Similar to the model inference stage in the model);

[0123] Next, the t-th i The coordinates of the unmatched target detection boxes in the frame and the predicted trajectories PL1 to PL2 d Secondary matching is performed based on IoU. The predicted positions of the predicted trajectories up to the matching point in the previous multiple time steps are embedded into the actual trajectories as actual positions, and the actual trajectories are updated to complete the broken trajectories caused by the sudden change in the target position.

[0124] Finally, for target points that still fail to match after the second matching, a new motion trajectory is created and merged with the trajectory from the first and second matching, becoming the current trajectory t. i The trajectory information of the frame.

[0125] Optionally, the training process of the joint detection-tracking model includes:

[0126] Aerial video images are acquired, each frame is extracted to form a video frame sequence, the video frame sequence is preprocessed (data cleaning, filtering out invalid frames, blurry frames, etc.) and target is labeled, and the resulting labeled file and the video frame sequence constitute the original dataset.

[0127] In this embodiment of the invention, each frame is annotated according to the annotation format of the public dataset for target tracking. The annotation information includes frame number, target ID number, target bounding box, target type, etc. The annotation information of each video forms a GT annotation file in txt format, which together with the video frame sequence constitutes the original dataset.

[0128] The video frame sequences from the original dataset are input into a multi-view feature generation fusion network, such as... Figure 8 As shown, the multi-view feature generation and fusion network first uses a target segmentation network (SAM2) to automatically extract and segment multiple target images (H1~H2) from the image. n A background image (M) is used, and for each target image, images (H) from different viewpoints are generated from a single viewpoint using a diffusion generation model. i1 ~H imThen, through a feature fusion network, target features from different perspectives are extracted and fused. Finally, the fused target features and the background features extracted directly from the background image through the DLA34 network are spliced ​​together to obtain the final fused features. The final fused features are then input into the joint detection-tracking model for feature learning training.

[0129] Optionally, such as Figure 9 As shown, the processing steps of the diffusion generation model include:

[0130] First, Gaussian noise is gradually added to the input single-view target image within a preset time step to form a noisy image;

[0131] Next, progressive reverse denoising is performed on the noisy image to gradually reconstruct a clear image consistent with the input viewpoint from a pure noise state;

[0132] Next, the reconstructed image is used as input, combined with the target viewpoint information described by the preset text prompts, to form a multi-view representation of the target object in the latent space. By repeatedly performing an iterative process of adding noise and reversing noise under different viewpoint conditions, a viewpoint transformation image H with consistent style and geometric rationality corresponding to the input viewpoint is progressively generated. i1 ~H im At the same time, the target image from the original viewpoint is defined as H. i0 H i0 ~H im There are a total of (m+1) target images.

[0133] Optionally, such as Figure 10 As shown, the processing procedure of the feature fusion network includes:

[0134] First, H i0 ~H im The target feature F is extracted using a feature extraction network (ResNet50). i0 ~F im ;

[0135] Then, based on the multi-head self-attention mechanism, F is calculated using multiple independent self-attention heads. i0 ~F im The similarity between each pair of viewpoint features is calculated, and a fusion weight is learned to capture the global dependencies between viewpoints. The complementarity and consistency of features between viewpoints are modeled, where the target feature F of the i-th target under the j-th viewpoint is... ij First calculate the query key value

[0136]

[0137] in, Both are learnable self-attention matrices, where h represents the h-th attention head;

[0138] Then, the attention weights are calculated, and the formula is as follows:

[0139]

[0140] in, For queries and keys from another perspective, d represents the attention weight of the h-th head between viewpoints j and l. k For feature dimensions;

[0141] Will The weighted values ​​are used as the output of the head, and the outputs of all heads are concatenated to obtain the dynamically weighted fusion feature F. attn The calculation formula is as follows:

[0142]

[0143] in, Let W be the output of viewpoint j at the h-th head, N be the number of attention heads, and W be the output of viewpoint j at the h-th head. O It is a learnable linear transformation matrix;

[0144] Next, the F i,attn The spatial transformation matrix T is learned by inputting it into a lightweight convolutional network, and F is transformed through T. i,attn Perform spatial alignment to obtain the aligned feature F' i,attn The calculation formula is as follows:

[0145] F′ i,attn =Γ(F i,attn ,T)

[0146] Where Γ represents a space transformation operation;

[0147] Next, the F' i,attn The DLA34 network is used to extract and fuse feature maps of different scales, ultimately obtaining the multi-scale semantic fusion feature F. i .

[0148] The training process of the fracture trajectory prediction and repair module in this embodiment of the invention is as follows:

[0149] By reading the ID labels of each target and their position coordinates in each frame of the generated ground truth (GT) file, a complete trajectory sequence is constructed. These time-series trajectories are then used as input to the model to update the model's weight parameters.

[0150] like Figure 11As shown, this embodiment of the invention also provides a target tracking system for aerial video images based on broken trajectory repair, the system comprising:

[0151] The acquisition module 1110 is used to acquire aerial video images to be tracked via drones or airplanes;

[0152] The segmentation module 1120 is used to input the aerial video images into the sliding window slicing module frame by frame. The sliding window slicing module divides each input frame into multiple small-resolution image slices P1 to P2. n ;

[0153] The detection and tracking module 1130 is used to detect and track the original image P0 of each frame and the images P1 to P2. n Input the trained joint detection-tracking model into the target. The joint detection-tracking model adopts the paradigm of detection before tracking, which detects the target first and then tracks it. It includes a target detection network and a Re-ID network that integrates a broken trajectory prediction and repair algorithm.

[0154] The target detection network, for P0 to P n A total of n+1 images are used for object detection, resulting in n+1 detection results. P1 to P... n The detection bounding box is restored from the coordinates on the image slice to the coordinates on the original image, and then for all P0 to P1... n After performing non-maximum suppression post-processing on the detection boxes, the same target is merged, and the final target detection result is output, including target category information, detection box coordinates, and ID identity vector.

[0155] The Re-ID network performs target tracking based on the target detection results of each target in each frame. The target tracking includes:

[0156] The detection box coordinates of each target in the first frame are initialized as the target trajectory. Starting from the second frame, the detection box coordinates of each target in the current frame are matched with the existing target trajectory in combination with the ID identity vector. If the match is successful, the detection box coordinates of each target in the current frame are updated to the target trajectory. If the match is unsuccessful, a new target trajectory is created.

[0157] The matching process includes using a broken trajectory prediction and repair module to predict the trajectory information of the target in the current frame in multiple subsequent time periods, forming multiple different predicted trajectory information, and associating and matching the correct predicted trajectory when a sudden change in spatial position occurs in a subsequent frame, using it as complete trajectory information to complete the target's actual discrete trajectory to form a continuous motion trajectory.

[0158] The aerial video image target tracking system based on broken trajectory repair provided in this embodiment of the invention has a functional structure that corresponds to the aerial video image target tracking method based on broken trajectory repair provided in this embodiment of the invention, and will not be described again here.

[0159] Figure 12 This is a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present invention. The electronic device 1200 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1201 and one or more memories 1202. The memory 1202 stores at least one instruction, which is loaded and executed by the processor 1201 to implement the steps of the above-described aerial video image target tracking method based on broken trajectory repair.

[0160] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the above-described aerial video image target tracking method based on broken trajectory repair. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0161] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for aerial video image target tracking based on broken trajectory repair, characterized in that, The method comprises: S1, collecting a to-be-tracked aerial video image by a drone or an airplane; S2, input the aerial video image in the form of single frame by frame into a sliding window slicing module, the sliding window slicing module divides each frame of the input image into a plurality of small resolution image slices P1~P n ; S3, combine the original image P0 of each frame with the images P1 to P2. n Input the trained joint detection-tracking model into the target. The joint detection-tracking model adopts the paradigm of detection before tracking, and performs target detection before tracking. It includes a target detection network and a Re-ID network that integrates a broken trajectory prediction and repair algorithm. The target detection network detects targets in P0~P n n+1 images to obtain n+1 detection results, and restores the detection boxes of P1~P n from the coordinates on the image slices to the coordinates on the original image, and then merges the same target after post-processing of non-maximum suppression on all the detection boxes of P0~P n 0~Pn, and outputs the final target detection result, including target category information, detection box coordinates and ID identity vector. The Re-ID network performs target tracking according to a target detection result of each target of each frame, and the target tracking comprises: The detection box coordinates of each target of the first frame are initialized as a target track, and from the second frame, the detection box coordinates of each target of the current frame are matched with the existing target track in combination with the ID identity vector, if the matching is successful, the detection box coordinates of each target of the current frame are updated to the target track, and if the matching is unsuccessful, a new target track is created. The matching comprises predicting, by a broken track prediction repair module, track information of the target in subsequent multiple time periods to form multiple different predicted track information, and when a spatial position mutation occurs in a subsequent frame, correctly matched predicted track information is associated and used as complete track information to complete a continuous motion track in the actual discrete track of the target.

2. The method of claim 1, wherein, The target detection network takes YOLOv11 as a main detector, and calculates the regression error of the target detection box by smooth L1 loss optimization, and the formula is: where b i represents the real coordinates of the i-th frame, represents the predicted frame coordinates, and the smooth L1 loss is a penalty on the regression error, avoiding the influence of large errors; The classification loss adopts cross-entropy loss to optimize the category prediction, and the formula is: where p i represents the real class label of the i-th target, is the predicted class probability, and the classification loss improves the classification accuracy of the model by minimizing the gap between the predicted class and the real class. Meanwhile, the model further comprises an ID embedding branch based on the main detector, and the core target of the ID embedding branch is to generate features capable of distinguishing different targets to ensure the consistency of target detection and the Re-ID network tracking, the ID embedding branch is a convolution layer with 128 kernels, which is used to extract unique identity embedding features from each position, these identity embedding features not only provide a recognizable representation for each target, but also help the model to better capture the identity information of the target, the ID embedding branch maps the feature map of the target to an identity vector with 128 dimensions through a convolution layer, as the target identity feature, these identity vectors are optimized together with the position and category information of the target during the training process, to ensure that the model can not only locate the target, but also generate accurate and strong distinguishing target features; During training, in order to enhance the consistency of the features of the same target in different frames and maximize the feature difference between different targets, an ID consistency loss is introduced, and the calculation formula is: Wherein, N is the total number of detection frames, K is the total number of IDs, L i (k) is the labeled ID number, is the target identity feature extracted by the network, and the loss function further improves the stability and distinguishability of the target identity by minimizing the distance between the features of the same target and increasing the distance between different targets. Finally, the model outputs the target category information and the target detection box output by the main detector, and the identity vector output by the ID embedding branch.

3. The method of claim 1, wherein, The processing process of the broken track prediction repair module comprises: First, the existing target trajectory is high-dimensional embedded, the spatial position information is mapped to a feature vector with semantic representation ability through a linear projection layer, and position coding is added to retain the timing structure to obtain the embedded trajectory feature z t ; Then, the embedded track feature z t In the input fusion motion-aware temporal modeling module, by introducing the speed and acceleration of the target, the model can identify the motion trend and change pattern while performing time modeling, thereby improving the perception ability for fast or irregular motion. The calculation formula is: wherein, is the trajectory feature sequence after fusing motion information, Enc(v t ), Enc(a t ) denotes embedding transformation of velocity and acceleration vectors through linear layers or small neural networks, respectively; Next, the state space model stacked N times is input input into the state space model stacked N times: At each time step t, the input features are normalized and a gated representation g is computed t followed by an update of the hidden state using a state space unit computed as follows: where h t is the obtained timing feature; W g , A, B are all learnable weight matrices, W g is a gating matrix, b g is a bias vector, h t , h t-1 respectively represent the hidden state of the current time step and the previous time step, A represents a state transition matrix, and B represents an input mapping matrix. Finally, the timing feature h t is decoded to output the position point where the target is likely to appear in the next time step.

4. The method of claim 3, wherein, The matching specifically comprises: For the t i th frame, first use the traditional SORT algorithm to match the t i th frame target detection box coordinates and the t i-1 th frame ID motion trajectory information, obtain matched trajectory information T1~T n and unmatched trajectory information L1~L m ; Then, for each unmatched trajectory L j , as time series data, input into the broken trajectory prediction repair module to first predict the coordinates of the next time step, obtaining a plurality of coordinate points P 11 ~P 1s , then splice P 11 ~P 1s to the trajectory L j respectively to form s predicted trajectories L 11 ~L 1s , then input these trajectories into the broken trajectory prediction repair module for prediction, and so on to obtain predicted trajectories PL1~PL d connected with the prediction results of multiple future time steps. Next, the t i frame and the predicted trajectory PL1~PL d Based on the second matching of IoU, the predicted positions of the matched predicted trajectory until the matching point at the previous multiple time steps are embedded into the actual trajectory as the actual positions, and the actual trajectory is updated to complete the broken trajectory caused by the sudden change of the target position. Finally, for the target points that still cannot be matched after the second matching, new motion trajectories are created and merged with the trajectories of the first matching and the second matching as the trajectory information of the current t i frame.

5. The method of claim 1, wherein, The training process of the joint detection-tracking model comprises: Collecting an aerial video image, extracting each frame to form a video frame sequence, pre-processing and target labeling are performed on the video frame sequence, and a labeling file obtained is combined with the video frame sequence to form an original data set; The video frame sequence in the original data set is input into a multi-view feature generation fusion network.

6. The method of claim 5, wherein, The processing process of the diffusion generation model includes: First, the input single-view target image is gradually added with Gaussian noise within a preset time step to form a noisy image; Next, step-by-step inverse denoising is performed on the noisy image to gradually reconstruct a clear image consistent with the input view from a pure noise state; Then, the reconstructed image is input as a condition, combined with the target view angle information described by the preset text prompt, and a multi-view representation of the target object is formed in the latent space. Through repeated execution of the iteration process of adding noise and reverse denoising under different view angle conditions, a view angle transformation image H consistent in style and reasonable in geometry corresponding to the input view angle is gradually generated i1 ~H im Meanwhile, the target image of the original view angle is defined as H i0 , and H i0 ~H im (m+1) target images are obtained.

7. The method of claim 6, wherein, The processing process of the feature fusion network includes: First, H i0 ~ H im The target feature F is extracted by the feature extraction network i0 ~ F im ; Then based on the multi-head self-attention mechanism, the F is calculated through multiple independent self-attention heads i0 ~F im The similarity between each pair of view features is calculated, and the fusion weight is learned to capture the global dependency between views, model the complementarity and consistency of the features between views, wherein the target feature F ij of the jth view of the ith target is calculated first Key Value wherein, are learnable self-attention matrices, h denotes the h-th attention head; Then, the attention weight is calculated, and the calculation formula is: wherein, query and key at another perspective l, represents the attention weight of the h-th head between perspective j and perspective l, d k is the feature dimension; Will The weighted values ​​are used as the output of the head, and the outputs of all heads are concatenated to obtain the dynamically weighted fusion feature F. attn The calculation formula is as follows: wherein, is the output of the h-th head for view j, N is the number of attention heads, W O is a learnable linear transformation matrix; Then, the F i,attn is input into a light convolutional network to learn a spatial transformation matrix T, and the F i,attn is spatially aligned by T to obtain an aligned feature F' i,attn , and the calculation formula is as follows: F' i,attn = Γ(F i,attn ,T) Where Γ represents a spatial transformation operation; Then, the F i,attn The feature maps of different scales are extracted and fused through the DLA34 network, and finally the multi-scale semantic fusion feature F i is obtained.

8. An aerial video image target tracking system based on broken trajectory repair, characterized in that, The system includes: The acquisition module is configured to acquire the to-be-tracked aerial video image by using a UAV or an airplane; The segmentation module is configured to input the aerial video images in the form of single frames into a sliding window slicing module, and the sliding window slicing module is configured to divide each input frame image into a plurality of small resolution image slices P1-Pn. n ; The detection tracking module is configured to input the original image P0 and the P1-Pn of each frame into the joint detection-tracking model. n The joint detection-tracking model is inputted respectively, adopts a paradigm of detection first and tracking later, and performs detection first and tracking later on the target, and includes a target detection network and a Re-ID network integrated with a broken track prediction repair algorithm. The target detection network detects targets in P0~P n n+1 images to obtain n+1 detection results, and restores the detection frames of P1~P n from the coordinates on the image slices to the coordinates on the original image, and then merges the same target after post-processing of non-maximum suppression on all the detection frames of P0~P n 0~Pn, and outputs the final target detection result, including target category information, detection frame coordinates, and ID identity vector. The Re-ID network performs target tracking according to the target detection result of each target of each frame, and the target tracking includes: The detection box coordinates of each target of the first frame are initialized as a target track, and from the second frame, the detection box coordinates of each target of the current frame are matched with the existing target track in combination with the ID identity vector, if the matching is successful, the detection box coordinates of each target of the current frame are updated to the target track, and if the matching is unsuccessful, a new target track is created. The matching includes predicting the track information of the current frame target in subsequent multiple time periods by using a broken track prediction repair module, forming multiple different predicted track information, and when a spatial position mutation occurs in a subsequent frame, correctly matching the predicted track is associated and matched, and the correctly matched predicted track is completed as complete track information to the actual discrete track of the target to form a continuous motion track. 9.An electronic device comprising a processor and a memory in which at least one instruction is stored, wherein, The at least one instruction is loaded and executed by the processor to implement the aerial video image target tracking method based on broken track repair according to any one of claims 1-7.

10. A computer-readable storage medium having stored therein at least one instruction, wherein The at least one instruction is loaded and executed by the processor to implement the aerial video image target tracking method based on broken track repair according to any one of claims 1-7.

Images