Unmanned aerial vehicle video multi-target tracking method based on motion decoupling and related device

Abstract

The application belongs to the technical field of computer vision and multi-target tracking, and discloses a UAV video multi-target tracking method based on motion decoupling and a related device; wherein the UAV video multi-target tracking method comprises: acquiring a to-be-tracked video sequence shot by a UAV; acquiring a target detection bounding box, an appearance feature map of each of two adjacent images, and a dense optical flow feature between the two images; decomposing the dense optical flow feature to obtain a background optical flow component and a foreground optical flow component; obtaining a target prediction bounding box based on the background optical flow component, obtaining an enhanced appearance feature map based on the foreground optical flow component, and obtaining a multi-target tracking result based on the target prediction bounding box and the enhanced appearance feature map. The technical scheme of the application solves the technical problem that the existing method is affected by the fast nonlinear motion of a camera, leading to difficult position prediction and appearance degradation, and further reducing the performance of the entire multi-target tracking association process.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multi-target tracking technology, and specifically relates to a method and related device for multi-target tracking of UAV videos based on motion decoupling. Background Technology

[0002] Multi-target tracking is a core foundational technology in computer vision. It aims to accurately locate multiple targets in consecutive image frames and maintain the uniqueness and continuity of each target's identity throughout the process. It has become a key support for intelligent upgrades in fields such as intelligent monitoring, autonomous driving, and drone inspection.

[0003] Currently, driven by the iteration of target detection and re-identification technologies, the performance of multi-target tracking has been significantly improved. However, most of the existing mainstream multi-target tracking methods use simple motion models such as Kalman filtering and appearance feature association technology based on re-identification, which are highly dependent on the premise of stable camera motion and are only suitable for static or low-dynamic camera scenarios. When facing high-dynamic camera scenarios such as drones, the tracking performance drops sharply or even fails due to the complexity of camera motion and the degradation of target appearance.

[0004] Further explanation reveals that in drone scenarios, camera motion is high-frequency and abrupt due to flight attitude adjustments and airflow interference, posing two major bottlenecks to multi-target tracking: First, traditional smooth motion models can only model simple motions and cannot adapt to the global nonlinear offsets introduced by drones, resulting in large deviations in target position prediction. Second, targets are easily occluded, and motion blur leads to appearance degradation, compromising the reliability of traditional appearance feature associations and causing target identity confusion. Furthermore, while keypoint matching compensation methods such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) are efficient in compensating for camera motion, they are prone to misselecting foreground targets as background keypoints, leading to motion parameter deviations. Dense optical flow compensation methods, while capable of capturing pixel-level motion, introduce invalid information due to interference between background and target motion, increasing errors. Moreover, when enhancing appearance features, existing technologies improve robustness by aggregating historical features, but in drone scenarios, inter-frame target features are prone to misalignment; direct aggregation introduces significant noise, exacerbating identity instability and affecting tracking accuracy.

[0005] In summary, UAV video multi-target tracking aims to achieve multi-target tracking on video data with rich camera motion, such as that of UAVs. Similar to most multi-target tracking tasks, the main challenge of UAV video multi-target tracking is how to obtain high-quality tracking results. Existing methods are affected by the rapid nonlinear motion of the camera, which leads to difficulties in position prediction and appearance degradation, thus reducing the performance of the entire multi-target tracking association process. Summary of the Invention

[0006] The purpose of this invention is to provide a method and related apparatus for multi-target tracking of UAV video based on motion decoupling, so as to solve the technical problem that existing methods are affected by the rapid nonlinear motion of the camera, which leads to difficulties in position prediction and appearance degradation, thereby reducing the performance of the entire multi-target tracking association process.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, this invention provides a method for multi-target tracking in UAV video based on motion decoupling, comprising the following steps:

[0009] Acquire the video sequence to be tracked captured by the drone;

[0010] Two adjacent frames of images in the video sequence to be tracked are acquired and their features are extracted respectively to obtain the target detection bounding box, appearance feature map, and dense optical flow features between the two frames.

[0011] The dense optical flow features are decomposed to obtain the background optical flow component reflecting camera motion and the foreground optical flow component reflecting the target's own motion;

[0012] The global camera motion transformation matrix is ​​calculated based on the background optical flow component, and the target prediction bounding box of the current frame is obtained by using the global camera motion transformation matrix based on the target detection bounding box of the previous frame.

[0013] Based on the foreground optical flow component and the appearance feature maps of the two frames, an enhanced appearance feature map is obtained using a feature fusion network.

[0014] Based on the predicted bounding box of the target and the enhanced appearance feature map, the association between the historical trajectory and the current detection is completed to obtain the multi-target tracking result; wherein, the multi-target tracking result includes at least the bounding box of each target in each frame of the video sequence, and a unique identifier used to distinguish different targets.

[0015] A further improvement to the technical solution of this invention lies in the step of decomposing the dense optical flow features to obtain the background optical flow component reflecting camera motion and the foreground optical flow component reflecting the target's own motion. Specifically, robust principal component analysis is used to decompose the dense optical flow features between two frames of images; wherein,

[0016] First, the dense optical flow features are reshaped into a matrix. ,matrix Each row corresponds to a horizontal or vertical displacement vector of one pixel;

[0017] Next, based on the motion characteristics of the drone scenario, a matrix factorization model is established, represented as: In the formula, The low-rank component matrix corresponds to the background motion; The sparse component matrix corresponds to the foreground motion;

[0018] Then, robust principal component analysis is used to solve the matrix decomposition model to obtain the decomposed matrix; wherein, in solving the matrix decomposition model, unsupervised separation of the matrix is ​​achieved through a convex optimization problem, and the optimization objective function is defined as:

[0019] ;

[0020] In the formula, The nuclear norm is used to constrain... The low rank ensures that the global motion of the background is represented; The L1 norm is used to constrain... The sparsity ensures that only the motion of the foreground target is characterized; This is a balancing coefficient used to coordinate the weights of low-rank constraints and sparse constraints;

[0021] Finally, the obtained decomposition matrices are reshaped to the same dimensions as the dense optical flow features to obtain the background optical flow component and the foreground optical flow component.

[0022] A further improvement to the technical solution of the present invention is that the step of calculating the global camera motion transformation matrix based on the background optical flow component, and using the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame includes:

[0023] Randomly sample the background optical flow components to construct a pseudo-corresponding point set;

[0024] The global camera motion transformation matrix is ​​obtained by processing the pseudo-corresponding point set using the least squares median method.

[0025] The global camera motion transformation matrix is ​​decomposed into linear components and translation vectors. The Kalman filter prediction state of the previous frame is updated using the decomposed components. Based on the updated target motion state and covariance, the target bounding box is updated using Kalman filtering to obtain the compensated target prediction bounding box of the current frame.

[0026] A further improvement to the technical solution of the present invention lies in that,

[0027] In the step of randomly sampling the background optical flow components and constructing a pseudo-corresponding point set, the constructed pseudo-corresponding point set... In the formula, For the first Pixel coordinates in a frame Let be the displacement vector of the pixel in the background optical flow. For the pixel in the first The corresponding predicted coordinates in the frame caused by camera motion. For background optical flow components;

[0028] Using the least squares median method to process pseudo-corresponding point sets In the step of obtaining the global camera motion transformation matrix, for each point in the corresponding point set... residual The actual coordinates of this point and the transformed matrix The Euclidean distance between the mapped coordinates is given by the formula: In the formula, Let Euclidean distance be the function used to calculate the distance; through iterative optimization, find the transformation matrix that minimizes the median of the squared residuals at all corresponding points. The optimization objective formula is expressed as: The final matrix For the first Frame to the The global camera motion transformation matrix of the frame;

[0029] In the step of decomposing the global camera motion transformation matrix into linear components and translation vectors, and then updating the Kalman filter prediction state of the previous frame using the decomposed components, the update formula is:

[0030] ;

[0031] ;

[0032] In the formula, The target motion state is predicted by Kalman filtering. For covariance; and This represents the updated target motion state and covariance.

[0033] A further improvement to the technical solution of the present invention lies in the step of obtaining the enhanced appearance feature map using a feature fusion network based on the foreground optical flow component and the appearance feature maps of the two frames.

[0034] Based on the foreground optical flow component The previous frame's appearance feature map Propagate to the current frame and generate a motion-compensated propagation feature map. To achieve spatial alignment between the target features of the previous frame and the current frame, the formula is: In the formula, The current frame pixel is taken from the foreground optical flow component. The foreground motion displacement vector corresponding to the position;

[0035] The obtained propagation feature map Compared with the appearance feature map of the current frame After channel-level concatenation, the input is a pre-trained feature fusion network, which outputs a fused feature map as the enhanced appearance feature map. The formula is: In the formula, The mapping function representing the feature fusion network; This indicates a channel-level splicing operation.

[0036] A further improvement of the technical solution of the present invention is that, in the feature fusion network, the original input first passes through a first convolutional layer to adjust the feature dimension, then performs channel attention weighting to dynamically adjust the importance weight of each channel feature, and uses a second convolutional layer to obtain the weighted result; the third convolutional layer adjusts the dimension of the weighted result, and the fourth convolutional layer adjusts the dimension of the original input to make the two sizes uniform; the results obtained from the third and fourth convolutional layers are residually connected and used as the output of the feature fusion network.

[0037] A further improvement to the technical solution of this invention lies in the step of associating historical trajectories with current detections based on the predicted bounding box of the target and the enhanced appearance feature map to obtain multi-target tracking results.

[0038] From the fused feature map Extract each predicted bounding box Corresponding target appearance feature vector Combine the current frame to detect bounding boxes appearance feature vector Calculate matching score This yields a matching score matrix; where the matching scores are... The calculation formula is: In the formula, All are weighting coefficients; Represents the predicted bounding box With detection bounding box The crossover ratio between them; The cosine similarity between appearance features and These are from the fused feature maps Neutralize the current frame appearance feature map The target feature vector extracted from it corresponds to the location of the target;

[0039] Based on the matching score matrix, the optimal match is obtained by using the Hungarian algorithm, realizing the identity association between the current frame detection result and the historical tracking trajectory, and obtaining multi-target tracking results.

[0040] A second aspect of the present invention provides a UAV video multi-target tracking system based on motion decoupling, comprising:

[0041] The data acquisition module is used to acquire the video sequence to be tracked captured by the drone;

[0042] The feature extraction module is used to acquire two adjacent frames of images in the video sequence to be tracked and extract features from them respectively, so as to obtain the target detection bounding box, appearance feature map and dense optical flow features between the two frames.

[0043] The scene motion decomposition module is used to decompose the dense optical flow features to obtain the background optical flow component reflecting the camera motion and the foreground optical flow component reflecting the target's own motion.

[0044] The background motion compensation module is used to calculate the global camera motion transformation matrix based on the background optical flow component, and use the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame.

[0045] The foreground-guided feature propagation module is used to obtain an enhanced appearance feature map based on the foreground optical flow component and the appearance feature maps of the two frames, using a feature fusion network.

[0046] The target association module is used to associate historical trajectories with current detections based on the predicted bounding boxes and enhanced appearance feature maps of the targets, thereby obtaining multi-target tracking results. The multi-target tracking results include at least the bounding boxes of each target in each frame of the video sequence, as well as unique identifiers used to distinguish different targets.

[0047] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the motion decoupling-based UAV video multi-target tracking method as described in any one of the first aspects of the present invention.

[0048] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the UAV video multi-target tracking method based on motion decoupling as described in any one of the first aspects of the present invention.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] Given that existing methods are affected by the rapid nonlinear motion of the camera, leading to difficulties in position prediction and appearance degradation, thus reducing the performance of the entire multi-target tracking and association process, this invention specifically discloses a motion decoupling-based UAV video multi-target tracking method. This method effectively solves the tracking challenges of nonlinear motion, target occlusion, and motion blur in UAV scenarios, significantly improving the accuracy and stability of multi-target tracking results in dynamic scenes. Specifically, this invention proposes a novel approach to compensate for position prediction and enhance appearance features using motion decoupling. Specifically, by decoupling dense optical flow, background optical flow reflecting camera motion and foreground optical flow reflecting target motion are obtained. This allows for the prediction and compensation of bounding boxes for historical trajectories, and the fusion and enhancement of the appearance feature map of the current frame. This improves the accuracy of the predicted bounding box position and the reliability of the appearance feature map, thereby enhancing the performance of the entire multi-target tracking process. In summary, the technical solution disclosed in this invention decomposes background optical flow and foreground optical flow through motion decoupling; it compensates for camera motion using background optical flow to solve the nonlinear motion problem in multi-target tracking of UAVs; it enhances feature temporal consistency through foreground-guided feature propagation to alleviate the problem of unreliable association caused by appearance degradation; finally, a large number of experimental results demonstrate the effectiveness and generalization of the technical solution of this invention, as well as its superior performance over existing methods on public datasets. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating a method for multi-target tracking of UAV video based on motion decoupling, as described in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall architecture of a UAV video multi-target tracking method based on motion decoupling in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the overall structure of the feature fusion network in an embodiment of the present invention;

[0055] Figure 4 This is a visualization diagram of the results of a UAV video multi-target tracking method based on motion decoupling in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of a UAV video multi-target tracking system based on motion decoupling, as described in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0059] Please see Figure 1 The present invention provides a method for multi-target tracking of UAV video based on motion decoupling, comprising the following steps:

[0060] Step 1: Obtain the video sequence to be tracked captured by the drone;

[0061] Step 2: Extract features from two adjacent frames obtained in Step 1 to obtain their respective detection bounding boxes, appearance feature maps, and dense optical flow between the two frames.

[0062] Step 3: Based on the dense optical flow features obtained in Step 2, robust principal component analysis is used to decompose the optical flow to obtain the background optical flow reflecting the camera motion and the foreground optical flow reflecting the target's own motion.

[0063] Step 4: Based on the background optical flow obtained in Step 3, calculate the global camera motion transformation matrix, and use this matrix to obtain the predicted bounding box of the target in the current frame through the target detection bounding box of the previous frame.

[0064] Step 5: Based on the foreground optical flow obtained in Step 3 and the appearance feature maps of the two frames obtained in Step 2, an enhanced appearance feature map is obtained using a pre-trained feature fusion network.

[0065] Step 6: Based on the predicted bounding box obtained in Step 4 and the appearance feature map obtained in Step 5, as well as the detected bounding box and appearance feature map of the current frame obtained in Step 2, the association between the historical trajectory and the current detection is completed to obtain the multi-target tracking result; wherein, the multi-target tracking result includes at least the bounding box of each target in each frame of the video sequence, and a unique identifier used to distinguish different targets.

[0066] In the technical solution provided by this invention, dense optical flow is decomposed into background and foreground optical flow through unsupervised learning, accurately separating static background and dynamic target motion information, laying the foundation for subsequent motion modeling. Background optical flow is used to estimate and compensate for the global camera motion parameters of the UAV, offsetting camera movement interference and achieving accurate target frame alignment under complex motion. During the foreground-guided feature propagation process, the target motion information from the foreground optical flow guides the features to propagate along a consistent target trajectory, mitigating feature degradation caused by occlusion and motion blur. This technical solution, centered on the core logic of optical flow decoupling, complementarily optimizes motion decomposition, camera motion compensation, and appearance feature enhancement dimensions to jointly strengthen the framework's video-level tracking representation capabilities, comprehensively improving the multi-target tracking performance of the UAV.

[0067] Please see Figures 2 to 4 In an exemplary embodiment of the present invention, a method for multi-target tracking of UAV video based on motion decoupling is provided, specifically including the following steps:

[0068] Step 1: Use pre-trained models YOLOX, ResNet-50, and FastFlowNet to extract detection bounding boxes, appearance feature maps, and dense optical flow between two frames, respectively.

[0069] The specific exemplary technical solution includes the following steps:

[0070] Step 1.1: Given a video sequence acquired by a drone camera. It is defined as containing consecutive video frames. , This indicates the total number of video frames contained in the video sequence.

[0071] Step 1.2 involves acquiring the detection bounding boxes, appearance feature maps, and dense optical flow between two frames of the video sequence for use in subsequent steps. In this embodiment of the invention, a YOLOX network is used to process each frame. Extracted Detection bounding box of each target Each frame of the image is processed using a ResNet-50 network. Extract the appearance feature map Processing two consecutive frames using the FastFlowNet network and Inter-frame dense optical flow was extracted. .

[0072] Step 2 is the process of decoupling the dense optical flow using robust principal component analysis, given the known optical flow feature sequence of the video (i.e., dense optical flow).

[0073] The specific exemplary technical solution includes the following steps:

[0074] Step 2.1, obtain the dense optical flow between the two frames. Remodeled into matrix form , This refers to the total number of pixels in the image, with each row of the matrix corresponding to the horizontal or vertical displacement vector of one pixel. Based on the motion characteristics of the drone scene, a matrix factorization model is established: In the formula, The low-rank component matrix corresponds to the background motion; It is a sparse component matrix, corresponding to the foreground motion.

[0075] Step 2.2: Solve the decomposition model using robust principal component analysis. Unsupervised matrix separation is achieved through a convex optimization problem. The objective function is defined as:

[0076] ;

[0077] in, The nuclear norm (the sum of the singular values ​​of a matrix) is used to constrain... The low-rank nature of the character ensures that it represents the global motion of the background; The L1 norm (the sum of the absolute values ​​of the matrix elements) is used to constrain... The sparsity ensures that it only represents the motion of foreground targets; This is the balance coefficient, and its value is [value missing]. The weights are used to coordinate low-rank constraints and sparse constraints.

[0078] Step 2.3: Reshape the decomposed matrices into the same form as the original dense optical flow. Consistent dimensions yield background optical flow components. and foreground optical flow component As input to downstream methods.

[0079] Step 3: Given the historical information of the tracked target (i.e., the bounding box of the historical trajectory), utilize the background optical flow obtained from decoupling. The process of performing camera motion compensation includes the following specific steps:

[0080] Step 3.1, adjust the background optical flow component. Perform random sampling to construct a pseudo-corresponding point set. In the formula, For the first Pixel coordinates in a frame This is the displacement vector of the pixel in the background optical flow. For this pixel in the th The corresponding predicted coordinates in the frame caused by camera motion.

[0081] Step 3.2: Use the least squares median method to process the pseudo-corresponding point set. Obtain the transformation matrix The specific steps include:

[0082] Step 3.2.1, for each point in the corresponding point set residual The actual coordinates of the point and the transformed matrix The Euclidean distance between the mapped coordinates is given by the formula: ;in, This represents the function for calculating Euclidean distance;

[0083] Step 3.2.2: Through iterative optimization, find the transformation matrix that minimizes the median of the squared residuals at all corresponding points. The optimization objective formula is: The final matrix obtained That is, the first Frame to the The transformation matrix of the global camera motion of the frame.

[0084] Step 3.3, using the transformation matrix Update and predict the target bounding box of the historical trajectory; the specific steps include:

[0085] Step 3.3.1, transform the matrix Decomposed into linear components Represents linear transformations such as rotation and scaling, and translation vectors. The Kalman filter prediction state of the previous frame is updated using the decomposed components. The update formula is as follows:

[0086] ;

[0087] ;

[0088] In the formula, The target motion state is predicted by Kalman filtering. For covariance, and The updated target motion state and covariance;

[0089] Step 3.3.2, simultaneously, the updated target motion state With covariance The target bounding box will be updated using Kalman filtering to obtain the compensated predicted bounding box for the current frame. .

[0090] Step 4: Given the characteristics of the video sequence (i.e., appearance feature map), utilize the foreground optical flow obtained from decoupling. The process of guiding the propagation and fusion of appearance features includes the following specific steps:

[0091] Step 4.1, based on the foreground optical flow component The previous frame's appearance feature map Propagate to the current frame to generate motion-compensated historical feature maps. To achieve spatial alignment between the target features of the previous frame and the current frame, the formula is:

[0092] ;

[0093] in, The current frame pixel is taken from the foreground optical flow component. The foreground motion displacement vector corresponding to the position;

[0094] In a specific exemplary technical solution, bilinear interpolation is used in the calculation process of step 4.1 to achieve sub-pixel sampling accuracy and ensure the spatial continuity of the appearance feature map after propagation;

[0095] Step 4.2, obtain the propagation feature map Compared with the appearance feature map of the current frame After channel-level concatenation, the input is fed into a pre-trained feature fusion network, which outputs a fused feature map. The formula is:

[0096] ;

[0097] in, The mapping function representing the feature fusion network; This indicates a channel-level splicing operation;

[0098] In a specific exemplary technical solution, the feature fusion network is as follows: Figure 3As shown, it mainly includes: four convolutional layers, each followed by a non-linear activation function; a channel attention module at the end of the network to adaptively adjust the weights of the fused feature channels; and a residual connection to directly connect the input spliced ​​feature map to the network output.

[0099] The specific processing steps include: concatenating the obtained propagation feature map with the appearance feature map of the current frame along the channel direction, and using the result as the input to the feature fusion network; the input first passes through the first convolutional layer to adjust the feature dimension, then uses SE Block features for channel attention weighting, dynamically adjusting the importance weight of each channel feature, and using the second convolutional layer to obtain the weighted result; the third convolutional layer is used to adjust the feature dimension, and the fourth convolutional layer is used to adjust the dimension of the original input to ensure that the two are of the same size; finally, the results obtained from the third and fourth convolutional layers are residually linked as the final output of the feature fusion network, i.e., the fused feature map.

[0100] The steps in the training phase of a pre-trained feature fusion network include:

[0101] Obtain the training sample set, which consists of the propagation feature map obtained from the foreground optical flow and the appearance feature map of the previous frame, as well as the appearance feature map of the current frame;

[0102] Using the propagation feature map obtained above and the appearance feature map of the current frame, the feature fusion network is trained in a supervised manner. In the exemplary technical solution, during supervised training, the predicted features extracted from the fused feature map at the target location and the actual features extracted from the feature map of the current frame at the target location are compared. The loss is calculated as the loss for each target, with the total loss being the average loss of all targets in each frame.

[0103] In the inference stage of the pre-trained feature fusion network, the trained feature fusion network is used to perform inference. That is, between every two frames of images, the propagation feature map and the appearance feature map of the current frame are used to perform feature fusion to obtain the enhanced appearance feature map.

[0104] To further illustrate, during network training and optimization, the feature fusion network minimizes its output feature map. With target feature map Between Loss is monitored, and the distance is evaluated at the object-level region; specifically, for each target... From the predicted feature map Extracting feature vectors From the current frame feature map Extract And calculate the loss:

[0105] ;

[0106] in, Indicates the number of target objects.

[0107] The embodiments of the present invention adopt Loss functions, because they are robust to outliers and can preserve feature changes, help maintain identity consistency across time.

[0108] Step 5: Given the compensated predicted bounding box and enhanced appearance feature map of the current frame, complete the target association based on the target bounding box and feature map of the historical trajectory; the specific steps include:

[0109] Step 5.1, from the fused feature map Extract each predicted bounding box Corresponding target appearance feature vector Combine the current frame to detect bounding boxes appearance feature vector Calculate matching score The formula is:

[0110] ;

[0111] In the formula, These are the weighting coefficients; Represents the prediction box With detection bounding box The crossover ratio between them This represents the cosine similarity between their appearance features; where, and These are from the global feature map neutralization The target feature vector extracted from it corresponds to the location of the target;

[0112] Step 5.2: Solve for the optimal match using the Hungarian algorithm to associate the detection result of the current frame with the identity of the historical tracking trajectory and obtain the multi-target tracking result; wherein, the multi-target tracking result includes the bounding box of each target in each frame of the video sequence, the category to which the target belongs, and a unique tracking identity identifier used to distinguish different targets, and the bounding box, category and identity identifier of each target form a complete and continuous tracking trajectory with consecutive frames.

[0113] Please see Figure 4In the data analysis of specific embodiments of the present invention, the method proposed in the embodiments of the present invention was evaluated using multiple object tracking accuracy (MOTA), identity recognition F1 score (IDF1), and identity switching count (IDs), and the results on the VisDrone2019 and UAVDT datasets are shown.

[0114] Table 1 shows the experimental results of different methods on the VisDrone2019 public UAV video dataset used to evaluate multi-target tracking. As can be seen from Table 1, the motion decoupling-based UAV video multi-target tracking method proposed in this embodiment of the invention has significant advantages.

[0115] Table 1. Results of the comparison of different methods on the VisDrone2019 dataset.

[0116]

[0117] Table 2 shows the evaluation results of the technical solution of the present invention on the UAVDT dataset; the experimental results show the effectiveness of the technical solution of the present invention in multi-target tracking capability in UAV video scenarios.

[0118] Table 2. Results of comparison of different methods on the UAVDT dataset

[0119]

[0120] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0121] Please see Figure 5 In this embodiment of the invention, a UAV video multi-target tracking system based on motion decoupling is provided, comprising:

[0122] The data acquisition module is used to acquire the video sequence to be tracked captured by the drone;

[0123] The feature extraction module is used to acquire two adjacent frames of images in the video sequence to be tracked and extract features from them respectively, so as to obtain the target detection bounding box, appearance feature map and dense optical flow features between the two frames.

[0124] The scene motion decomposition module is used to decompose the dense optical flow features to obtain the background optical flow component reflecting the camera motion and the foreground optical flow component reflecting the target's own motion.

[0125] The background motion compensation module is used to calculate the global camera motion transformation matrix based on the background optical flow component, and use the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame.

[0126] The foreground-guided feature propagation module is used to obtain an enhanced appearance feature map based on the foreground optical flow component and the appearance feature maps of the two frames, using a feature fusion network.

[0127] The target association module is used to associate historical trajectories with the current detection based on the target predicted bounding box and the enhanced appearance feature map to obtain multi-target tracking results; wherein, the multi-target tracking results include at least the bounding box of each target in each frame of the video sequence, and a unique identifier used to distinguish different targets.

[0128] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to execute the operation of a motion-decoupling-based UAV video multi-target tracking method.

[0129] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the motion decoupling-based UAV video multi-target tracking method in the above embodiments.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for multi-target tracking in UAV video based on motion decoupling, characterized in that, Includes the following steps: Acquire the video sequence to be tracked captured by the drone; Two adjacent frames of images in the video sequence to be tracked are acquired and feature extraction is performed on them respectively to obtain the target detection bounding box, appearance feature map and dense optical flow feature between the two frames. The dense optical flow features are decomposed to obtain the background optical flow component reflecting camera motion and the foreground optical flow component reflecting the target's own motion; The global camera motion transformation matrix is ​​calculated based on the background optical flow component, and the target prediction bounding box of the current frame is obtained by using the global camera motion transformation matrix based on the target detection bounding box of the previous frame. Based on the foreground optical flow component and the appearance feature maps of the two frames, an enhanced appearance feature map is obtained using a feature fusion network. Based on the predicted bounding boxes of the targets and the enhanced appearance feature maps, the association between historical trajectories and the current detection is completed to obtain multi-target tracking results; wherein, the multi-target tracking results include at least the bounding boxes of each target in each frame of the video sequence, as well as unique identifiers used to distinguish different targets; The steps of calculating the global camera motion transformation matrix based on the background optical flow component, and using the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame, include: Random sampling is performed on the background optical flow components to construct a pseudo-corresponding point set; wherein, the constructed pseudo-corresponding point set... In the formula, For the first Pixel coordinates in a frame This is the displacement vector of the pixel in the background optical flow. For the pixel in the first The corresponding predicted coordinates in the frame caused by camera motion. For background optical flow components; The pseudo-corresponding point set is processed using the least squares median method to obtain the global camera motion transformation matrix; where, for each point in the corresponding point set... residual The actual coordinates of the point and the transformed matrix The Euclidean distance between the mapped coordinates is given by the formula: In the formula, Let Euclidean distance be the function used to calculate the distance; through iterative optimization, find the transformation matrix that minimizes the median of the squared residuals of all corresponding points. The optimization objective formula is expressed as: The final matrix For the first Frame to the The global camera motion transformation matrix of the frame; The global camera motion transformation matrix is ​​decomposed into linear components and translation vectors. The Kalman filter prediction state of the previous frame is updated using these decomposed components. Based on the updated target motion state and covariance, the target bounding box is updated using Kalman filtering to obtain the compensated target prediction bounding box for the current frame. The update formula is as follows: ； ； In the formula, The target motion state is predicted by Kalman filtering. For covariance; and This represents the updated target motion state and covariance.

2. The UAV video multi-target tracking method based on motion decoupling according to claim 1, characterized in that, In the step of decomposing the dense optical flow features to obtain the background optical flow component reflecting camera motion and the foreground optical flow component reflecting the target's own motion, the dense optical flow features between two frames are specifically decomposed using a robust principal component analysis method; wherein, First, the dense optical flow features are reshaped into a matrix. ,matrix Each row corresponds to a horizontal or vertical displacement vector of one pixel; Next, based on the motion characteristics of the drone scenario, a matrix factorization model is established, represented as: In the formula, The low-rank component matrix corresponds to the background motion; The sparse component matrix corresponds to the foreground motion; Then, robust principal component analysis is used to solve the matrix decomposition model to obtain the decomposed matrix; wherein, in solving the matrix decomposition model, unsupervised separation of the matrix is ​​achieved through a convex optimization problem, and the optimization objective function is defined as: ； In the formula, The nuclear norm is used to constrain... The low rank ensures that the global motion of the background is represented; The L1 norm is used to constrain... The sparsity ensures that only the motion of the foreground target is characterized; This is a balancing coefficient used to coordinate the weights of low-rank constraints and sparse constraints; Finally, the obtained decomposition matrices are reshaped to the same dimensions as the dense optical flow features to obtain the background optical flow component and the foreground optical flow component.

3. The UAV video multi-target tracking method based on motion decoupling according to claim 1, characterized in that, In the step of obtaining the enhanced appearance feature map based on the foreground optical flow component and the appearance feature maps of the two frames, a feature fusion network is used. Based on the foreground optical flow component The previous frame's appearance feature map Propagate to the current frame and generate a motion-compensated propagation feature map. To achieve spatial alignment between the target features of the previous frame and the current frame, the formula is: In the formula, The current frame pixel is taken from the foreground optical flow component. The foreground motion displacement vector corresponding to the position; The obtained propagation feature map Compared with the appearance feature map of the current frame After channel-level concatenation, the input is a pre-trained feature fusion network, which outputs a fused feature map as the enhanced appearance feature map. The formula is: In the formula, The mapping function representing the feature fusion network; This indicates a channel-level splicing operation.

4. A method for multi-target tracking of UAV video based on motion decoupling according to claim 1 or 3, characterized in that, In the feature fusion network, the original input first passes through a first convolutional layer to adjust the feature dimensions, then performs channel attention weighting to dynamically adjust the importance weights of each channel feature, and uses a second convolutional layer to obtain the weighted result; the third convolutional layer adjusts the dimension of the weighted result, and the fourth convolutional layer adjusts the dimension of the original input to make the two sizes uniform; the results obtained from the third and fourth convolutional layers are residually connected and used as the output of the feature fusion network.

5. The UAV video multi-target tracking method based on motion decoupling according to claim 3, characterized in that, In the step of associating historical trajectories with current detections based on the predicted bounding box and enhanced appearance feature map to obtain multi-object tracking results, From the fused feature map Extract each predicted bounding box Corresponding target appearance feature vector Combine the current frame to detect bounding boxes appearance feature vector Calculate matching score This yields a matching score matrix; where the matching scores are... The calculation formula is: In the formula, All are weighting coefficients; Represents the predicted bounding box With detection bounding box The crossover ratio between them; The cosine similarity between appearance features and These are from the fused feature maps Neutralize the current frame appearance feature map The target feature vector extracted from it corresponds to the location of the target; Based on the matching score matrix, the optimal match is obtained by using the Hungarian algorithm, realizing the identity association between the current frame detection result and the historical tracking trajectory, and obtaining multi-target tracking results.

6. A UAV video multi-target tracking system based on motion decoupling, characterized in that, include: The data acquisition module is used to acquire the video sequence to be tracked captured by the drone; The feature extraction module is used to acquire two adjacent frames of images in the video sequence to be tracked and extract features from them respectively, so as to obtain the target detection bounding box, appearance feature map and dense optical flow features between the two frames. The scene motion decomposition module is used to decompose the dense optical flow features to obtain the background optical flow component reflecting the camera motion and the foreground optical flow component reflecting the target's own motion. The background motion compensation module is used to calculate the global camera motion transformation matrix based on the background optical flow component, and use the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame. The foreground-guided feature propagation module is used to obtain an enhanced appearance feature map based on the foreground optical flow component and the appearance feature maps of the two frames, using a feature fusion network. The target association module is used to associate historical trajectories with the current detection based on the predicted bounding boxes and enhanced appearance feature maps of the targets, so as to obtain multi-target tracking results; wherein, the multi-target tracking results include at least the bounding boxes of each target in each frame of the video sequence, as well as unique identifiers used to distinguish different targets; The background motion compensation module includes the following steps: calculating a global camera motion transformation matrix based on the background optical flow components, and using the global camera motion transformation matrix to obtain the target prediction bounding box of the current frame based on the target detection bounding box of the previous frame: Random sampling is performed on the background optical flow components to construct a pseudo-corresponding point set; wherein, the constructed pseudo-corresponding point set... In the formula, For the first Pixel coordinates in a frame This is the displacement vector of the pixel in the background optical flow. For the pixel in the first The corresponding predicted coordinates in the frame caused by camera motion. For background optical flow components; The pseudo-corresponding point set is processed using the least squares median method to obtain the global camera motion transformation matrix; where, for each point in the corresponding point set... residual The actual coordinates of the point and the transformed matrix The Euclidean distance between the mapped coordinates is given by the formula: In the formula, Let Euclidean distance be the function used to calculate the distance; through iterative optimization, find the transformation matrix that minimizes the median of the squared residuals of all corresponding points. The optimization objective formula is expressed as: The final matrix For the first Frame to the The global camera motion transformation matrix of the frame; The global camera motion transformation matrix is ​​decomposed into linear components and translation vectors. The Kalman filter prediction state of the previous frame is updated using these decomposed components. Based on the updated target motion state and covariance, the target bounding box is updated using Kalman filtering to obtain the compensated target prediction bounding box for the current frame. The update formula is as follows: ； ； In the formula, The target motion state is predicted by Kalman filtering. For covariance; and This represents the updated target motion state and covariance.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the UAV video multi-target tracking method based on motion decoupling as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV video multi-target tracking method based on motion decoupling as described in any one of claims 1 to 5.

Images