Unmanned aerial vehicle multi-target tracking method based on CNN-Transformer-Mamba network and space-time Mamba motion model

Abstract

The present application relates to the field of artificial intelligence and computer vision, in particular to a UAV multi-target tracking method based on a CNN-Transformer-Mamba network and a space-time Mamba motion model, comprising: acquiring a video data set of multi-target tracking collected by a UAV; constructing a CNN-Transformer-Mamba network as a target detector to acquire the bounding box and spatial position information of multiple targets on the ground in a single frame of image; constructing a multi-target space-time trajectory based on the spatial position information in the continuous space-time; constructing a space-time Mamba motion model to complete nonlinear modeling of the multi-target space-time trajectory, realize multi-target tracking; constructing a joint loss function to train the model, and evaluating the model performance through precision evaluation. The present application effectively fuses local-global-long-range dependent features by aggregating the advantages of CNN, Transformer and Mamba network, improves the positioning accuracy of multi-target position, and improves the accuracy and robustness of multi-target tracking through nonlinear modeling of space-time trajectory, thereby further improving the adaptability to complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, and in particular to a method for multi-target tracking of unmanned aerial vehicles based on a CNN-Transformer-Mamba network and a spatiotemporal Mamba motion model. Background Technology

[0002] Multi-object tracking (MOT) is a hot topic in the fields of artificial intelligence and computer vision. With the rapid development of drone technology and the continuous advancement of computer vision, drone-based MOT technology shows broad application prospects in intelligent transportation, public space management, marine monitoring, and military reconnaissance. However, MOT-based MOT primarily employs a "detection-tracking" paradigm, which still faces numerous challenges:

[0003] (1) Two main problems are usually faced in the target detection stage. First, the frequent changes in the altitude and viewpoint of the UAV cause significant changes in the appearance and scale of the target, which increases the difficulty of target interpretation. Second, the similar visual features of ground objects in complex backgrounds amplify the interference with target recognition. Current mainstream target detection networks usually rely on the following three methods to enhance visual features: 1) Local feature enhancement based on CNN networks, such as the YOLO series; 2) Global feature enhancement based on Transformer, such as FocusTrack; 3) Long-range dependency enhancement based on Mamba, such as VisionMamba. The above methods further enhance the model's visual perception and expressive ability by adopting different approaches, thereby improving the accuracy of target detection.

[0004] (2) In the target tracking stage, current mainstream networks rely on motion estimation algorithms to capture the spatiotemporal correlation of ground objects, thereby achieving robust tracking over a continuous time range. These mainstream algorithms rely on linear motion estimation methods, including the Kalman filter (KF) and its variants, such as SORT, OC-SORT, Bytetrack, U2MOT, and TRBS-Net. These methods demonstrate that achieving robust spatiotemporal correlation is beneficial for further enhancing multi-target tracking capabilities.

[0005] In summary, the current research still has the following shortcomings:

[0006] (1) For the target detection stage, the current method has insufficient visual representation ability. The single deep neural network architecture limits the representation ability of multiple visual features and lacks an effective way to aggregate multiple visual features.

[0007] (2) For the target tracking stage, the spatiotemporal relationship of ground targets associated with the linear motion assumption is difficult to characterize the complex nonlinear motion patterns of ground targets in real-world scenarios.

[0008] Based on the shortcomings of the aforementioned research, the inventors developed a multi-target tracking method for unmanned aerial vehicles (UAVs) based on a CNN-Transformer-Mamba network and a spatiotemporal Mamba motion model. Summary of the Invention

[0009] To address the aforementioned technical challenges, this application provides a UAV multi-target tracking method based on a CNN-Transformer-Mamba network and a spatiotemporal Mamba motion model. This method enhances local detail perception through dynamic convolution, strengthens Mamba's long-range dependencies through multi-directional scanning, effectively fuses these features from a global perspective through a Transformer-based cross-attention mechanism, and improves the accuracy of multi-target detection within a single frame through multi-scale aggregation. Furthermore, by constructing a spatiotemporal Mamba motion model, it achieves nonlinear modeling of multi-target spatiotemporal trajectories, thereby improving the accuracy and efficiency of multi-target tracking.

[0010] The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model provided in this application adopts the following technical solution:

[0011] A method for multi-target tracking of unmanned aerial vehicles based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model includes the following steps:

[0012] S1. Obtain the video dataset of multi-target tracking collected by the UAV, and divide the video dataset into training set, test set and validation set. The training set is used for network fitting, the validation set is used for hyperparameter tuning, and the test set is used for accuracy evaluation.

[0013] S2, construct a CNN-Transformer-Mamba network as an object detector, and obtain the bounding boxes and spatial location information of multiple ground objects in a single frame image;

[0014] S3, constructing multi-target spatiotemporal trajectories based on spatial location information within continuous spatiotemporal space;

[0015] S4. Construct a spatiotemporal Mamba motion model to complete the nonlinear modeling of the spatiotemporal trajectories of multiple targets and realize multi-target tracking;

[0016] S5. Construct a joint loss function to train the model, and evaluate the model performance through accuracy evaluation.

[0017] Preferably, S1 includes the following steps: based on open-source video data, obtain a multi-target tracking video dataset collected by a drone, and divide the video dataset into a training set, a validation set, and a test set according to a preset ratio;

[0018] The training set is used for fitting network parameters, the validation set is used for tuning hyperparameters, and the test set is used for evaluating accuracy.

[0019] Preferably, the CNN-Transformer-Mamba network in S2 includes a backbone network, a neck module, and an object detection head. Constructing the CNN-Transformer-Mamba network includes the following steps:

[0020] S21. Construct the Backbone network, which includes a Stem module, multiple CNN-Transformer-Mamba modules, and a downsampling module;

[0021] The Stem module is used to project the input image onto a high-dimensional feature space. The Stem module includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0022] The CNN-Transformer-Mamba module is used to aggregate local, global, and long-range dependent features.

[0023] The CNN-Transformer-Mamba module includes a multi-scale perceptual convolutional block, a multi-directional scanning Mamba block, and a Transformer-based global aggregation block, and includes the following steps:

[0024] S211, Constructing a multi-scale perceptual convolutional block includes the following steps: Inputting the input feature map F into three parallel paths, each path including: a normalization layer, a global average pooling layer, a linear layer, an activation function layer, and a dynamic convolutional layer. Each path uses a dynamic convolutional layer with different windows, namely 3×3, 5×5, and 7×7. The calculation formula for dynamic convolution is as follows:

[0025] (1)

[0026] in, Indicates input features, Indicates output features, k represents the scale of the convolution kernel, w and h represent the positions of the convolution kernel at position (i,j) on the horizontal and vertical axes respectively, W(·) represents the dynamic weights, and then the dynamic features at different scales are fused through the Cat function, and a 1×1 convolutional layer is used to output multi-scale local features F. local;

[0027] S212, Constructing a multi-directional scanning Mamba block includes the following steps: Dividing the input feature F into branch one and branch two. Branch one includes a linear layer, a 1×1 convolutional layer, a SiLU activation function, and an SSM block; branch two includes a linear layer, a 1×1 convolutional layer, and a SiLU activation function. Branch one and branch two are aggregated using the Cat function, and a normalization layer is used to obtain the output long-range dependent feature F. long The calculation formula is as follows:

[0028] (2)

[0029] Where Conv(·) represents a convolutional layer, Linear(·) is a linear layer, SiLU(·) is the sigmoid activation function, Cat(·) is the concatenate function, and SSM(·) is the SSM module. It employs a multi-directional aggregation scanning method to obtain richer contextual features, uses linear layers to obtain continuous parameters, and converts them into discrete parameters based on a zero-order hold method, as shown in the following formula:

[0030] (3)

[0031] Where A, B, and C are the learnable parameters. , and Here are the transformed discretized parameters, where △ represents the time scale parameter, △A and △B represent the corresponding discrete time terms within a given time interval, and I represents the identity matrix. After transformation, the parameters are calculated using a linear recursive method, as shown in the following expression:

[0032] (4)

[0033] Where, x t This represents the input 1-dimensional feature, y t Represents the output feature, h t Representing the learnable intermediate states, the entire sequence is represented as a convolutional form, expressed as:

[0034] (5)

[0035] in, It is represented as a structured convolutional kernel, where M represents the length of the input sequence, x represents the input features, and y represents the output discretized features. It is trained by convolution and performs autoregressive inference using a linear recursive approach.

[0036] S213, Construct a global aggregation block based on Transformer, including the following steps:

[0037] A linear layer is used for the input feature F to obtain the query feature Q. For the output feature F in S211 local , two parallel linear layers are used to transform it into local value and key-value feature V C and K C , for the output feature F in S212 long , two parallel linear layers are used to transform it into long-range value and key-value feature V M and K M , and cross-attention is used to achieve effective dynamic fusion. The formula is as follows:

[0038] ; (6)

[0039] Among them, TGFB(·) represents the Transformer-based global aggregation block, Cat(·) represents the concatenate function, softmax(·) is the softmax function, (·) T represents the transpose of the feature, F O represents the output feature, and D represents the number of feature channels;

[0040] The downsampling module is used to achieve feature compression to obtain multi-scale features;

[0041] S22. Build the neck module. The neck module includes multiple cross-scale interaction modules and a scale aggregation module, including the following steps:

[0042] S221. Build the cross-scale interaction module. The formula is as follows:

[0043] ; (7)

[0044] Among them, SEB(·) is the cross-scale interaction module, (a, b, c, r) ∈ [1, 4] represents the indexes of different-scale features, r represents regarding the feature of the r-th scale as the main feature, and the remaining features are regarded as auxiliary features. z is the auxiliary feature of the z-th scale. If z > r, UDconv(·) uses the downsampling convolutional layer; if z < r, UDConv(·) uses the upsampling convolutional layer. Cat(·) is the concatenate function, and CTMB(·) represents the CNN-Transformer-Mamba module. The cross-scale interaction module will perform cross-scale interaction step by step from the low-resolution feature to the high-resolution feature as the main feature, and the output feature will replace the original resolution feature to participate in the next interaction;

[0045] S222. Build the scale aggregation module to aggregate the features after cross-scale interaction. The formula is as follows:

[0046] ; (8)

[0047] Where Uconv(·) represents upsampling convolution, Cat(·) is the concatenate function, CTMB(·) is the CNN-Transformer-Mamba module, and O1, O2, O3, and O4 are the output results of the cross-scale interaction module in S221 at different scales.

[0048] S23, Construct the target detection head module. The target detection head module contains three parallel detection heads. Each detection head includes a 3×3 convolutional layer with 256 channels and a 1×1 convolutional layer. The three parallel detection heads are a heatmap head, a bounding box offset head, and a bounding box size head. The heatmap head is used to estimate the center position of the object in each frame, while the bounding box offset head and the bounding box size head are used to improve the positioning accuracy.

[0049] Preferably, step S3 includes the following steps:

[0050] A CNN-Transformer-Mamba network built on S2 is used to identify multiple targets in a single frame image and obtain detection boxes, represented as follows: B u Let u represent the detection box, and let u represent the u-th target. This represents the x and y coordinates of the top-left corner of the detection box. Represents the horizontal and vertical coordinates of the bottom right corner of the detection box;

[0051] The Hungarian algorithm is used to perform spatiotemporal correlation of objects in consecutive frames. For each object's detection bounding box, based on the detection in each frame, k spatiotemporal trajectories of the object are obtained. ,in This represents the spatiotemporal trajectory of the e-th object. Let f represent the bounding box of the f-th object at frame t, where p and q represent the horizontal and vertical positions of the center point, and a and h represent the aspect ratio and height of the center point of the bounding box.

[0052] Preferably, step S4 includes the following steps:

[0053] For the spatiotemporal trajectory of the input target, it is expanded to a high-dimensional space through convolutional layers, and the output is continuous spatiotemporal parameters, as shown in the formula:

[0054] (9)

[0055] Wherein, Conv(·) is a convolutional layer, Linear(·) is a linear projection layer, and SSM(·) is the SSM module in step S212, which is used to perform powerful spatiotemporal modeling while maintaining the linear computational complexity of equations (3) to (5), and adds a multilayer perceptron layer MLP and a detection head to output the object prediction detection box to achieve target tracking.

[0056] Preferably, step S5 includes the following steps:

[0057] A global loss is used to optimize the detection and tracking tasks, as follows:

[0058] (10)

[0059] in, These are the learning weights used to balance detection and tracking tasks. It is the target detection loss. It is target tracking loss;

[0060] S51, construct the object detection loss to optimize the output of the heatmap header, bounding box offset header, and bounding box size header to improve the accuracy of object localization;

[0061] The heatmap header generates a heatmap of size H×W×1, assigning a value of 1 to the center of the ground truth target, and defining the ground truth bounding box as... The target center is defined as The center position on the feature map output can be calculated as The heatmap response value at location (m,n) is calculated as follows:

[0062] (11)

[0063] Where K represents the total number of targets in the current frame, σ is the standard deviation, and u represents the u-th target. The center point localization is optimized in the following way:

[0064] (12)

[0065] in, For the predicted heatmap, α and β are parameter variables. To compensate for the loss of positional accuracy caused by the step size of the output features, an L1 loss optimization objective is used, which is:

[0066] (13)

[0067] Where λ is a weight set to 0.15. and These represent the offset and size of the actual value, respectively. and These represent the dimensions of the true and predicted values, respectively, and an overall detection loss is applied by setting weight parameters. To adjust L heat and L box ,Right now:

[0068] (14)

[0069] S52, employs a multi-task loss function combining L1 loss and GIoU loss, weighted... To monitor and track tasks, the expression is as follows:

[0070] (15)

[0071] in, These are the learning weights used to balance the tracking task. It is L1 loss. It is a GIoU loss.

[0072] In summary, compared with the prior art, the present invention has the following beneficial technical effects:

[0073] 1. This invention enhances local detail perception by embedding dynamic convolution, strengthens the long-range dependent contextual information of Mamba through multi-directional scanning, achieves effective fusion of the above features from a global perspective through a Transformer-based cross-attention mechanism, enhances the generalization ability for multi-scale targets through multi-scale aggregation, and improves the accuracy of target detection by achieving effective aggregation of global-local-long-range dependent visual features. At the same time, it achieves nonlinear modeling of multi-target spatiotemporal trajectories by constructing a spatiotemporal Mamba motion model, thereby improving the accuracy and efficiency of multi-target tracking.

[0074] 2. In the target detection stage, this invention enhances local details by aggregating multi-scale features through dynamic convolution, compensates for the loss of long-range dependent contextual information in the Mamba model by employing a multi-directional scanning strategy, and effectively aggregates the enhanced local details and long-range dependencies in global vision through a cross-attention mechanism. Then, it fuses multi-scale features through a multi-scale aggregation strategy to enhance the visual representation ability of ground objects, effectively improving the detection capability and recognition accuracy of multi-target recognition within a single frame image.

[0075] 3. In the target tracking stage, this invention constructs a robust spatiotemporal trajectory and uses the spatiotemporal Mamba model to perform nonlinear modeling of the target's spatiotemporal trajectory across frames. This enables the adaptive capture of various complex motion dynamics, ensuring more accurate and objective motion estimation of ground objects and improving the accuracy of multi-target tracking. At the same time, thanks to the efficient computational method of the Mamba model with linear complexity, the computational efficiency for long-range spatiotemporal data is improved.

[0076] 4. In terms of model optimization, this invention enhances the model's convergence ability and inference efficiency by integrating multiple loss functions to construct a joint optimization objective. Through multi-loss collaborative optimization, it not only promotes robust convergence of model training but also leverages the efficient computational efficiency of the Mamba model to improve overall operational efficiency while ensuring improved accuracy in multi-target tracking. Attached Figure Description

[0077] Figure 1 This is a flowchart from an embodiment of the present invention.

[0078] Figure 2 This is an architecture diagram of the overall network in an embodiment of the present invention.

[0079] Figure 3 This is an architecture diagram of the CNN-Transformer-Mamba network as a target detector in an embodiment of the present invention.

[0080] Figure 4 These are schematic diagrams of (a) the CNN-Transformer-Mamba module and (b) the architecture diagram of the multi-scale perceptual convolution module, as given in the embodiments of the present invention.

[0081] Figure 5 This is an architecture diagram of multi-directional scanning Mamba blocks given in an embodiment of the present invention.

[0082] Figure 6 This is an architecture diagram of the spatiotemporal Mamba motion model given in this embodiment of the invention.

[0083] Figure 7 This is a schematic diagram of the multi-target tracking results in VisDrone data, as given in an embodiment of the present invention. Detailed Implementation

[0084] The present invention will be further described in detail below with reference to the accompanying drawings.

[0085] This invention discloses a method for multi-target tracking of unmanned aerial vehicles based on a CNN-Transformer-Mamba network and a spatiotemporal Mamba motion model.

[0086] This embodiment takes multi-object tracking based on drone video as an example and uses the open-source VisDrone dataset.

[0087] Reference Figure 1 and Figure 2 The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model includes the following steps:

[0088] S1. Obtain the video dataset of multi-target tracking collected by the UAV, and divide the video dataset into training set, test set and validation set. The training set is used for network fitting, the validation set is used for hyperparameter tuning, and the test set is used for accuracy evaluation.

[0089] Specifically, S1 includes the following steps:

[0090] The selected VisDrone dataset is a large-scale benchmark video dataset captured from a drone platform for multi-object tracking tasks. This dataset consists of 79 video clips totaling 33,366 frames, divided into training (56 videos), validation (7 videos), and test (16 videos) subsets. The dataset covers diverse urban and suburban environments with complex backgrounds, significant scale variations, frequent occlusion, and dense crowds. This dataset provides realistic testing for evaluating multi-object tracking algorithms under typical drone conditions.

[0091] S2, construct a CNN-Transformer-Mamba network as an object detector to obtain the bounding boxes and spatial location information of multiple ground objects within a single frame of imagery, referring to... Figure 3 ;

[0092] Constructing a CNN-Transformer-Mamba network as an object detector involves the following steps: The CNN-Transformer-Mamba network includes a backbone network, a neck module, and an object detection head.

[0093] S21. Construct the Backbone network, which includes a Stem module, multiple CNN-Transformer-Mamba (CTMB) modules, and a downsampling module;

[0094] The Stem module is used to project the input image onto a high-dimensional feature space. The Stem module includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0095] Reference Figure 4(a) The CNN-Transformer-Mamba module is used to aggregate local-global-long-range dependent features. The CNN-Transformer-Mamba module includes a multi-scale perceptual convolutional block, a multi-directional scanning Mamba block, and a Transformer-based global aggregation block, including the following steps:

[0096] S211, construct multi-scale perceptual convolutional blocks, referring to... Figure 4 (b) includes the following steps: inputting the input feature map F into three parallel paths, each path including: a normalization layer, a global average pooling layer, a linear layer, an activation function layer, and a dynamic convolutional layer. Each path uses a dynamic convolutional layer with different windows, namely 3×3, 5×5, and 7×7. The calculation formula for dynamic convolution is as follows:

[0097] (1)

[0098] in, Indicates input features, Indicates output features, k represents the scale of the convolution kernel, w and h represent the positions of the convolution kernel at position (i,j) on the horizontal and vertical axes respectively, W(·) represents the dynamic weights, and then the dynamic features at different scales are fused through the Cat function, and a 1×1 convolutional layer is used to output multi-scale local features F. local ;

[0099] S212, Construct a multi-directional scan of Mamba blocks, refer to... Figure 5 ;

[0100] Specifically, the steps include: dividing the input feature F into branch one and branch two; branch one includes a linear layer, a 1×1 convolutional layer, a SiLU activation function, and an SSM block; branch two includes a linear layer, a 1×1 convolutional layer, and a SiLU activation function; a Cat function is used to aggregate branch one and branch two; and a normalization layer is used to obtain the output long-range dependent feature F. long The calculation formula is as follows:

[0101] (2)

[0102] Where Conv(·) represents a convolutional layer, Linear(·) is a linear layer, SiLU(·) is the sigmoid activation function, Cat(·) is the concatenate function, and SSM(·) is the SSM module. It employs a multi-directional aggregation scanning method to obtain richer contextual features, uses linear layers to obtain continuous parameters, and converts them into discrete parameters based on a zero-order hold method, as shown in the following formula:

[0103] (3)

[0104] Where A, B, and C are the learnable parameters. , and Here are the transformed discretized parameters, where △ represents the time scale parameter, △A and △B represent the corresponding discrete time terms within a given time interval, and I represents the identity matrix. After transformation, the parameters are calculated using a linear recursive method, as shown in the following expression:

[0105] (4)

[0106] Where, x t This represents the input 1-dimensional feature, y t Represents the output feature, h t Representing the learnable intermediate states, the entire sequence is represented as a convolutional form, expressed as:

[0107] (5)

[0108] in, It is represented as a structured convolutional kernel, where M represents the length of the input sequence, x represents the input features, and y represents the output discretized features. It is trained by convolution and performs autoregressive inference using a linear recursive approach.

[0109] S213, Construct a global aggregation block based on Transformer, including the following steps:

[0110] A linear layer is applied to the input feature F to obtain the query feature Q. For the output feature F in S211... local It employs two parallel linear layers to transform into local value and key value features V. C and K C For the output feature F in S212 long Two parallel linear layers are used to transform long-range value and key-value features V. M and K M And cross-attention is used to achieve effective dynamic fusion, as shown in the following formula:

[0111] (6)

[0112] Where TGFB(·) represents a global aggregation block based on Transformer, Cat(·) represents the concatenate function, softmax(·) is the softmax function, and (·) T F represents the transpose of a feature. O This represents the output feature, where D represents the number of feature channels;

[0113] The downsampling module is used to achieve feature compression to obtain multi-scale features;

[0114] S22. Construct the neck module. The neck module includes multiple cross-scale interaction modules and one scale aggregation module, which includes the following steps:

[0115] S221. Construct the cross-scale interaction module, the formula is as follows:

[0116] ; (7)

[0117] where SEB(·) is the cross-scale interaction module, (a, b, c, r) ∈ [1, 4] represents the indices of different-scale features, r represents regarding the feature of the r-th scale as the main feature, and the remaining features are regarded as auxiliary features. z is the auxiliary feature of the z-th scale. If z > r, UDconv(·) uses a downsampling convolutional layer; if z < r, UDConv(·) uses an upsampling convolutional layer. Cat(·) is the concatenate function, and CTMB(·) represents the CNN-Transformer-Mamba module. The cross-scale interaction module performs cross-scale interaction step by step from low-resolution features to high-resolution features as the main feature, and the output feature will replace the original-resolution feature to participate in the next interaction;

[0118] S222. Construct the scale aggregation module to aggregate the features after cross-scale interaction, the formula is as follows:

[0119] ; (8)

[0120] where Uconv(·) represents an upsampling convolution, Cat(·) is the concatenate function, CTMB(·) is the CNN-Transformer-Mamba module, and O1, O2, O3, O4 are the output results of different scales of the cross-scale interaction module in S221 respectively;

[0121] S23. Construct the object detection head module. The object detection head module contains three parallel detection heads. Each detection head includes a 3×3 convolutional layer with 256 channels and a 1×1 convolutional layer. The three parallel detection heads are the heatmap head, the bounding box offset head, and the bounding box size head. The heatmap head is used to estimate the center position of each frame of the object, while the bounding box offset head and the bounding box size head are used to improve the positioning accuracy.

[0122] S3. Construct multi-object spatio-temporal trajectories based on the spatial position information in continuous space-time;

[0123] Specifically, S3 includes the following steps:

[0124] A CNN-Transformer-Mamba network built on S2 is used to identify multiple targets in a single frame image and obtain detection boxes, which is represented as... B u Let u represent the detection box, and let u represent the u-th target. This represents the x and y coordinates of the top-left corner of the detection box. Represents the horizontal and vertical coordinates of the bottom right corner of the detection box;

[0125] The Hungarian algorithm is used to perform spatiotemporal correlation of objects in consecutive frames. For each object's detection bounding box, based on the detection in each frame, k spatiotemporal trajectories of the object are obtained. ,in This represents the spatiotemporal trajectory of the e-th object. Let f represent the bounding box of the f-th object at frame t, where p and q represent the horizontal and vertical positions of the center point, and a and h represent the aspect ratio and height of the center point of the bounding box.

[0126] S4, construct a spatiotemporal Mamba motion model to complete the nonlinear modeling of multi-target spatiotemporal trajectories, achieving multi-target tracking, referencing... Figure 6 ;

[0127] Specifically, S4 includes the following steps:

[0128] For the input target spatiotemporal trajectory, it is expanded to a high-dimensional space through convolutional layers, and the output is continuous spatiotemporal parameters, as shown in the formula:

[0129] (9)

[0130] Wherein, Conv(·) is a convolutional layer, Linear(·) is a linear projection layer, and SSM(·) is the SSM module in step S212, which is used to perform powerful spatiotemporal modeling while maintaining the linear computational complexity of equations (3) to (5), and adds a multilayer perceptron layer MLP and a detection head to output the object prediction detection box to achieve target tracking.

[0131] S5. Construct a joint loss function to train the model, and evaluate the model performance through accuracy evaluation.

[0132] Specifically, S5 includes the following steps:

[0133] A global loss is used to optimize the detection and tracking tasks, as follows:

[0134] (10)

[0135] in, These are the learning weights used to balance detection and tracking tasks. It is the target detection loss. It is target tracking loss;

[0136] S51, construct the object detection loss to optimize the output of the heatmap header, bounding box offset header, and bounding box size header to improve the accuracy of object localization;

[0137] The heatmap header generates a heatmap of size H×W×1, assigning a value of 1 to the center of the ground truth target, and defining the ground truth bounding box as... The target center is defined as The center position on the feature map output can be calculated as The heatmap response value at location (m,n) is calculated as follows:

[0138] (11)

[0139] Where K represents the total number of targets in the current frame, σ is the standard deviation, and u represents the u-th target. The center point localization is optimized in the following way:

[0140] (12)

[0141] in, For the predicted heatmap, α and β are parameter variables. To compensate for the loss of positional accuracy caused by the step size of the output features, an L1 loss optimization objective is used, which is:

[0142] (13)

[0143] Where λ is a weight set to 0.15. and These represent the offset and size of the actual value, respectively. and These represent the dimensions of the true and predicted values, respectively, and an overall detection loss is applied by setting weight parameters. To adjust L heat and L box ,Right now:

[0144] (14)

[0145] S52, employs a multi-task loss function combining L1 loss and GIoU loss, weighted... To monitor and track tasks, the expression is as follows:

[0146] (15)

[0147] in, These are the learning weights used to balance the tracking task. It is L1 loss. It is a GIoU loss.

[0148] Reference Figure 7 This is to achieve the results for multi-target tracking tasks on the VisDrone dataset.

[0149] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for multi-target tracking of unmanned aerial vehicles based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model, characterized in that, Includes the following steps: S1. Obtain the video dataset of multi-target tracking collected by the UAV, and divide the video dataset into training set, test set and validation set. The training set is used for network fitting, the validation set is used for hyperparameter tuning, and the test set is used for accuracy evaluation. S2, construct a CNN-Transformer-Mamba network as an object detector, and obtain the bounding boxes and spatial location information of multiple ground objects in a single frame image; The CNN-Transformer-Mamba network in S2 includes a backbone network, a neck module, and an object detection head. Constructing the CNN-Transformer-Mamba network includes the following steps: S21. Construct the Backbone network, which includes a Stem module, multiple CNN-Transformer-Mamba modules, and a downsampling module; The Stem module is used to project the input image onto a high-dimensional feature space. The Stem module includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function. The CNN-Transformer-Mamba module is used to aggregate local, global, and long-range dependent features. The CNN-Transformer-Mamba module includes a multi-scale perceptual convolutional block, a multi-directional scanning Mamba block, and a Transformer-based global aggregation block, and includes the following steps: S211, Constructing a multi-scale perceptual convolutional block includes the following steps: Inputting the input feature map F into three parallel paths, each path including: a normalization layer, a global average pooling layer, a linear layer, an activation function layer, and a dynamic convolutional layer. Each path uses a dynamic convolutional layer with different windows, namely 3×3, 5×5, and 7×7. The calculation formula for dynamic convolution is as follows: ； (1) in, Indicates input features, Indicates output features, k represents the scale of the convolution kernel, w and h represent the positions of the convolution kernel at position (i,j) on the horizontal and vertical axes respectively, W(·) represents the dynamic weights, and then the dynamic features at different scales are fused through the Cat function, and a 1×1 convolutional layer is used to output multi-scale local features F. local ; S212, Constructing a multi-directional scanning Mamba block includes the following steps: Dividing the input feature F into branch one and branch two. Branch one includes a linear layer, a 1×1 convolutional layer, a SiLU activation function, and an SSM block; branch two includes a linear layer, a 1×1 convolutional layer, and a SiLU activation function. Branch one and branch two are aggregated using the Cat function, and a normalization layer is used to obtain the output long-range dependent feature F. long The calculation formula is as follows: ； (2) Where Conv(·) represents a convolutional layer, Linear(·) is a linear layer, SiLU(·) is the sigmoid activation function, Cat(·) is the concatenate function, and SSM(·) is the SSM module. It employs a multi-directional aggregation scanning method to obtain richer contextual features, uses linear layers to obtain continuous parameters, and converts them into discrete parameters based on a zero-order hold method, as shown in the following formula: ； (3) Where A, B, and C are the learnable parameters. , and Here are the transformed discretized parameters, where △ represents the time scale parameter, △A and △B represent the corresponding discrete time terms within a given time interval, and I represents the identity matrix. After transformation, the parameters are calculated using a linear recursive method, as shown in the following expression: ； (4) Where, x t This represents the input 1-dimensional feature, y t Represents the output feature, h t Representing the learnable intermediate states, the entire sequence is represented as a convolutional form, expressed as: ； (5) in, It is represented as a structured convolutional kernel, where M represents the length of the input sequence, x represents the input features, and y represents the output discretized features. It is trained by convolution and performs autoregressive inference using a linear recursive approach. S213, Construct a global aggregation block based on Transformer, including the following steps: A linear layer is applied to the input feature F to obtain the query feature Q. For the output feature F in S211... local It employs two parallel linear layers to transform into local value and key value features V. C and K C For the output feature F in S212 long Two parallel linear layers are used to transform long-range value and key-value features V. M and K M And cross-attention is used to achieve effective dynamic fusion, as shown in the following formula: ； (6) Where TGFB(·) represents a global aggregation block based on Transformer, Cat(·) represents the concatenate function, softmax(·) is the softmax function, and (·) T F represents the transpose of a feature. O This represents the output feature, where D represents the number of feature channels; The downsampling module is used to compress features to obtain multi-scale features; S3, constructing multi-target spatiotemporal trajectories based on spatial location information within continuous spatiotemporal space; S4. Construct a spatiotemporal Mamba motion model to complete the nonlinear modeling of the spatiotemporal trajectories of multiple targets and realize multi-target tracking; S5. Construct a joint loss function to train the model, and evaluate the model performance through accuracy evaluation.

2. The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model according to claim 1, characterized in that: The S1 includes the following steps: based on open-source video data, obtain a multi-target tracking video dataset collected by a drone, and divide the video dataset into a training set, a validation set, and a test set according to a preset ratio; Among them, the training set is used for fitting network parameters, the validation set is used for adjusting hyperparameters, and the test set is used for performing accuracy evaluation.

3. The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model according to claim 1, characterized in that: The S2 further includes the following steps: S22. Construct a neck module. The neck module includes multiple cross-scale interaction modules and a scale aggregation module, which includes the following steps: S221. Construct a cross-scale interaction module, and the formula is as follows: ； (7) Among them, SEB(·) is the cross-scale interaction module, (a, b, c, r) ∈ [1, 4] represents the indexes of features at different scales, r represents regarding the feature at the r-th scale as the main feature, and the remaining features are regarded as auxiliary features. z is the auxiliary feature at the z-th scale. If z > r, UDconv(·) uses a downsampling convolutional layer; if z < r, UDConv(·) uses an upsampling convolutional layer. Cat(·) is the concatenate function, and CTMB(·) represents the CNN-Transformer-Mamba module. The cross-scale interaction module performs cross-scale interaction step by step from low-resolution features to high-resolution features as the main feature, and the output feature will replace the original resolution feature to participate in the next interaction; S222. Construct a scale aggregation module to aggregate the features after cross-scale interaction, and the formula is as follows: ； (8) Among them, Uconv(·) represents an upsampling convolution, Cat(·) is the concatenate function, CTMB(·) is the CNN-Transformer-Mamba module, and O1, O2, O3, and O4 are the output results of different scales of the cross-scale interaction module in S221 respectively; S23. Construct a target detection head module. The target detection head module contains three parallel detection heads. Each detection head includes a 3×3 convolutional layer with 256 channels and a 1×1 convolutional layer. The three parallel detection heads are the heatmap head, the bounding box offset head, and the bounding box size head. The heatmap head is used to estimate the center position of each frame of the object, while the bounding box offset head and the bounding box size head are used to improve the positioning accuracy.

4. The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model according to claim 1, characterized in that: The S3 includes the following steps: A CNN-Transformer-Mamba network built on S2 is used to identify multiple targets in a single frame image and obtain detection boxes, represented as follows: B u This represents the detection box, where u represents the u-th target. This represents the x and y coordinates of the top-left corner of the detection box. Represents the horizontal and vertical coordinates of the bottom right corner of the detection box; The Hungarian algorithm is used to perform spatiotemporal correlation of objects in consecutive frames. For each object's detection bounding box, based on the detection in each frame, k spatiotemporal trajectories of the object are obtained. ,in This represents the spatiotemporal trajectory of the e-th object. Let f represent the bounding box of the f-th object at frame t, where p and q represent the horizontal and vertical positions of the center point, and a and h represent the aspect ratio and height of the center point of the bounding box.

5. The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model according to claim 3, characterized in that: The S4 includes the following steps: For the spatio-temporal trajectory of the input target, it is extended to a high-dimensional space through a convolutional layer and output as continuous spatio-temporal parameters, as shown in the formula: ； (9) Among them, Conv(·) is the convolutional layer, Linear(·) is the linear projection layer, SSM(·) is the SSM module in step S212, which is used to perform the excellent ability of powerful spatio-temporal modeling while maintaining the linear computational complexity of equations (3) to (5), and a multi-layer perceptron layer MLP and a detection head are added to output the detection box of the object prediction to achieve target tracking.

6. The UAV multi-target tracking method based on CNN-Transformer-Mamba network and spatiotemporal Mamba motion model according to claim 3, characterized in that: The S5 includes the following steps: A global loss is used to optimize the detection and tracking tasks, and the process is as follows: ； (10) in, These are the learning weights used to balance detection and tracking tasks. It is the target detection loss. It is target tracking loss; S51. Construct a target detection loss, which is used to optimize the outputs of the heatmap head, the bounding box offset head, and the bounding box size head to improve the accuracy of object positioning; The heatmap header generates a heatmap of size H×W×1, assigning a value of 1 to the center of the ground truth target, and defining the ground truth bounding box as... The target center is defined as The center position on the feature map output can be calculated as The heatmap response value at location (m,n) is calculated as follows: ； (11) Among them, K represents the total number of targets in the current frame, σ is the standard deviation, u represents the u-th target, and the following method is used to optimize the center point positioning: ； (12) in, For the predicted heatmap, α and β are parameter variables. To compensate for the loss of positional accuracy caused by the step size of the output features, an L1 loss optimization objective is used, which is: ； (13) Where λ is a weight set to 0.

15. and These represent the offset and size of the actual value, respectively. and These represent the dimensions of the true and predicted values, respectively, and an overall detection loss is applied by setting weight parameters. To adjust L heat and L box ,Right now: ； (14) S52, employs a multi-task loss function combining L1 loss and GIoU loss, weighted... To monitor and track tasks, the expression is as follows: ； (15) in, These are the learning weights used to balance the tracking task. It is L1 loss. It is a GIoU loss.

Images