Embedded aerial photography-based multi-target tracking algorithm and system

Abstract

The invention discloses an embedded aerial photography multi-target tracking system, and belongs to the technical field of image processing. The aerial photography multi-target tracking system comprises a remote end system and a ground end system. The remote end system is composed of an image acquisition and preprocessing module, a target detection module and a target tracking module. The image acquisition and preprocessing module acquires aerial image data by calling a camera and performs preprocessing operation on the data to enable the data to accord with the input format of the target detection module. And then, outputting coordinates of a detection target in the picture through a lightweight target detection module. And inputting the detected target coordinates into a target tracking module, tracking the target through the target tracking module, and distributing a unique ID number. The remote end pushes a video stream to the ground end through an RTSP protocol, the ground end analyzes and displays the video stream, and the ground end can also control parameters of the image acquisition and preprocessing module, the target detection module and the target tracking module of the remote end and transmit the parameters to the remote end through a TCP protocol. The method can be deployed in an unmanned aerial vehicle airborne system, and compared with a traditional target tracking algorithm, the method has lower performance consumption and higher detection performance in a target tracking task.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and digital image processing, and in particular relates to an embedded-based aerial photography multi-target tracking system. Background Art

[0002] In the current field of image processing technology, aerial multi-target tracking systems are increasingly used in airborne systems such as drones. In the fields of real-time monitoring, search and rescue, security monitoring, etc., it is necessary to efficiently and accurately detect and track multiple targets in dynamic scenes.

[0003] Aerial multi-target tracking includes two parts: multi-target detection and multi-target tracking. The classic deep learning target detection network is divided into two-stage network and single-stage network. Among them, the two-stage detection algorithm has high detection accuracy but slow running speed; the single-stage detection algorithm runs fast but has a high false alarm rate. Two-stage networks, such as RCNN, FastR-CNN, FasterR-CNN and CascadeR-CNN, first need to generate candidate areas, and then classify and locate the candidate areas, which are suitable for applications with higher detection accuracy requirements; single-stage networks, such as SSD, YOLO series and CenterNet, directly generate coordinate positions and class probabilities, which are faster than two-stage networks. Multi-target tracking methods based on deep learning can be divided into two categories: tracking based detection (TBD) and joint detection tracking (JDT). The JDT algorithm integrates the detection module and the tracking module to achieve a faster reasoning speed, but there is a problem of collaborative training of each module. The multi-stage design structure of the TBD strategy is clear, and detection and tracking can be optimized separately, which is more flexible and more suitable for complex scenes. The classic TBD tracker SORT proposed a simple and real-time data association method. DeepSORT adds appearance information to the framework of SORT to improve the performance of the algorithm, so as to achieve target tracking with long-term occlusion. StrongSORT has upgraded it from various aspects such as detection, embedding and association, and optimized DeepSORT with the latest components and training tricks; it proposed an appearance-free link model (AFLink), which only uses spatiotemporal information to associate short trajectories to complete trajectories, and then proposed Gaussian smoothing interpolation (GSI) to compensate for missing detections. BoT-SORT proposed camera motion compensation and more accurate Kalman filter state vectors to achieve better bounding box positioning, as well as a new fusion method based on cosine distance of IoU and re-id. The KF-based motion model is not robust to occlusion, nonlinear motion, and low frame rate video. OC-SORT attempts to optimize the motion model (i.e., KF filtering) and emphasizes the role of detection (Observation) in recovering lost trajectories and reducing the error accumulation of the KF model during the loss period, improving the robustness to situations such as occlusion and nonlinear motion, but there are still problems such as blurred background interference, occlusion and disappearance of objects.

[0004] In summary, existing target tracking technologies perform poorly when dealing with complex backgrounds, target occlusion or fast movement, which limits the effectiveness and reliability of aerial photography systems in practical applications. Existing aerial photography target tracking methods based on embedded systems have high requirements for algorithm complexity and processor performance. At the same time, these systems have challenges in terms of high power consumption, expensive hardware costs, and real-time and stability, especially in resource-constrained environments such as drones. Summary of the invention

[0005] In view of the shortcomings and deficiencies of the above systems and algorithms, the present invention proposes an embedded aerial photography multi-target tracking system. The steps are:

[0006] Step S1, construct and train the YOLOv8-FasterAttention target detection network, which is composed of a feature extraction backbone network, a multi-scale feature fusion network, and a detection head network.

[0007] Step S2, construct and train the CA-OSNet feature re-identification network. CA-OSNet consists of a CA attention module, an Omni-Scale module, and a hierarchical feature integration module.

[0008] Step S3: construct and train the Transformer-Track trajectory prediction network. The Transformer-Track trajectory prediction network consists of an input layer, a Transformer encoder, and an output layer.

[0009] Step S4, implement the multi-target tracking module. The multi-target tracking module matches the output results of the target detection network, feature re-identification network and trajectory predictor through a cascade matching algorithm, identifies the detected target and assigns a unique ID number. The target frame with the assigned ID is merged with the detected video and output as a video stream.

[0010] Step S5, realizing the remote end system, the remote end system runs the image acquisition and preprocessing module, the target detection module and the target tracking module, and sends the video stream output by the multi-target tracking module to the ground end system through the RTSP protocol.

[0011] Step S6, implement the ground system, the ground system runs the video display module and the parameter control module. Receive the tracking image of the remote system, and use the video display module to display the video. Use the TCP protocol to communicate with the ground, and the ground sends the command to control the remote using the parameter control module.

[0012] Further preferably, the YOLOv8-FasterAttention target detection network in step S1 specifically includes the following steps:

[0013] S11, introduce Pconv convolution, construct Faster-Conv module and Faster-C2f module to replace the original standard convolution block and original C2f module respectively. Pconv convolution can reduce redundant calculation and memory access at the same time, and extract spatial features more efficiently. The calculation formula of PConv convolution at each position is defined as follows:

[0014]

[0015] In the above formula represents the eigenvalue within the sliding window, and is the corresponding binary mask, where 0 in the mask represents the hole area and 1 in the mask represents the non-hole area. is the element-wise product, is a vector of all ones of the same size as the input, and The definition is as follows:

[0016]

[0017] After each PConv convolution operation, the mask is updated. If the sliding window contains a valid value, the mask at that position is a valid value, which is defined as follows:

[0018]

[0019] This operation dynamically makes masks at different layers contribute differently to the loss.

[0020] The Faster-Conv block consists of a PartialConvolution (PConv), a BN layer, and a RELU activation function. Compared with the original model, the proposed Faster-Conv module can improve the model accuracy and efficiency. The Faster-C2f module consists of a Conv layer and a Faster-Bottleneck network after the original standard convolution Conv is replaced by the PConv convolution. Compared with the original model, the proposed Faster-C2f module can reduce the computational complexity and improve the efficiency of the model.

[0021] S12, build Faster-BackBone network. Build Faster-BackBone feature extraction network to replace the original BackBone network. The Faster-BackBone network consists of Faster-Conv module and Faster-C2f module. Its network definition is as follows:

[0022] Take an image of size 640*640*3 as input, the first layer is the Faster-Conv layer for downsampling, the output feature map size is 320*320*64, the length and width become 1 / 4 of the initial image, and the number of output channels is 64; the second layer is still the Faster-Conv layer for downsampling, the output feature map size is 160*160*128, the length and width become 1 / 4 of the initial image, and the number of output channels becomes 128; the third layer is the Faster-C2f layer, which mainly passes through the Conv layer, the Faster-Bottleneck layer and the connected Conv layer, and its output feature map size is still 160*160*128, the length and width are 1 / 4 of the input image, and the number of output channels is 128; the fourth layer is the Faster-Conv layer for downsampling, the output feature map size is 80*80*256, the length and width become 1 / 8, the number of output channels becomes 256; the fifth layer is the Faster-C2f layer, the structure is the same as the third layer, and the output feature map size is still 80*80*256; the sixth layer is the Faster-Conv layer for downsampling, the output feature map size is 40*40*512, the length and width become 1 / 16 of the initial image, and the number of output channels becomes 512; the seventh layer is the Faster-C2f layer, the structure is the same as the third layer, the output feature map size is still 40*40*512; the eighth layer Faster-Conv layer is downsampled, the output feature map size is 20*20*1024, the length and width become 1 / 32 of the initial image, and the number of output channels becomes 1024; the ninth layer is the Faster-C2f layer, the structure is the same as the third layer, the output feature map size is still 20*20*1024; the length and width are still 1 / 32 of the initial image, and the number of output channels is still 1024. Compared with the original model, the proposed Faster-BackBone network can further improve the accuracy of the target detection network and accelerate the model reasoning speed while optimizing the complexity of the target detection network.

[0023] S13, introduce the cascade group attention mechanism CGAttention module, build the CGAttention-SPPF module to replace the original SPPF module, and place the module after the Faster-BackBone network. The CGAttention-SPPF module is composed of the cascade group attention mechanism CGAttention module and the SPPF module. The feature map is first extracted and fused at multiple scales by the SPPF module, and then the CGAttention module performs attention calculation to increase the network's attention to the key areas of the image. The core idea is to enhance the diversity of features input to the attention head. The definition of the CGAttention module is as follows:

[0024] First, the global context is encoded as follows:

[0025]

[0026] In the above formula, X is the input feature map, and G is the global context vector obtained by the global pooling operation.

[0027] Secondly, local feature guidance is performed, and the formula is as follows:

[0028]

[0029] In the above formula, Represents the feature vector at the i-th position of the input feature map X, represents the vector inner product, The local attention weight is calculated by the softmax function, which is used to measure the similarity between each position and the global context.

[0030] Finally, feature enhancement is performed, and the formula is as follows:

[0031]

[0032] In the above formula, the local features of each position are The weighted sum is performed to obtain the final enhanced feature map Y.

[0033] S14, construct a bidirectional multi-scale feature fusion network BiFPN, and construct an effective bidirectional cross-scale connection and weighted feature fusion path through the BiFPN network. First, the deep features are upsampled and fused with the shallow features. After the upsampling is completed, the shallow features are downsampled and fused with the deep features to achieve cross-scale fusion of deep features and shallow features. Compared with the original model PANet, the BiFPN network enhances the information transfer between feature maps of different network levels, and effectively improves the problem of image feature information loss caused by changes in feature map resolution.

[0034] S15, using the Head layer of YOLOv8, adds a 160*160 small target detection head, which consists of an upsampling layer, a connection layer, a C2f layer, and a Detect layer. The Head layer further encodes the features of the multi-scale feature fusion network and outputs the detection box to be processed. The specific steps are as follows:

[0035] First, the upsampling layer enlarges the feature map, then the connection layer merges the features from different layers, and then the C2f layer further processes the merged features, and finally the Detect layer outputs the detection box.

[0036] S16, using the Visdrone2019 dataset, select The loss function and Adam optimizer are used to train the YOLOv8-FasterAttention target detection network. The loss function is the weighted sum of positioning loss, classification loss and confidence loss, which is defined as follows:

[0037]

[0038] In the above formula, represents the positioning loss, represents the classification loss, Represents the confidence loss.

[0039] S17, use the NMS algorithm to process the redundant detection frame and output the detection target frame.

[0040] S18, use rknn-toolkit to convert the Pytorch model to the RKNN model

[0041] Further preferably, the CA-OSNet feature re-identification network in step S2 specifically includes the following steps:

[0042] S21, build the CA attention module, the CA attention mechanism is defined as follows:

[0043] First, the feature map is globally averaged horizontally and vertically pooled. This process is defined as follows:

[0044]

[0045]

[0046] In the above formula, It represents a feature map with C channels, H length and W width. Represents horizontal and vertical characteristics.

[0047] The horizontal and vertical features are spliced ​​together to fuse the horizontal and vertical information. This process is defined as follows:

[0048]

[0049] In the above formula, Represents the generated feature map, represents a 1D convolutional layer, and σ represents the Sigmoid activation function.

[0050] The shared convolutional layer is used to extract features from the horizontal and vertical pooling information. This process is defined as follows:

[0051]

[0052]

[0053] In the above formula, and represents a shared 1D convolutional layer, and σ represents the Sigmoid activation function. , Represents the generated horizontal and vertical attention weights. , They respectively represent the results after the Split operation on feature F.

[0054] Finally, the result of the attention weight calculation is applied to the input feature map. This process is defined as follows:

[0055]

[0056] In the above formula Feature map representing the output of the CA attention module.

[0057] S22, add a CA attention module before each BottleNeck layer in OS-Net. The modified CA-OSNet feature re-identification network is defined as follows:

[0058] The first layer consists of a 7×7 convolutional layer with a stride of 2 and an output channel number of 64; the second layer consists of a 3×3 maximum pooling layer with a stride of 2 and an output channel number of 64; the third layer is a CA attention module with an output channel number of 64; the fourth layer is composed of two OS-Net BottleNecks with an output channel number of 256; the fifth layer consists of a 1×1 convolutional layer and a 2×2 average pooling layer with a stride of 2 and an output channel number of 256; the sixth layer is a CA attention module with an output channel number of 256; the seventh layer is composed of two OS-Net BottleNecks The eighth layer consists of a 1×1 convolutional layer and a 2×2 average pooling layer with a stride of 2, with an output channel number of 384; the ninth layer is a CA attention module, with an output channel number of 384; the tenth layer consists of two OS-Net BottleNecks, with an output channel number of 512; the eleventh layer is a 1×1 convolutional layer with an output channel number of 512; the twelfth layer is a global adaptive average pooling layer with an output channel number of 512; the thirteenth layer is a fully connected layer with an output channel number of 512.

[0059] S23, use the ImageNet dataset to train the CA-OSNet feature re-identification network. The number of network training rounds is set to 50 rounds, and the optimizer is Adma.

[0060] S24, use rknn-toolkit to convert the Pytorch model to the RKNN model

[0061] Further preferably, the Transformer-Track trajectory prediction network in step S3 specifically includes the following steps:

[0062] S31, obtain the Visdrone-2019-MOT dataset, extract the trajectory data, and perform preprocessing. The format is:

[0063]

[0064] in, Represents the center coordinates of the object on the image, Indicates width and height. Represents a historical trajectory of length T.

[0065] S32, encode the bounding box to obtain high-dimensional information. The process is defined as follows:

[0066]

[0067] in, Represents a sinusoidal position encoding function that expands 4 dimensions to 512 dimensions. Represents a vector after dimension expansion.

[0068] S33, adding a prediction tag to the sequence vector to indicate the predicted bounding box in the current frame t. This stage is defined as follows:

[0069]

[0070] S34, adding time position coding, the process is defined as follows:

[0071]

[0072] in, Represents the trajectory sequence after adding the time position encoding. Indicates that the dimension of the time series is expanded to 512; Represents a natural number sequence from 1 to T+1.

[0073] S35, build the Transformerr-Track trajectory prediction network, which consists of a 4-layer Transformer encoder, which uses 6-head multi-head self-attention and two fully connected layers and ReLu activation function. The network reasoning process is as follows:

[0074]

[0075] In the above formula, Represents the encoding operation of the Transformer encoder. Represents the output of the encoder.

[0076] extract In , use the fully connected layer to reason about it and map it to the coordinates of the detection box. The process is as follows:

[0077]

[0078] In the above formula , represents the Sigmoid function, , Represent two fully connected layers, Represents the Relu activation function.

[0079] S36, define the loss function for training the Transformerr-Track trajectory prediction network. The loss function is defined as follows:

[0080]

[0081] S37, train the Transformerr-Track trajectory prediction network, set the number of training rounds to 75, use the Adam optimizer, set the batch size to 256, and the trajectory length T to 20.

[0082] S38, use rknn-toolkit to convert the Pytorch model to the RKNN model

[0083] Further preferably, the multi-target tracking module in step S4 specifically includes the following steps:

[0084] S41, obtain the output of the target detection module, perform feature extraction on each detected target area, and use CA-OSNet to extract the target's appearance feature vector.

[0085] S42, for each tracker, use the target position in the next frame of the trajectory predictor. The trajectory predictor consists of a Kalman filter and a Transformer-Track trajectory prediction network. For a target trajectory in a determined state, the number of trajectory frames saved in the tracker is greater than 20 frames, and the target detection result of the target disappears, that is, when the target is in an occluded state, the prediction of the subsequent trajectory of the target is made by the Transformer-Track trajectory prediction network until the target detection network recognizes the target again.

[0086] S43, calculate the distance matrix between the detection result in the current frame and the prediction result of the Kalman filter. The distance metric usually takes into account the IoU (Intersection over Union) of the bounding box and the Euclidean distance of the appearance feature. The Hungarian algorithm is used to solve the matching problem and find the best match between the detection result and the existing tracker.

[0087] S44, for each matched tracker, use the corresponding detection result to update the state of the Kalman filter. If a detection result does not match an existing tracker, create a new tracker. If a tracker does not match a detection result within 100 frames, use the Transformer-Track trajectory prediction network prediction result to update the state of the Kalman filter, and delete it after more than 100 frames.

[0088] S45, updating the appearance model of each tracker to adapt to the change of the target appearance.

[0089] S46, output the trajectory of each tracked target, including the target's ID, bounding box, and confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0091] Figure 1 It is a structural diagram of an embedded-based aerial photography multi-target tracking system in an example of the present invention;

[0092] Figure 2 It is a YOLOv8-FasterAttention target detection network structure diagram in the example of the present invention;

[0093] Figure 3 It is a CA-OSNet network structure diagram in an example of the present invention;

[0094] Figure 4 is a Transformer-Track trajectory prediction network structure diagram in an example of the present invention;

[0095] Figure 5 It is a flow chart of the target tracking algorithm operation in the example of the present invention. DETAILED DESCRIPTION

[0096] In order to make the objectives, technical solutions and advantages of the present invention clearer, the specific implementation process of the present invention will be described in more detail below in combination with specific implementation methods and drawings, so as to facilitate technical personnel to more accurately understand the present invention and apply it in various specific fields.

[0097] See also Figure 1 The present invention discloses an embedded aerial photography multi-target tracking system, including a remote end system and a ground end system. The steps are:

[0098] Step S1, build and train the YOLOv8-FasterAttention target detection network, such as Figure 2As shown in the figure, the YOLOv8-FasterAttention target detection network is based on a feature extraction backbone network, a multi-scale feature fusion network, and a detection head network.

[0099] Pconv convolution is introduced to construct Faster-Conv module and Faster-C2f module to replace the original standard convolution block and original C2f module respectively. The calculation formula of PConv convolution at each position is defined as follows:

[0100]

[0101] In the above formula represents the eigenvalue within the sliding window, is the corresponding binary mask, where 0 in the mask represents the hole area and 1 in the mask represents the non-hole area. is the element-wise product, is a vector of all ones of the same size as the input, The definition is as follows:

[0102]

[0103] After each PConv convolution operation, the mask is updated. If the sliding window contains a valid value, the mask at that position is a valid value. This process is defined as follows:

[0104]

[0105] This operation dynamically makes masks at different layers contribute differently to the loss.

[0106] The Faster-Conv block consists of a PartialConvolution (PConv), a BN layer, and a RELU activation function. Compared with the original model, the proposed Faster-Conv module can improve the model accuracy and efficiency. The Faster-C2f module consists of a Conv layer and a Faster-Bottleneck network after introducing PConv convolution to replace the original standard convolution Conv.

[0107] Construct a Faster-BackBone feature extraction network to replace the original BackBone network. The Faster-BackBone network consists of a Faster-Conv module and a Faster-C2f module. The network definition is as follows:

[0108] Take an image of size 640*640*3 as input, the first layer is the Faster-Conv layer for downsampling, the output feature map size is 320*320*64, the length and width become 1 / 4 of the initial image, and the number of output channels is 64; the second layer is still the Faster-Conv layer for downsampling, the output feature map size is 160*160*128, the length and width become 1 / 4 of the initial image, and the number of output channels becomes 128; the third layer is the Faster-C2f layer, which mainly passes through the Conv layer, the Faster-Bottleneck layer and the connected Conv layer, and its output feature map size is still 160*160*128, the length and width are 1 / 4 of the input image, and the number of output channels is 128; the fourth layer is the Faster-Conv layer for downsampling, the output feature map size is 80*80*256, the length and width become 1 / 8, the number of output channels becomes 256; the fifth layer is the Faster-C2f layer, the structure is the same as the third layer, and the output feature map size is still 80*80*256; the sixth layer is the Faster-Conv layer for downsampling, the output feature map size is 40*40*512, the length and width become 1 / 16 of the initial image, and the number of output channels becomes 512; the seventh layer is the Faster-C2f layer, the structure is the same as the third layer, the output feature map size is still 40*40*512; the eighth layer Faster-Conv layer is downsampled, the output feature map size is 20*20*1024, the length and width become 1 / 32 of the initial image, and the number of output channels becomes 1024; the ninth layer is the Faster-C2f layer, the structure is the same as the third layer, the output feature map size is still 20*20*1024; the length and width are still 1 / 32 of the initial image, and the number of output channels is still 1024.

[0109] The cascaded group attention mechanism CGAttention module is introduced, and the CGAttention-SPPF module is constructed to replace the original SPPF module, and the module is placed after the Faster-BackBone network. The CGAttention-SPPF module is composed of the cascaded group attention mechanism CGAttention module and the SPPF module. The feature map is first extracted and fused by the SPPF module at multiple scales, and then the CGAttention module performs attention calculation to increase the network's attention to the key areas of the image. The definition of the CGAttention module is as follows:

[0110] First, the global context is encoded, which is defined as follows:

[0111]

[0112] In the above formula, X is the input feature map, and G is the global context vector obtained by the global pooling operation.

[0113] Secondly, local feature guidance is performed, and this process is defined as follows:

[0114]

[0115] In the above formula, Represents the feature vector at the i-th position of the input feature map X, represents the vector inner product, The local attention weight is calculated by the softmax function, which is used to measure the similarity between each position and the global context.

[0116] Finally, feature enhancement is performed, and this process is defined as follows:

[0117]

[0118] In the above formula, the local features of each position are The weighted sum is performed to obtain the final enhanced feature map Y.

[0119] A bidirectional multi-scale feature fusion network BiFPN is constructed. An effective bidirectional cross-scale connection and weighted feature fusion path are constructed through the BiFPN network. First, the deep features are upsampled and fused with the shallow features. After the upsampling is completed, the shallow features are downsampled and fused with the deep features to achieve cross-scale fusion of deep features and shallow features.

[0120] A 160*160 small target detection head is added by using the Head layer of YOLOv8, which consists of an upsampling layer, a connection layer, a C2f layer, and a Detect layer. The Head layer further encodes the features of the multi-scale feature fusion network and outputs the detection box to be processed. The specific steps are as follows:

[0121] First, the upsampling layer enlarges the feature map, then the connection layer merges the features from different layers, and then the C2f layer further processes the merged features, and finally the Detect layer outputs the detection box.

[0122] Using the Visdrone2019 dataset, select The loss function and Adam optimizer are used to train the YOLOv8-FasterAttention target detection network. The loss function is the weighted sum of positioning loss, classification loss and confidence loss, which is defined as follows:

[0123]

[0124] In the above formula, represents the positioning loss, represents the classification loss, Represents the confidence loss.

[0125] The NMS algorithm is used to process redundant detection frames and output the detection target frame.

[0126] Use rknn-toolkit to convert the Pytorch model to the RKNN model.

[0127] Step S2, such as Figure 3 As shown in the figure, a CA-OSNet feature re-identification network is constructed and trained. CA-OSNet consists of a CA attention module, an Omni-Scale module, and a hierarchical feature integration module.

[0128] Construct the CA attention module. The CA attention mechanism is defined as follows:

[0129] First, the feature map is globally averaged horizontally and vertically pooled. This process is defined as follows:

[0130]

[0131]

[0132] In the above formula, It represents a feature map with C channels, H length and W width. Represents horizontal and vertical characteristics.

[0133] The horizontal and vertical features are spliced ​​together to fuse the horizontal and vertical information. This process is defined as follows:

[0134]

[0135] In the above formula, Represents the generated feature map, represents a 1D convolutional layer, and σ represents the Sigmoid activation function.

[0136] The shared convolutional layer is used to extract features from the horizontal and vertical pooling information. This process is defined as follows:

[0137]

[0138]

[0139] In the above formula, and represents a shared 1D convolutional layer, and σ represents the Sigmoid activation function. , Represents the generated horizontal and vertical attention weights. , They respectively represent the results after the Split operation on feature F.

[0140] Finally, the result of the attention weight calculation is applied to the input feature map. This process is defined as follows:

[0141]

[0142] In the above formula, Feature map representing the output of the CA attention module.

[0143] A CA attention module is added before each BottleNeck layer in OS-Net. The modified CA-OSNet feature re-identification network is defined as follows:

[0144] The first layer consists of a 7×7 convolutional layer with a stride of 2 and an output channel number of 64; the second layer consists of a 3×3 maximum pooling layer with a stride of 2 and an output channel number of 64; the third layer is a CA attention module with an output channel number of 64; the fourth layer is composed of two OS-Net BottleNecks with an output channel number of 256; the fifth layer consists of a 1×1 convolutional layer and a 2×2 average pooling layer with a stride of 2 and an output channel number of 256; the sixth layer is a CA attention module with an output channel number of 256; the seventh layer is composed of two OS-Net BottleNecks The eighth layer consists of a 1×1 convolutional layer and a 2×2 average pooling layer with a stride of 2, with an output channel number of 384; the ninth layer is a CA attention module, with an output channel number of 384; the tenth layer consists of two OS-Net BottleNecks, with an output channel number of 512; the eleventh layer is a 1×1 convolutional layer with an output channel number of 512; the twelfth layer is a global adaptive average pooling layer with an output channel number of 512; the thirteenth layer is a fully connected layer with an output channel number of 512.

[0145] The CA-OSNet feature re-identification network is trained using the ImageNet dataset. The number of network training rounds is set to 50 rounds, and the optimizer is Adma.

[0146] Convert the Pytorch model to the RKNN model using rknn-toolkit

[0147] Step S3: construct and train the Transformer-Track trajectory prediction network. The Transformer-Track trajectory prediction network consists of an input layer, a Transformer encoder, and an output layer.

[0148] Get the Visdrone-2019-MOT dataset, extract the trajectory data, and preprocess it. The format is:

[0149]

[0150]

[0151] in, Represents the center coordinates of the object on the image, Indicates width and height. Represents a historical trajectory of length T.

[0152] Encode the bounding box to obtain high-dimensional information. The process is defined as follows:

[0153]

[0154] in, Represents a sinusoidal position encoding function that expands 4 dimensions to 512 dimensions. Represents a vector after dimension expansion.

[0155] A prediction marker is added to the sequence vector to indicate the predicted bounding box in the current frame t. This stage is defined as follows:

[0156]

[0157] Adding time position coding, the process is defined as follows:

[0158]

[0159] in, Represents the trajectory sequence after adding the time position encoding. Indicates that the dimension of the time series is expanded to 512; Represents a natural number sequence from 1 to T+1.

[0160] Construct the Transformerr-Track trajectory prediction network, such as Figure 4 As shown in the figure, the network consists of a 4-layer Transformer encoder, which uses 6-head multi-head self-attention and two fully connected layers and ReLu activation function. The network reasoning process is as follows:

[0161]

[0162] In the above formula, Represents the encoding operation of the Transformer encoder. Represents the output of the encoder.

[0163] extract In , use the fully connected layer to reason about it and map it to the coordinates of the detection box. The process is as follows:

[0164]

[0165] In the above formula , represents the Sigmoid function, , Represent two fully connected layers, Represents the Relu activation function.

[0166] Define the loss function for training the Transformerr-Track trajectory prediction network. The loss function is defined as follows:

[0167]

[0168] Train the Transformerr-Track trajectory prediction network, set the number of training rounds to 75, use the Adam optimizer, set the batch size to 256, and set the trajectory length T to 20.

[0169] Convert the Pytorch model to the RKNN model using rknn-toolkit

[0170] Step S4, such as Figure 5 As shown in the figure, a multi-target tracking module is implemented. The multi-target tracking module matches the output results of the target detection network, feature re-identification network and trajectory predictor through a cascade matching algorithm, identifies the detected target and assigns a unique ID number. The target frame with the assigned ID is merged with the detection video and output as a video stream.

[0171] Get the output of the target detection module, perform feature extraction on each detected target area, and use CA-OSNet to extract the target's appearance feature vector.

[0172] The target position in the next frame of the trajectory predictor is used for each tracker. The trajectory predictor consists of a Kalman filter and a Transformer-Track trajectory prediction network. For a target trajectory in a determined state, the number of trajectory frames saved in the tracker is greater than 20 frames, and the target detection result of the target disappears, that is, when the target is in an occluded state, the prediction of the subsequent trajectory of the target is made by the Transformer-Track trajectory prediction network until the target detection network recognizes the target again.

[0173] Compute the distance matrix between the detection results in the current frame and the Kalman filter prediction results. The distance metric usually takes into account the IoU (Intersection over Union) of the bounding box and the Euclidean distance of the appearance features. Use the Hungarian algorithm to solve the matching problem and find the best match between the detection results and the existing tracker.

[0174] For each matched tracker, the corresponding detection result is used to update the state of the Kalman filter. If a detection result does not match an existing tracker, a new tracker is created. If a tracker does not match a detection result within 100 frames, the state of the Kalman filter is updated using the prediction result of the Transformer-Track trajectory prediction network, and it is deleted after more than 100 frames.

[0175] The appearance model of each tracker is updated to adapt to changes in the target appearance.

[0176] Outputs the trajectory of each tracked object, including the object's ID, bounding box, and confidence.

[0177] Step S5, realizing the remote end system, the remote end system runs the image acquisition and preprocessing module, the target detection module and the target tracking module, and sends the video stream output by the multi-target tracking module to the ground end system through the RTSP protocol.

[0178] Step S6, implement the ground system, the ground system runs the video display module and the parameter control module. Receive the tracking image of the remote system, and use the video display module to display the video. Use the TCP protocol to communicate with the ground, and the ground sends the command to control the remote using the parameter control module.

Claims

1. An embedded aerial photography multi-target tracking system, characterized in that: include: The image acquisition and preprocessing module is used to obtain and preprocess the video stream data of the onboard camera and adjust the image size to meet the 640*640 input requirements of the target detection module.

2. The target detection module is used to extract features and identify targets from the input image and output the target coordinates.

3. The target tracking module performs cascade matching operations based on the target coordinates output by the target detection module, assigns a unique ID to the target, and continues to track it.

4. Video display module, obtains the RTSP video stream data output by the remote end, parses and processes it, and displays the video stream.

5. Parameter control module, which controls the recognition and tracking related hyper parameters of the remote end.

6. The target detection module according to claim 1, characterized in that: The YOLOv8-FasterAttention network uses Faster-Backbone as the feature extraction network, and then the CGAttention-SPPF module further extracts and fuses the features. The neck network uses the BiFPN structure to perform multi-scale fusion of the features of the feature extraction network. Finally, the multi-scale fused features are input into the detection heads of 20*20, 40*40, 80*80, and 160*160. After that, the NMS algorithm is used to process the redundant detection boxes output by the network. The network is trained using the Visdrone2019 dataset, and the Pytorch model is converted to the RKNN model using rknn-toolkit.

7. The Faster-Backbone feature extraction network according to claim 2, characterized in that: It consists of Faster-Conv block and Faster-C2f module. Faster-Conv block consists of Partial Convolution (PConv), BN layer, RELU activation function. Faster-C2f module consists of Conv and Faster-Bottleneck. Faster-Bottleneck consists of Faster-Conv block.

8. The CGAttention-SPPF module according to claim 2, characterized in that It consists of a cascaded group attention mechanism CGAttention module and an SPPF module. The feature map is first extracted and fused at multiple scales by the SPPF module, and then the CGAttention module performs attention calculation to increase the network's attention to key areas of the image.

9. The target tracking module according to claim 1, characterized in that: It consists of a lightweight feature re-identification network, a trajectory prediction algorithm, a cascade matching algorithm, and a camera motion compensation algorithm. The lightweight feature re-identification network is responsible for encoding the target output by the feature detection module. The trajectory prediction algorithm is used to predict the trajectory of the target at the next moment based on the trajectory of the target in the past. The cascade matching algorithm is responsible for determining the state of the tracked target based on the output results of the lightweight feature re-identification network and the trajectory prediction algorithm. The camera motion compensation algorithm uses the CMC algorithm to reduce the target position drift caused by camera motion.

10. The lightweight feature re-identification network according to claim 4, characterized in that: The OSNet network is improved by adding a CA attention mechanism before the BottleNeck of each layer of the network to enhance the encoding effect of the feature re-identification network on the target. The improved feature re-identification network is trained using the ImageNet dataset.

11. The trajectory prediction algorithm according to claim 4, characterized in that: The Transformer-Track algorithm is combined with the Kalman filter method for trajectory prediction. When a target in a certain state is blocked, the Transformer is used to continuously predict the blocked target, and the Kalman filter is updated using the Transformer's prediction value until the target appears again, and the Kalman filter method is continued for trajectory prediction.

12. The Transformer-Track algorithm of claim 6, wherein: It consists of an input layer, a Transformer encoder, and an output layer. The input layer converts the input detection coordinates into feature representations of the model dimension. The Transformer encoder consists of multiple self-attention layers, which can capture the temporal relationship in the input sequence. The output layer converts the output of the Transformer back to the original coordinate dimension. The mean square error loss (MSE) is used to evaluate the difference between the prediction result and the target sequence. The Adam optimizer is used to update the model parameters. Finally, the Pytorch model is converted to an RKNN model using rknn-toolkit.

13. The parameter control module according to claim 1, characterized in that: The ground-side program controls the remote end through the TCP protocol. The controllable parameters include: IOU threshold of target detection, confidence threshold, target detection model, and feature re-identification network model.

Images