A method for multi-target tracking of infrared UAVs

Abstract

This invention discloses a multi-target tracking method for infrared unmanned aerial vehicles (UAVs), relating to the field of computer vision. The method includes: reading infrared camera video and using YOLOv8 to detect targets and acquire information; introducing camera motion compensation to correct the position, and using a Kalman filter algorithm to predict the trajectory and determine the state; marking low-matching targets using a pre-matching strategy; extracting the appearance features of the targets using RepO-LightMBN, calculating the cost matrix by combining appearance feature distance and motion information, and associating the detection boxes and trajectories through cascaded matching and B-IoU+ matching algorithms; for targets and trajectories that mismatch in cascaded matching, matching the last observation and detection values ​​of the trajectory using B-IoU+ and OCR matching algorithms; finally, updating the state of the matched trajectories using Kalman filtering, deleting unmatched trajectories, and outputting the results. This invention effectively improves the speed and accuracy of multi-target tracking of UAVs in infrared mode, possessing innovation and practicality, and can be used in fields such as anti-UAV monitoring.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method for multi-target tracking of infrared unmanned aerial vehicles. Background Technology

[0002] In the field of modern security and military defense, the widespread application of drones has brought numerous potential risks and challenges. With the rapid development of drone technology, its usage frequency in both civilian and military scenarios has increased dramatically. Infrared-based anti-drone multi-target tracking technology has become an important research direction that urgently needs development and improvement.

[0003] In the field of target tracking technology, traditional visible light-based methods have limitations in certain scenarios. For example, in low light, inclement weather, and at night, the performance of visible light imaging devices is significantly reduced, making it difficult to effectively detect and track targets. In contrast, infrared technology, based on the principle of thermal radiation imaging of objects themselves, has unique advantages. It can detect targets in complete darkness or complex weather conditions, and is not directly affected by light intensity and visibility, providing a reliable technical approach for multi-target detection and tracking against drones.

[0004] Multi-object tracking comprises two parts: multi-object detection and multi-object tracking. Classic deep learning object detection networks are divided into two-stage networks and single-stage networks. Two-stage detection algorithms offer high detection accuracy but are slow; single-stage detection algorithms are fast but have a high false positive rate. Two-stage networks, such as R-CNN, FastR-CNN, FasterR-CNN, and CascadeR-CNN, first generate candidate regions, then classify and locate them, making them suitable for applications requiring higher detection accuracy. Single-stage networks, such as SSD, YOLO series, and CenterNet, directly generate coordinate positions and class probabilities, making them faster than two-stage networks. Deep learning-based multi-object tracking methods are mainly divided into two categories: Tracking Based Detection (TBD) and Joint Detection Tracking (JDT). JDT algorithms attempt to fuse detection and tracking modules to improve inference speed, but in practical applications, they face difficulties in co-training the modules, leading to unstable overall performance. The TBD strategy, with its clear multi-stage design structure, allows for separate optimization of detection and tracking, demonstrating good adaptability to complex scenes. For example, the classic TBD tracker SORT proposed a simple and real-time data concatenation method. DeepSORT, building upon the SORT framework, adds appearance information to improve algorithm performance, enabling tracking of targets with prolonged occlusion. StrongSORT further upgrades it in terms of detection, embedding, and concatenation, employing the latest components and training techniques to optimize DeepSORT. It proposes an appearance-free linking model (AFLink) that uses only spatiotemporal information to concatenate short trajectories into complete trajectories, and Gaussian smooth interpolation (GSI) to compensate for missing detections. BoT-SORT achieves better bounding box localization through camera motion compensation and more accurate Kalman filter state vectors, as well as a novel fusion method based on IoU and re-id cosine distance. However, KF-based motion models lack robustness in handling occlusion, nonlinear motion, and low frame rate video. Although OC-SORT attempts to optimize the motion model, emphasizing the role of observation in recovering lost trajectories and reducing the accumulation of errors in the KF model during the loss period, and improving robustness to occlusion and nonlinear motion, it still suffers from inaccurate tracking and difficulty in re-locking the target after it is lost when faced with blurred background interference, object occlusion and disappearance problems.

[0005] In summary, existing multi-target tracking technologies still face numerous challenges when applied to infrared-based counter-drone scenarios, particularly in terms of target detection accuracy in complex environments, tracking stability, and adaptability to resource-constrained conditions. Therefore, developing a novel infrared multi-target tracking method for drones is urgently needed to overcome the shortcomings of current technologies, effectively address drone threats, and ensure the safety and stability of related fields. Summary of the Invention

[0006] To address the shortcomings and deficiencies of the aforementioned methods, this invention proposes an infrared UAV multi-target tracking method. The steps are as follows:

[0007] Step S1: Input infrared camera video stream data, use YOLOv8 target detector to detect the target in the current frame of the infrared video, and obtain target information; the YOLOv8 target detection network is composed of a feature extraction backbone network, a multi-scale feature fusion network, and a detection head.

[0008] Step S2: Correct the target's position information based on the camera's motion state using a camera motion compensation method.

[0009] Step S3: Use the Kalman filter algorithm to predict the trajectory of all detected targets and determine whether the trajectory is in a confirmed state or an unconfirmed state.

[0010] Step S4 proposes a pre-matching strategy: before extracting appearance features, a pre-match is performed, and safe targets and dangerous targets are marked according to the matching degree.

[0011] Step S5: A lightweight feature re-identification network, RepO-LightMBN, is proposed. The RepO-LightMBN network is constructed and trained, and appearance features of low-matching targets in video frames are extracted through RepO-LightMBN.

[0012] Step S6 proposes the B-IoU+ matching algorithm, which combines the B-IoU+ and OCM matching algorithms to perform cascade matching of the detection box and trajectory based on the appearance features of the target.

[0013] Step S7: For targets and trajectories that mismatched in the cascade matching stage, an OCR matching algorithm is used to match the last observation and detection values ​​of the trajectory.

[0014] Step S8: Use the Kalman filter algorithm to update the state of the matching trajectories obtained by cascaded matching and OCR matching, delete the unmatched trajectories in the mismatched trajectories obtained by OCR matching, and output the target trajectory prediction box and target ID after state update.

[0015] A further preferred embodiment of the YOLOv8 object detection network in step S1 includes the following steps:

[0016] S11, Constructing the YOLOv8 object detection network includes constructing a feature extraction backbone network, constructing a multi-scale feature fusion network, and constructing a detection head.

[0017] S12, using the collected dataset, select L total The loss function and Adam optimizer are used to train the YOLOv8 object detection network. The above L... total The loss function is a weighted sum of the localization loss, classification loss, and confidence loss, and its definition is as follows:

[0018] L total =L loc +L cls +L conf

[0019] In the above formula, L loc L represents the localization loss. cls L represents the classification loss. conf This represents the confidence loss.

[0020] S13 uses the NMS algorithm to process redundant detection boxes and outputs the detection target boxes.

[0021] A further preferred embodiment of the camera motion compensation method in step S2 includes the following steps:

[0022] S21, for the infrared video, before processing each frame, determine the motion state of the camera;

[0023] S22, If the camera is detected to be in motion, a camera motion compensation method is introduced. Based on the CMC (Camera Motion Compensation, enhanced correlation coefficient) method, image registration or alignment is performed on the offset or distortion caused by camera motion to correct the target position information.

[0024] S23, Calculate the affine transformation matrix of the camera movement using the RANSAC algorithm. This process is defined as follows:

[0025]

[0026]

[0027] In the above formula, Denotes the affine transformation matrix. This represents the scaling and rotation matrix. This represents the translation matrix.

[0028] S24, using an affine transformation matrix, transforms the predicted bounding box from the coordinate system of frame k-1 to the coordinate system of the next frame k. This process is defined as follows:

[0029]

[0030] In the above formula, , These represent the state vectors before and after compensation, respectively. , Let represent the covariance matrices of the Kalman filter before and after compensation, respectively.

[0031] More preferably, in step S3, the Kalman filter algorithm is used to predict the trajectory of all detected targets and determine whether the trajectory is in a confirmed or unconfirmed state. This specifically includes the following steps:

[0032] S31. For each detected target, obtain the state vector and covariance matrix of the compensated Kalman filter.

[0033] S32, based on the target's motion model, predict the target's state at the current moment according to the state estimate of the previous moment.

[0034] S33, calculate the correlation metric between the predicted state and the currently detected target location information. By setting a threshold, compare the correlation metric with the threshold to determine whether the trajectory is in a confirmed or unconfirmed state. If the correlation metric is less than the threshold, it is in a confirmed state; otherwise, it is in an unconfirmed state.

[0035] A further preferred approach is the pre-matching strategy proposed in step S4, which consists of a pre-matching algorithm and a feature update process. Pre-matching uses motion features to segment targets; for targets with obvious motion features, the calls to the feature extraction module are reduced, and vice versa. Specifically, it includes the following steps:

[0036] S41 uses the IoU of the predicted and observed values ​​to represent the strength of the target's motion characteristics.

[0037] S42, before performing feature extraction, perform an IoU match between the trajectory and the detection box.

[0038] S43, by setting a threshold, the only detection box with a high matching degree is set as a safe target, and all other targets are set as dangerous targets.

[0039] Further preferred, the lightweight feature re-identification network RepO-LightMBN proposed in step S5 is used to construct the RepO-LightMBN feature re-identification network to extract the appearance features of targets with low matching degree, and the feature re-identification network is trained using the collected data, specifically including the following steps:

[0040] S51 constructs the Rep-OSNET backbone network, introduces the reparameterized structural unit SAG into the OSNET network, and effectively improves the network's ability to extract and utilize features in infrared anti-drone multi-target tracking scenarios through a unique multi-branch convolutional structure.

[0041] S52, SAG has a multi-branch convolutional structure during the training phase, including an input layer, an average pooling layer, a multilayer perceptron (MLP), a sigmoid activation function, and an output layer.

[0042] The input layer receives feature map information from the corresponding stage in the Rep-OSNET network;

[0043] Average pooling layers reduce feature dimensionality and enhance feature robustness by averaging features within local regions. This ensures that the extracted features remain relatively stable and representative even when faced with potential local interference from UAV targets in infrared images, laying the groundwork for further feature transformations.

[0044] MLP, with its multi-layered structure composed of multiple neurons, performs deep nonlinear transformations on input features, thereby uncovering more complex and abstract feature information;

[0045] The Sigmoid activation function maps the output value to the 0-1 range, giving the features appropriate non-linear characteristics. This allows the features to have a more reasonable numerical range and a distribution pattern that better matches the actual scene, enabling the network to learn and utilize these features more effectively and optimize the feature extraction and utilization performance in infrared anti-drone multi-target tracking.

[0046] The output features will be passed to subsequent network layers to provide discriminative feature information, thereby improving the accuracy and stability of tracking;

[0047] During the inference phase, the multi-branch structure is converted into a single path through reparameterization, which greatly reduces computational complexity and resource consumption, resulting in a significant speedup.

[0048] S53 further processes the output of OSNet by extracting local, global, and channel features of the target through three branches, thereby enhancing the network's ability to extract information from different dimensions.

[0049] S54, in the part branch of RepO-LightMBN, divides the input features into three parts: p1, p2, and pg, which represent the upper half, lower half, and overall features of the target, respectively. The part branch improves the acquisition of detailed information about the target.

[0050] In S55, within the global branch of RepO-LightMBN, different operations are performed on the high-interest and low-interest regions of the network, using... , These represent the features of high-attention and low-attention regions, respectively, and the global branch further enhances the network's ability to distinguish between background and target information.

[0051] In S56, the feature map is transformed into a one-dimensional vector through global pooling. Then, the one-dimensional vector is divided into two one-dimensional vectors on an equal basis. After that, the length of the two vectors is expanded to the length before the division through a shared convolutional layer. c1 and c2 represent the two channel vectors. The channel branch further enhances the interaction between the channel information of the network.

[0052] S57, the network uses BNNeck output mapping for the output of each branch, where BNNeck consists of BN layers and fully connected layers. RepO-LightMBN classifies the results of different branches into two categories, a process defined as follows:

[0053]

[0054]

[0055] In the above formula, This indicates the result after BNNeck. , These represent identity and rank spaces, respectively.

[0056] LightMBN pairs , Design different loss functions to enhance , The feature representations at different scales show good recognition performance for targets of various scales.

[0057] S58. The RepO-LightMBN feature re-identification network was trained using the collected data. The network training epochs were set to 50 epochs, and the optimizer was Adma.

[0058] In a further preferred embodiment, during the cascaded matching in step S6, a B-IoU+ matching algorithm is proposed. This algorithm combines the B-IoU+ algorithm with the OCM matching algorithm to match the target's appearance features, specifically including the following steps:

[0059] S61, using the CMC algorithm to process the observations at the two time steps required to calculate the direction of motion in the OCR algorithm. The compensation process is defined as follows:

[0060]

[0061] In the formula, denoted as the estimated value of the target feature obtained after compensation, and c represents the original observation value. Represents the motion compensation matrix. This represents the motion compensation term, which compensates for the last observation before the target is lost.

[0062] S62 proposes the B-IoU+ matching algorithm to match detection boxes and trajectories. The specific process is as follows:

[0063] Obtain the detection bounding box information and existing trajectory information of the current frame.

[0064] For each detection box and trajectory prediction box, the buffer is constructed as follows, with the B-IoU+ matching algorithm bounding box and buffer size transformation method as shown in the formula below:

[0065]

[0066]

[0067] In the above formula, o represents the border before expansion, x represents the x-coordinate of the top-left corner of the border, y represents the y-coordinate of the top-left corner of the border, w represents the width of the border, and h represents the length of the border. 'b' represents the expanded border, and 'b' represents the expansion coefficient.

[0068] By calculating the crossover ratio of the buffer instead of the original crossover ratio of the bounding box, a larger intersection area is achieved compared to the ordinary IoU, further reducing the problem of low intersection area caused by the perturbation of small-area targets.

[0069] For each trajectory prediction box buffer and each detection box buffer, calculate their intersection-over-union ratio (IoU).

[0070] Let the coordinates of the trajectory prediction box buffer be... The coordinates of the detection box buffer are ;

[0071] Calculate the intersection area This process is defined as follows:

[0072]

[0073]

[0074]

[0075] Calculate the area of ​​the union This process is defined as follows:

[0076]

[0077]

[0078]

[0079] The process of calculating the B - IoU+ value is defined as follows:

[0080]

[0081] Set an appropriate B-IoU+ threshold T;

[0082] For each trajectory, calculate its B-IoU+ value with all bounding boxes. When the B-IoU+ value of a bounding box with respect to a trajectory is greater than a threshold T, the bounding box is considered a candidate matching box for that trajectory.

[0083] For each detection box, calculate its B-IoU+ value with all trajectories. When the B-IoU+ value of a trajectory with respect to the detection box is greater than a threshold T, the trajectory is considered a candidate matching trajectory for that detection box.

[0084] A detection box is considered to be a successful match for a trajectory only when it is a candidate matching box for a trajectory in forward matching and the trajectory is also a candidate matching trajectory for the same detection box in backward matching.

[0085] More preferably, in step S7, for targets and trajectories that mismatch during the cascade matching stage, an OCR matching algorithm is used to match the last observation and detection values ​​of the trajectory. Specifically, this includes the following steps:

[0086] S71, obtain the target and trajectory of mismatch in the cascade matching stage.

[0087] S72, the CMC algorithm is used to update the observations in the OCR algorithm, and the observations are updated accordingly. The CMC algorithm is applied for compensation, and this process is defined as follows:

[0088]

[0089] In the above formula, denoted as the estimated value of the target feature obtained after compensation, and c represents the original observation value. Represents the motion compensation matrix. This represents the motion compensation term, which compensates for the last observation before the target is lost.

[0090] S73, Introduce the B-IoU+ algorithm into the OCR matching algorithm. For the specific process formula, please refer to step S52.

[0091] More preferably, in step S8, the Kalman filter algorithm is used to update the state of the matched trajectories obtained from cascaded matching and OCR matching, the unmatched trajectories in the mismatched trajectories obtained from OCR matching are deleted, and the updated target trajectory prediction box and target ID are output. Specifically, this includes the following steps:

[0092] S81 updates the status of the matching trajectories obtained from cascaded matching and OCR matching.

[0093] S82, delete the unmatched trajectory in the mismatched trajectory obtained by OCR matching.

[0094] S83 updates the appearance model of each tracker to adapt to changes in the target's appearance.

[0095] S84 outputs the trajectory of each tracked target, including the target's ID, bounding box, and confidence level.

[0096] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0097] This invention provides an infrared UAV multi-target tracking method with strong adaptability. It can stably and accurately track multiple targets in various complex scenarios, such as dense targets, occlusion, variable movement, and poor image quality, significantly improving the overall system performance and providing strong support for security monitoring and anti-UAV operations.

[0098] 1. The pre-matching strategy proposed in this invention uses the IoU between predicted and observed values ​​to quantify the target motion features, selects safe targets with high matching degree, reduces feature extraction calls, significantly reduces computational costs, significantly improves tracking efficiency in multi-target scenarios, better meets the real-time tracking needs with limited resources, and is superior to traditional technologies.

[0099] 2. This invention proposes the Rep-OSNET network, which introduces a reparameterized structure into OSNET. During training, it uses multiple branches to deeply mine features and accurately capture key features of UAVs, making the network features highly discriminative. During inference, it is converted to a single path, reducing computational complexity and resource consumption, achieving real-time multi-target tracking and reducing errors. Compared with networks that do not use this structure, the network performance is superior.

[0100] 3. The RepO-LightMBN proposed in this invention uses Rep-OSNET as its backbone network. Its three-branch design effectively extracts multi-dimensional features of the target, enhancing the network's adaptability to changes in targets in infrared anti-drone scenarios. The reparameterized structure optimizes feature learning and computation efficiency during the training and inference phases, respectively, significantly improving both feature extraction performance and running speed, surpassing traditional networks.

[0101] 4. The B-IoU+ matching algorithm proposed in this invention effectively solves the problem of small-area target disturbance by calculating the intersection-union ratio of the buffer through unique coordinate transformation, improves the matching accuracy in complex scenes, and accurately matches the target and trajectory in infrared anti-UAV tracking, reducing errors and loss, and outperforming the traditional IoU algorithm. Attached Figure Description

[0102] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0103] Figure 1 This is a flowchart of an infrared UAV multi-target tracking method according to an example of the present invention;

[0104] Figure 2 This is a structural diagram of the pre-matching strategy in an example of the present invention;

[0105] Figure 3 This is a diagram of the RepO-LightMBN network structure in an example of the present invention;

[0106] Figure 4 This is a diagram of the Rep-OSNET backbone network structure in an example of the present invention;

[0107] Figure 5 This is a diagram of the reparameterized SAG structure in an example of the present invention;

[0108] Figure 6 This is a structural diagram of the B-IoU+ matching algorithm in an example of the present invention. Detailed Implementation

[0109] To make the objectives, technical solutions, and advantages of this invention clearer, the specific implementation process of this invention will be described in more detail below with reference to specific embodiments and accompanying drawings, so as to facilitate those skilled in the art to more accurately understand this invention and apply it to various specific fields.

[0110] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0111] See Figure 1 This invention discloses a multi-target tracking method for infrared unmanned aerial vehicles (UAVs). The steps are as follows:

[0112] Step S1: Input infrared camera video stream data, and use the YOLOv8 target detector to detect targets in the current frame of the infrared video to obtain target information; the YOLOv8 target detection network is composed of a feature extraction backbone network, a multi-scale feature fusion network, and a detection head. The specific steps are as follows:

[0113] Step S11, constructing the YOLOv8 object detection network includes constructing a feature extraction backbone network, constructing a multi-scale feature fusion network, and constructing a detection head.

[0114] Step S12, using the collected dataset, select The loss function and Adam optimizer are used to train the YOLOv8 object detection network. (The above...) The loss function is a weighted sum of the localization loss, classification loss, and confidence loss, and its definition is as follows:

[0115]

[0116] In the above formula, Indicates location loss. Represents classification loss, This represents the confidence loss.

[0117] Step S13: Use the NMS algorithm to process redundant detection boxes and output the detection target boxes.

[0118] Step S2: Using a camera motion compensation method, the position information of the target is corrected according to the camera's motion state, and the Kalman filter algorithm is used to predict the trajectory of all detected targets to determine whether the trajectory is in a confirmed or unconfirmed state. The specific steps are as follows:

[0119] Step S21: For the infrared video, before processing each frame, determine the motion state of the camera;

[0120] If camera motion is detected, a camera motion compensation method is introduced. Based on the CMC (Camera Motion Compensation, enhanced correlation coefficient) method, image registration or alignment is performed on the offset or distortion caused by camera motion to correct the target position information.

[0121] Step S22: Calculate the affine transformation matrix of the camera movement using the RANSAC algorithm. This process is defined as follows:

[0122]

[0123]

[0124] In the above formula, Denotes the affine transformation matrix. This represents the scaling and rotation matrix. This represents the translation matrix.

[0125] The predicted bounding box is transformed from the coordinate system of frame k-1 to the coordinate system of the next frame k using an affine transformation matrix. This process is defined as follows:

[0126]

[0127] In the above formula, , These represent the state vectors before and after compensation, respectively. , Let represent the covariance matrices of the Kalman filter before and after compensation, respectively.

[0128] Step S3: Use the Kalman filter algorithm to predict the trajectory of all detected targets and determine whether the trajectory is in a confirmed or unconfirmed state. The specific steps are as follows:

[0129] Step S31: For each detected target, obtain the state vector and covariance matrix of the compensated Kalman filter.

[0130] Step S32: Based on the target's motion model, predict the target's state at the current moment according to the state estimate from the previous moment.

[0131] Step S33: Calculate the correlation metric between the predicted state and the currently detected target location information, and set a threshold. If the correlation metric is less than the threshold, it is considered that the currently detected target and the predicted trajectory have a high correlation, and the trajectory is marked as confirmed. Conversely, if the correlation metric is greater than the threshold, it indicates that the current detection may have a large deviation or be a newly appeared target, and the corresponding trajectory is marked as unconfirmed.

[0132] Step S4 proposes a pre-matching strategy. Before extracting appearance features, a pre-match is performed to mark targets with low matching scores. The pre-matching strategy process is as follows: Figure 2 As shown. The specific steps are as follows:

[0133] Step S41: Use the IoU of the predicted value and the observed value to represent the strength of the target's motion characteristics.

[0134] Step S42: Before feature extraction, perform an IoU match between the trajectory and the detection box.

[0135] Step S43: By setting a threshold, the only detection box with a high matching degree is set as a safe target, and all other targets are set as dangerous targets.

[0136] In step S44, the Reid network extracts features from targets designated as dangerous targets and updates the appearance feature encoding of safe targets using the appearance encoding stored in the matched trajectory.

[0137] Step S5: Propose a lightweight feature re-identification network, RepO-LightMBN, and construct and train the RepO-LightMBN network, as follows: Figure 3 As shown, RepO-LightMBN is used to extract the appearance features of low-match targets in video frames. The specific steps are as follows:

[0138] Step S51, construct the Rep-OSNET backbone network, such as Figure 4 As shown. The specific process is as follows:

[0139] Introducing the reparameterized structural unit SAG in the OSNET network, such as Figure 5 As shown, through a unique multi-branch convolutional structure, the network's ability to extract and utilize features in infrared anti-drone multi-target tracking scenarios is effectively improved.

[0140] During the training phase, SAG has a multi-branch convolutional structure, including an input layer, an average pooling layer, a multilayer perceptron (MLP), an activation function sigmoid, and an output layer.

[0141] The input layer receives feature map information from the corresponding stage in the Rep-OSNET network;

[0142] Average pooling layers reduce feature dimensionality and enhance feature robustness by averaging features within local regions. This ensures that the extracted features remain relatively stable and representative even when faced with potential local interference from UAV targets in infrared images, laying the groundwork for further feature transformations.

[0143] MLP, with its multi-layered structure composed of multiple neurons, performs deep nonlinear transformations on input features, thereby uncovering more complex and abstract feature information;

[0144] The Sigmoid activation function maps the output value to the 0-1 range, giving the features appropriate non-linear characteristics. This allows the features to have a more reasonable numerical range and a distribution pattern that better matches the actual scene, enabling the network to learn and utilize these features more effectively and optimize the feature extraction and utilization performance in infrared anti-drone multi-target tracking.

[0145] The output features will be passed to subsequent network layers, providing discriminative feature information to improve the accuracy and stability of tracking;

[0146] During the inference phase, SAG effectively transforms the multi-branch structure into a single-branch structure, reducing the computational load and model complexity of the network without losing the feature information learned during the training phase.

[0147] Step S52 further processes the OSNet output by extracting local, global, and channel features of the target through three branches, thereby enhancing the network's ability to extract information from different dimensions. The specific process is as follows:

[0148] In RepO-LightMBN, the input features are divided into three parts: p1, p2, and pg, which represent the upper half, lower half, and overall features of the target, respectively. The local branch improves the acquisition of detailed information about the target.

[0149] In the global branch of RepO-LightMBN, different operations are performed on the high-interest and low-interest regions of the network, using g, These represent the features of high-attention and low-attention regions, respectively, and the global branch further enhances the network's ability to distinguish between background and target information.

[0150] In the channel branch, the feature map is transformed into a one-dimensional vector through global pooling, and then the one-dimensional vector is divided into two one-dimensional vectors on an equal basis. After that, the length of the two vectors is expanded to the length before the division through a shared convolutional layer. c1 and c2 represent the two channel vectors respectively. The channel branch further enhances the interaction between the channel information of the network.

[0151] Step S53: The network maps the output of each branch using BNNeck, which consists of BN layers and fully connected layers. RepO-LightMBN categorizes the results of different branches into two classes, defined as follows:

[0152]

[0153]

[0154] In the above formula, This indicates the result after BNNeck. , These represent identity and rank spaces, respectively.

[0155] LightMBN pairs , Design different loss functions to enhance , The feature representations at different scales show good recognition performance for targets of various scales.

[0156] Step S54: Train the RepO-LightMBN feature re-identification network using the collected data. The network training epochs are set to 50 epochs, and the optimizer is Adma.

[0157] Step S6: The B-IoU+ matching algorithm is proposed. This algorithm combines B-IoU+ with the OCM matching algorithm, using the target's appearance features to match the bounding box and trajectory. The specific steps are as follows:

[0158] Step S61: Use the CMC algorithm to process the observations at the two time steps required to calculate the direction of motion in the OCM algorithm. The compensation process is defined as follows:

[0159]

[0160] In the above formula, denoted as the estimated value of the target feature obtained after compensation, and c represents the original observation value. Represents the motion compensation matrix. This represents the motion compensation term, which compensates for the observations at the two time steps required to calculate the direction of motion.

[0161] Step S62, the B-IoU+ matching algorithm is proposed to match the detection box and the trajectory. The structure of the matching algorithm is as follows: Figure 6 As shown. The specific process is as follows:

[0162] Obtain the detection bounding box information and existing trajectory information of the current frame.

[0163] For each detection box and trajectory prediction box, the buffer is constructed as follows, with the B-IoU+ matching algorithm bounding box and buffer size transformation method as shown in the formula below:

[0164]

[0165]

[0166] In the above formula, o represents the border before expansion, x represents the x-coordinate of the top-left corner of the border, y represents the y-coordinate of the top-left corner of the border, w represents the width of the border, and h represents the length of the border. 'b' represents the expanded border, and 'b' represents the expansion coefficient.

[0167] By calculating the crossover ratio of the buffer instead of the original crossover ratio of the bounding box, a larger intersection area is achieved compared to the ordinary IoU, further reducing the problem of low intersection area caused by the perturbation of small-area targets.

[0168] For each trajectory prediction box buffer and each detection box buffer, calculate the intersection-union ratio (IUGR) between them.

[0169] Let the coordinates of the trajectory prediction box buffer be... The coordinates of the detection box buffer are ;

[0170] Calculate the intersection area This process is defined as follows:

[0171]

[0172]

[0173]

[0174] Calculate the area of ​​the union This process is defined as follows:

[0175]

[0176]

[0177]

[0178] The process of calculating the B - IoU+ value is defined as follows:

[0179]

[0180] Set an appropriate B-IoU+ threshold T;

[0181] For each trajectory, calculate its B-IoU+ value with all bounding boxes. When the B-IoU+ value of a bounding box with respect to a trajectory is greater than a threshold T, the bounding box is considered a candidate matching box for that trajectory.

[0182] For each detection box, calculate its B-IoU+ value with all trajectories. When the B-IoU+ value of a trajectory with respect to the detection box is greater than a threshold T, the trajectory is considered a candidate matching trajectory for the detection box.

[0183] A detection box is considered to be successfully matched with a trajectory only when it is a candidate matching box for a trajectory in forward matching and the trajectory is also a candidate matching trajectory for the same detection box in reverse matching.

[0184] Step S7: For targets and trajectories that mismatched during the cascade matching stage, an OCR matching algorithm is used to match the last observation and detection values ​​of the trajectory. The specific steps are as follows:

[0185] Step S71: Obtain the target and trajectory of the mismatch in the cascade matching stage.

[0186] Step S72: Update the observations in the OCR algorithm using the CMC algorithm. The CMC algorithm is applied for compensation, and this process is defined as follows:

[0187]

[0188] In the above formula, denoted as the estimated value of the target feature obtained after compensation, and c represents the original observation value. Represents the motion compensation matrix. This represents the motion compensation term, which compensates for the last observation before the target is lost.

[0189] Step S73: Introduce the B-IoU+ algorithm into the OCR matching algorithm. For the specific process formula, please refer to step S5.

[0190] Step S8: The Kalman filter algorithm is used to update the state of the matched trajectories obtained from cascaded matching and OCR matching. Unmatched trajectories in the mismatched trajectories obtained from OCR matching are deleted, and the updated target trajectory prediction box and target ID are output. The specific steps are as follows:

[0191] Step S81: For each matching trajectory obtained from cascaded matching and OCR matching, initialize the state vector and covariance matrix of the Kalman filter.

[0192] Step S82: The state prediction equation of the matching trajectory is used for state prediction. The state prediction equation of the Kalman filter is:

[0193]

[0194] In the above formula, Is The state vector at the predicted time. yes The optimal estimated state vector at time t. It is the state transition matrix.

[0195] Step S83: Predict the covariance matrix using the covariance prediction equation:

[0196]

[0197] In the above formula, It is a prediction The covariance matrix at time t, yes The covariance matrix at time t, It is the process noise covariance matrix.

[0198] Step S84: Update measurements based on the observations corresponding to the matched trajectory. First, calculate the Kalman gain:

[0199] in It is a measurement matrix that links the state vector with the measurement vector. It is the measurement noise covariance matrix.

[0200] Step S85, update the status:

[0201]

[0202] Get In Optimal state estimation at time 1 and covariance update , where I is the identity matrix.

[0203] Step S86: In the mismatched trajectories obtained from OCR matching, identify the trajectories that do not match the detected target. These unmatched trajectories represent that the target has left the monitoring area, is occluded, or cannot be detected for other reasons. Delete these unmatched trajectories.

[0204] Step S87: Update the appearance model of each tracker to adapt to changes in the target's appearance.

[0205] Step S88: Output the trajectory of each tracked target, including the target's ID, bounding box, and confidence level.

[0206] Obviously, the above steps of the present invention are merely examples to clearly illustrate the technical solution of the present invention, and are not intended to limit the specific embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for multi-target tracking of an infrared unmanned aerial vehicle (UAV), characterized in that, include: Use the YOLOv8 target detector to obtain target information in the current frame of the infrared camera video stream; The target's position information is corrected based on the camera's motion state using a camera motion compensation method. The Kalman filter algorithm is used to predict the trajectory of all detected targets and determine whether the trajectory is in a confirmed or unconfirmed state. By employing a pre-matching strategy, a pre-matching process is performed before appearance feature extraction to label targets with low matching scores. The appearance features of low-match targets in video frames are extracted using the lightweight feature re-identification network RepO-LightMBN. The B-IoU+ matching algorithm is used to combine the detected target with the confirmed trajectory and the target's appearance features, distance and motion information to calculate the cost matrix for cascade matching. The B-IoU+ and OCM matching algorithms are used to combine the target's appearance features to match the detection box and trajectory. For targets and trajectories that mismatch during the cascade matching stage, an OCR matching algorithm is used to match the last observation and detection values ​​of the trajectory. The Kalman filter algorithm is used to update the state of the matched trajectories obtained by cascaded matching and OCR matching. The unmatched trajectories in the mismatched trajectories obtained by OCR matching are deleted, and the target trajectory prediction box and target ID after state update are output.

2. The infrared UAV multi-target tracking method as described in claim 1, characterized in that: The pre-matching strategy consists of a pre-matching algorithm and a feature update process. Pre-matching uses motion features to divide the target. For targets with obvious motion features, the calls to the feature extraction module are reduced, and vice versa. The specific process includes: The IoU between predicted and observed values ​​represents the strength of the target's motion characteristics; Before feature extraction, perform an IoU match between the trajectory and the detection box; By setting a threshold, the only detection box with a high matching degree is set as a safe target, and all other targets are set as dangerous targets.

3. The infrared UAV multi-target tracking method as described in claim 1, characterized in that: The lightweight feature re-identification network is constructed by using the RepO-LightMBN feature re-identification network to extract the appearance features of targets with low matching degree, and the feature re-identification network is trained using the collected data.

4. The infrared UAV multi-target tracking method as described in claim 3, characterized in that: The RepO-LightMBN construction uses Rep-OSNET as the backbone network and introduces a reparameterization structure to further process the output of OSNet. Through three branches, it extracts the local, global, and channel features of the target, thereby enhancing the network's ability to extract information from different dimensions.

5. The infrared UAV multi-target tracking method as described in claim 4, characterized in that: The introduction of a reparameterization structure, by setting a reparameterization unit SAG in the Rep-OSNET network, effectively improves the network's ability to extract and utilize features in infrared anti-drone multi-target tracking scenarios through a unique multi-branch convolutional structure. The SAG has a multi-branch convolutional structure during the training phase, including an input layer, an average pooling layer, an MLP, an activation function Sigmoid, and an output layer.

6. The infrared UAV multi-target tracking method as described in claim 1, characterized in that: The B-IoU+ matching algorithm includes coordinate transformation methods for the border and buffer. It replaces the original border's intersection-union ratio by calculating the intersection-union ratio of the buffer, resulting in a larger intersection area and further reducing the low intersection area problem caused by disturbances of small-area targets.

7. The infrared UAV multi-target tracking method as described in claim 6, characterized in that: The B-IoU+ matching algorithm's bounding box and buffer size transformation method is specifically defined as follows: o = (x, y, w, h) In the above formula, o represents the border before expansion, x represents the horizontal coordinate of the upper left corner of the border, y represents the vertical coordinate of the upper left corner of the border, w represents the width of the border, h represents the length of the border, o' represents the border after expansion, b represents the expansion coefficient, the size of the buffer is expanded by the length of the largest side, and it has the same expansion distance in both the vertical and horizontal directions, which has a good matching effect for different movement directions of the target.

Images