Infrared unmanned aerial vehicle multi-target tracking method

Abstract

The invention discloses an infrared unmanned aerial vehicle multi-target tracking method, and relates to the field of computer vision. The method comprises the following steps: reading a video of an infrared camera, and detecting a target by using YOLOv8 to obtain information; a camera motion compensation correction position is introduced, and a Kalman filtering algorithm is adopted to predict a track and determine a state; marking a low-matching-degree target through a pre-matching strategy; extracting appearance characteristics of the target by using RepO-LightMBN, calculating a cost matrix by combining an appearance characteristic distance and motion information, and associating a detection frame and a track through cascade matching and a B-IoU + matching algorithm; matching the last observation value and detection value of the track by adopting B-IoU + and OCR matching algorithms according to the mismatched target and track in cascade matching; and finally, updating a matched track state by using Kalman filtering, deleting an unmatched track and outputting a result. The method effectively improves the speed and precision of multi-target tracking of the unmanned aerial vehicle in the infrared mode, is innovative and practical, and can be used in the fields of anti-unmanned aerial vehicle monitoring and the like.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an infrared unmanned aerial vehicle multi-target tracking method. Background Art

[0002] In the field of modern security and military defense, the widespread use of drones has brought many potential risks and challenges. With the rapid development of drone technology, the frequency of its use in civil and military scenarios has increased dramatically. Infrared-based anti-drone multi-target tracking technology has become an important research direction that needs to be developed and improved.

[0003] In the field of target tracking technology, traditional visible light-based methods have limitations in certain scenarios. For example, in low light, bad weather, and nighttime environments, the performance of visible light imaging equipment will be greatly reduced, making it difficult to effectively detect and track targets. In contrast, infrared technology has unique advantages based on the object's own thermal radiation imaging principle. It can detect targets in complete darkness or complex meteorological conditions, and is not directly affected by light intensity and visibility, providing a reliable technical approach for anti-UAV multi-target detection and tracking.

[0004] Multi-target tracking includes two parts: multi-target detection and multi-target tracking. Classic deep learning target detection networks are divided into two-stage networks and single-stage networks. Among them, the two-stage detection algorithm has high detection accuracy but slow running speed; the single-stage detection algorithm runs fast but has a high false alarm rate. Two-stage networks, such as RCNN, FastR-CNN, FasterR-CNN and CascadeR-CNN, first need to generate candidate regions, and then classify and locate the candidate regions, which are suitable for applications with higher detection accuracy requirements; single-stage networks, such as SSD, YOLO series and CenterNet, directly generate coordinate positions and class probabilities, which are faster than two-stage networks. Multi-target tracking methods based on deep learning are mainly divided into two categories: Tracking Based Detection (TBD) and Joint Detection Tracking (JDT). The JDT algorithm attempts to integrate the detection module and the tracking module to improve the inference speed, but in practical applications, it faces the problem of difficulty in co-training of each module, resulting in unstable overall performance. The TBD strategy has a clear multi-stage design structure and can optimize detection and tracking separately, and performs well in terms of adaptability to complex scenes. For example, the classic TBD tracker SORT proposed a simple and real-time data cascading method. DeepSORT adds appearance information based on the SORT framework to improve the algorithm performance and achieve tracking of targets that are occluded for a long time. StrongSORT further upgrades it in terms of detection, embedding and cascading, optimizes DeepSORT with the latest components and training techniques, and proposes an appearance-free link model (AFLink) that only uses spatiotemporal information to cascade short trajectories into complete trajectories, as well as Gaussian smoothing interpolation (GSI) to compensate for missing detections. BoT-SORT achieves better bounding box positioning through camera motion compensation and more accurate Kalman filter state vectors, as well as a new fusion method based on cosine distance of IoU and re-id. However, the KF-based motion model is not robust enough when dealing with occlusion, nonlinear motion, and low frame rate video. Although OC-SORT attempts to optimize the motion model and emphasizes the role of detection (Observation) in recovering lost trajectories and reducing the error accumulation of the KF model during the loss period, thereby improving the robustness to occlusion and nonlinear motion, it still has defects such as inaccurate tracking and difficulty in re-locking the target after it is lost when faced with blurred background interference, occlusion and disappearance of objects.

[0005] In summary, the existing multi-target tracking technology still has many problems when applied to infrared-based anti-UAV scenarios, especially in terms of target detection accuracy, tracking stability, and adaptability to resource-constrained environments in complex environments. Therefore, it is urgent to develop a new infrared UAV multi-target tracking method to overcome the shortcomings of existing technologies, effectively respond to UAV threats, and ensure the safety and stability of related fields. Summary of the invention

[0006] In view of the shortcomings and deficiencies of the above methods, the present invention proposes an infrared UAV multi-target tracking method. The steps are:

[0007] Step S1, input infrared camera video stream data, use the target detector YOLOv8 to detect the target in the current frame of the infrared video, and obtain target information; the YOLOv8 target detection network is based on a feature extraction backbone network, a multi-scale feature fusion network, and a detection head.

[0008] Step S2: Correcting the position information of the target according to the motion state of the camera by using a camera motion compensation method.

[0009] Step S3, using the Kalman filter algorithm to predict the trajectories of all detected targets, and determining whether the state of the trajectory is a confirmed state or an unconfirmed state.

[0010] Step S4, a pre-matching strategy is proposed. Before extracting the appearance features, a pre-matching is performed to mark safe targets and dangerous targets according to the matching degree.

[0011] Step S5, propose a lightweight feature re-identification network RepO-LightMBN, build and train the RepO-LightMBN network, and extract the appearance features of low-matching targets in video frames through RepO-LightMBN.

[0012] Step S6, a B-IoU+ matching algorithm is proposed, and the detection box and trajectory are cascade matched by using the B-IoU+ and OCM matching algorithms in combination with the appearance features of the target.

[0013] Step S7: For the mismatched targets and trajectories in the cascade matching stage, an OCR matching algorithm is used to match the last observed value and the detected value of the trajectory.

[0014] Step S8, using the Kalman filter algorithm to update the state of the matching tracks obtained by cascade matching and OCR matching, deleting the unmatched tracks in the mismatch tracks obtained by OCR matching, and outputting the target track prediction box and target ID after the state is updated.

[0015] Further preferably, the YOLOv8 target detection network in step S1 specifically includes the following steps:

[0016] S11, building a YOLOv8 target detection network includes building a feature extraction backbone network, building a multi-scale feature fusion network, and building a detection head.

[0017] S12, using the collected data set, select L total The loss function and Adam optimizer are used to train the YOLOv8 target detection network. total The loss function is the weighted sum of positioning loss, classification loss and confidence loss, which is defined as follows:

[0018] L total =L loc +L cls +L conf

[0019] In the above formula, L loc represents the positioning loss, L cls represents the classification loss, L conf Represents the confidence loss.

[0020] S13, use the NMS algorithm to process the redundant detection frame and output the detection target frame.

[0021] Further preferably, the camera motion compensation method in step S2 specifically comprises the following steps:

[0022] S21, determining the motion state of the camera before processing each frame of the infrared video;

[0023] S22, if it is detected that the camera is in motion, a camera motion compensation method is introduced, and based on a CMC (Camera Motion Compensation, enhanced correlation coefficient) method, image registration or alignment is performed on the offset or distortion caused by the camera motion to achieve correction of the target position information;

[0024] S23, using the RANSAC algorithm to calculate the affine transformation matrix of camera movement, this process is defined as follows:

[0025]

[0026]

[0027] In the above formula, represents the affine transformation matrix, represents the scaling rotation matrix, Represents the translation matrix.

[0028] S24, using the affine transformation matrix to transform the predicted bounding box from the coordinate system of the k-1 frame to the coordinate system of the next frame k. This process is defined as follows:

[0029]

[0030] In the above formula, , represent the state vectors before and after compensation, , Represent the covariance matrix of the Kalman filter before and after compensation respectively.

[0031] Further preferably, in step S3, the trajectories of all detected targets are predicted using a Kalman filter algorithm to determine whether the state of the trajectory is a confirmed state or an unconfirmed state, which specifically includes the following steps:

[0032] S31, for each detected target, obtaining the state vector and covariance matrix of the compensated Kalman filter.

[0033] S32, predicting the state of the target at the current moment according to the motion model of the target and the state estimation value at the previous moment.

[0034] S33, calculate the correlation metric between the predicted state and the currently detected target position information, set a threshold, compare the correlation metric and the threshold to determine whether the trajectory is confirmed or unconfirmed. If it is less than the threshold, it is confirmed, otherwise it is unconfirmed.

[0035] Further preferably, a pre-matching strategy proposed in step S4 is composed of a pre-matching algorithm and a feature updating process. The pre-matching uses motion features to divide the target, and reduces the call of the feature extraction module for targets with obvious motion features, and increases it otherwise. Specifically, it includes the following steps:

[0036] S41, the IoU of the predicted value and the observed value is used to represent the strength of the target's motion features.

[0037] S42, before feature extraction, perform an IoU match between the trajectory and the detection box.

[0038] S43, by setting a threshold, the only detection box with a high matching degree is set as a safe target, and the other targets are set as dangerous targets

[0039] Further preferably, a lightweight feature re-identification network RepO-LightMBN proposed in step S5, constructing a RepO-LightMBN feature re-identification network to extract target appearance features with low matching degree, and training the feature re-identification network using the collected data, specifically includes the following steps:

[0040] S51, builds the Rep-OSNET backbone network and introduces the heavy parameterized structural unit SAG into the OSNET network. Through the unique multi-branch convolutional structure, it effectively improves the network's ability to extract and utilize features in the infrared anti-UAV multi-target tracking scenario.

[0041] S52, SAG has a multi-branch convolutional structure in the training stage, including input layer, average pooling layer, multi-layer perceptron (MLP), activation function Sigmoid and output layer.

[0042] Input layer, receiving feature graph information from the corresponding stage in the Rep-OSNET network;

[0043] The average pooling layer averages the features in the local area, reducing the feature dimension while enhancing the robustness of the features. This ensures that the extracted features remain relatively stable and representative when facing possible local interference of drone targets in infrared images, paving the way for further feature transformation.

[0044] MLP, with its multi-layer structure consisting of multiple neurons, performs deep nonlinear transformation on input features and mines more complex and abstract feature information;

[0045] The Sigmoid activation function maps the output value to the range of 0-1, giving the feature a suitable nonlinear characteristic, so that the feature has a more reasonable numerical range and a distribution law that is more in line with the actual scene, so that the network can better learn and use these features, and optimize the network's feature extraction and utilization effect in infrared anti-UAV multi-target tracking;

[0046] The output features will be passed to the subsequent network layers to provide discriminative feature information to improve the accuracy and stability of tracking;

[0047] During the inference phase, the multi-branch structure is converted into a single path through re-parameterization, which greatly reduces the computational complexity and resource consumption, and significantly speeds up the process.

[0048] S53 further processes the output results of OSNet and extracts the local, global and channel features of the target through three branches, thereby enhancing the network's ability to extract information of different dimensions.

[0049] S54, in the part branch of RepO-LightMBN, the input features are divided into three parts: p1, p2, and pg, which represent the upper half, lower half, and overall features of the target, respectively. The local branch improves the acquisition of target detail information.

[0050] S55, in the global branch of RepO-LightMBN, different operations are performed on the high-attention area and low-attention area of ​​the network, using , They represent the features of high-attention areas and low-attention areas respectively. The global branch further improves the network's recognition of background information and target information.

[0051] S56, in the channel branch, the feature map is converted into a one-dimensional vector through global pooling, and then the one-dimensional vector is evenly divided into two one-dimensional vectors. After that, the length of the two vectors is expanded to the length before the division through a shared convolutional layer. c1 and c2 are used to represent the two channel vectors. The channel branch further improves the interaction between the channel information of the network.

[0052] S57, the network uses BNNeck output mapping for the output of each branch, and BNNeck consists of a BN layer and a fully connected layer. RepO-LightMBN divides the results of different branches into two categories. This process is defined as follows:

[0053]

[0054]

[0055] In the above formula, Indicates the result after BNNeck. , They represent identity and rank spaces respectively.

[0056] LightMBN , Design different loss functions to enhance , The feature expressions of different scales show good recognition effects for targets of different scales.

[0057] S58, using the collected data to train the RepO-LightMBN feature re-identification network. The number of network training rounds is set to 50 rounds, and the optimizer is Adma.

[0058] Further preferably, during the cascade matching in step S6, a B-IoU+ matching algorithm is proposed, and the detection frame and the trajectory are matched using the B-IoU+ and OCM matching algorithms combined with the appearance features of the target, specifically including the following steps:

[0059] S61, using the CMC algorithm to calculate the two time-step observations required for the OCR algorithm to calculate the direction of motion To compensate, this process is defined as follows:

[0060]

[0061] In the formula, represents the estimated value of the target feature after compensation, c represents the original observed value, represents the motion compensation matrix, Represents the motion compensation term, which is used to compensate for the last observation value before the target is lost.

[0062] S62, propose the B-IoU+ matching algorithm to match the detection box and the trajectory. The specific process is as follows:

[0063] Get the detection box information and existing trajectory information of the current frame.

[0064] For each detection box and trajectory prediction box, the buffer is constructed in the following way. The B-IoU+ matching algorithm border and buffer size transformation method is as follows:

[0065]

[0066]

[0067] In the above formula, o represents the border before expansion, x represents the horizontal coordinate of the upper left corner of the border, y represents the vertical coordinate of the upper left corner of the border, w represents the width of the border, and h represents the length of the border. represents the border after expansion, and b represents the expansion coefficient.

[0068] By calculating the IoU of the buffer instead of the IoU of the original border, it has a larger intersection area than the ordinary IoU, further reducing the problem of low intersection area caused by the disturbance of small area targets;

[0069] For the buffer of each trajectory prediction box and the buffer of each detection box, calculate the intersection-and-union ratio between them;

[0070] Assume the trajectory prediction box buffer coordinates are , the detection box buffer coordinates are ;

[0071] Calculate the intersection area , this process is defined as follows:

[0072]

[0073]

[0074]

[0075] Calculate the area of ​​the union , this process is defined as follows:

[0076]

[0077]

[0078]

[0079] The process of calculating B - IoU+ value is defined as follows:

[0080]

[0081] Set a suitable B-IoU+ threshold T;

[0082] For each track, calculate the B-IoU+ value between it and all detection boxes. When the B-IoU+ value between a detection box and the track is greater than the threshold T, the detection box is considered as a candidate matching box for the track;

[0083] For each detection box, calculate the B-IoU+ value between it and all tracks. When the B-IoU+ value between a track and the detection box is greater than the threshold T, the track is considered as a candidate matching track for the detection box;

[0084] When a detection box is a candidate matching box of a certain track in the forward matching, and the track is also a candidate matching track of the detection box in the reverse matching, it is determined that the detection box matches the track successfully;

[0085] Further preferably, in step S7, for the mismatched target and trajectory in the cascade matching stage, an OCR matching algorithm is used to match the last observed value and the detected value of the trajectory. Specifically, the following steps are included:

[0086] S71, obtaining mismatched targets and trajectories in the cascade matching stage.

[0087] S72, using the CMC algorithm to update the observation value in the OCR algorithm, The CMC algorithm is used for compensation, and the process is defined as follows:

[0088]

[0089] In the above formula, represents the estimated value of the target feature after compensation, c represents the original observed value, represents the motion compensation matrix, Represents the motion compensation term, which is used to compensate for the last observation value before the target is lost.

[0090] S73, introducing the B-IoU+ algorithm into the OCR matching algorithm. For the specific process formula, see step S52.

[0091] Further preferably, in step S8, the Kalman filter algorithm is used to update the state of the matching tracks obtained by cascade matching and OCR matching, the unmatched tracks in the mismatched tracks obtained by OCR matching are deleted, and the target track prediction box and target ID after the state is updated are output. Specifically, the following steps are included:

[0092] S81, updating the status of the matching track obtained by cascade matching and OCR matching.

[0093] S82, deleting the unmatched tracks in the mismatched tracks obtained by the OCR matching.

[0094] S83, update the appearance model of each tracker to adapt to the change of target appearance

[0095] S84, output the trajectory of each tracked target, including the target's ID, bounding box, and confidence.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] The present invention provides an infrared UAV multi-target tracking method with strong adaptability. It can stably and accurately track multiple targets in various complex scenarios, such as dense targets, occlusions, variable motion, and poor image quality, significantly improving the overall performance of the system and providing strong support for security monitoring and anti-UAV operations.

[0098] 1. The pre-matching strategy proposed in this paper uses the IoU of the predicted value and the observed value to quantify the target motion characteristics, screen out safe targets with high matching degree, reduce feature extraction calls, greatly reduce computing costs, and significantly improve tracking efficiency in multi-target scenarios. It is more in line with the real-time tracking needs of resource constraints and is superior to traditional technologies.

[0099] 2. The present invention proposes a Rep-OSNET network, which introduces a heavy parameterized structure into OSNET. During training, multiple branches are used to deeply mine features, accurately capture the key features of drones, and make network features highly discriminative. During reasoning, it is converted to a single path, which reduces computational complexity and resource consumption, realizes real-time multi-target tracking and reduces errors. Compared with networks that do not adopt this structure, the performance is excellent.

[0100] 3. The proposed RepO-LightMBN uses Rep-OSNET as the backbone network. Its three-branch design effectively extracts multi-dimensional features of the target and enhances the network's adaptability to changes in infrared anti-UAV scene targets. The re-parameterized structure optimizes feature learning and computational efficiency in the training and inference stages, respectively, significantly improving both feature extraction effect and running speed, surpassing traditional networks.

[0101] 4. The B-IoU+ matching algorithm proposed in this invention calculates the intersection-over-union ratio of the buffer zone through a unique coordinate transformation, effectively solves the problem of small-area target disturbance, improves the matching accuracy of complex scenes, and accurately matches targets and trajectories in infrared anti-UAV tracking, reducing errors and losses, and performs better than the traditional IoU algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0103] Figure 1 This is a flow chart of a multi-target tracking method of an infrared UAV in an example of the present invention;

[0104] Figure 2 is a pre-matching strategy structure diagram in an example of the present invention;

[0105] Figure 3 It is a RepO-LightMBN network structure diagram in the example of the present invention;

[0106] Figure 4 It is a Rep-OSNET backbone network structure diagram in an example of the present invention;

[0107] Figure 5 is a structural diagram of a re-parameterized structure SAG in an example of the present invention;

[0108] Figure 6 It is a structural diagram of the B-IoU+ matching algorithm in the example of the present invention. DETAILED DESCRIPTION

[0109] In order to make the objectives, technical solutions and advantages of the present invention clearer, the specific implementation process of the present invention will be described in more detail below in combination with specific implementation methods and drawings, so as to facilitate technical personnel to more accurately understand the present invention and apply it in various specific fields.

[0110] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0111] See also Figure 1 The present invention discloses a method for tracking multiple targets of an infrared UAV. The steps are:

[0112] Step S1, input infrared camera video stream data, use target detector YOLOv8 to detect the target in the current frame of the infrared video, and obtain target information; YOLOv8 target detection network is based on feature extraction backbone network, multi-scale feature fusion network, and detection head. The specific steps are as follows:

[0113] Step S11, constructing a YOLOv8 target detection network includes constructing a feature extraction backbone network, constructing a multi-scale feature fusion network, and constructing a detection head.

[0114] Step S12, using the collected data set, select The loss function and Adam optimizer are used to train the YOLOv8 target detection network. The loss function is the weighted sum of positioning loss, classification loss and confidence loss, which is defined as follows:

[0115]

[0116] In the above formula, represents the positioning loss, represents the classification loss, Represents the confidence loss.

[0117] Step S13: Use the NMS algorithm to process the redundant detection frame and output the detection target frame.

[0118] Step S2, through the camera motion compensation method, the position information of the target is corrected according to the motion state of the camera, and the trajectory of all detected targets is predicted using the Kalman filter algorithm to determine whether the state of the trajectory is confirmed or unconfirmed. The specific steps are as follows:

[0119] Step S21, for the infrared video, before processing each frame, determining the motion state of the camera;

[0120] If it is detected that the camera is in motion, a camera motion compensation method is introduced, and based on the CMC (Camera Motion Compensation, enhanced correlation coefficient) method, image registration or alignment is performed on the offset or distortion caused by the camera motion to achieve correction of the target position information;

[0121] Step S22, using the RANSAC algorithm to calculate the affine transformation matrix of the camera movement, this process is defined as follows:

[0122]

[0123]

[0124] In the above formula, represents the affine transformation matrix, represents the scaling rotation matrix, Represents the translation matrix.

[0125] The affine transformation matrix is ​​used to transform the predicted bounding box from the coordinate system of the k-1 frame to the coordinate system of the next frame k. This process is defined as follows:

[0126]

[0127] In the above formula, , Respectively represent the state vectors before and after compensation, , Represent the covariance matrix of the Kalman filter before and after compensation respectively.

[0128] Step S3, using the Kalman filter algorithm to predict the trajectories of all detected targets, and determining whether the state of the trajectory is a confirmed state or an unconfirmed state, the specific steps are as follows:

[0129] Step S31, for each detected target, obtaining the state vector and covariance matrix of the compensated Kalman filter.

[0130] Step S32, predicting the state of the target at the current moment according to the motion model of the target and the state estimation value at the previous moment.

[0131] Step S33, calculate the correlation measure between the predicted state and the currently detected target position information, set a threshold, if the correlation measure is less than the threshold, it is considered that the currently detected target has a high correlation with the predicted trajectory, and the trajectory is marked as confirmed; conversely, if the correlation measure is greater than the threshold, it means that the current detection may have a large deviation or is a newly appeared target, and the corresponding trajectory is marked as unconfirmed.

[0132] Step S4, a pre-matching strategy is proposed. Before extracting the appearance features, a pre-matching is performed to mark the targets with low matching degree. The pre-matching strategy process is as follows: Figure 2 The specific steps are as follows:

[0133] In step S41, the IoU of the predicted value and the observed value is used to represent the strength of the motion feature of the target.

[0134] Step S42: Before feature extraction, perform an IoU match between the trajectory and the detection box.

[0135] Step S43, by setting a threshold, the only detection box with a high matching degree is set as a safe target, and the other targets are set as dangerous targets.

[0136] Step S44, the Reid network extracts features from the target set as a dangerous target, and updates the appearance feature coding of the safe target using the appearance coding saved in the matched trajectory.

[0137] Step S5, propose a lightweight feature re-identification network RepO-LightMBN, build and train the RepO-LightMBN network, such as Figure 3 As shown in the figure, RepO-LightMBN is used to extract the appearance features of low-matching targets in video frames. The specific steps are as follows:

[0138] Step S51, constructing the Rep-OSNET backbone network, such as Figure 4 The specific process is as follows:

[0139] Introduce the heavy parameterized structural unit SAG in OSNET network, such as Figure 5 Through the unique multi-branch convolution structure, the network's ability to extract and utilize features in infrared anti-UAV multi-target tracking scenarios is effectively improved.

[0140] In the training phase, SAG has a multi-branch convolutional structure, including an input layer, an average pooling layer, a multi-layer perceptron (MLP), an activation function Sigmoid, and an output layer.

[0141] Input layer, receiving feature graph information from the corresponding stage in the Rep-OSNET network;

[0142] The average pooling layer averages the features in the local area, reducing the feature dimension while enhancing the robustness of the features. This ensures that the extracted features remain relatively stable and representative when facing possible local interference of drone targets in infrared images, paving the way for further feature transformation.

[0143] MLP, with its multi-layer structure consisting of multiple neurons, performs deep nonlinear transformation on input features and mines more complex and abstract feature information;

[0144] The Sigmoid activation function maps the output value to the range of 0-1, giving the feature a suitable nonlinear characteristic, so that the feature has a more reasonable numerical range and a distribution law that is more in line with the actual scene, so that the network can better learn and use these features, and optimize the network's feature extraction and utilization effect in infrared anti-UAV multi-target tracking;

[0145] The output features will be passed to the subsequent network layers to provide discriminative feature information to improve the accuracy and stability of tracking;

[0146] In the inference phase, SAG converts the multi-branch structure into a single-branch structure, reducing the computational effort and model complexity of the network without losing the feature information learned in the training phase.

[0147] Step S52: further process the output of OSNet, extract the local, global and channel features of the target through three branches, and strengthen the network's ability to extract information of different dimensions. The specific process is as follows:

[0148] In the part branch of RepO-LightMBN, the input features are divided into three parts: p1, p2, and pg, which represent the upper half, lower half, and overall features of the target, respectively. The local branch improves the acquisition of target detail information.

[0149] In the global branch of RepO-LightMBN, different operations are performed on the high-attention area and low-attention area of ​​the network, using g, They represent the features of high-attention areas and low-attention areas respectively. The global branch further improves the network's recognition of background information and target information.

[0150] In the channel branch, the feature map is converted into a one-dimensional vector through global pooling, and then the one-dimensional vector is evenly divided into two one-dimensional vectors. After that, the length of the two vectors is expanded to the length before the division through a shared convolutional layer. c1 and c2 are used to represent the two channel vectors respectively. The channel branch further improves the interaction between the channel information of the network.

[0151] Step S53, the network uses BNNeck output mapping for the output of each branch, and BNNeck consists of a BN layer and a fully connected layer. RepO-LightMBN divides the results of different branches into two categories. This process is defined as follows:

[0152]

[0153]

[0154] In the above formula, Indicates the result after BNNeck. , They represent identity and rank spaces respectively.

[0155] LightMBN , Design different loss functions to enhance , The feature expressions of different scales show good recognition effects for targets of different scales.

[0156] Step S54: Use the collected data to train the RepO-LightMBN feature re-identification network. The number of network training rounds is set to 50 rounds, and the optimizer is Adma.

[0157] Step S6: propose a B-IoU+ matching algorithm, and use the B-IoU+ and OCM matching algorithms to match the detection box and the trajectory in combination with the appearance features of the target. The specific steps are as follows:

[0158] Step S61, using the CMC algorithm to calculate the observation values ​​of the two time steps required for the motion direction in the OCM algorithm To compensate, this process is defined as follows:

[0159]

[0160] In the above formula, represents the estimated value of the target feature after compensation, c represents the original observed value, represents the motion compensation matrix, represents the motion compensation term, which compensates for the observations at two time steps required to calculate the motion direction.

[0161] Step S62, a B-IoU+ matching algorithm is proposed to match the detection box and the trajectory. The matching algorithm structure is as follows: Figure 6 The specific process is as follows:

[0162] Get the detection box information and existing trajectory information of the current frame.

[0163] For each detection box and trajectory prediction box, the buffer is constructed in the following way. The B-IoU+ matching algorithm border and buffer size transformation method is as follows:

[0164]

[0165]

[0166] In the above formula, o represents the border before expansion, x represents the horizontal coordinate of the upper left corner of the border, y represents the vertical coordinate of the upper left corner of the border, w represents the width of the border, and h represents the length of the border. represents the border after expansion, and b represents the expansion coefficient.

[0167] By calculating the IoU of the buffer instead of the IoU of the original border, it has a larger intersection area than the ordinary IoU, further reducing the problem of low intersection area caused by the disturbance of small area targets;

[0168] For the buffer of each trajectory prediction box and the buffer of each detection box, calculate the intersection-over-union ratio between them.

[0169] Assume the trajectory prediction box buffer coordinates are , the detection box buffer coordinates are ;

[0170] Calculate the intersection area , this process is defined as follows:

[0171]

[0172]

[0173]

[0174] Calculate the area of ​​the union , this process is defined as follows:

[0175]

[0176]

[0177]

[0178] The process of calculating B - IoU+ value is defined as follows:

[0179]

[0180] Set a suitable B-IoU+ threshold T;

[0181] For each track, calculate the B-IoU+ value between it and all detection boxes. When the B-IoU+ value between a detection box and the track is greater than the threshold T, the detection box is considered as a candidate matching box for the track.

[0182] For each detection box, calculate the B-IoU+ value between it and all tracks. When the B-IoU+ value between a track and the detection box is greater than the threshold T, the track is considered as a candidate matching track for the detection box.

[0183] When a detection box is a candidate matching box of a certain track in the forward matching, and the track is also a candidate matching track of the detection box in the reverse matching, it is determined that the detection box matches the track successfully.

[0184] Step S7: For the mismatched targets and trajectories in the cascade matching stage, the OCR matching algorithm is used to match the last observed value and the detected value of the trajectory. The specific steps are as follows:

[0185] Step S71, obtaining mismatched targets and trajectories in the cascade matching stage.

[0186] Step S72, using the CMC algorithm to update the observed values ​​in the OCR algorithm. The CMC algorithm is used for compensation, and the process is defined as follows:

[0187]

[0188] In the above formula, represents the estimated value of the target feature after compensation, c represents the original observed value, represents the motion compensation matrix, Represents the motion compensation term, which is used to compensate for the last observation value before the target is lost.

[0189] Step S73: introduce the B-IoU+ algorithm into the OCR matching algorithm. For the specific process formula, see step S5.

[0190] Step S8: Use the Kalman filter algorithm to update the status of the matching tracks obtained by cascade matching and OCR matching, delete the unmatched tracks in the mismatched tracks obtained by OCR matching, and output the target track prediction box and target ID after the status update. The specific steps are as follows:

[0191] Step S81 : for each matching trajectory obtained from cascade matching and OCR matching, initialize the state vector and covariance matrix of the Kalman filter.

[0192] Step S82, using the state prediction equation of Kalman filtering to predict the state of the matching trajectory. The state prediction equation of Kalman filtering is:

[0193]

[0194] In the above formula, is The state vector at the moment of prediction, yes The optimal estimated state vector at time , is the state transition matrix.

[0195] Step S83, predicting the covariance matrix using the covariance prediction equation:

[0196]

[0197] In the above formula, It is predicted The covariance matrix at time , yes The covariance matrix at time , is the process noise covariance matrix.

[0198] Step S84, perform measurement update according to the observation value corresponding to the matching trajectory, and first calculate the Kalman gain:

[0199] in is the measurement matrix, which relates the state vector to the measurement vector, is the measurement noise covariance matrix.

[0200] Step S85, update the status:

[0201]

[0202] Get in The optimal state estimate at time , and the covariance update , where I is the identity matrix.

[0203] Step S86, find the tracks that do not match the detection target in the mismatched tracks obtained by OCR matching. These mismatched tracks represent that the target has left the monitoring area, is blocked, or cannot be detected due to other reasons. Delete these mismatched tracks.

[0204] Step S87, updating the appearance model of each tracker to adapt to the changes in the target appearance.

[0205] Step S88, output the trajectory of each tracked target, including the target's ID, bounding box and confidence.

[0206] Obviously, the above steps of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modification, equivalent substitution and improvement made within the spirit and principle of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A multi-target tracking method for infrared unmanned aerial vehicles, characterized in that: include: Use the YOLOv8 target detector to obtain the target information in the current frame of the infrared camera video stream; Correcting the position information of the target according to the motion state of the camera by using a camera motion compensation method; The Kalman filter algorithm is used to predict the trajectories of all detected targets and determine whether the states of the trajectories are confirmed or unconfirmed; Through the pre-matching strategy, a pre-matching is performed before the appearance feature extraction to mark the targets with low matching degree; Through the lightweight feature re-identification network RepO-LightMBN, the appearance features of low-matching targets in video frames are extracted; The detected target and the confirmed trajectory are combined with the target's appearance feature distance and motion information through the B-IoU+ matching algorithm to calculate the cost matrix for cascade matching. The detection box and trajectory are matched using the B-IoU+ and OCM matching algorithms combined with the target's appearance features. For the mismatched targets and trajectories in the cascade matching stage, the OCR matching algorithm is used to match the last observation value and detection value of the trajectory; The Kalman filter algorithm is used to update the status of the matching tracks obtained by cascade matching and OCR matching, delete the unmatched tracks in the mismatched tracks obtained by OCR matching, and output the target track prediction box and target ID after the status update.

2. The infrared unmanned aerial vehicle multi-target tracking method as claimed in claim 1, characterized in that: The proposed pre-matching strategy consists of a pre-matching algorithm and a feature updating process. The pre-matching uses motion features to divide the target, and reduces the call of the feature extraction module for targets with obvious motion features, and vice versa. The specific process includes: The IoU between the predicted value and the observed value is used to represent the strength of the target's motion characteristics; Before feature extraction, perform an IoU match between the trajectory and the detection box; By setting a threshold, the only detection box with a high matching degree is set as a safe target, and the other targets are set as dangerous targets.

3. The infrared unmanned aerial vehicle multi-target tracking method as claimed in claim 1, characterized in that: The proposed lightweight feature re-identification network constructs the RepO-LightMBN feature re-identification network to extract target appearance features with low matching degree, and uses the collected data to train the feature re-identification network.

4. The infrared unmanned aerial vehicle multi-target tracking method as claimed in claim 3, characterized in that: The construction of RepO-LightMBN adopts Rep-OSNET as the backbone network, introduces a heavy parameterized structure, and further processes the output results of OSNet. Through three branches, the local, global, and channel characteristics of the target are extracted respectively, thereby enhancing the network's ability to extract information of different dimensions.

5. The infrared unmanned aerial vehicle multi-target tracking method as claimed in claim 4, characterized in that: The re-parameterized structure is introduced, and a re-parameterized unit SAG is set in the Rep-OSNET network. Through a unique multi-branch convolutional structure, the network's ability to extract and utilize features in infrared anti-UAV multi-target tracking scenarios is effectively improved. SAG has a multi-branch convolutional structure in the training stage, including an input layer, an average pooling layer, an MLP, an activation function Sigmoid and an output layer.

6. The infrared unmanned aerial vehicle multi-target tracking method according to claim 1, characterized in that: The proposed B-IoU+ matching algorithm includes a coordinate change method of a border and a buffer zone, and calculates the intersection-over-union ratio of the buffer zone to replace the intersection-over-union ratio of the original border. The intersection area is larger, and the low intersection area problem caused by the disturbance of a small-area target is further reduced.

7. The infrared unmanned aerial vehicle multi-target tracking method as claimed in claim 6, characterized in that: The B-IoU+ matching algorithm border and buffer size transformation method, this process is specifically defined as follows: o=(x,y,w,h) In the above formula, o represents the border before expansion, x represents the horizontal coordinate of the upper left corner of the border, y represents the vertical coordinate of the upper left corner of the border, w represents the width of the border, h represents the length of the border, and o represents the horizontal coordinate of the upper left corner of the border. ’ represents the border after expansion, b represents the expansion coefficient, the buffer size is expanded by the length of the largest side, and has the same expansion distance in the vertical and horizontal directions, which has a good matching effect for different movement directions of the target.

Images