Continuous frame dense infrared small target pseudo sample generation and hierarchical correlation tracking method

Abstract

The invention belongs to the technical field of target tracking, particularly relates to a continuous frame dense infrared small target pseudo sample generation and hierarchical association tracking method, and provides an efficient infrared pseudo sample generation normal form. A heterogeneous interactive degradation model (HIDM) and a random trajectory generation strategy are innovatively designed to generate a large amount of pseudo sample data simulating real infrared degradation, and serious dependence on manual data acquisition is avoided. The invention further provides a hierarchical association fusion tracking framework, YOLOV13 is used as a strong detector, a dynamic target density regulator, a mixed feature association engine and a trajectory continuity enhancement module are introduced into BoT-SORT, and the continuity and stability of a target id are kept; under the conditions that the hierarchical association tracking framework does not provide an initial target position and a Reid function is closed, various indexes of the hierarchical association tracking framework are superior to those of an existing method.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking. Background Art

[0002] Dense small target tracking in thermal infrared is a core task of intelligent perception and is urgently needed in areas such as border security, airspace control, and anti-UAV warfare. The difficulty in achieving robust recognition and continuous trajectory association of high-density, high-speed, maneuverable small UAV targets in complex airspace environments lies not only in the low resolution and low contrast inherent in infrared imaging, but also in the coupling of multiple sources of interference such as frequent target occlusion, strong motion nonlinearity, and high appearance similarity. Although various advanced trackers have made significant progress in visible light scenes, there are still two key bottlenecks in dense infrared band scenes. First, due to the cost of infrared sensors and the difficulty of data acquisition, the lack of real labeled samples leads to insufficient model generalization ability. Second, the problems of trajectory breakage and ID jump caused by frequent cross-occlusion of targets in dense scenes have not yet been systematically solved in the existing tracking paradigm.

[0003] To address these challenges, the present invention first provides an efficient infrared pseudo-sample generation paradigm, innovatively designs a heterogeneous interactive degradation model (HIDM) and a random trajectory generation strategy to generate a large amount of pseudo-sample data simulating real infrared degradation, thus getting rid of the severe dependence on artificial data acquisition; the present invention also proposes a hierarchical association fusion tracking framework (EnBoT-SORT). Specifically, we use YOLOV13 as a strong detector, introduce a dynamic target density regulator, a hybrid feature association engine, and trajectory continuity enhancement into BoT-SORT to maintain the continuity and stability of the target ID; and provide a closed-loop solution from data to algorithm for infrared dense small target tracking. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking, which solves the problems of scarcity of real labeled samples and high collection costs at the data level; and solves the difficult problem of frequent switching of identity tags caused by high-density target aggregation, frequent occlusion and reappearance at the algorithm level.

[0006] (2) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] The method for generating pseudo samples of dense infrared small targets in continuous frames and hierarchical association tracking specifically includes the following steps:

[0009] S1. Prepare background and target samples: Prepare IRT-B background sample set 1, IRC-B background sample set 2, and IRT-T target sample set. IRT-B and IRC-B basically cover a variety of real-world scenes such as the sky, forests, lakes, buildings, fences, and grasslands. IRT-T covers rotorcraft drones, helicopters, civil aircraft, and fighter jets. Therefore, background samples and target samples can be synthesized into pseudo samples with strong generalization properties.

[0010] S2, design a heterogeneous interactive degradation model (HIDM): including target degradation (HIDM-T), background degradation (HIDM-B) and local contrast interaction (HIDM-I); input the background samples and target samples prepared in step S1 into the HIDM to construct degraded samples; it should be noted that this step does not directly obtain the final degraded samples, but is combined with steps S4 and S5 to generate pseudo samples.

[0011] S3, design a multi-target trajectory generation mechanism: including designing a trajectory generation module and a boundary control function; the trajectory generation module is used to generate multi-target motion trajectories, and the boundary control function is used to prevent the target from transitioning out of bounds and disappearing from view for a long time.

[0012] S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2; adaptive weighted fusion strategy is used to fuse target and background samples with high fidelity.

[0013] S5, build a hierarchical correlation tracking framework: including adapting the YOLOV13 network and FastReid network, and designing a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancement module ; YOLOV13 is used to obtain motion features; FastReid is used to obtain appearance features; 、 and Embedded in BoT-SORT to build a hierarchical correlation tracking framework.

[0014] S6, network training and model solidification: including training the YOLOv13 network and the FastReid network, and solidifying the model parameters; inputting the pseudo samples obtained in step S4 into the YOLOv13 network for training to obtain the target detection model; inputting the IRT-T in step S1 into the FastReid network for training to obtain the appearance feature extraction model.

[0015] S7: Prepare test datasets: Prepare six test datasets for comprehensively testing the robustness and generalization performance of the pseudo sample generation method and tracking framework of the present invention; dataset one and dataset two are the IRT-B and IRC-B infrared datasets synthesized by the present invention, dataset three is the MultiUAV infrared dataset, dataset four is the Anti-UAV multimodal dataset, dataset five is the DroneDetection multimodal dataset, and dataset six is ​​the UAVSwarm multimodal dataset; multimodality refers to infrared and visible light modalities.

[0016] S8, testing and evaluating the hierarchical association tracking framework: including testing the hierarchical association tracking framework and evaluating the hierarchical association tracking framework; inputting the six continuous frame data sets prepared in step S7 into the hierarchical association framework, and outputting the coordinates, id and confidence of each target in each frame; the present invention uses high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity conversion IDs, trajectory fragmentation Frag, association accuracy AssA and association recall rate AssR to jointly evaluate the performance of the hierarchical association tracking framework.

[0017] S9, evaluate the quality of pseudo samples: the pseudo sample dataset generated in step S7 is then used to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

[0018] Furthermore, in S1, IRT-B background sample set 1 is a sample set taken by the present invention itself; IRC-B background sample set 2 is an existing background sample set; and IRT-T target sample set is a visible light sample set first obtained by the present invention in Unreal Engine 5, and then an infrared target sample set generated by the infrared sample generation algorithm of the present invention.

[0019] Furthermore, in said S1, said IRT-B background sample set 1 is a sample set taken by oneself; said IRC-B background sample set 2 is an existing background sample set; and said IRT-T target sample set is obtained by a designed infrared target sample generation algorithm;

[0020] The establishment process of the infrared target sample generation algorithm is as follows: first, 500 types of drone-type visible light target samples are collected from Unreal Engine 5, with the number of samples of each type of aircraft ranging from 20 to 25 and the resolution ranging from 20 to 60; then, random flipping and random resizing are used to increase the sample size of each type by 10 times; finally, Gaussian blur kernel and mask are used to simulate the random hot spot effect of infrared drones, so as to achieve the target generation algorithm from visible light. The conversion from infrared to infrared can be expressed as follows:

[0021]

[0022]

[0023] in, represents the binary mask generated by adaptive thresholding; express Random hot spot matrix; represents the Gaussian blur kernel; Represents element-wise multiplication; Indicates high fuzzy operation; Represents the input visible light samples; Represents a grayscale image in Gray value at ; Indicates the tolerance threshold, set to 10;

[0024] The final IRT-T target sample set contains 16,775 samples; in addition, in order to train FastReid, the IRT-T data format is edited into the standard Reid dataset file format, which contains 84,699 training samples, 16,038 test samples and 16,038 identity samples.

[0025] Furthermore, in S2, the heterogeneous interactive degradation model HIDM includes: target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I;

[0026] After the target sample is input into HIDM-T, it undergoes three degradation stages in sequence:

[0027] 1) Scale degradation: Generate a scale degradation curve based on the imaging distance between the target and the camera, so that the apparent size of the target in the image continuously decreases as the distance increases;

[0028] 2) Motion blur degradation: Calculating the camera's motion speed , based on which the motion blur effect of the target is generated. When the camera moves faster, this process will simulate the jagged effect of the target edge. The size and direction of the blur kernel are determined by Decide;

[0029] 3) Visibility degradation: The background variance and target variance of the local area where the target is located are calculated. Based on this, the apparent brightness of the target is adaptively adjusted so that it interacts with the background brightness. This ensures that the target brightness approaches the background brightness in brighter areas and is significantly higher than the background brightness in darker areas. However, the contrast is limited by physical constraints.

[0030] After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted to first analyze the temporal characteristics of the continuous background frame sequence and generate a description of the overall brightness and contrast of the scene over time. The degradation curve of the change is used to adaptively adjust the background sequence in segments to simulate the effects of light gradient, heat source movement, etc. Then, the same camera motion speed as HIDM-T is used. The corresponding global motion blur effect is applied to each frame of the background image based on the information; finally, the noise degradation that conforms to the characteristics of the infrared detector is superimposed on the blurred background frame.

[0031] Furthermore, in said S3, said multi-target trajectory generation mechanism includes: a trajectory generation module and a boundary control function;

[0032] The trajectory generation function of the trajectory generation module is defined as:

[0033]

[0034] in, Indicates the initial position of the target; Indicates the segment length of an image sequence; Indicates the image size; represents the set of real numbers; Represents a two-dimensional real space, where the input is the initial position coordinates on the two-dimensional plane ; T represents the length of the time series, Represents the two-dimensional coordinates output at each time point, specifically a T×2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be interpreted as: function The initial two-dimensional position and parameters Mapped into a trajectory sequence of time length T, the output is a two-dimensional array, each row corresponds to a time point coordinate;

[0035] We embed the moving target in each small sequence segment, and finally splice the data segments to obtain the complete image sequence data.

[0036] In order to ensure the complexity of the motion trajectory, each trajectory is defined as randomly selected from the linear model and the quadratic model. The linear model and the quadratic model are expressed using the formulas:

[0037]

[0038]

[0039] in, Express obedience Uniformly distributed simulated velocity components; represents the curvature parameter, which obeys evenly distributed;

[0040] The boundary control function , can be expressed as:

[0041]

[0042] Trigger elastic collision rules when detecting out of bounds , so that the target The relative speed of the reverse movement; finally generate a set of trajectories , each trajectory There are a series of moments Corresponding position .

[0043] Furthermore, in said S4, said target embedding and fusion strategy includes: an adaptive weighted fusion strategy and the HIDM-I implementation in step S2;

[0044] The target-background adaptive weighting strategy is specifically as follows: goals , first use it to normalize; then construct the spatial variation fusion coefficient matrix:

[0045]

[0046] in, Represents the target image In position The pixel value on ; Represents the indicator function, performing 0 and 1 operations; the fusion of target and background can be expressed as follows:

[0047]

[0048] in, Indicates the current trajectory position of the target;

[0049] The HIDM-I implementation is specifically as follows: when the background area variance Below target variance When a linear transformation is applied to the target ,in and denote contrast gain and brightness offset respectively.

[0050] Furthermore, in S5, the hierarchical association tracking framework includes: adapting the YOLOV13 network and the FastReid network; designing a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancer modules Embedded in BoT-SORT to build a hierarchical correlation tracking framework;

[0051] The dynamic target density adjuster : First, calculate the average distance of the scene target or the scene target density , and Represent the area of ​​the region of interest and the number of targets, respectively, and according to Setting a dynamic confidence threshold Filtering test results; dynamic threshold It can be defined by the following formula:

[0052]

[0053] in, Indicates the basic confidence threshold; adjustment coefficient pass and The interval segment is calculated together;

[0054] The hybrid feature association engine Aims to improve Bot-SORT and Direct correlation method, introducing trajectory reliability factor To achieve the most robust fusion of appearance and motion information in time sequence, the composite similarity matrix can be expressed as , where the trajectory reliability factor It can be expressed as:

[0055]

[0056] in, and represents the weight boundary; and They are used to control the transition boundaries respectively; Represents the sigmoid function; it can be calculated that Frame tracking initial ;exist The robust tracking stage of the frame , which shows that such a control process is particularly suitable for the tracking task in this paper;

[0057] The trajectory continuity enhancer module Use hybrid prediction model to predict lost targets; specifically, given the trajectory history position sequence ,in Indicates the The target position coordinates of the frame, calculate the average motion velocity vector :

[0058]

[0059] For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration Through the historical 5-frame trajectory fitting estimation, Express obedience Random perturbations of distribution; It is considered as a short-term frame loss. It is considered as a long-term frame loss when The target can be inapplicable Restore in case of .

[0060] Furthermore, in S6, the network training and model solidification include: training the YOLOV13 network and the FastReid network, and solidifying the model parameters.

[0061] Furthermore, in S7, the preparing of six test data sets includes: Dataset 1 and Dataset 2 are synthetic IRT-B and IRC-B infrared data sets, Dataset 3 is a MultiUAV infrared data set, Dataset 4 is an Anti-UAV multimodal data set, Dataset 5 is a DroneDetection multimodal data set, and Dataset 6 is a UAVSwarm multimodal data set;

[0062] The multimodality refers to infrared and visible light modalities.

[0063] Furthermore, in said S8, said testing and evaluating hierarchical correlation tracking framework includes a testing hierarchical correlation tracking framework and an evaluating hierarchical correlation tracking framework;

[0064] The evaluation indicators of the tracking framework include: high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR.

[0065] Furthermore, in S9, the evaluation of the quality of the pseudo samples is intended to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

[0066] (3) Beneficial effects

[0067] Compared with the existing technology, the present invention provides a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking, which has the following beneficial effects:

[0068] 1. This paper proposes a heterogeneous interactive degradation model (HIDM), which innovatively designs a three-stage heterogeneous interactive degradation chain of background and target in scale, motion, and visibility, and efficiently simulates the complex degradation process of dense infrared small target imaging in the real world.

[0069] 2. This paper proposes a complete pseudo-sample generation scheme for dense infrared small target continuous frames, which integrates the random trajectory generation model and the target adaptive embedding strategy to synthesize a high-fidelity dense infrared small target continuous frame dataset, making up for the problems of data scarcity and sample unreality in this field.

[0070] 3. This paper proposes a hierarchical association framework, innovatively introducing a dynamic density regulator, a hybrid feature association engine, and a trajectory continuity enhancement module. Without relying on the ReID module and initial labeling, it still maintains stable ID continuity and lost target recovery performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of the method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking of the present invention;

[0072] Figure 2 Generate a schematic diagram for the infrared target sample designed for the present invention;

[0073] Figure 3 Schematic diagram of the heterogeneous interactive degradation model (HIDM) designed for the present invention;

[0074] Figure 4 Four different multi-target trajectory maps generated on a set of image sequences for visualization in the present invention;

[0075] Figure 5 The following are the generalization experimental results of the present invention on two multimodal datasets;

[0076] Figure 6 This is the quantitative comparison result between the present invention and various existing tracking methods. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0078] Example:

[0079] like Figure 1FIG. 1 is a flowchart of a method for generating dense infrared small target pseudo samples in consecutive frames and performing hierarchical association tracking according to an embodiment of the present invention. The method specifically includes the following steps:

[0080] S1, prepare background and target sample sets: prepare IRT-B background sample set 1, IRC-B background sample set 2 and IRT-T target sample set.

[0081] The IRT-B sample set is a collection of ground-to-air images captured by the present invention using a long-wave infrared camera. It includes 10 scenes, including sky, forest, lake, building, fence, and grassland. The sky area accounts for the largest proportion in each scene. The IRT-B is divided into training, validation, and test sets in a ratio of 6:2:2; the image resolution is 640×512. The total number of images in the original sequence is approximately 80,000. The present invention removes samples without targets in the long time series. The final IRT-B contains 25 segments of data, a total of 33,606 real images for training, 8,067 for testing, and 9,655 for testing.

[0082] The IRC-B sample set is used to expand the background sample library. The existing generative infrared point target dataset is used. Since it meets the following three conditions, it can also be used for the second background sample library of the present invention: 1) The scenes are very rich, including various scenes such as sky, water surface, vegetation, buildings, etc.; 2) It includes multi-perspective embedded drone conditions for looking up, looking straight, and looking down; 3) The weak point targets are basically invisible after HIDM degradation and can be regarded as equivalent to clean background sample images; the present invention keeps the 175 training sequences of the existing IRC-B unchanged, and uses the first 150 sequences of the IRC-B test sequence as the verification sequence, and the last 25 sequences as the test sequence; IRC-B contains 350 data sequences, and the training subset and test subset each contain 175 data sequences, totaling 150,185 frames of images.

[0083] The IRT-T target sample set is a simulated target sample. Figure 2 The target sample set establishment process shown in the figure can be used as an example of infrared target sample generation task. Specifically, the present invention first collects 500 types of drone-type visible light target samples from Unreal Engine 5. The number of samples of each type of aircraft is between 20-25 and the resolution is between 20-60. Then, the samples of each type are enlarged by 10 times using random flipping and random resizing. Finally, the random hot spot effect of the infrared drone is simulated using Gaussian blur kernel and mask to achieve the target sample generation task from visible light. The conversion from infrared to infrared can be expressed as follows:

[0084]

[0085]

[0086] in, represents the binary mask generated by adaptive thresholding; express Random hot spot matrix; represents the Gaussian blur kernel; Represents element-wise multiplication; Indicates high fuzzy operation; Represents the input visible light samples; Represents a grayscale image in Gray value at ; Indicates the tolerance threshold, set to 10.

[0087] The final IRT-T target sample set contains 16,775 samples; in addition, in order to train FastReid, the present invention edits the IRT-T data format into the standard Reid dataset file format, which contains 84,699 training samples, 16,038 test samples and 16,038 identity samples.

[0088] S2, design heterogeneous interactive degradation model (HIDM): including target degradation (HIDM-T), background degradation (HIDM-B) and local contrast interaction (HIDM-I); Figure 3 As shown, unlike the traditional single-frame image degradation process, the present invention adopts a segmented degradation strategy for continuous video frames and achieves physically consistent degradation effects through key interactions.

[0089] After the target sample is input into HIDM-T, it undergoes three degradation stages in sequence: 1) Scale degradation: The present invention generates a scale degradation curve based on the imaging distance between the target and the camera, and controls the apparent size of the target in the image to continuously decrease with increasing distance; 2) Motion blur degradation: The present invention calculates the motion speed of the camera. , based on which the motion blur effect of the target is generated. When the camera moves faster, this process will simulate the jagged effect of the target edge. The size and direction of the blur kernel are determined by 3) Visibility degradation: Calculate the background variance and target variance of the local area where the target is located, and adaptively adjust the apparent brightness of the target based on this, so that it interacts with the background brightness, ensuring that the target brightness is close to the background brightness in areas with brighter backgrounds and significantly higher than the background brightness in areas with darker backgrounds, but the contrast is subject to physical limitations.

[0090] After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted. First, the temporal characteristics of the continuous background frame sequence are analyzed to generate a description of the overall brightness and contrast of the scene over time. The degradation curve of the change is used to adaptively adjust the background sequence in segments to simulate the effects of light gradient, heat source movement, etc. Then, the same camera motion speed as HIDM-T is used. The HIDM algorithm uses the information from the image to apply a corresponding global motion blur effect to each background frame. Finally, it superimposes noise degradation that matches the characteristics of the infrared detector on the blurred background frame. HIDM simulates the most realistic degradation of consecutive frames of real infrared images.

[0091] S3, design a multi-target trajectory generation mechanism: including designing a trajectory generation module and a boundary control function; the number of targets generated by the present invention, the starting point of the target trajectory and the target library ID are all random; Figure 4 Shown are four target trajectories visualized on the same video frame sequence by the present invention, which are complex and random.

[0092] First, define the trajectory generation function as:

[0093]

[0094] in, Indicates the initial position of the target; Indicates the segment length of an image sequence; Indicates the image size; represents the set of real numbers; Represents a two-dimensional real space, where the input is the initial position coordinates on the two-dimensional plane ; T represents the length of the time series, Represents the two-dimensional coordinates output at each time point, specifically a T×2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be interpreted as: function The initial two-dimensional position and parameters Mapped into a trajectory sequence of time length T, the output is a two-dimensional array, each row corresponds to a time point coordinate;

[0095] We embed the moving target in each small sequence segment, and finally splice the data segments to obtain the complete image sequence data.

[0096] In order to ensure the complexity of the motion trajectory, the present invention defines that each trajectory is randomly selected and generated from a linear model and a quadratic model. The linear model and the quadratic model are respectively expressed using the formula:

[0097]

[0098]

[0099] in, Express obedience Uniformly distributed simulated velocity components; represents the curvature parameter, which obeys Evenly distributed.

[0100] In order to prevent the target from crossing the boundary and disappearing from the field of view for a long time, the present invention uses a dynamic boundary detection function To suppress this phenomenon, the formula can be expressed as:

[0101]

[0102] Trigger elastic collision rules when detecting out of bounds , so that the target The relative speed of the reverse movement; finally generate a set of trajectories , each trajectory There are a series of moments Corresponding position .

[0103] S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2;

[0104] In the target embedding stage, the present invention adopts the target-background adaptive weighting strategy; goals , first normalize using . Then construct the spatial variation fusion coefficient matrix:

[0105]

[0106] in, Represents the target image In position The pixel value on ; Represents the indicator function, performing 0 and 1 operations; the fusion of target and background can be expressed as follows:

[0107]

[0108] in, Indicates the current trajectory position of the target.

[0109] In order to enhance the physical reality, the present invention introduces the local contrast comparison mechanism HIDM-I in combination with HIDM. Below target variance When a linear transformation is applied to the target ,in and denote contrast gain and brightness offset respectively.

[0110] S5, build a hierarchical correlation tracking framework: including adapting the YOLOV13 network and FastReid network; design a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancer modules ; YOLOV13 is used to obtain motion features; FastReid is used to obtain appearance features; 、 and Embedded in BoT-SORT to build a hierarchical correlation tracking framework.

[0111] The BoT-SORT algorithm uses a static confidence threshold that is difficult to adapt to the non-uniformity of target spatial distribution, which will lead to a high rate of missed detection in sparse target areas and a large number of false detections in high-density areas. This rigid filtering strategy essentially ignores the inherent relationship between the dynamic complexity of the scene and the filtering confidence. Therefore, the present invention adopts An adaptive mapping relationship between scene complexity and confidence is established; specifically, the average distance of scene targets or the density of scene targets is first calculated. , and Represent the area of ​​the region of interest and the number of targets, respectively, and according to Setting a dynamic confidence threshold Filtering test results; dynamic threshold It can be defined by the following formula:

[0112]

[0113] in, Indicates the basic confidence threshold; adjustment coefficient pass and The interval segment is calculated together; When the target density is high, increase To avoid too many false positive detections; when the target density is low, reduce To ensure that the target is fully detected, this dynamic adjustment improves the tracking performance of the tracker under different scene conditions.

[0114] In the case of BoT-SORT opening reid, through the motion similarity matrix The composite similarity matrix constructed with the appearance similarity matrix can be expressed as ,in, and It is a fixed fusion parameter. Since the appearance characteristics of small targets of infrared UAV do not change significantly, we need to pay more attention to the appearance characteristics of short distances in the initial tracking period and the long-distance motion characteristics in the tracking stability period. and Not suitable for infrared small drone tracking tasks.

[0115] Therefore, the present invention improves the Bot-SORT and Direct correlation method, introducing trajectory reliability factor To achieve the most robust fusion of appearance and motion information in time sequence, the composite similarity matrix can be expressed as , where the trajectory reliability factor It can be expressed as:

[0116]

[0117] in, and represents the weight boundary; and They are used to control the transition boundaries respectively; Represents the sigmoid function; it can be calculated that Frame tracking initial ;exist The robust tracking stage of the frame , indicating that such a control process is particularly suitable for the tracking task in this paper.

[0118] The target loss problem caused by the recovery of high-speed moving targets after occlusion is inevitable, and the single linear interpolation method of Kalman filtering will introduce significant trajectory association errors and state estimation bias in high-density target scenes; therefore, the present invention uses a hybrid prediction model to predict lost targets. Specifically, given a trajectory history position sequence ,in Indicates the The target position coordinates of the frame, calculate the average motion velocity vector :

[0119]

[0120] For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration Through the historical 5-frame trajectory fitting estimation, Express obedience Random perturbations of distribution; It is considered as a short-term frame loss. It is considered as a long-term frame loss when The target can be inapplicable Restore in case of .

[0121] S6, network training and model solidification: including training the YOLOv13 network and the FastReid network, and solidifying the model parameters; inputting the pseudo samples obtained in step S4 into the YOLOv13 network for training to obtain the target detection model; inputting the IRT-T in step S1 into the FastReid network for training to obtain the appearance feature extraction model.

[0122] S7: Prepare test datasets: Prepare six test datasets for comprehensively testing the robustness and generalization performance of the pseudo sample generation method and tracking framework of the present invention; dataset one and dataset two are the IRT-B and IRC-B infrared datasets synthesized by the present invention, dataset three is the MultiUAV infrared dataset, dataset four is the Anti-UAV multimodal dataset, dataset five is the DroneDetection multimodal dataset, and dataset six is ​​the UAVSwarm multimodal dataset; multimodality refers to infrared and visible light modalities.

[0123] The Anti-UAV dataset contains multimodal real-world drone tracking image sequences. Using a variety of commercially available small drone models, it collects and annotates over 300 visible light / infrared video pairs (containing over 580,000 targets), covering both daytime and nighttime conditions, including complex backgrounds such as clouds, buildings, and jungles, as well as challenges posed by false targets such as flying birds and airborne objects. This dataset is highly suitable for verifying the generalization capabilities of the tracker model trained on pseudo samples.

[0124] The DroneDetection dataset contains infrared images, visible light images, and audio data of four different types of objects: airplanes, birds, drones, and helicopters. This paper assumes that, except for birds, all other flying targets can be considered small drones if their size is small enough, and the method of this paper can be generalized to tracking these targets.

[0125] The UAVSwarm dataset collects many drone swarm videos for drone swarm detection and tracking. UAVSwarm records 13 different scenes and more than 19 types of drones, including 12,598 annotated images. The number of drones in each sequence ranges from 3 to 23, which is very suitable for testing the performance of the tracking method proposed in this paper on visible light datasets.

[0126] S8, testing and evaluating the hierarchical association tracking framework: including testing the hierarchical association tracking framework and evaluating the hierarchical association tracking framework; inputting the six continuous frame data sets prepared in step S7 into the hierarchical association framework, and outputting the coordinates, id and confidence of each target in each frame; the present invention uses high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, multi-target tracking accuracy MOTP, association accuracy AssA and association recall rate AssR to jointly evaluate the performance of the hierarchical association tracking framework.

[0127] MOTA, FN, FP, IDs, and Frag can be collectively categorized as the CLEAR metric. HOTA more evenly considers detection accuracy and identity preservation. MOTA accounts for all object tracking errors (including FP, FN, and IDs) incurred by the tracker across all frames. IDs are a key metric for measuring the consistency of the tracker's target identity. MOTP measures the accuracy of the algorithm's estimation of the target location (e.g., bounding box, center point), specifically the quality of the match between the predicted box and the ground-truth box.

[0128] S9, evaluate the quality of pseudo samples: the pseudo sample dataset generated in step S7 is then used to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

[0129] This paper studies the two difficult problems of data and algorithm in the field of thermal infrared dense UAV tracking, and proposes a continuous frame dense infrared small target pseudo sample generation and hierarchical association tracking method; in the pseudo sample generation method, the heterogeneous interactive degradation model (HIDM) interactively reconstructs the infrared background and target real degradation through the three-stage physical degradation chain of scale-motion-visibility, and combines random multi-target trajectory generation and adaptive image fusion to generate IRT-B and IRC-B high-fidelity pseudo sample datasets. In the hierarchical association framework, the dynamic target density regulator, hybrid feature association engine (with re-ID enabled) and trajectory continuity enhancement are innovatively integrated, and the relevant indicators of various existing tracking methods are compared. Figure 6 As shown in Figure 2, with re-id disabled, HOTA achieved 68.7% / 67.3% (2.9 / 3.0 higher than BoT-SORT) and MOTA achieved 76.2% / 74.6% (2.8 / 3.1 higher than BoT-SORT). Figure 5As shown, the present invention also verifies the universality and reliability of pseudo sample data and hierarchical association tracking framework through generalization experiments on multiple cross-modal datasets, providing a new paradigm of data-algorithm collaborative optimization for dense UAV tracking. The meaning of the annotations above the detection box is: id: 1 indicates that the number of targets in this frame is 1, tar indicates the target category, and the last digit indicates the confidence level.

[0130] At the same time, the pseudo-sample generation method of the present invention still needs to be improved in the physical modeling of simulating extreme weather conditions (such as heavy rain and dense fog), and the mining of temporal motion features needs to be improved. The next step is to introduce an atmospheric scattering model to enhance the degradation authenticity and integrate optical flow information to enhance the short-term occlusion processing capability.

[0131] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for generating pseudo samples of dense infrared small targets in continuous frames and hierarchical association tracking, characterized by: The following steps are involved: S1, prepare background and target samples: prepare IRT-B background sample set 1, IRC-B background sample set 2 and IRT-T target sample set; S2, design heterogeneous interactive degradation model HIDM: including target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I; S3, design of multi-target trajectory generation mechanism: including design of trajectory generation module and boundary control function; S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2; S5, build a hierarchical correlation tracking framework: including adapting the YOLOV13 network and FastReid network, and designing a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancement module ; S6, network training and model solidification: including training YOLOV13 network and FastReid network, and solidifying model parameters; S7: Prepare test datasets: Prepare six test datasets; S8, test and evaluate the hierarchical association tracking framework: use high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity conversion IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR to jointly test and evaluate the performance of the hierarchical association tracking framework; S9, evaluate the quality of pseudo samples: use motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8 to measure the quality of the generated pseudo samples.

2. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In S1, the IRT-B background sample set 1 is a self-photographed sample set; the IRC-B background sample set 2 is an existing background sample set; the IRT-T target sample set is obtained by a designed infrared target sample generation algorithm; The establishment process of the infrared target sample generation algorithm is as follows: first, 500 types of drone-type visible light target samples are collected from Unreal Engine 5, with the number of samples of each type of aircraft ranging from 20 to 25 and the resolution ranging from 20 to 60; then, random flipping and random resizing are used to increase the sample size of each type by 10 times; finally, Gaussian blur kernel and mask are used to simulate the random hot spot effect of infrared drones, so as to achieve the target generation algorithm from visible light. The conversion from infrared to infrared can be expressed as follows: ； ； in, represents the binary mask generated by adaptive thresholding; express Random hot spot matrix; represents the Gaussian blur kernel; Represents element-wise multiplication; Indicates high fuzzy operation; Represents the input visible light samples; Represents a grayscale image in Gray value at ; Indicates the tolerance threshold, set to 10; The final IRT-T target sample set contains 16,775 samples; in addition, in order to train FastReid, the IRT-T data format is edited into the standard Reid dataset file format, which contains 84,699 training samples, 16,038 test samples and 16,038 identity samples.

3. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In S2, the heterogeneous interactive degradation model HIDM includes: target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I; After the target sample is input into HIDM-T, it undergoes three degradation stages in sequence: 1) Scale degradation: Generate a scale degradation curve based on the imaging distance between the target and the camera, so that the apparent size of the target in the image continuously decreases as the distance increases; 2) Motion blur degradation: Calculating the camera's motion speed , based on which the motion blur effect of the target is generated. When the camera moves faster, this process will simulate the jagged effect of the target edge. The size and direction of the blur kernel are determined by Decide; 3) Visibility degradation: The background variance and target variance of the local area where the target is located are calculated. Based on this, the apparent brightness of the target is adaptively adjusted so that it interacts with the background brightness. This ensures that the target brightness approaches the background brightness in brighter areas and is significantly higher than the background brightness in darker areas. However, the contrast is limited by physical constraints. After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted to first analyze the temporal characteristics of the continuous background frame sequence and generate a description of the overall brightness and contrast of the scene over time. The degradation curve of the change is used to adaptively adjust the background sequence in segments to simulate the effects of light gradient, heat source movement, etc. Then, the same camera motion speed as HIDM-T is used. The corresponding global motion blur effect is applied to each frame of the background image based on the information; finally, the noise degradation that conforms to the characteristics of the infrared detector is superimposed on the blurred background frame.

4. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In said S3, said multi-target trajectory generation mechanism includes: a trajectory generation module and a boundary control function; The trajectory generation function of the trajectory generation module is defined as: ； in, Indicates the initial position of the target; Indicates the segment length of an image sequence; Indicates the image size; represents the set of real numbers; Represents a two-dimensional real space, where the input is the initial position coordinates on the two-dimensional plane ; T represents the length of the time series, Represents the two-dimensional coordinates output at each time point, specifically a T×2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be interpreted as: function The initial two-dimensional position and parameters Mapped into a trajectory sequence of time length T, the output is a two-dimensional array, each row corresponds to a time point coordinate; We embed the moving target in each small sequence segment, and finally splice the data segments to obtain the complete image sequence data; In order to ensure the complexity of the motion trajectory, each trajectory is defined as randomly selected from the linear model and the quadratic model. The linear model and the quadratic model are expressed using the formulas: ； ； in, Express obedience Uniformly distributed simulated velocity components; represents the curvature parameter, which obeys evenly distributed; The boundary control function , can be expressed as: ； Trigger elastic collision rules when detecting out of bounds , so that the target The relative speed of the reverse movement; finally generate a set of trajectories , each trajectory There are a series of moments Corresponding position .

5. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In said S4, said target embedding and fusion strategy includes: an adaptive weighted fusion strategy and the HIDM-I implementation in step S2; The target-background adaptive weighting strategy is specifically as follows: goals , first use it to normalize; then construct the spatial variation fusion coefficient matrix: ； in, Represents the target image In position The pixel value on ; Represents the indicator function, performing 0 and 1 operations; the fusion of target and background can be expressed as follows: ； in, Indicates the current trajectory position of the target; The HIDM-I implementation is specifically as follows: when the background area variance Below target variance When a linear transformation is applied to the target ,in and denote contrast gain and brightness offset respectively.

6. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In S5, the hierarchical association tracking framework includes: adapting the YOLOV13 network and the FastReid network; designing a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancer modules Embedded in BoT-SORT to build a hierarchical correlation tracking framework; The dynamic target density adjuster : First, calculate the average distance of the scene target or the scene target density , and Represent the area of ​​the region of interest and the number of targets, respectively, and according to Setting a dynamic confidence threshold Filtering test results; dynamic threshold It can be defined by the following formula: ； in, Indicates the basic confidence threshold; adjustment coefficient pass and The interval segment is calculated together; The hybrid feature association engine Aims to improve Bot-SORT and Direct correlation method, introducing trajectory reliability factor To achieve the most robust fusion of appearance and motion information in time sequence, the composite similarity matrix can be expressed as , where the trajectory reliability factor It can be expressed as: ； in, and represents the weight boundary; and They are used to control the transition boundaries respectively; Represents the sigmoid function; it can be calculated that Frame tracking initial ;exist The robust tracking stage of the frame , which shows that such a control process is particularly suitable for the tracking task in this paper; The trajectory continuity enhancer module Use hybrid prediction model to predict lost targets; specifically, given the trajectory history position sequence ,in Indicates the The target position coordinates of the frame, calculate the average motion velocity vector : ； For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration Through the historical 5-frame trajectory fitting estimation, Express obedience Random perturbations of distribution; It is considered as a short-term frame loss. It is considered as a long-term frame loss when The target can be inapplicable Restore in case of .

7. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In S6, the network training and model solidification include: training the YOLOV13 network and the FastReid network, and solidifying the model parameters.

8. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In S7, the six test datasets are prepared: dataset 1 and dataset 2 are synthetic IRT-B and IRC-B infrared datasets, dataset 3 is a MultiUAV infrared dataset, dataset 4 is an Anti-UAV multimodal dataset, dataset 5 is a DroneDetection multimodal dataset, and dataset 6 is a UAVSwarm multimodal dataset; The multimodality refers to infrared and visible light modalities.

9. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In said S8, said testing and evaluating hierarchical correlation tracking framework includes a testing hierarchical correlation tracking framework and an evaluating hierarchical correlation tracking framework; The evaluation indicators of the tracking framework include: high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR.

10. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In said S9, the evaluation of the quality of the pseudo samples is intended to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

Images