Continuous frame dense infrared small target pseudo sample generation and hierarchical correlation tracking method

Abstract

The present application belongs to the technical field of target tracking, and especially is a continuous frame dense infrared small target pseudo sample generation and hierarchical correlation tracking method. The present application is a continuous frame dense infrared small target pseudo sample generation and hierarchical correlation tracking method, which firstly provides an efficient infrared pseudo sample generation paradigm, and innovatively designs a heterogeneous interactive degradation model (HIDM) and a random trajectory generation strategy to generate a large number of pseudo sample data simulating real infrared degradation, thereby getting rid of the serious dependence on artificial data acquisition. The present application also proposes a hierarchical correlation fusion tracking framework, uses YOLOV13 as a strong detector, introduces a dynamic target density regulator, a mixed feature correlation engine and a trajectory continuity enhancement module into BoT-SORT to maintain the continuity and stability of the target id, and the hierarchical correlation tracking framework is superior to the existing method in various indicators without providing the initial target position and closing the Reid function.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking. Background Art

[0002] Dense small target tracking in thermal infrared is a core task of intelligent perception and is urgently needed in areas such as border security, airspace control, and anti-UAV warfare. The difficulty in achieving robust recognition and continuous trajectory association of high-density, high-speed, maneuverable small UAV targets in complex airspace environments lies not only in the low resolution and low contrast inherent in infrared imaging, but also in the coupling of multiple sources of interference such as frequent target occlusion, strong motion nonlinearity, and high appearance similarity. Although various advanced trackers have made significant progress in visible light scenes, there are still two key bottlenecks in dense infrared band scenes. First, due to the cost of infrared sensors and the difficulty of data acquisition, the lack of real labeled samples leads to insufficient model generalization ability. Second, the problems of trajectory breakage and ID jump caused by frequent cross-occlusion of targets in dense scenes have not yet been systematically solved in the existing tracking paradigm.

[0003] To address these challenges, the present invention first provides an efficient infrared pseudo-sample generation paradigm, innovatively designs a heterogeneous interactive degradation model (HIDM) and a random trajectory generation strategy to generate a large amount of pseudo-sample data simulating real infrared degradation, thus getting rid of the severe dependence on artificial data acquisition; the present invention also proposes a hierarchical association fusion tracking framework (EnBoT-SORT). Specifically, we use YOLOV13 as a strong detector, introduce a dynamic target density regulator, a hybrid feature association engine, and trajectory continuity enhancement into BoT-SORT to maintain the continuity and stability of the target ID; and provide a closed-loop solution from data to algorithm for infrared dense small target tracking. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking, which solves the problems of scarcity of real labeled samples and high collection costs at the data level; and solves the difficult problem of frequent switching of identity tags caused by high-density target aggregation, frequent occlusion and reappearance at the algorithm level.

[0006] (2) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] The method for generating pseudo samples of dense infrared small targets in continuous frames and hierarchical association tracking specifically includes the following steps:

[0009] S1. Prepare background and target samples: Prepare IRT-B background sample set 1, IRC-B background sample set 2, and IRT-T target sample set. IRT-B and IRC-B basically cover a variety of real-world scenes such as the sky, forests, lakes, buildings, fences, and grasslands. IRT-T covers rotorcraft drones, helicopters, civil aircraft, and fighter jets. Therefore, background samples and target samples can be synthesized into pseudo samples with strong generalization properties.

[0010] S2, design a heterogeneous interactive degradation model (HIDM): including target degradation (HIDM-T), background degradation (HIDM-B) and local contrast interaction (HIDM-I); input the background samples and target samples prepared in step S1 into the HIDM to construct degraded samples; it should be noted that this step does not directly obtain the final degraded samples, but is combined with steps S4 and S5 to generate pseudo samples.

[0011] S3, design a multi-target trajectory generation mechanism: including designing a trajectory generation module and a boundary control function; the trajectory generation module is used to generate multi-target motion trajectories, and the boundary control function is used to prevent the target from transitioning out of bounds and disappearing from view for a long time.

[0012] S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2; adaptive weighted fusion strategy is used to fuse target and background samples with high fidelity.

[0013] S5, build a hierarchical correlation tracking framework: including adapting the YOLOV13 network and FastReid network, and designing a dynamic target density regulator , hybrid feature association engine and trajectory continuity enhancement module ; YOLOV13 is used to obtain motion features; FastReid is used to obtain appearance features; 、 and Embedded in BoT-SORT to build a hierarchical correlation tracking framework.

[0014] S6, network training and model solidification: including training the YOLOv13 network and the FastReid network, and solidifying the model parameters; inputting the pseudo samples obtained in step S4 into the YOLOv13 network for training to obtain the target detection model; inputting the IRT-T in step S1 into the FastReid network for training to obtain the appearance feature extraction model.

[0015] S7: Prepare test data sets: Prepare six test data sets for comprehensive testing of the robustness and generalization performance of the pseudo sample generation method and tracking framework of the present application; data set one and data set two are IRT-B and IRC-B infrared data sets synthesized by the present application, data set three is a MultiUAV infrared data set, data set four is an Anti-UAV multi-modal data set, data set five is a DroneDetection multi-modal data set, and data set six is a UAVSwarm multi-modal data set; multi-modal refers to infrared and visible light modalities.

[0016] S8, test and evaluate the hierarchical association tracking framework: including testing the hierarchical association tracking framework and evaluating the hierarchical association tracking framework; inputting the six consecutive frame data sets prepared in step S7 into the hierarchical association framework, and outputting the coordinates, id and confidence of each target in each frame; the present application uses high-order tracking precision HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transition IDs, track fragmentation Frag, association accuracy AssA and association recall AssR to jointly evaluate the performance of the hierarchical association tracking framework.

[0017] S9, evaluate the quality of pseudo samples: further use motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and part of the tracking indicators in step S8 to measure the quality of the generated pseudo samples.

[0018] Further, in S1, the IRT-B background sample set one is a sample set self-shot by the present application; the IRC-B background sample set two is an existing background sample set; the IRT-T target sample set is a visible light sample set first obtained in Unreal Engine 5, and then an infrared target sample set generated by the infrared sample generation algorithm of the present application.

[0019] Further, in S1, the IRT-B background sample set one is a sample set self-shot by the present application; the IRC-B background sample set two is an existing background sample set; the IRT-T target sample set is a designed infrared target sample generation algorithm;

[0020] The establishment process of the infrared target sample generation algorithm is as follows: first, 500 types of unmanned aerial vehicle visible light target samples are collected from Unreal Engine 5, with 20-25 samples for each type of aircraft and a resolution of 20-60; then, each type of sample is augmented by 10 times using random flipping and random size transformation; finally, the random hot spot effect of infrared unmanned aerial vehicles is simulated using a Gaussian blur kernel and a mask to realize the conversion from visible light to infrared. This process can be represented by the formula:

[0021]

[0022]

[0023] in, represents the binary mask generated by adaptive thresholding; express Random hot spot matrix; represents the Gaussian blur kernel; Represents element-wise multiplication; Indicates high fuzzy operation; Represents the input visible light samples; Represents a grayscale image in Gray value at ; Indicates the tolerance threshold, set to 10;

[0024] The final IRT-T target sample set contains 16,775 samples; in addition, in order to train FastReid, the IRT-T data format is edited into the standard Reid dataset file format, which contains 84,699 training samples, 16,038 test samples and 16,038 identity samples.

[0025] Furthermore, in S2, the heterogeneous interactive degradation model HIDM includes: target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I;

[0026] After the target sample is input into HIDM-T, it undergoes three degradation stages in sequence:

[0027] 1) Scale degradation: Generate a scale degradation curve based on the imaging distance between the target and the camera, so that the apparent size of the target in the image continuously decreases as the distance increases;

[0028] 2) Motion blur degradation: Calculating the camera's motion speed , based on which the motion blur effect of the target is generated. When the camera moves faster, this process will simulate the jagged effect of the target edge. The size and direction of the blur kernel are determined by Decide;

[0029] 3) Visibility degradation: The background variance and target variance of the local area where the target is located are calculated. Based on this, the apparent brightness of the target is adaptively adjusted so that it interacts with the background brightness. This ensures that the target brightness approaches the background brightness in brighter areas and is significantly higher than the background brightness in darker areas. However, the contrast is limited by physical constraints.

[0030] After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted to first analyze the temporal characteristics of the continuous background frame sequence and generate a description of the overall brightness and contrast of the scene over time. The degradation curve is changed, and the background sequence is adaptively adjusted in segments to simulate the effects of light gradient and heat source movement, etc. Information, the corresponding global motion blur effect is applied to each frame of background image; finally, the noise degradation conforming to the characteristics of infrared detector is superimposed on the blurred background frame.

[0031] Further, in the S3, the multi-target trajectory generation mechanism comprises a trajectory generation module and a boundary control function.

[0032] The trajectory generation function of the trajectory generation module is defined as:

[0033]

[0034] Wherein, represents the initial position of the target; represents the segment length of an image sequence; represents the image size; represents a set of real numbers; represents a two-dimensional real space, which specifically refers to the initial position coordinates on a two-dimensional plane ; T represents the length of the time sequence, represents the two-dimensional coordinates output at each time point, which specifically refers to a Tx2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be explained as: the function maps the initial two-dimensional position and the parameter to a trajectory sequence with a time length of T, and the output is a two-dimensional array, each row of which corresponds to the coordinates of a time point;

[0035] We embed the moving target in each small sequence segment, and finally splice the data segments to obtain complete image sequence data.

[0036] In order to ensure the complexity of the motion trajectory, each trajectory is randomly selected from a linear model and a quadratic model, and the linear model and the quadratic model are represented by the following formulas respectively:

[0037]

[0038]

[0039] Wherein, represents a simulated speed component conforming to uniform distribution; represents a curvature parameter, which conforms to uniform distribution;

[0040] The boundary control function , can be expressed as:

[0041]

[0042] Triggering elastic collision rules when detecting out-of-bound , the target moves in reverse at a relative speed of ; finally generate a set of track trajectories , each track has a series of time corresponding positions .

[0043] Further, in the S4, the target embedding and fusion strategy includes: adaptive weighted fusion strategy and HIDM-I implementation in step S2;

[0044] The adaptive weighted strategy of the target-background is specifically: for the first target , first use to normalize; then construct a spatially varying fusion coefficient matrix:

[0045]

[0046] Where, denotes the pixel value of the target image at position ; denotes the indicator function, which performs 0 and 1 operations; the fusion of the target and the background can be expressed as:

[0047]

[0048] Where, denotes the current track position of the target;

[0049] The HIDM-I implementation is specifically: when the background region variance is lower than the target variance , a linear transformation is applied to the target, where and respectively represent the contrast gain and brightness offset.

[0050] Further, in the S5, the hierarchical correlation tracking framework includes: adapting YOLOV13 network and FastReid network; design dynamic target density regulator , mixed feature correlation engine and track continuity enhancer module embedded in BoT-SORT to build a hierarchical correlation tracking framework;

[0051] The dynamic target density adjuster : First, calculate the average distance of the scene target or the scene target density , and Represent the area of ​​the region of interest and the number of targets, respectively, and according to Setting a dynamic confidence threshold Filtering test results; dynamic threshold It can be defined by the following formula:

[0052]

[0053] in, Indicates the basic confidence threshold; adjustment coefficient pass and The interval segment is calculated together;

[0054] The hybrid feature association engine Aims to improve Bot-SORT and Direct correlation method, introducing trajectory reliability factor To achieve the most robust fusion of appearance and motion information in time sequence, the composite similarity matrix can be expressed as , where the trajectory reliability factor It can be expressed as:

[0055]

[0056] in, and represents the weight boundary; and They are used to control the transition boundaries respectively; Represents the sigmoid function; it can be calculated that Frame tracking initial ;exist The robust tracking stage of the frame , which shows that such a control process is particularly suitable for the tracking task in this paper;

[0057] The trajectory continuity enhancer module Use hybrid prediction model to predict lost targets; specifically, given the trajectory history position sequence ,in Indicates the The target position coordinates of the frame, calculate the average motion velocity vector :

[0058]

[0059] For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration Through the historical 5-frame trajectory fitting estimation, Express obedience Random perturbations of distribution; It is considered as a short-term frame loss. It is considered as a long-term frame loss when The target can be inapplicable Restore in case of .

[0060] Furthermore, in S6, the network training and model solidification include: training the YOLOV13 network and the FastReid network, and solidifying the model parameters.

[0061] Furthermore, in S7, the preparing of six test data sets includes: Dataset 1 and Dataset 2 are synthetic IRT-B and IRC-B infrared data sets, Dataset 3 is a MultiUAV infrared data set, Dataset 4 is an Anti-UAV multimodal data set, Dataset 5 is a DroneDetection multimodal data set, and Dataset 6 is a UAVSwarm multimodal data set;

[0062] The multimodality refers to infrared and visible light modalities.

[0063] Furthermore, in said S8, said testing and evaluating hierarchical correlation tracking framework includes a testing hierarchical correlation tracking framework and an evaluating hierarchical correlation tracking framework;

[0064] The evaluation indicators of the tracking framework include: high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR.

[0065] Furthermore, in S9, the evaluation of the quality of the pseudo samples is intended to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

[0066] (3) Beneficial effects

[0067] Compared with the existing technology, the present invention provides a method for generating dense infrared small target pseudo samples in continuous frames and hierarchical association tracking, which has the following beneficial effects:

[0068] 1. The application proposes a heterogeneous interactive degradation model (HIDM), which innovatively designs a scale-motion-visibility three-stage background and target heterogeneous interactive degradation chain, and efficiently simulates the complex degradation process of real-world dense infrared small target imaging.

[0069] 2. The application proposes a complete dense infrared small target continuous frame pseudo sample generation scheme, which combines a random trajectory generation model and a target adaptive embedding strategy, synthesizes a high-fidelity dense infrared small target continuous frame dataset, and makes up for the problems of data scarcity and sample unreality in this field.

[0070] 3. The application proposes a hierarchical association framework, which innovatively introduces a dynamic density regulator, a mixed feature association engine and a trajectory continuity enhancement module, and still maintains stable id continuity and lost target recovery performance without relying on a ReID module and initial labeling. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 A flowchart of the continuous frame dense infrared small target pseudo sample generation and hierarchical association tracking method of the application;

[0072] Figure 2 An infrared target sample generation principle diagram designed by the application;

[0073] Figure 3 A heterogeneous interactive degradation model (HIDM) principle diagram designed by the application;

[0074] Figure 4 Four different multi-target trajectory graphs generated on a group of image sequences by the application for visualization;

[0075] Figure 5 Generalization experiment results of the application on two multi-modal datasets;

[0076] Figure 6 Quantitative comparison results of the application with a variety of existing tracking methods. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0078] Embodiment:

[0079] As Figure 1FIG. 1 is a flowchart of a method for generating dense infrared small target pseudo samples in consecutive frames and performing hierarchical association tracking according to an embodiment of the present invention. The method specifically includes the following steps:

[0080] S1, prepare background and target sample sets: prepare IRT-B background sample set 1, IRC-B background sample set 2 and IRT-T target sample set.

[0081] The IRT-B sample set is a collection of ground-to-air images captured by the present invention using a long-wave infrared camera. It includes 10 scenes, including sky, forest, lake, building, fence, and grassland. The sky area accounts for the largest proportion in each scene. The IRT-B is divided into training, validation, and test sets in a ratio of 6:2:2; the image resolution is 640×512. The total number of images in the original sequence is approximately 80,000. The present invention removes samples without targets in the long time series. The final IRT-B contains 25 segments of data, a total of 33,606 real images for training, 8,067 for testing, and 9,655 for testing.

[0082] The IRC-B sample set is used to expand the background sample library. The existing generative infrared point target dataset is used. Since it meets the following three conditions, it can also be used for the second background sample library of the present invention: 1) The scenes are very rich, including various scenes such as sky, water surface, vegetation, buildings, etc.; 2) It includes multi-perspective embedded drone conditions for looking up, looking straight, and looking down; 3) The weak point targets are basically invisible after HIDM degradation and can be regarded as equivalent to clean background sample images; the present invention keeps the 175 training sequences of the existing IRC-B unchanged, and uses the first 150 sequences of the IRC-B test sequence as the verification sequence, and the last 25 sequences as the test sequence; IRC-B contains 350 data sequences, and the training subset and test subset each contain 175 data sequences, totaling 150,185 frames of images.

[0083] The IRT-T target sample set is a simulated target sample. Figure 2 The target sample set establishment process shown in the figure can be used as an example of infrared target sample generation task. Specifically, the present invention first collects 500 types of drone-type visible light target samples from Unreal Engine 5. The number of samples of each type of aircraft is between 20-25 and the resolution is between 20-60. Then, the samples of each type are enlarged by 10 times using random flipping and random resizing. Finally, the random hot spot effect of the infrared drone is simulated using Gaussian blur kernel and mask to achieve the target sample generation task from visible light. The conversion from infrared to infrared can be expressed as follows:

[0084]

[0085]

[0086] wherein, represents a binary mask generated by adaptive thresholding; represents a random speckle matrix; represents a Gaussian blur kernel; represents element-wise multiplication; represents a high-pass blur operation; represents an input visible light sample; represents the grayscale value of a grayscale image at ; represents a tolerance threshold, set to 10.

[0087] The final IRT-T target sample set contains 16775 samples; in addition, in order to train FastReid, the present application edits the IRT-T data format into a standard Reid dataset file format, which contains 84699 training samples, 16038 test samples and 16038 identity samples.

[0088] S2, design a heterogeneous interactive degradation model (HIDM): including target degradation (HIDM-T), background degradation (HIDM-B) and local contrast interaction (HIDM-I); as shown in Figure 3 , unlike the traditional single-frame image degradation process, the present application adopts a segmented degradation strategy for consecutive video frames, and achieves physically consistent degradation effects through key interactions.

[0089] After the target sample is input into HIDM-T, it successively experiences three degradation stages: 1) scale degradation: the present application generates a scale degradation curve based on the imaging distance of the target and the camera, controls the apparent size of the target in the image to continuously decrease with the increase of the distance; 2) motion blur degradation: the present application calculates the motion speed of the camera, and accordingly generates the motion blur effect of the target, when the camera motion speed is fast, this process will simulate the sawtooth effect of the target edge, and the size and direction of the blur kernel are determined by ; 3) visibility degradation: calculate the background variance and target variance of the local region where the target is located, based on this, adaptively adjust the apparent brightness of the target, so that it interacts with the background brightness, ensure that the target tends to the background brightness in the relatively bright background area, and the target brightness is significantly higher than the background brightness in the relatively dark background area, but the contrast is limited by physics.

[0090] After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted, first analyze the time characteristics of the consecutive background frame sequence, generate a description of the overall brightness and contrast of the scene over time The degradation curve is changed, the background sequence is adaptively adjusted in segments, and effects such as light gradient and heat source movement are simulated information, a corresponding global motion blur effect is applied to each frame of background image; finally, noise degradation conforming to the characteristics of the infrared detector is superimposed on the blurred background frame.

[0091] S3, a multi-target trajectory generation mechanism is designed: a trajectory generation module and a boundary control function are designed; the number of generated targets, the starting point of the target trajectory and the target library id are random; as shown in Figure 4 Fig. 4 shows four kinds of target trajectories visualized on the same video frame sequence, which have complexity and randomness.

[0092] First, the trajectory generation function is defined as:

[0093]

[0094] wherein, represents the initial position of the target; represents the segment length of an image sequence; represents the image size; represents a real number set; represents a two-dimensional real number space, which specifically refers to the initial position coordinates on a two-dimensional plane ; T represents the length of the time sequence, represents the two-dimensional coordinates output at each time point, which specifically refers to a T*2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be explained as: the function maps the initial two-dimensional position and the parameter into a trajectory sequence with a time length of T, and the output is a two-dimensional array, each row of which corresponds to the coordinates of a time point;

[0095] We embed moving targets in each small sequence segment, and finally splice the data segments to obtain complete image sequence data.

[0096] In order to ensure the complexity of the motion trajectory, the present application defines that each trajectory is randomly selected and generated from a linear model and a quadratic model, and the linear model and the quadratic model are respectively represented by the following formulas:

[0097]

[0098]

[0099] wherein, represents a uniform distribution Uniformly distributed simulated velocity components; represents the curvature parameter, which obeys Evenly distributed.

[0100] In order to prevent the target from crossing the boundary and disappearing from the field of view for a long time, the present invention uses a dynamic boundary detection function To suppress this phenomenon, the formula can be expressed as:

[0101]

[0102] Trigger elastic collision rules when detecting out of bounds , so that the target The relative speed of the reverse movement; finally generate a set of trajectories , each trajectory There are a series of moments Corresponding position .

[0103] S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2;

[0104] In the target embedding stage, the present invention adopts the target-background adaptive weighting strategy; goals , first normalize using . Then construct the spatial variation fusion coefficient matrix:

[0105]

[0106] in, Represents the target image In position The pixel value on ; Represents the indicator function, performing 0 and 1 operations; the fusion of target and background can be expressed as follows:

[0107]

[0108] in, Indicates the current trajectory position of the target.

[0109] In order to enhance the physical reality, the present invention introduces the local contrast comparison mechanism HIDM-I in combination with HIDM. Below target variance When a linear transformation is applied to the target ,in and denote contrast gain and brightness offset respectively.

[0110] S5, build a hierarchical correlation tracking framework: including adapting the YOLOV13 network and FastReid network; design a dynamic target density regulator , hybrid feature association engine and the Trajectory Continuity Enhancer module ; YOLOV13 is used to obtain motion features; FastReid is used to obtain appearance features; 、 and Embedded in BoT-SORT to build a hierarchical correlation tracking framework.

[0111] The BoT-SORT algorithm uses a static confidence threshold that is difficult to adapt to the non-uniformity of target spatial distribution, which will lead to a high rate of missed detection in sparse target areas and a large number of false detections in high-density areas. This rigid filtering strategy essentially ignores the inherent relationship between the dynamic complexity of the scene and the filtering confidence. Therefore, the present invention adopts An adaptive mapping relationship between scene complexity and confidence is established; specifically, the average distance of scene targets or the density of scene targets is first calculated. , and Represent the area of ​​the region of interest and the number of targets, respectively, and according to Setting a dynamic confidence threshold Filtering test results; dynamic threshold It can be defined by the following formula:

[0112]

[0113] in, Indicates the basic confidence threshold; adjustment coefficient pass and The interval segment is calculated together; When the target density is high, increase To avoid too many false positive detections; when the target density is low, reduce To ensure that the target is fully detected, this dynamic adjustment improves the tracking performance of the tracker under different scene conditions.

[0114] In the case of BoT-SORT opening reid, through the motion similarity matrix The composite similarity matrix constructed with the appearance similarity matrix can be expressed as ,in, and It is a fixed fusion parameter. Since the appearance characteristics of small targets of infrared UAV do not change significantly, we need to pay more attention to the appearance characteristics of short distances in the initial tracking period and the long-distance motion characteristics in the tracking stability period. and Not suitable for infrared small drone tracking tasks.

[0115] Therefore, the present invention improves the Bot-SORT and Direct correlation method, introducing trajectory reliability factor To achieve the most robust fusion of appearance and motion information in time sequence, the composite similarity matrix can be expressed as , where the trajectory reliability factor It can be expressed as:

[0116]

[0117] in, and represents the weight boundary; and They are used to control the transition boundaries respectively; Represents the sigmoid function; it can be calculated that Frame tracking initial ;exist The robust tracking stage of the frame , indicating that such a control process is particularly suitable for the tracking task in this paper.

[0118] The target loss problem caused by the recovery of high-speed moving targets after occlusion is inevitable, and the single linear interpolation method of Kalman filtering will introduce significant trajectory association errors and state estimation bias in high-density target scenes; therefore, the present invention uses a hybrid prediction model to predict lost targets. Specifically, given a trajectory history position sequence ,in Indicates the The target position coordinates of the frame, calculate the average motion velocity vector :

[0119]

[0120] For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration Through the historical 5-frame trajectory fitting estimation, Express obedience Random perturbations of distribution; It is considered as a short-term frame loss. It is considered as a long-term frame loss when The target can be inapplicable Restore in case of .

[0121] S6, network training and model solidification: including training the YOLOv13 network and the FastReid network, and solidifying the model parameters; inputting the pseudo samples obtained in step S4 into the YOLOv13 network for training to obtain the target detection model; inputting the IRT-T in step S1 into the FastReid network for training to obtain the appearance feature extraction model.

[0122] S7: Prepare test datasets: Prepare six test datasets for comprehensively testing the robustness and generalization performance of the pseudo sample generation method and tracking framework of the present invention; dataset one and dataset two are the IRT-B and IRC-B infrared datasets synthesized by the present invention, dataset three is the MultiUAV infrared dataset, dataset four is the Anti-UAV multimodal dataset, dataset five is the DroneDetection multimodal dataset, and dataset six is ​​the UAVSwarm multimodal dataset; multimodality refers to infrared and visible light modalities.

[0123] The Anti-UAV dataset contains multimodal real-world drone tracking image sequences. Using a variety of commercially available small drone models, it collects and annotates over 300 visible light / infrared video pairs (containing over 580,000 targets), covering both daytime and nighttime conditions, including complex backgrounds such as clouds, buildings, and jungles, as well as challenges posed by false targets such as flying birds and airborne objects. This dataset is highly suitable for verifying the generalization capabilities of the tracker model trained on pseudo samples.

[0124] The DroneDetection dataset contains infrared images, visible light images, and audio data of four different types of objects: airplanes, birds, drones, and helicopters. This paper assumes that, except for birds, all other flying targets can be considered small drones if their size is small enough, and the method of this paper can be generalized to tracking these targets.

[0125] The UAVSwarm dataset collects many drone swarm videos for drone swarm detection and tracking. UAVSwarm records 13 different scenes and more than 19 types of drones, including 12,598 annotated images. The number of drones in each sequence ranges from 3 to 23, which is very suitable for testing the performance of the tracking method proposed in this paper on visible light datasets.

[0126] S8, testing and evaluating the hierarchical association tracking framework: including testing the hierarchical association tracking framework and evaluating the hierarchical association tracking framework; inputting the six continuous frame data sets prepared in step S7 into the hierarchical association framework, and outputting the coordinates, id and confidence of each target in each frame; the present invention uses high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, multi-target tracking accuracy MOTP, association accuracy AssA and association recall rate AssR to jointly evaluate the performance of the hierarchical association tracking framework.

[0127] MOTA, FN, FP, IDs, and Frag can be collectively categorized as the CLEAR metric. HOTA more evenly considers detection accuracy and identity preservation. MOTA accounts for all object tracking errors (including FP, FN, and IDs) incurred by the tracker across all frames. IDs are a key metric for measuring the consistency of the tracker's target identity. MOTP measures the accuracy of the algorithm's estimation of the target location (e.g., bounding box, center point), specifically the quality of the match between the predicted box and the ground-truth box.

[0128] S9, evaluate the quality of pseudo samples: the pseudo sample dataset generated in step S7 is then used to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

[0129] This paper studies the two difficult problems of data and algorithm in the field of thermal infrared dense UAV tracking, and proposes a continuous frame dense infrared small target pseudo sample generation and hierarchical association tracking method; in the pseudo sample generation method, the heterogeneous interactive degradation model (HIDM) interactively reconstructs the infrared background and target real degradation through the three-stage physical degradation chain of scale-motion-visibility, and combines random multi-target trajectory generation and adaptive image fusion to generate IRT-B and IRC-B high-fidelity pseudo sample datasets. In the hierarchical association framework, the dynamic target density regulator, hybrid feature association engine (with re-ID enabled) and trajectory continuity enhancement are innovatively integrated, and the relevant indicators of various existing tracking methods are compared. Figure 6 As shown in Figure 2, with re-id disabled, HOTA achieved 68.7% / 67.3% (2.9 / 3.0 higher than BoT-SORT) and MOTA achieved 76.2% / 74.6% (2.8 / 3.1 higher than BoT-SORT). Figure 5As shown, the present invention also verifies the universality and reliability of pseudo sample data and hierarchical association tracking framework through generalization experiments on multiple cross-modal datasets, providing a new paradigm of data-algorithm collaborative optimization for dense UAV tracking. The meaning of the annotations above the detection box is: id: 1 indicates that the number of targets in this frame image is 1, tar indicates the target category, and the last digit indicates the confidence level.

[0130] At the same time, the pseudo-sample generation method of the present invention still needs to be improved in the physical modeling of simulating extreme weather conditions (such as heavy rain and dense fog), and the mining of temporal motion features needs to be improved. The next step is to introduce an atmospheric scattering model to enhance the degradation authenticity and integrate optical flow information to enhance the short-term occlusion processing capability.

[0131] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for generating pseudo samples of dense infrared small targets in continuous frames and hierarchical association tracking, characterized by: The following steps are involved: S1, prepare background and target samples: prepare IRT-B background sample set 1, IRC-B background sample set 2 and IRT-T target sample set; In S1, the IRT-B background sample set 1 is a self-photographed sample set; the IRC-B background sample set 2 is an existing background sample set; the IRT-T target sample set is obtained by a designed infrared target sample generation algorithm; The establishment process of the infrared target sample generation algorithm is as follows: first, 500 types of drone-type visible light target samples are collected from Unreal Engine 5, with the number of samples of each type of aircraft ranging from 20 to 25 and the resolution ranging from 20 to 60; then, random flipping and random resizing are used to increase the sample size of each type by 10 times; finally, Gaussian blur kernel and mask are used to simulate the random hot spot effect of infrared drones, so as to achieve the infrared target sample generation algorithm from visible light G σ The conversion from infrared to infrared is expressed as follows: Among them, M bin represents the binary mask generated by adaptive thresholding; B rand represents an N×N random hot spot matrix; G σ represents the Gaussian blur kernel; ⊙ represents element-wise multiplication; Indicates high fuzzy operation; I VI Represents the input visible light sample; I grary (x,y) represents the grayscale value of the grayscale image at (x,y); τ represents the tolerance threshold, which is set to 10; The final IRT-T target sample set contains 16,775 samples. In addition, in order to train FastReid, the IRT-T data format is edited into the standard Reid dataset file format, which contains 84,699 training samples, 16,038 test samples, and 16,038 identity samples. S2, design heterogeneous interactive degradation model HIDM: including target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I; S3, design a multi-target trajectory generation mechanism: including designing a trajectory generation module and a boundary control function; S4, design target embedding and fusion strategy: including adaptive weighted fusion strategy and HIDM-I implementation in step S2; S5, build a hierarchical association tracking framework: including adapting the YOLOV13 network and FastReid network, and designing a dynamic target density regulator Φ d , hybrid feature association engine Φ f and trajectory continuity enhancement module Φ c ; S6, network training and model solidification: including training YOLOV13 network and FastReid network, and solidifying model parameters; S7: Prepare test datasets: Prepare six test datasets; S8, test and evaluate the hierarchical association tracking framework: use high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity conversion IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR to jointly test and evaluate the performance of the hierarchical association tracking framework; S9, evaluate the quality of pseudo samples: use motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8 to measure the quality of the generated pseudo samples.

2. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In S2, the heterogeneous interactive degradation model HIDM includes: target degradation HIDM-T, background degradation HIDM-B and local contrast interaction HIDM-I; After the target sample is input into HIDM-T, it undergoes three degradation stages in sequence: 1) Scale degradation: Generate a scale degradation curve based on the imaging distance between the target and the camera, and control the apparent size of the target in the image to continuously decrease as the distance increases; 2) Motion blur degradation: Calculate the camera's motion speed Based on this, the motion blur effect of the target is generated. When the camera moves faster, this process will simulate the jagged effect of the target edge. The size and direction of the blur kernel are determined by Decide; 3) Visibility degradation: The background variance and target variance of the local area where the target is located are calculated. Based on this, the apparent brightness of the target is adaptively adjusted to make it interact with the background brightness. This ensures that the target brightness approaches the background brightness in brighter areas and is significantly higher than the background brightness in darker areas. However, the contrast is limited by physical constraints. After the background sample is input into HIDM-B, a segmented brightness / contrast degradation strategy is adopted. First, the temporal characteristics of the continuous background frame sequence are analyzed to generate a degradation curve describing the change of the overall brightness and contrast of the scene with time t. The background sequence is adaptively adjusted in segments to simulate effects such as gradual illumination change and heat source movement. Then, the same camera motion speed as HIDM-T is used. The corresponding global motion blur effect is applied to each frame of the background image based on the information; finally, the noise degradation that conforms to the characteristics of the infrared detector is superimposed on the blurred background frame.

3. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In said S3, said multi-target trajectory generation mechanism includes: a trajectory generation module and a boundary control function; The trajectory generation function of the trajectory generation module is defined as: Where p0 = (x0, y0) represents the initial position of the target; L represents the segment length of an image sequence; (W, H) represents the image size; represents the set of real numbers; Represents a two-dimensional real space, where the input is the initial position coordinate p0 on the two-dimensional plane; T represents the length of the time series, represents the two-dimensional coordinates output at each time point, specifically a T×2 real number matrix, that is, a trajectory coordinate sequence containing T time points; the above formula can be interpreted as: the function Γ maps the initial two-dimensional position p0 and parameters (L, W, H) into a trajectory sequence of time length T, and the output is a two-dimensional array, where each row corresponds to the (x, y) coordinates of a time point; The moving target is embedded in each small sequence segment, and finally the data segments are spliced ​​together to obtain the complete image sequence data. In order to ensure the complexity of the motion trajectory, each trajectory is defined as randomly selected from the linear model and the quadratic model. The linear model and the quadratic model are expressed using the formulas: Among them, v x ,v y represents the simulated velocity component that obeys the uniform distribution of u(-2,2); a represents the curvature parameter, which obeys the uniform distribution of u(-0.005,0.005); The boundary control function The formula is: When the detection is out of bounds, the elastic collision rule v is triggered new =-ξv old , so that the target moves in the opposite direction at a relative speed of ξ; finally, a set of trajectories is generated Each trajectory Γ m There are a series of positions p corresponding to time t t .

4. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In said S4, said target embedding and fusion strategy includes: an adaptive weighted fusion strategy and the HIDM-I implementation in step S2; The target-background adaptive weighting strategy is specifically as follows: for the mth target First, use it to normalize; then construct the spatial variation fusion coefficient matrix: in, Represents the target image T m The pixel value at position (u, v); f represents the indicator function, performing 0 and 1 operations; the fusion of the target and the background is expressed as follows: I out (y+u,x+v)=α uv T m (u,v)+(1-α uv )·I b (y+u,x+v) Among them, (x, y) represents the current trajectory position of the target; The HIDM-I implementation is specifically as follows: when the background area variance Below target variance When applying a linear transformation T to the target m′ =γT m +β, where γ and β represent contrast gain and brightness offset, respectively.

5. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1 is characterized by: In S5, the hierarchical association tracking framework includes: adapting the YOLOV13 network and the FastReid network; designing a dynamic target density regulator Φ d , hybrid feature association engine Φ f and trajectory continuity enhancer module Φ c Embedded in BoT-SORT to build a hierarchical correlation tracking framework; The dynamic target density regulator Φ d : First, calculate the average distance of the scene target or the scene target density A roi and N t Represent the area of ​​the region of interest and the number of targets respectively, and according to ρ t Set a dynamic confidence threshold τ t Filter detection results; dynamic threshold τ t It is defined by the following formula: t t =t base +l·r t Among them, τ base Represents the basic confidence threshold; the adjustment coefficient λ is adjusted by τ base and ρ t The interval segment is calculated together; The hybrid feature association engine Φ f Aims to improve Bot-SORT and The direct association method of the trajectory reliability factor α(t) is introduced to achieve the most robust fusion of appearance and motion information in time sequence, and the composite similarity matrix can be expressed as The trajectory reliability factor α(t) is expressed as: Among them, γ min and γ max represents the weight boundary; t0 and τ are used to control the transition boundary respectively; σ(.) represents the sigmoid function; it can be calculated that α(t)≈0.642 in the initial tracking stage when t<20 frames; α(t)≈0.776 in the robust tracking stage when t>50 frames, indicating that such a control process is particularly suitable for the tracking task in this paper; The trajectory continuity enhancer module Φ c Use hybrid prediction model to predict lost targets; specifically, given the trajectory history position sequence where p k =(x k ,y k ) represents the target position coordinates of the kth frame, and calculates the average motion velocity vector For short-term loss trajectories, the Kalman linear model is used For long-term loss of trajectory, a quadratic model based on motion trajectory curvature analysis is used Prediction, where acceleration a t Estimated by fitting the historical 5-frame trajectory, ∈ means obeying Random disturbance of distribution; when Δt≤10, it is considered as short-term frame loss, and when 10<Δt≤30, it is considered as long-term frame loss. Experiments show that when Δt>30, the target can be inapplicable to Φ c Restore in case of .

6. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In S6, the network training and model solidification include: training the YOLOV13 network and the FastReid network, and solidifying the model parameters.

7. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In S7, the six test datasets are prepared: dataset 1 and dataset 2 are synthetic I RT-B and IRC-B infrared datasets, dataset 3 is a MultiUAV infrared dataset, dataset 4 is an Anti-UAV multimodal dataset, dataset 5 is a DroneDetection multimodal dataset, and dataset 6 is a UAVSwarm multimodal dataset; The multimodality refers to infrared and visible light modalities.

8. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In said S8, said testing and evaluating hierarchical correlation tracking framework includes a testing hierarchical correlation tracking framework and an evaluating hierarchical correlation tracking framework; The evaluation indicators of the tracking framework include: high-order tracking accuracy HOTA, multi-target tracking accuracy MOTA, false negative FN, false positive FP, harmonic mean IDF1, identity transformation IDs, trajectory fragmentation Frag, association accuracy AssA and association recall AssR.

9. The method for generating pseudo samples of dense infrared small targets in consecutive frames and tracking them by hierarchical association according to claim 1, characterized in that: In said S9, the evaluation of the quality of the pseudo samples is intended to measure the quality of the generated pseudo samples using motion complexity MMSAO and MMSO, image average signal-to-noise ratio SCR, structural similarity SSIM and some tracking indicators in step S8.

Images