Unmanned aerial vehicle target detection method and system based on infrared modal privileged information

Abstract

This invention discloses a method and system for UAV target detection based on infrared modal privileged information. The method trains a multi-scale infrared feature codebook using pixel-aligned RGB and infrared images during the training phase. During the inference phase, corresponding prediction codebook indices at different scales are obtained by inputting an RGB image. The multi-scale infrared feature map is then reconstructed using these indices and fused with the multi-scale RGB feature map to generate a multi-scale fused feature map containing dual-modal information. Finally, this fused feature map is input to a detection head for classification and regression tasks to obtain the detection result. This invention achieves high-precision UAV target detection even without infrared input during the inference phase, effectively reducing deployment costs and improving detection robustness.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and UAV target detection technology, and in particular to a UAV target detection method and system based on infrared modal privileged information. Background Technology

[0002] Unmanned aerial vehicle (UAV) target detection is a core technology in fields such as security patrol, disaster monitoring, and low-altitude traffic management. Its detection accuracy and deployment feasibility directly affect the effectiveness of application scenarios. Current mainstream detection methods fall into two categories: RGB single-modal detection and RGB-infrared cross-modal detection. The former relies on visible light images for detection, and in complex environments such as low light and fog, insufficient image contrast easily leads to missed or false detections, resulting in a sharp decline in performance. The latter achieves complementarity by fusing texture details from RGB images with thermal semantic information from infrared images, theoretically improving adaptability to complex environments. However, in practical applications, it faces several fundamental limitations:

[0003] First, the dual constraints of infrared image quality defects and modal misalignment prevent training optimization from being transferred to practical applications.

[0004] RGB and infrared images inherently suffer from pixel misalignment due to differences in their imaging principles. Existing cross-modal fusion methods rely on manual selection of paired samples or fixed transformation matrices to obtain aligned data. This process is not only cumbersome and time-consuming but also fails to completely eliminate the effects of misalignment, leading to feature alignment deviations and misleading model learning. More critically, the hardware characteristics and cost limitations of infrared cameras create a vicious cycle: high-resolution, low-noise infrared cameras are far more expensive than ordinary RGB cameras. In most scenarios, low-cost infrared equipment can only be used during the training phase, and the images acquired by these cameras generally suffer from low resolution, lack of detail, and susceptibility to noise caused by temperature interference. In actual deployment, cost constraints make it even more difficult to guarantee the quality of infrared images, further exacerbating the difficulty of cross-modal fusion. Although these problems can be mitigated by carefully selecting high-quality aligned samples during the training phase, real-time manual alignment is impossible in practical applications, and it is difficult to guarantee stable infrared image quality, ultimately leading to a significant decline in the performance of the fusion model.

[0005] Secondly, the high cost and low availability of infrared devices severely limit the deployment of dual-modal models.

[0006] The hardware manufacturing cost of infrared cameras is significantly higher than that of ordinary RGB cameras. Furthermore, due to limitations in technology adoption, most civilian and commercial drones only carry basic RGB imaging modules and cannot be equipped with infrared lenses. This creates a core contradiction: during the training phase, the model's infrared input requirements can be met by configuring specialized dual-modal acquisition equipment and manually aligning the dataset; however, in actual deployment, most scenarios can only acquire RGB images. Fusion models relying on dual-modal input cannot be implemented due to the lack of infrared data support, making it difficult to translate their technological advantages into practical application value. This situation of "infrared available during training, but no infrared available during deployment" has become a key bottleneck restricting the implementation of cross-modal detection technology.

[0007] Third, existing feature transfer methods are inefficient and difficult to adapt to the accurate detection needs of real-world scenarios.

[0008] To address the issue of missing infrared input during the inference phase, some "feature illusion" methods have attempted to directly regress infrared features from RGB images. However, the semantic differences between RGB and infrared modalities are significant. This continuous feature regression approach easily generates noisy, semantically inconsistent pseudo-infrared features and lacks effective abstraction and discretization of infrared semantics, resulting in low infrared knowledge transfer efficiency and insufficient model robustness. Even if the generation effect can be optimized with high-quality infrared samples during the training phase, the reliability of the generated features further decreases in real-world applications facing complex scene changes and the diversity of RGB images, making it difficult to meet the dual requirements of real-time performance and accuracy for UAV target detection.

[0009] In summary, existing technologies cannot systematically solve the collaborative challenges of "infrared image quality and modal misalignment interference," "deployment limitations due to the cost of infrared equipment," and "inefficient cross-modal feature transfer." There is a significant gap between the ideal conditions during training and the stringent scenarios of actual deployment. An innovative solution is urgently needed that fully leverages the complementary advantages of RGB-infrared dual-modal images during training, while simultaneously achieving high-precision detection results by relying solely on RGB image input during inference. This would overcome cost, quality, and deployment limitations, making it suitable for various practical application scenarios. Summary of the Invention

[0010] In view of the above, the main objective of this invention is to propose a method and system for detecting unmanned aerial vehicles (UAVs) based on infrared modal privileged information, so as to solve the above-mentioned technical problems.

[0011] This invention proposes a method for UAV target detection based on infrared modal privileged information, the method comprising the following steps:

[0012] Step 1: Obtain the training sample set, which includes several RGB images and their corresponding infrared images;

[0013] Step 2: Input the infrared image into the codebook to construct branches, extract multi-scale infrared feature maps from the infrared image, and use hierarchical residual vectorization to vectorize the multi-scale infrared feature maps to construct a multi-scale infrared feature codebook.

[0014] Step 3: Input the RGB image into the target detection branch to extract the multi-scale RGB feature map of the RGB image, and predict the multi-scale RGB feature map to obtain the prediction codebook index at different scales;

[0015] Step 4: Based on the prediction codebook indexes at different scales, find the corresponding multi-scale infrared feature codebooks and reconstruct them to obtain reconstructed infrared feature maps at different scales.

[0016] Step 5: Fuse the reconstructed infrared feature maps at different scales with the corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information.

[0017] Step 6: Perform classification and regression operations on the multi-scale fused feature map to predict the target bounding box and category, and obtain the detection results;

[0018] Step 7: Train the codebook construction branch and the object detection branch in stages. After training, input the image to be tested into the trained object detection branch to obtain the final detection result.

[0019] This invention also proposes a UAV target detection system based on infrared modal privileged information, wherein the system applies the UAV target detection method based on infrared modal privileged information as described above, and the system includes:

[0020] The data acquisition module is used for:

[0021] Acquire a training sample set and the image to be tested. The training sample set includes several RGB images and their corresponding infrared images.

[0022] The infrared feature map extraction module is used for:

[0023] Infrared images are input into the codebook to construct branches, in order to extract multi-scale infrared feature maps from the infrared images, and the multi-scale infrared feature maps are vectorized by hierarchical residual vectorization to construct a multi-scale infrared feature codebook.

[0024] The RGB feature map extraction module is used for:

[0025] The RGB image is input into the object detection branch to extract multi-scale RGB feature maps of the RGB image;

[0026] The infrared feature index prediction module is used for:

[0027] Predicting multi-scale RGB feature maps yields prediction codebook indices at different scales;

[0028] The infrared feature reconstruction module is used for:

[0029] Based on the prediction codebook indexes at different scales, the corresponding multi-scale infrared feature codebooks are found and reconstructed to obtain reconstructed infrared feature maps at different scales.

[0030] The cross-modal feature fusion module is used for:

[0031] Reconstructed infrared feature maps at different scales are fused with corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information.

[0032] The training optimization module is used for:

[0033] The codebook construction branch and the object detection branch are trained in stages.

[0034] The test result output module is used for:

[0035] During the training phase, classification and regression operations are performed on the multi-scale fused feature map to predict the target bounding box and category, and the detection results are obtained. After training is completed, the image to be tested is input into the trained target detection branch to obtain the final detection results.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention achieves efficient and stable transfer of infrared semantic knowledge to RGB modality through vectorization and codebook index prediction mechanism, enabling the model to achieve performance comparable to dual-modal fusion without infrared input during the inference stage. This fundamentally overcomes the dependence of existing dual-modal models on infrared sensors and broadens application scenarios.

[0038] 2. This invention transforms the goal of cross-modal knowledge transfer from continuous feature regression, which is difficult to align, to discrete index classification, significantly reducing the requirement for pixel-level alignment in training data, effectively suppressing the interference of modal misalignment on model performance, and improving stability in complex real-world scenarios.

[0039] 3. The hierarchical residual vectorization strategy adopted in this invention can adaptively capture infrared semantic information at different scales and naturally fits with the multi-scale detection framework, ensuring the integrity of cross-modal knowledge transfer from details to semantics and improving detection accuracy.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0041] Figure 1 This is a flowchart of the UAV target detection method based on infrared modal privileged information proposed in this invention;

[0042] Figure 2 This is the overall framework of the UAV target detection method based on infrared modal privileged information proposed in this invention;

[0043] Figure 3 This is a schematic diagram of the structure of the UAV target detection system based on infrared modal privileged information proposed in this invention. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0045] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0046] Please see Figure 1 and Figure 2 This embodiment provides a method for detecting unmanned aerial vehicle (UAV) targets based on infrared modal privileged information, the method comprising the following steps:

[0047] Step 1: Obtain the training sample set, which includes several RGB images and their corresponding infrared images, i.e., the RGB images and infrared images are pixel-aligned.

[0048] In this step, the method for obtaining the training sample set specifically includes the following steps:

[0049] The DroneVehicle bimodal dataset was used for both main training and testing. This dataset contains 19,459 pixel-aligned RGB-IR image pairs, of which 17,990 pairs were used for training and 1,469 pairs were used for testing. The labeled categories cover five vehicle targets: Car, Truck, Bus, Van, and Freighter. At the same time, the VisDrone dataset was used to verify the cross-dataset generalization ability. This dataset is an RGB unimodal dataset that contains multiple categories of targets such as pedestrians, cars, bicycles, and tricycles, and is used to simulate real-world deployment scenarios without infrared devices.

[0050] The original DroneVehicle data was manually screened and alignment accuracy verified to remove misaligned samples caused by hardware jitter and environmental interference, and a high-quality multimodal training sample set was constructed to ensure the alignment basis of RGB-infrared features during the training phase.

[0051] All RGB and infrared images are uniformly cropped and scaled to 256×256 pixels. This size is suitable for the typical resolution of images acquired by drones and is perfectly matched with the multi-scale feature extraction downsampling rhythm of the subsequent backbone network, ensuring the consistency of the feature map space size.

[0052] During the training phase, data augmentation operations such as random rotation and random pruning are employed to enrich sample diversity and enhance the model's robustness to changes in target attitude and scale from the UAV's perspective, providing sufficient generalization support for subsequent cross-modal knowledge transfer.

[0053] Step 2: Input the infrared image into the codebook to construct branches, extract multi-scale infrared feature maps from the infrared image, and use hierarchical residual vectorization to vectorize the multi-scale infrared feature maps, so as to discretize the continuous infrared feature space into a semantic dictionary represented by the codebook index, and construct a multi-scale infrared feature codebook.

[0054] In this step, the infrared image is input into the codebook to construct branches, thereby extracting multi-scale infrared feature maps from the infrared image. These features are then vectorized using a hierarchical residual vectorization method. The construction of the multi-scale infrared feature codebook specifically includes the following steps:

[0055] First, set up an infrared image feature extraction network in the codebook construction branch. The infrared image feature extraction network is CSPDarknet53.

[0056] Infrared images are input into a codebook to construct branches, and an infrared image feature extraction network is used to extract multi-scale features from the infrared images to obtain multi-scale infrared feature maps. ;in, This represents the underlying infrared feature map. This represents the mid-infrared feature map. Represents infrared signatures of high-altitude areas;

[0057] The underlying infrared feature map By directly applying basic vectorization operations, the underlying quantized features are obtained. ;

[0058] Quantize the underlying features Through downsampling and dimension adjustment functions Adjust to match the mid-infrared feature map The dimensions and number of channels are kept consistent to obtain the adjusted bottom-layer infrared feature map; then the adjusted bottom-layer infrared feature map and the middle-layer infrared feature map are calculated. The residuals between them correspond to the following process relationship:

[0059] ;

[0060] in, Indicates the characteristics of the middle layer residuals. This represents the downsampling and dimension adjustment function, which specifically involves first downsampling the feature map using a 3×3 convolution with a stride of 2, and then adjusting the number of channels using a 1×1 convolution.

[0061] Mid-layer residual characteristics Perform basic vectorization to obtain mid-level quantized features. ;

[0062] Quantize the mid-level features Through downsampling and dimension adjustment functions Adjust to match the high-level feature map By ensuring that the dimensions and number of channels are consistent, an adjusted mid-infrared feature map is obtained; then, the adjusted mid-infrared feature map and the high-level feature map are calculated. The residuals between them correspond to the following process relationship:

[0063] ;

[0064] in, Indicates the characteristics of high-rise residuals;

[0065] Perform basic vectorization operations on the high-level residual features to obtain the high-level quantized features. ;

[0066] Several underlying quantization features Constructing the underlying codebook Several mid-level quantization features Constructing the middle-level codebook Several high-level quantitative characteristics Constructing a high-level codebook ; underlying codebook Middle-level codebook and high-level codebook Constructing a multi-scale infrared feature codebook;

[0067] As a preferred embodiment, the basic vectorization operation is as follows: for each feature vector in an infrared feature map of any scale, the corresponding quantized codebook index is found in a learnable codebook at the corresponding scale through nearest neighbor search. The quantization function is:

[0068] ;

[0069] in, This represents the quantized codebook index. Infrared feature map One of the feature vectors in, , ; This represents the Euclidean norm (L2 norm), used to calculate the distance between two vectors; Represents the quantization function. Represents a learnable codebook The k-th entry in the codebook, where k represents the index of the codebook entry. Represents the learnable codebook at the current scale. H, W, M, and N represent the height, width, number of channels, and codebook capacity of the feature map, respectively. Indicates the feature map scale. ;

[0070] Based on the quantized codebook index The corresponding codebook vector is found from the learnable codebook to obtain the reconstructed feature vector. The corresponding process has the following relationship:

[0071] ;

[0072] in, This represents a reconstruction function that searches for features in the codebook based on an index. Indicates a learnable codebook The index is codebook vector, This represents the reconstructed feature vector;

[0073] All reconstructed feature vectors of the infrared feature map at the current scale These constitute the quantitative characteristics of the corresponding level.

[0074] As a preferred embodiment, the underlying codebook Middle-level codebook and high-level codebook The capacity is configured in a stepped decreasing manner; among which, the underlying codebook Size is 4N, middle-level codebook Size is 2N, high-level codebook The size is N, where the baseline value N=64. This baseline value was determined through multiple sets of comparative experiments: during the model development phase, after testing and screening the candidate values ​​{16,32,64,128,256}, it was found that when N=64, the model can achieve the best balance between ensuring effective representation of high-level infrared semantics and controlling the codebook size to reduce the difficulty of index prediction, thereby comprehensively optimizing the model's detection accuracy and inference efficiency.

[0075] Step 3: Input the RGB image into the target detection branch to extract the multi-scale RGB feature map of the RGB image, and predict the multi-scale RGB feature map to obtain the prediction codebook index at different scales;

[0076] In this step, an RGB image feature extraction network is first set up in the object detection branch, and the infrared image feature extraction network is CSPDarknet53. Here, the object detection branch and the codebook construction branch use the same CSPDarknet53 backbone network structure to extract multi-scale RGB feature maps. Comparing with multi-scale infrared feature maps The spatial dimensions are strictly aligned, corresponding to downsampling rates of 8, 16, and 32.

[0077] Furthermore, an illusion head is designed at each layer of the RGB image feature extraction network to predict the prediction codebook index at different scales.

[0078] This scale alignment design based on downsampling rate is the technical foundation for subsequent implementation of hierarchical residual vectorization, codebook index prediction, and cross-modal feature fusion. It fundamentally ensures that infrared privileged information can be accurately and efficiently transferred to the RGB modality, avoiding feature matching errors caused by scale misalignment.

[0079] As a preferred embodiment, predicting multi-scale RGB feature maps to obtain prediction codebook indexes at different scales specifically includes the following steps:

[0080] The goal of this section is to learn the mapping from RGB features to infrared codebook indices. The process also follows a hierarchical residual structure, as detailed below:

[0081] One is configured for each scale of the multi-scale RGB feature map, and each illusion head consists of a 3×3 convolutional layer and a pixel-level classifier;

[0082] The underlying RGB feature map Through the first illusion head Perform prediction to obtain the underlying codebook index classification logical value. The corresponding process has the following relationship:

[0083] ;

[0084] in, This represents the logical value of the underlying codebook index classification. Indicates the first hallucination head;

[0085] Calculate the RGB feature map of the middle layer The residual between the sampled and downsampled underlying RGB features is then input into the second illusion head. Perform prediction to obtain the classification logical value of the middle-level codebook index. The corresponding process has the following relationship:

[0086] ;

[0087] in, This represents the logical value for classifying the middle-level codebook index. Indicates the second hallucination head;

[0088] Calculate high-level RGB feature maps The residual between the downsampled mid-layer RGB features and the third illusion head is then input into the third illusion head. Perform predictions to obtain the high-level codebook index classification logical values. The corresponding process has the following relationship:

[0089] ;

[0090] in, This represents the logical value for classifying the high-level codebook index. Indicates the third hallucination head;

[0091] Based on the classification logic values ​​of the bottom-level codebook index, the middle-level codebook index, and the high-level codebook index, the corresponding level of the predicted codebook index is determined, resulting in predicted codebook indices at different scales.

[0092] In a preferred embodiment, determining the corresponding level of predicted codebook index based on the classification logic values ​​of the bottom-level codebook index, the middle-level codebook index, and the high-level codebook index specifically involves: determining the predicted logic value for each scale. Executing the argmax operation yields the prediction codebook index for the corresponding level. The process follows the following relationship:

[0093] ;

[0094] in, This refers to the argmax function. This represents the logical value of the codebook index classification at the current level. This represents the index of the prediction codebook at the current level.

[0095] Step 4: Based on the prediction codebook indexes at different scales, find the corresponding multi-scale infrared feature codebooks and reconstruct them to obtain reconstructed infrared feature maps at different scales.

[0096] In a preferred embodiment, reconstructed infrared feature maps at different scales are obtained using a hierarchical residual structure. The acquisition process of the bottom-level reconstructed infrared features follows the following relationship:

[0097] ;

[0098] in, This indicates the reconstructed infrared features at the bottom layer. Indicates the underlying prediction codebook index;

[0099] The process of acquiring mid-layer reconstructed infrared features follows the following relationship:

[0100] ;

[0101] in, This indicates the reconstructed infrared features in the middle layer. Indicates the index of the intermediate prediction codebook;

[0102] The process of acquiring high-level reconstructed infrared features is governed by the following relationship:

[0103] ;

[0104] in, This indicates the reconstructed infrared signature at higher levels. This indicates the high-level prediction codebook index.

[0105] Step 5: Fuse the reconstructed infrared feature maps at different scales with the corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information.

[0106] In this step, in order to combine the rich details of the RGB mode with the complementary information of the illusionized infrared mode, the reconstructed infrared features are... Compared with the original RGB features of the same scale The fusion process is performed. The fusion strategy employs a "channel concatenation followed by convolutional compression" method, which can be expressed as follows:

[0107] ;

[0108] in, This indicates a splicing operation along the channel dimension. This represents the fused feature map at the current scale. This indicates a 1×1 convolutional layer, used to compress the doubled number of channels after splicing.

[0109] Step 6: Perform classification and regression operations on the multi-scale fused feature map to predict the target bounding box and category, and obtain the detection results;

[0110] Step 7: Train the codebook construction branch and the object detection branch in stages. After training, input the image to be tested into the trained object detection branch to obtain the final detection result.

[0111] As a preferred embodiment, the training of the codebook construction branch and the object detection branch using a staged training method specifically includes the following steps:

[0112] Phase 1:

[0113] Obtain the reconstructed feature vectors corresponding to the feature vectors in the multi-scale infrared feature map, and construct a vectorized loss function based on the feature vectors in the multi-scale infrared feature map and the corresponding reconstructed feature vectors. The corresponding process has the following relationship:

[0114] ;

[0115] in, This indicates that the gradient operation is stopped. It represents the square of the Euclidean norm (L2 norm), used to calculate the squared distance between two vectors;

[0116] The codebook construction branch is trained with minimizing the vectorization loss as the pre-training objective. After pre-training, all parameters of the codebook construction branch and the learnable codebook are frozen to obtain the multi-scale infrared feature codebook.

[0117] Phase Two:

[0118] Obtain the true classification labels corresponding to the training set, and use the quantized codebook index as the true index label;

[0119] Construct the detection loss using real classification labels and detection results;

[0120] We construct the cross-entropy loss for codebook index prediction using real index labels and index classification logical values;

[0121] The detection loss and the cross-entropy loss from codebook index prediction are weighted and combined to obtain the total loss. The corresponding process has the following relationship:

[0122] ;

[0123] In this formula: It is the original detection loss in the YOLOv8m detection head; It is the cross-entropy loss predicted by the codebook index; It is the index classification logical value output by the illusion head at the current scale; It is the quantized codebook index corresponding to the current scale; It is a weighting hyperparameter that balances the two losses;

[0124] The object detection branch is trained by minimizing the total loss. After training, the trained object detection branch is obtained.

[0125] In this step, hyperparameters The value of α was determined through cross-validation. Specifically, during model development, cross-validation experiments were conducted within the candidate value range of {2.5, 5.0, 10.0, 20.0, 40.0} to evaluate model performance. Experiments showed that when α is relatively low, the codebook index prediction task is not sufficiently learned, leading to a decrease in the quality of reconstructed infrared features; when α is relatively high, it overemphasizes index prediction accuracy while suppressing the learning of the detection task itself. Ultimately, α=10.0 was determined to be the value that achieves the optimal balance between detection accuracy and index prediction accuracy.

[0126] Finally, during the inference phase, only the RGB image to be detected is input into the trained target detection branch. The process automatically executes RGB feature extraction, infrared feature codebook index prediction, infrared feature reconstruction, and cross-modal feature fusion, inputting these into the detection head and outputting the final detection result. At this stage, since there is no infrared modal image input, the codebook construction branch pre-trained during the training phase has been completely removed; only the pre-trained multi-scale vectorized codebook is retained for infrared feature reconstruction. This simplifies the inference process and avoids unnecessary modules consuming computational resources.

[0127] This embodiment also provides a UAV target detection system based on infrared modal privileged information to implement the above method. The system includes:

[0128] The data acquisition module is used for:

[0129] Acquire a training sample set and the image to be tested. The training sample set includes several RGB images and their corresponding infrared images.

[0130] The infrared feature map extraction module is used for:

[0131] Infrared images are input into the codebook to construct branches, in order to extract multi-scale infrared feature maps from the infrared images, and the multi-scale infrared feature maps are vectorized by hierarchical residual vectorization to construct a multi-scale infrared feature codebook.

[0132] The RGB feature map extraction module is used for:

[0133] The RGB image is input into the object detection branch to extract multi-scale RGB feature maps of the RGB image;

[0134] The infrared feature index prediction module is used for:

[0135] Predicting multi-scale RGB feature maps yields prediction codebook indices at different scales;

[0136] The infrared feature reconstruction module is used for:

[0137] Based on the prediction codebook indexes at different scales, the corresponding multi-scale infrared feature codebooks are found and reconstructed to obtain reconstructed infrared feature maps at different scales.

[0138] The cross-modal feature fusion module is used for:

[0139] Reconstructed infrared feature maps at different scales are fused with corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information.

[0140] The training optimization module is used for:

[0141] The codebook construction branch and the object detection branch are trained in stages.

[0142] The test result output module is used for:

[0143] During the training phase, classification and regression operations are performed on the multi-scale fused feature map to predict the target bounding box and category, and the detection results are obtained. After training is completed, the image to be tested is input into the trained target detection branch to obtain the final detection results.

[0144] To verify the effectiveness of this invention and its advancements compared to existing technologies, this embodiment was developed and verified in a high-performance computing environment. Specifically, experiments were conducted on two authoritative public datasets, DroneVehicle and VisDrone, using the Python 3.7 programming language and the PyTorch 1.10.1 deep learning platform. In terms of hardware configuration, an Intel® Core™ i9-14900KF (32 cores) was used as the central processing unit, equipped with two NVIDIA RTX4090 graphics processors for accelerated computation. The operating environment of this embodiment was an Ubuntu 20.04 LTS 64-bit operating system, ensuring stable and efficient model training and inference processes.

[0145] During the model training phase, all configurations are kept consistent to ensure the fairness and reproducibility of the results. The network uses models such as CSPDarknet53, ResNet50, and RemDet (adapted to different comparison methods), and the size of all input images is uniformly adjusted to 256×256 pixels. The training process employs a joint loss function (including detection loss and cross-entropy loss) for optimization. The hyperparameter α balancing the two losses is set to 10.0, determined through multiple cross-validation experiments. After testing candidate values ​​of α = 2.5, 5.0, 10.0, 20.0, and 40.0, it was found that α = 10.0 achieves the best balance between detection accuracy and codebook index prediction accuracy. The optimizer type, hyperparameters (such as momentum and weight decay), initial learning rate, and learning rate scheduling strategy all follow the standard configuration of the YOLOv8m framework to ensure consistency with the training logic of the benchmark model. To ensure the consistency of experimental results, all experiments use parameters selected within a random seed range of 1–10. Testing showed that all seed settings within this range achieve similar model performance; specific seed values ​​can be selected based on the actual training environment.

[0146] To comprehensively and specifically evaluate the performance of this invention in infrared privileged UAV target detection tasks, this invention uses the mean average accuracy (mAP@0.5) as the core evaluation index. This index calculates the mean average accuracy (AP) of all target categories under the condition that the intersection-over-union (IoU) threshold is 0.5, which can comprehensively reflect the model's ability to detect different types of targets and adapt to the needs of multiple target coexistence in UAV scenarios.

[0147] Table 1 shows a performance comparison between the method of this invention and existing methods on the DroneVehicle dataset. Comprehensive analysis reveals that this invention exhibits significant advantages in the following three aspects: First, this invention verifies the effectiveness of the infrared privileged learning paradigm. As shown in the table, infrared privileged group methods (such as FeatHallu and M2D-LIF†) generally outperform RGB single-modality groups, indicating that introducing infrared modalities as privileged information during the training phase can effectively improve model performance. Second, under the same base detector, the method of this invention demonstrates superior infrared knowledge transfer capability. Using YOLOv8m as a benchmark, the mAP@0.5 of "this invention + YOLOv8m" reaches 67.6%, which is 1.6% and 2.3% higher than M2D-LIF† (66.0%) and FeatHallu (65.3%), both infrared privileged methods. This fully demonstrates that the transfer mechanism based on codebook index prediction is superior to traditional feature regression or knowledge distillation methods.

[0148] Furthermore, this invention demonstrates broad adaptability to different detector architectures. When combined with different backbone networks such as CEASC, DQ-DETR, and RemDet, the proposed method consistently improves their baseline performance (e.g., mAP@0.5 increases by 3.2%, 3.4%, and 5.4%, respectively), proving the good versatility and scalability of the proposed cross-modal vectorization framework. In addition, this invention approaches the performance ceiling of bimodal fusion under single-modal inference conditions. Although RGB images are used only during inference, the optimal configuration (this invention + RemDet) achieves an mAP@0.5 of 71.3%, approximately 87.6% of the performance of the strongest bimodal input method, M2D-LIF (81.4%). This achieves near-fully supervised bimodal fusion system performance with extremely low deployment cost, demonstrating significant application value.

[0149] Table 1: Performance comparison of the proposed model and existing methods on the DroneVehicle dataset

[0150]

[0151] To verify the generalization ability and practicality of this invention in real-world scenarios equipped only with RGB sensors, a cross-dataset evaluation was conducted on the VisDrone dataset (the model was pre-trained on DroneVehicle and then fine-tuned on VisDrone). When fine-tuning the infrared privileged method, the parameters of the backbone network, the illusion head in the object detection branch, and the codebook construction branch were frozen to avoid degradation of transferred infrared semantic knowledge. Table 2 presents the comparison results under this setting and draws the following key conclusions: First, since the VisDrone dataset does not provide infrared image input, all RGB-IR benchmark methods that rely on bimodal input (such as CALNet, ODAF, etc.) cannot be tested and compared in this scenario, highlighting the fundamental limitations of existing bimodal fusion technologies in practical deployments. Second, infrared privileged methods generally outperform pure RGB methods. For example, the average accuracy (AVG) of M2D-LIF† is 59.6%, which is 7.6 percentage points higher than YOLOv8m (52.0%). This proves that even in unfamiliar environments with different datasets, the knowledge transferred from infrared privileged information can still effectively compensate for the lack of information in the RGB modality. Finally, the method of this invention exhibits the best generalization performance on various backbone networks. Among them, the "this invention + RemDet" configuration achieved an AVG of 65.9%, which not only significantly surpasses the baseline YOLOv8m by 13.9 percentage points, but also outperforms other infrared privileged methods (such as being 6.3 percentage points higher than M2D-LIF†). This strongly demonstrates that the cross-modal vectorization framework proposed in this invention can learn and transfer more general and robust infrared semantic representations, thereby achieving stable and accurate detection in novel practical application scenarios containing only RGB images.

[0152] Table 2: Performance comparison of the proposed model and existing methods on the VisDrone dataset

[0153]

[0154] To quantitatively verify the contribution of each core technical module of this invention, a systematic ablation experiment was conducted on the DroneVehicle dataset based on the YOLOv8m detector. The results are shown in Table 3. The experimental design adopted the controlled variable method, introducing each core module one by one to evaluate its contribution in isolation: the first row uses the original RGB single-modal architecture as the baseline; the second row adds a phantom head structure only at each scale of the target detection branch, and the RGB features are fused with the reconstructed infrared features output by the phantom head structure before being input into the detection head; the third row enables the cross-entropy loss predicted by the codebook index based on the second row (…). The codebook indexes at each scale are predicted using the illusion head; the fourth line further uses a hierarchical residual vectorization method to predict the codebook indexes based on the third line.

[0155] The experimental results clearly reveal the independent contributions and synergistic effects of each module: By comparing rows 2 and 1, it can be seen that adding only the illusion head structure (without enabling cross-entropy loss, no codebook index prediction task) only brings a small performance improvement (+0.6%), indicating that the performance gain does not stem from a simple increase in the number of parameters; comparing rows 3 and 2, enabling cross-entropy loss ( After driving the codebook index prediction task, the performance was significantly improved (+4.9%), which fully demonstrates that knowledge transfer through discrete codebook indexes can more effectively and robustly convey complementary information of infrared modes compared with traditional continuous feature regression. Comparing the fourth and third rows, the performance was further significantly improved (+1.9%) after adopting hierarchical residual vectorization, which verifies that the strategy can capture multi-scale infrared semantics in a refined manner, avoid semantic redundancy between levels, and thus achieve cross-modal knowledge transfer more completely.

[0156] Table 3: Ablation experimental results of the model of this invention on the DroneVehicle dataset

[0157]

[0158] To verify the robustness of the proposed framework, this invention was statistically compared with the most competitive baseline method, YOLOv8m. On the DroneVehicle test set, both models were independently trained and evaluated 10 times each (each time using different random seeds ranging from 1 to 10), and their mAP@0.5 performance was compared using a paired-samples t-test at a significance level of 0.05. The results are shown in Table 4.

[0159] The test results show that the average performance of the method of this invention (67.7 ± 0.4) is significantly better than that of the baseline model YOLOv8m (63.9 ± 0.53), and the calculated p-value (4.67e-13) is far below the significance threshold of 0.05. This provides strong statistical evidence for the superior performance of the method of this invention, proving that it is not due to random experimental variation, and thus further verifying the effectiveness and reliability of the cross-modal vector quantization mechanism proposed in this invention in infrared privileged UAV target detection tasks from a statistical perspective.

[0160] Table 4: Statistical Analysis of the Model of this Invention and the Strongest Baseline Model YOLOv8m

[0161]

[0162] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0163] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for detecting unmanned aerial vehicle (UAV) targets based on infrared modal privileged information, characterized in that, The method includes the following steps: Step 1: Obtain the training sample set, which includes several RGB images and their corresponding infrared images; Step 2: Input the infrared image into the codebook to construct branches, extract multi-scale infrared feature maps from the infrared image, and use hierarchical residual vectorization to vectorize the multi-scale infrared feature maps to construct a multi-scale infrared feature codebook. Step 3: Input the RGB image into the target detection branch to extract the multi-scale RGB feature map of the RGB image, and predict the multi-scale RGB feature map to obtain the prediction codebook index at different scales; Step 4: Based on the prediction codebook indexes at different scales, find the corresponding multi-scale infrared feature codebooks and reconstruct them to obtain reconstructed infrared feature maps at different scales. Step 5: Fuse the reconstructed infrared feature maps at different scales with the corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information. Step 6: Perform classification and regression operations on the multi-scale fused feature map to predict the target bounding box and category, and obtain the detection results; Step 7: Train the codebook construction branch and the object detection branch in stages. After training, input the image to be tested into the trained object detection branch to obtain the final detection result.

2. The UAV target detection method based on infrared modal privileged information as described in claim 1, characterized in that, In step 2, the infrared image is input into the codebook to construct branches, thereby extracting multi-scale infrared feature maps from the infrared image. These features are then vectorized using a hierarchical residual vectorization method. The construction of the multi-scale infrared feature codebook specifically includes the following steps: Infrared images are input into a codebook to construct branches, and an infrared image feature extraction network is used to extract multi-scale features from the infrared images to obtain multi-scale infrared feature maps. ;in, This is the underlying infrared feature map. This is a mid-infrared feature map. Infrared feature map of high altitude; The underlying infrared feature map By directly applying basic vectorization operations, the underlying quantized features are obtained. ; Quantize the underlying features Through downsampling and dimension adjustment functions Adjust to match the mid-infrared feature map The dimensions and number of channels are kept consistent to obtain the adjusted bottom-layer infrared feature map; then the adjusted bottom-layer infrared feature map and the middle-layer infrared feature map are calculated. The residuals between them correspond to the following process relationship: ； in, Indicates the characteristics of the middle layer residuals. This represents the downsampling and dimension adjustment function, which specifically involves first downsampling the feature map using a 3×3 convolution with a stride of 2, and then adjusting the number of channels using a 1×1 convolution. Mid-layer residual characteristics Perform basic vectorization to obtain mid-level quantized features. ; Quantize the mid-level features Through downsampling and dimension adjustment functions Adjust to match the infrared signature of high-rise buildings By ensuring that the dimensions and number of channels are consistent, an adjusted mid-infrared feature map is obtained; then, the adjusted mid-infrared feature map and the upper-infrared feature map are calculated. The residuals between them correspond to the following process relationship: ； in, Indicates the characteristics of high-rise residuals; Perform basic vectorization operations on the high-level residual features to obtain the high-level quantized features. ; Several underlying quantization features Constructing the underlying codebook Several mid-level quantization features Constructing the middle-level codebook Several high-level quantitative characteristics Constructing a high-level codebook ; underlying codebook Middle-level codebook and high-level codebook Constructing a multi-scale infrared feature codebook; Among them, the underlying codebook Middle-level codebook and high-level codebook The capacity is configured in a stepped decreasing manner; among which, the underlying codebook Size is 4N, middle-level codebook Size is 2N, high-level codebook The size is N, where the base value N=64.

3. The UAV target detection method based on infrared modal privileged information as described in claim 2, characterized in that, In step 2, the basic vectorization operation is as follows: for each feature vector in an infrared feature map of arbitrary scale, the corresponding quantized codebook index is found in a learnable codebook at the corresponding scale through nearest neighbor search. The quantization function is: ； in, This represents the quantized codebook index. Infrared feature map One of the feature vectors in, , ; This represents the Euclidean norm, used to calculate the distance between two vectors; Represents the quantization function. Represents a learnable codebook The k-th entry in the codebook, where k represents the index of the codebook entry. Represents the learnable codebook at the current scale. H, W, M, and N represent the height, width, number of channels, and codebook capacity of the feature map, respectively. Indicates the feature map scale. ; Based on the quantized codebook index The corresponding codebook vector is found from the learnable codebook to obtain the reconstructed feature vector. The corresponding process has the following relationship: ； in, This represents a reconstruction function that searches for features in the codebook based on an index. Indicates a learnable codebook The index is codebook vector, This represents the reconstructed feature vector; All reconstructed feature vectors of the infrared feature map at the current scale These constitute the quantitative characteristics of the corresponding level.

4. The UAV target detection method based on infrared modal privileged information as described in claim 3, characterized in that, In step 3, the object detection branch and the codebook construction branch use the same CSPDarknet53 backbone network structure, and the extracted multi-scale RGB feature maps... Comparing with multi-scale infrared feature maps The spatial dimensions are strictly aligned, corresponding to downsampling rates of 8, 16, and 32. Represents the underlying RGB feature map. This represents the middle-layer RGB feature map. This represents the high-level RGB feature map.

5. The UAV target detection method based on infrared modal privileged information as described in claim 4, characterized in that, In step 3, predicting the multi-scale RGB feature maps to obtain prediction codebook indices at different scales specifically includes the following steps: For each scale of the multi-scale RGB feature map, an illusion head is configured. Each illusion head consists of a 3×3 convolutional layer and a pixel-level classifier. The underlying RGB feature map Through the first illusion head Perform prediction to obtain the underlying codebook index classification logical value. The corresponding process has the following relationship: ； in, This represents the logical value of the underlying codebook index classification. Indicates the first hallucination; Calculate the RGB feature map of the middle layer The residual between the sampled and downsampled underlying RGB features is then input into the second illusion head. Perform prediction to obtain the classification logical value of the middle-level codebook index. The corresponding process has the following relationship: ； in, This represents the logical value for classifying the middle-level codebook index. Indicates the second hallucination head; Calculate high-level RGB feature maps The residual between the downsampled mid-layer RGB features and the third illusion head is then input into the third illusion head. Perform predictions to obtain the high-level codebook index classification logical values. The corresponding process has the following relationship: ； in, This represents the logical value for classifying the high-level codebook index. Indicates the third hallucination head; Based on the classification logic values ​​of the bottom-level codebook index, the middle-level codebook index, and the high-level codebook index, the corresponding level of the predicted codebook index is determined, resulting in predicted codebook indices at different scales.

6. The UAV target detection method based on infrared modal privileged information as described in claim 5, characterized in that, The prediction codebook index for the corresponding level is determined based on the classification logical values ​​of the bottom-level codebook index, the middle-level codebook index, and the high-level codebook index. Specifically, this involves determining the prediction logical value for each scale. Executing the argmax operation yields the prediction codebook index for the corresponding level. The process follows the following relationship: ； in, This refers to the argmax function. This represents the logical value of the codebook index classification at the current level. This represents the index of the prediction codebook at the current level.

7. The UAV target detection method based on infrared modal privileged information as described in claim 6, characterized in that, In step 4, the reconstructed infrared feature maps at different scales are obtained using a hierarchical residual structure. The acquisition process of the bottom-level reconstructed infrared features follows the following relationship: ； in, This indicates the underlying reconstructed infrared features. Indicates the underlying prediction codebook index; The process of acquiring mid-layer reconstructed infrared features follows the following relationship: ； in, This indicates the reconstructed infrared features in the middle layer. Indicates the index of the intermediate prediction codebook; The process of acquiring high-level reconstructed infrared features is governed by the following relationship: ； in, Indicates high-level reconstructed infrared features. This indicates the high-level prediction codebook index.

8. The UAV target detection method based on infrared modal privileged information as described in claim 7, characterized in that, In step 5, the process of fusing reconstructed infrared feature maps at different scales with corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information is governed by the following relationship: ； in, This indicates a splicing operation along the channel dimension. This represents the fused feature map at the current scale. It is a convolutional layer with a 1×1 kernel, used to compress the doubled number of channels after splicing.

9. The UAV target detection method based on infrared modal privileged information as described in claim 8, characterized in that, In step 7, the training of the codebook construction branch and the object detection branch using a phased training approach specifically includes the following steps: Phase 1: Obtain the reconstructed feature vectors corresponding to the feature vectors in the multi-scale infrared feature map, and construct a vectorized loss function based on the feature vectors in the multi-scale infrared feature map and the corresponding reconstructed feature vectors. The corresponding process has the following relationship: ； in, This indicates that the gradient operation is stopped. The square of the Euclidean norm is used to calculate the squared distance between two vectors. The codebook construction branch is trained with minimizing the vectorization loss as the pre-training objective. After pre-training, all parameters of the codebook construction branch and the learnable codebook are frozen to obtain a multi-scale infrared feature codebook. Phase Two: Obtain the true classification labels corresponding to the training set, and use the quantized codebook index as the true index label; Construct the detection loss using real classification labels and detection results; The cross-entropy loss of codebook index prediction is constructed using real index labels and index classification logical values; The detection loss and the cross-entropy loss from codebook index prediction are weighted and combined to obtain the total loss. The corresponding process has the following relationship: ； in, This represents the original detection loss in the YOLOv8m detection head; This represents the cross-entropy loss predicted by the codebook index; This represents the index classification logical value output by the illusion head corresponding to the current scale; This represents the quantized codebook index corresponding to the current scale; The weighting hyperparameters represent the factors that balance the two losses; The object detection branch is trained by minimizing the total loss. After training, the trained object detection branch is obtained.

10. A UAV target detection system based on infrared modal privileged information, used to implement the UAV target detection method based on infrared modal privileged information as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used for: Acquire a training sample set and the image to be tested. The training sample set includes several RGB images and their corresponding infrared images. The infrared feature map extraction module is used for: Infrared images are input into the codebook to construct branches, in order to extract multi-scale infrared feature maps from the infrared images, and the multi-scale infrared feature maps are vectorized by hierarchical residual vectorization to construct a multi-scale infrared feature codebook. The RGB feature map extraction module is used for: The RGB image is input into the object detection branch to extract multi-scale RGB feature maps of the RGB image; The infrared feature index prediction module is used for: Predicting multi-scale RGB feature maps yields prediction codebook indices at different scales; The infrared feature reconstruction module is used for: Based on the prediction codebook indexes at different scales, the corresponding multi-scale infrared feature codebooks are found and reconstructed to obtain reconstructed infrared feature maps at different scales. The cross-modal feature fusion module is used for: Reconstructed infrared feature maps at different scales are fused with corresponding RGB feature maps to generate a multi-scale fused feature map containing dual-modal information. The training optimization module is used for: The codebook construction branch and the object detection branch are trained in stages. The test result output module is used for: During the training phase, classification and regression operations are performed on the multi-scale fused feature map to predict the target bounding box and category, and the detection results are obtained. After training is completed, the image to be tested is input into the trained target detection branch to obtain the final detection results.

Images