An unmanned aerial vehicle target detection optimization method and system based on privilege information knowledge transfer
By generating a three-channel privileged information graph aligned with RGB images and employing knowledge distillation techniques, the problem of insufficient feature fusion in UAV target detection is solved, achieving efficient and low-cost improvement in target detection accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2026-01-28
- Publication Date
- 2026-06-05
AI Technical Summary
In existing UAV target detection technologies, feature fusion methods fail to fully integrate global contextual information, resulting in limited detection performance. Furthermore, they rely on external sensors or complex modalities, increasing costs and potentially leading to overfitting and reduced generalization ability.
We employ a privileged information-based knowledge transfer approach, generating a three-channel privileged information map that is pixel-aligned with the RGB image. This map is then combined with the YOLOv11 model and knowledge distillation techniques to perform feature fusion and model training, thereby improving the model's generalization and robustness.
Without increasing model complexity or relying on additional modalities, it significantly improves the accuracy and robustness of object detection, reduces costs, and requires no changes to the basic network structure.
Smart Images
Figure CN122157032A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target detection technology in computer vision, specifically to an optimization method and system for UAV target detection based on privileged information knowledge transfer. Background Technology
[0002] Unmanned aerial vehicle (UAV) target detection is a perception technology in the field of computer vision specifically designed for low-altitude aerial photography. It uses images or video streams collected by UAVs as input data, and its core task is to use algorithms to automatically analyze visual content and quickly filter and confirm objects of interest such as vehicles, pedestrians, and buildings in the scene.
[0003] In UAV target detection tasks, the location and category of a target often depend on contextual information, and effective fusion of contextual information can help the model better understand the background. However, many feature fusion methods in existing target detection tasks fail to fully integrate global contextual information, focusing only on local features. For example, Chinese patent CN119723272A, entitled "A Target Detection Fusion Method for Improved YOLOv10," only uses simple weighted summation, concatenation, or layer-by-layer stacking for feature fusion. These methods cannot fully capture the complex relationships between multiple layers of features, resulting in limited feature fusion performance. In addition, many target detection studies still limit the improvement of detection accuracy to the modification and replacement of modules in the model itself. For example, Chinese patent CN119693940A, entitled "Intelligent Recognition System and Method for Fruit Varieties Based on Improved YOLO Model," only introduces more pooling layers or attention mechanism layers into the YOLO model, which has limited improvement in model accuracy but increases model training time and cost.
[0004] Against this backdrop, the concept of Privileged Information (PI) offers a new approach to overcoming performance bottlenecks. PI is information available during the training phase but not during the testing phase. Proposed by Vapnik et al., this theory advocates introducing additional auxiliary information during training to enhance the model's feature learning ability. In recent years, Privileged Information has demonstrated significant advantages in classification and regression tasks. For example, the innovative IR-RGB image translation model HalluciDet utilizes Privileged Information to enhance the performance of the RGB detector, thereby improving the accuracy of pedestrian detection in cross-modal tasks; Alessandro et al. used depth information as Privileged Information to further improve the accuracy of 3D human pose estimation based on RGB images. However, existing research often relies on external sensors or complex modalities, such as the Chinese patent with publication number CN119832219A, entitled "Lightweight Aerial Image Target Detection Method Based on Improved YOLOv11," which results in high costs and limited practical deployment. In addition, most studies directly input privileged information into the model, such as the Chinese patent application with publication number CN117612226A, entitled "Facial Expression Recognition Method and System Enhanced by Embedding Privileged Information," which may lead to overfitting and reduce the generalization ability in the testing phase. Summary of the Invention
[0005] The purpose of this invention is to provide an optimization method and system for UAV target detection based on privileged information knowledge transfer. Without increasing model complexity or relying on additional modalities, this method fully explores the inherent potential of label information, improves the generalization and robustness of the target detection model, provides an efficient and low-cost solution for target detection in complex scenarios, and promotes the further development of multimodal information fusion technology.
[0006] To solve the above technical problems, the present invention adopts the following technical solution:
[0007] An optimization method for UAV target detection based on privileged information knowledge transfer includes the following steps:
[0008] S1. Parse the annotation file of the UAV target detection dataset, extract the category, bounding box and instance segmentation label, and use the privileged information processing method to generate a three-channel privileged information map that is pixel-aligned with the RGB image in the UAV target detection dataset;
[0009] S2. Using the YOLOv11 model as the backbone network, a privileged information fusion layer is added to the input of the YOLOv11 model to obtain the teacher model; the RGB image and its three-channel privileged information map from step S1 are stitched together in the channel dimension to form a six-channel image.
[0010] S3. Using a six-channel image as input, train the teacher model in an end-to-end manner to obtain the trained teacher model;
[0011] S4. Freeze the parameters of the trained teacher model. Based on the RGB image in step S1, train the student model using the knowledge distillation loss function to obtain the trained student model.
[0012] S5. Use the trained student model to automatically detect objects in the RGB image to be detected, and output the object category and bounding box position.
[0013] Furthermore, in step S1, generating the three-channel privileged information graph includes the following:
[0014] S101. Normalize the coordinates in the annotation file of the UAV target detection dataset. To convert to absolute pixel coordinates, the specific formula is:
[0015] ;
[0016] ;
[0017] ;
[0018] ;
[0019] in, This indicates the coordinates of the top-left corner of the bounding box. The x and y coordinates of the center point of the target bounding box are represented. Indicates the width of the image size. Indicates the height of the image size. This indicates the coordinates of the bottom right corner of the bounding box. This represents the actual width of the image. Indicates the actual height of the image;
[0020] A bounding box is drawn on the target in the image based on absolute pixel coordinates, and the width of the bounding box is adaptive to the size of the target. The category name of the target inside the box is marked in the upper left corner of each bounding box, and the drawing color is assigned according to the category name, with the font size adaptive to the size of the bounding box. The drawn bounding box is used as the bounding box privilege information.
[0021] S102. Input the RGB images from the UAV target detection dataset into SAM2 (Segment Anything Model 2, image segmentation all-around model 2nd generation). Using the bounding box privileged information as a cue, generate a segmentation mask for the target in real time. Cover the masked area with white on the original image to obtain the synthesized image. Add Gaussian noise to the masked area in the image, using the following formula:
[0022] ;
[0023] in, This represents the image after adding Gaussian noise to the masked region. Indicates a composite image. Indicates Gaussian noise. , Indicates a Gaussian distribution. Indicates standard deviation;
[0024] right By adding global random noise, we obtain the segmentation label privileged information, using the following formula:
[0025] ;
[0026] in, Indicates to Image after adding global random noise, Indicates global random noise. , Indicates a uniform distribution;
[0027] S103. Extract the category names and bounding box coordinates from the annotation file to generate text information. The specific expression is as follows:
[0028] ;
[0029] in, Indicates the first in the annotation file The category name of the target within the bounding box. Indicates the first The x-coordinate of the top left corner of the bounding box Indicates the first The y-coordinate of the top left corner of the bounding box. Indicates the first The x-coordinate of the bottom right corner of the bounding box. Indicates the first The ordinate of the bottom right corner of the bounding box;
[0030] Organize all the bounding box text information into a text file to obtain the label text privilege information;
[0031] We use a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to perform deep semantic encoding on the privileged information of the labeled text, extracting high-order text feature vectors. We use a YOLOv11 model to perform multi-level convolution and pooling on RGB images in the UAV target detection dataset, extracting image feature maps layer by layer. Using a cross-attention fusion module, we use the high-order text feature vector as the query and the image feature map as the key-value pair to adaptively retrieve and weight key visual cues, generating multimodal fusion features. We then upsample and align the multimodal fusion features to the same vector space of the RGB images to obtain the privileged information of the text.
[0032] S104. Using bilinear interpolation, the RGB images in the UAV target detection dataset are enlarged in the spatial domain to the same resolution as the target. For each pixel in the enlarged image, the RGB channel values are calculated using a standard bilinear kernel. The specific formula is as follows:
[0033] ;
[0034] in, This represents the pixels in the enlarged image. All represent interpolation weight coefficients. Represents the four adjacent pixels of a pixel in an RGB image;
[0035] The magnified image is processed through multi-level convolution and pooling operations to obtain a feature map of the same size as the main branch resolution features, which serves as high-resolution privileged information.
[0036] S105. Input the RGB image from the UAV target detection dataset into the graph-to-graph link of the stable diffusion model, and use VAE (Variational Autoencoder) to compress the image into the latent space to obtain the initial latent code z0; according to the target magnification factor... Latent interpolation amplification is performed on the initial latent code z0 to obtain the high-resolution latent code zh. In the denoising loop of DDPM (Denoising Diffusion Probabilistic Models), U-Net is used to progressively predict and remove latent code noise, based on the text prompt words, to generate the final latent code. ;
[0037] The specific formula for predicting and removing latent code noise is as follows:
[0038] ;
[0039] in, Indicates by parameters Defined noise estimation, This represents the latent representation at time step t in the diffusion process. Indicates text prompt words, Indicates channel-level splicing;
[0040] Using VAE-Decoder (Variational Autoencoder-Decoder) to... Mapping to pixel space yields the diffusion generation privileged information;
[0041] The bounding box privilege information, segmentation label privilege information, text privilege information, high resolution privilege information, and diffusion generation privilege information constitute a three-channel privilege information map that is pixel-aligned with the RGB images in the UAV target detection dataset.
[0042] Furthermore, in step S2, the privileged information fusion layer is structurally configured as a dual-modal input interface, with a multi-scale feature aggregation module and an adaptive weight generation module set in parallel; wherein, the multi-scale feature aggregation module includes parallel average pooling units and max pooling units, which are used to capture global and local feature descriptions, respectively; the adaptive weight generation module includes a cascaded multilayer perceptron and a sigmoid activation unit; and a residual structure is set at the output of the privileged information fusion layer.
[0043] The weights of the privileged information fusion layer are initialized using He, with the specific formula as follows:
[0044] ;
[0045] ;
[0046] in, Indicates weight, This indicates the number of units in the input tensor. Indicates the number of input channels. Indicates the height of the convolution kernel. Indicates the width of the convolution kernel;
[0047] The privileged information fusion layer uses a six-channel stitching structure to physically concatenate the RGB images and their three-channel privileged information maps in the UAV target detection dataset along the channel dimension. The specific formula is as follows:
[0048] ;
[0049] in, Represents an RGB image. This represents the concatenation function. This represents a three-channel privileged information diagram. This represents a six-channel image.
[0050] Furthermore, in step S3, the trained teacher model includes the following:
[0051] The six-channel images are input into the teacher model according to a fixed batch size, and forward propagation is performed to output the target detection results. Based on the real labels in the annotation file, the classification loss and regression loss are calculated, and the two losses are weighted and summed according to predetermined weights to form the total loss function. The gradient is calculated through the backpropagation algorithm, and the parameters of the teacher model are updated using a stochastic gradient descent class optimizer.
[0052] After each update, the performance of the teacher model is evaluated; training is stopped when the performance no longer improves or the set number of training rounds is reached; the optimal teacher model parameters are saved as the final weights of the trained teacher model.
[0053] Furthermore, in step S4, the completed student model includes the following:
[0054] S401. Load the trained teacher model, fix all weights, and disable gradient backpropagation;
[0055] S402. In the knowledge distillation stage, the RGB images and six-channel images from the UAV target detection dataset are used as inputs to the student model and the trained teacher model, respectively, while forward propagation is performed. A softmax function with a temperature coefficient is used to smooth the classification score and bounding box prediction; the specific formula is as follows:
[0056] ;
[0057] ;
[0058] in, This represents the classification score and bounding box prediction of the smoothed a-th six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image a in the student model. This represents the classification score and bounding box prediction of the a-th six-channel image in the trained teacher model. Represents an exponential function. This represents the classification score and bounding box prediction of the j-th six-channel image in the trained teacher model. Let represent the classification score and bounding box prediction of the a-th RGB image in the student model. Let represent the classification score and bounding box prediction of the j-th RGB image in the student model. Indicates the temperature coefficient;
[0059] S403. Using KL divergence as the supervision signal, calculate the classification scores and bounding box prediction distillation loss functions of the trained teacher and student models. The specific formula is as follows:
[0060] ;
[0061] ;
[0062] in, This represents the classification score and bounding box prediction of the smoothed six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image in the student model;
[0063] Using cross-entropy loss function The specific formula for supervising the student model's ability to predict the true labels is as follows:
[0064] ;
[0065] in, This represents the true label of the a-th RGB image;
[0066] Total loss Depend on and It is composed of weighted combinations, and the specific formula is as follows:
[0067] ;
[0068] in, The weight coefficients representing the cross-entropy loss; Weighting coefficients representing distillation losses; This indicates the actual label.
[0069] Furthermore, in step S5, the output target category and bounding box location include the following:
[0070] The RGB image to be detected is input into the trained student model, and standardized preprocessing operations are performed to adjust the RGB image to a preset resolution, resulting in a preprocessed image tensor. This image tensor is then forward-propagated, and features are extracted using a convolutional network. The detection head outputs the classification confidence and bounding box regression parameters in parallel. These regression parameters are then post-processed and decoded into actual coordinates. Invalid parameters are filtered out using a confidence threshold, and overlapping redundant detection boxes are removed using a non-maximum suppression algorithm. This yields the target category and high-precision bounding box position, completing the automatic target detection.
[0071] Furthermore, this invention also proposes an optimized UAV target detection system based on privileged information knowledge transfer, comprising:
[0072] The privileged information map acquisition module is used to parse the annotation files of the UAV target detection dataset, extract the category, bounding box and instance segmentation label, and generate a three-channel privileged information map that is pixel-aligned with the RGB images in the UAV target detection dataset using privileged information processing methods.
[0073] The model building module is used to obtain the teacher model by using the YOLOv11 model as the backbone network and adding a privileged information fusion layer at the input of the YOLOv11 model; the RGB image and its three-channel privileged information image in the privileged information image acquisition module are stitched together in the channel dimension to form a six-channel image.
[0074] The model training module is used to train the teacher model in an end-to-end manner using a six-channel image as input to obtain the trained teacher model; the parameters of the trained teacher model are frozen, and the student model is trained using the knowledge distillation loss function based on the RGB image in the privileged information graph acquisition module to obtain the trained student model;
[0075] The results output module is used to automatically detect objects in the RGB image to be detected using the trained student model, and output the object category and bounding box position.
[0076] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the UAV target detection optimization method based on privileged information knowledge transfer.
[0077] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to perform the aforementioned UAV target detection optimization method based on privileged information knowledge transfer.
[0078] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:
[0079] 1. This invention enables label information to participate in more stages of model training, going beyond the traditional limitation of merely serving as a supervisory signal.
[0080] 2. In the training phase, this invention transfers the geometric perception and semantic understanding capabilities of privileged information to a pure visual student model, so that only RGB input is needed in the inference phase to overcome the perception limitations of traditional single-modal detectors.
[0081] 3. The privileged information used in this invention is easy to obtain, avoiding dependence on complex multimodal hardware, and is simple and low in cost.
[0082] 4. This invention does not require significant modifications to the basic network structure of the model, and improves the target detection performance of the model without changing the computational complexity. Attached Figure Description
[0083] Figure 1 This is a flowchart illustrating the overall implementation of the present invention.
[0084] Figure 2 This is the three-channel privileged information diagram of the present invention.
[0085] Figure 3 This is a flowchart of the process for generating a six-channel image according to the present invention.
[0086] Figure 4 This is a flowchart of the student model trained by the present invention.
[0087] Figure 5 This is a comparison chart of detection results obtained by different methods in the embodiments of the present invention. Detailed Implementation
[0088] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0089] To achieve the above objectives, this invention proposes an optimization method for UAV target detection based on privileged information knowledge transfer, such as... Figure 1 As shown, the specific steps are as follows:
[0090] S1, such as Figure 2 As shown, the annotation files of the UAV target detection dataset are parsed to extract category, bounding box, and instance segmentation labels. A privileged information processing method is then used to generate a three-channel privileged information map that is pixel-aligned with the RGB images in the UAV target detection dataset. Specifically:
[0091] S101. Normalize the coordinates in the annotation file of the UAV target detection dataset. To convert to absolute pixel coordinates, the specific formula is:
[0092] ;
[0093] ;
[0094] ;
[0095] ;
[0096] in, This indicates the coordinates of the top-left corner of the bounding box. The x and y coordinates of the center point of the target bounding box are represented. Indicates the width of the image size. Indicates the height of the image size. This indicates the coordinates of the bottom right corner of the bounding box. This represents the actual width of the image. Indicates the actual height of the image;
[0097] A bounding box is drawn on the target in the image based on absolute pixel coordinates, and the width of the bounding box is adaptive to the size of the target. The category name of the target inside the box is marked in the upper left corner of each bounding box, and the drawing color is assigned according to the category name, with the font size adaptive to the size of the bounding box. The drawn bounding box is used as the bounding box privilege information.
[0098] S102. Input the RGB images from the UAV target detection dataset into SAM2 (Segment Anything Model 2, image segmentation all-around model 2nd generation). Using the privileged bounding box information as a cue, generate a segmentation mask for the target in real time. Cover the masked area with white (RGB 255,255,255) over the original image to obtain the synthesized image. Add Gaussian noise to the masked area in the image to prevent the model from overly relying on shortcuts in the privileged information. The specific formula is as follows:
[0099] ;
[0100] in, This represents the image after adding Gaussian noise to the masked region. Indicates a composite image. Indicates Gaussian noise. , Indicates a Gaussian distribution. Indicates standard deviation, ;
[0101] right Global random noise is added to enhance robustness, and the segmentation label privileged information is obtained. The specific formula is as follows:
[0102] ;
[0103] in, Indicates to Image after adding global random noise, Indicates global random noise. , Indicates a uniform distribution;
[0104] S103. Extract the category names and bounding box coordinates from the annotation file to generate text information. The specific expression is as follows:
[0105] ;
[0106] in, Indicates the first in the annotation file The category name of the target within the bounding box. Indicates the first The x-coordinate of the top left corner of the bounding box Indicates the first The y-coordinate of the top left corner of the bounding box. Indicates the first The x-coordinate of the bottom right corner of the bounding box. Indicates the first The ordinate of the bottom right corner of the bounding box;
[0107] Organize all the bounding box text information into a text file to obtain the label text privilege information;
[0108] We utilize a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to perform deep semantic encoding on privileged information in labeled text, extracting high-order text feature vectors with rich syntactic and semantic associations. We then employ a YOLOv11 model to perform multi-level convolution and pooling on RGB images in the UAV target detection dataset, extracting image feature maps that combine spatial structure and visual semantics layer by layer. Using a cross-attention fusion module, we use the high-order text feature vectors as queries and the image feature maps as key-value pairs to adaptively retrieve and weight key visual cues, generating a set of multimodal fusion features that retain original modal characteristics while containing complementary textual information. Finally, we upsample and align these multimodal fusion features to the same vector space of the RGB images to obtain the privileged text information.
[0109] S104. Using bilinear interpolation, the RGB images in the UAV target detection dataset are enlarged in the spatial domain to the same resolution as the target. For each pixel in the enlarged image, the RGB channel values are calculated using a standard bilinear kernel. The specific formula is as follows:
[0110] ;
[0111] in, This represents the pixels in the enlarged image. All represent interpolation weight coefficients. Represents the four adjacent pixels of a pixel in an RGB image;
[0112] The magnified image is processed through multi-level convolution and pooling operations to obtain a feature map of the same size as the main branch resolution features, which serves as high-resolution privileged information.
[0113] S105. Input the RGB images from the UAV target detection dataset into the graph-to-graph link of the stable diffusion model, and use VAE (Variational Autoencoder) to compress the images into the latent space to obtain an initial latent code z0 with a spatial size of 1 / 8 and 4 channels; according to the target magnification factor... Perform latent interpolation amplification on the initial latent code z0 to obtain H / 8× The high-resolution latent code zh (W / 8×4) is used in the DDPM (Denoising Diffusion Probabilistic Models) denoising loop. Using text cue words as conditions, U-Net is used to progressively predict and remove noise from the latent code, generating the final latent code. ;
[0114] The specific formula for predicting and removing latent code noise is as follows:
[0115] ;
[0116] in, Indicates by parameters Defined noise estimation, This represents the latent representation at time step t in the diffusion process. Indicates text prompt words, Indicates channel-level splicing;
[0117] Using VAE-Decoder (Variational Autoencoder-Decoder) to... Mapped to pixel space, diffusion-generated privileged information is obtained, and its texture and edge details have been significantly enhanced by the diffusion prior.
[0118] The bounding box privilege information, segmentation label privilege information, text privilege information, high resolution privilege information, and diffusion generation privilege information constitute a three-channel privilege information map that is pixel-aligned with the RGB images in the UAV target detection dataset.
[0119] S2, such as Figure 3 As shown, a teacher model is obtained by using the YOLOv11 model as the backbone network and adding a privileged information fusion layer at the input of the YOLOv11 model; the RGB image and its three-channel privileged information map from step S1 are stitched together along the channel dimension to form a six-channel image; specifically:
[0120] The privileged information fusion layer is structurally configured with a dual-modal input interface, and sets up a multi-scale feature aggregation module and an adaptive weight generation module in parallel. The multi-scale feature aggregation module includes parallel average pooling units and max pooling units, which are used to capture global and local feature descriptions, respectively. The adaptive weight generation module includes a cascaded multilayer perceptron and a sigmoid activation unit. The output of the privileged information fusion layer is set with a residual structure.
[0121] The weights of the privileged information fusion layer are initialized using He, with the specific formula as follows:
[0122] ;
[0123] ;
[0124] in, Indicates weight, This indicates the number of units in the input tensor. Indicates the number of input channels. Indicates the height of the convolution kernel. Indicates the width of the convolution kernel;
[0125] The privileged information fusion layer uses a six-channel stitching structure to physically concatenate the RGB images and their three-channel privileged information maps in the UAV target detection dataset along the channel dimension. The specific formula is as follows:
[0126] ;
[0127] in, Represents an RGB image. This represents the concatenation function. This represents a three-channel privileged information diagram. This represents a six-channel image.
[0128] S3. Using a six-channel image as input, train the teacher model in an end-to-end manner to obtain the trained teacher model; specifically:
[0129] The six-channel images are input into the teacher model according to a fixed batch size, and forward propagation is performed to output the target detection results. Based on the real labels in the annotation file, the classification loss and regression loss are calculated, and the two losses are weighted and summed according to predetermined weights to form the total loss function. The gradient is calculated through the backpropagation algorithm, and the parameters of the teacher model are updated using a stochastic gradient descent class optimizer.
[0130] After each update, the performance of the teacher model is evaluated; training is stopped when the performance no longer improves or the set number of training rounds is reached; the optimal teacher model parameters are saved as the final weights of the trained teacher model.
[0131] S4, such as Figure 4 As shown, the parameters of the trained teacher model are frozen. Based on the RGB image in step S1, the student model is trained using the knowledge distillation loss function to obtain the trained student model; specifically:
[0132] S401. Load the trained teacher model, fix all weights, prohibit gradient backpropagation, and ensure the stability of teacher knowledge during the distillation process.
[0133] S402. In the knowledge distillation stage, the RGB images and six-channel images from the UAV target detection dataset are used as inputs to the student model and the trained teacher model, respectively, while forward propagation is performed. To enhance the learning of inter-category relationships and alleviate the overconfidence problem of the teacher model's output, a softmax function with a temperature coefficient is used to smooth the classification score and bounding box prediction. The specific formula is as follows:
[0134] ;
[0135] ;
[0136] in, This represents the classification score and bounding box prediction of the smoothed a-th six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image a in the student model. This represents the classification score and bounding box prediction of the a-th six-channel image in the trained teacher model. Represents an exponential function. This represents the classification score and bounding box prediction of the j-th six-channel image in the trained teacher model. Let represent the classification score and bounding box prediction of the a-th RGB image in the student model. Let represent the classification score and bounding box prediction of the j-th RGB image in the student model. Indicates the temperature coefficient. ;
[0137] S403. Using KL divergence as the supervision signal, calculate the classification scores and bounding box prediction distillation loss functions of the trained teacher and student models. The specific formula is as follows:
[0138] ;
[0139] ;
[0140] in, This represents the classification score and bounding box prediction of the smoothed six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image in the student model;
[0141] Using cross-entropy loss function The specific formula for supervising the student model's ability to predict the true labels is as follows:
[0142] ;
[0143] in, This represents the true label of the a-th RGB image;
[0144] Total loss Depend on and It is composed of weighted combinations, and the specific formula is as follows:
[0145] ;
[0146] in, The weights represent the cross-entropy loss coefficients, ensuring that the model always prioritizes learning the true labeled information and avoids distillation noise interfering with basic performance. ; The weighting coefficients representing the distillation loss balance the strength of knowledge transfer and prevent the teacher model's predictions from dominating the training process. ; This indicates the actual label.
[0147] S5. Using the trained student model, perform automatic object detection on the RGB image to be detected, and output the object category and bounding box location; specifically:
[0148] The RGB image to be detected is input into the trained student model, and standardized preprocessing operations are performed to adjust the RGB image to a preset resolution, resulting in a preprocessed image tensor. This image tensor is then forward-propagated, and features are extracted using a convolutional network. The detection head outputs the classification confidence and bounding box regression parameters in parallel. These regression parameters are then post-processed and decoded into actual coordinates. Invalid parameters are filtered out using a confidence threshold, and overlapping redundant detection boxes are removed using a non-maximum suppression algorithm. This yields the target category and high-precision bounding box position, completing the automatic target detection.
[0149] This invention innovatively employs privileged label information for cross-modal knowledge distillation, fully exploring the inherent potential of label information to guide feature learning and avoiding the information waste caused by using it merely as a supervisory signal. Guided solely by privileged information during the training phase, the performance ceiling of a pure visual detector can be significantly improved without relying on complex multimodal hardware, thus significantly reducing experimental costs. Simultaneously, this invention does not require modification of the basic network structure, keeping the computational complexity unchanged.
[0150] Example:
[0151] To verify the effectiveness of the present invention, the method proposed in this invention was applied to an urban traffic monitoring scenario using drones, aiming to improve the detection accuracy of "pedestrians" and "vehicles" by drones at long distances and in complex backgrounds.
[0152] The VisDrone drone target detection dataset was selected, which contains a large number of low-resolution objects captured by drones at high altitudes.
[0153] The training phase used an NVIDIA RTX 3090 GPU and the PyTorch deep learning framework. The initial learning rate was set to 0.01, and the AdamW optimizer was used. The training epochs were set to 200.
[0154] For an image of an urban intersection in VisDrone, generate a three-channel privileged information map that is pixel-aligned with the RGB images in the UAV target detection dataset.
[0155] Taking bounding box privileged information as an example, experiments showed that, since the input includes the geometric contour of the target, the teacher model achieved an average accuracy of 63.2% after 50 epochs of training, demonstrating a very strong feature extraction capability.
[0156] The trained student model learned to still notice the originally blurred vehicle edges in the image even without a mask reference.
[0157] All privileged information processing branches are closed, and only 640×640 resolution RGB images transmitted in real time from the drone camera are received. A visualization comparison of the detection results of this invention with the baseline model detection results is provided below. Figure 5 As shown, the present invention effectively solves the problem of false detection and false negative detection in the model. Simulation tests show that the average accuracy reaches 26.7%.
[0158] This invention also proposes an optimized UAV target detection system based on privileged information knowledge transfer, including a privileged information graph acquisition module, a model building module, a model training module, a result output module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.
[0159] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.
[0160] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that when the computer program is executed by a processor, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.
[0161] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An optimization method for UAV target detection based on privileged information knowledge transfer, characterized in that, include: S1. Parse the annotation file of the UAV target detection dataset, extract the category, bounding box and instance segmentation label, and use the privileged information processing method to generate a three-channel privileged information map that is pixel-aligned with the RGB image in the UAV target detection dataset; S2. Using the YOLOv11 model as the backbone network, a privileged information fusion layer is added to the input of the YOLOv11 model to obtain the teacher model; the RGB image and its three-channel privileged information map from step S1 are stitched together in the channel dimension to form a six-channel image. S3. Using a six-channel image as input, train the teacher model in an end-to-end manner to obtain the trained teacher model; S4. Freeze the parameters of the trained teacher model. Based on the RGB image in step S1, train the student model using the knowledge distillation loss function to obtain the trained student model. S5. Use the trained student model to automatically detect objects in the RGB image to be detected, and output the object category and bounding box position.
2. The UAV target detection optimization method based on privileged information knowledge transfer according to claim 1, characterized in that, In step S1, generating the three-channel privileged information graph includes the following: S101. Normalize the coordinates in the annotation file of the UAV target detection dataset. To convert to absolute pixel coordinates, the specific formula is: ； ； ； ； in, This indicates the coordinates of the top-left corner of the bounding box. The x and y coordinates of the center point of the target bounding box are represented. Indicates the width of the image size. Indicates the height of the image size. This indicates the coordinates of the bottom right corner of the bounding box. This represents the actual width of the image. Indicates the actual height of the image; A bounding box is drawn on the target in the image based on absolute pixel coordinates, and the width of the bounding box is adaptive to the size of the target. The category name of the target inside the box is marked in the upper left corner of each bounding box, and the drawing color is assigned according to the category name, with the font size adaptive to the size of the bounding box. The drawn bounding box is used as the bounding box privilege information. S102. Input the RGB images from the UAV target detection dataset into SAM2, and use the bounding box privileged information as a cue to generate a segmentation mask for the target in real time; cover the masked area with white on the original image to obtain the synthesized image; add Gaussian noise to the masked area in the image, using the following formula: ； in, This represents the image after adding Gaussian noise to the masked region. Indicates a composite image. Indicates Gaussian noise. , Indicates a Gaussian distribution. Indicates standard deviation; right By adding global random noise, we obtain the segmentation label privileged information, using the following formula: ； in, Indicates to Image after adding global random noise, Indicates global random noise. , Indicates a uniform distribution; S103. Extract the category names and bounding box coordinates from the annotation file to generate text information. The specific expression is as follows: ； in, Indicates the first in the annotation file The category name of the target within the bounding box. Indicates the first The x-coordinate of the top left corner of the bounding box Indicates the first The y-coordinate of the top left corner of the bounding box. Indicates the first The x-coordinate of the bottom right corner of the bounding box. Indicates the first The ordinate of the bottom right corner of the bounding box; Organize all the bounding box text information into a text file to obtain the label text privilege information; We use a pre-trained BERT model to perform deep semantic encoding on the privileged information of the labeled text and extract high-order text feature vectors. We use a YOLOv11 model to perform multi-level convolution and pooling on the RGB images in the UAV target detection dataset to extract image feature maps layer by layer. Using a cross-attention fusion module, we use the high-order text feature vector as the query and the image feature map as the key-value pair to adaptively retrieve and weight key visual cues to generate multimodal fusion features. We then upsample and align the multimodal fusion features to the same vector space of the RGB images to obtain the privileged information of the text. S104. Using bilinear interpolation, the RGB images in the UAV target detection dataset are enlarged in the spatial domain to the same resolution as the target. For each pixel in the enlarged image, the RGB channel values are calculated using a standard bilinear kernel. The specific formula is as follows: ； in, This represents the pixels in the enlarged image. All represent interpolation weight coefficients. Represents the four adjacent pixels of a pixel in an RGB image; The magnified image is processed through multi-level convolution and pooling operations to obtain a feature map of the same size as the main branch resolution features, which serves as high-resolution privileged information. S105. Input the RGB image from the UAV target detection dataset into the graph-generated graph link of the stable diffusion model, and use VAE to compress the image into the latent space to obtain the initial latent code z0; according to the target magnification factor... Latent interpolation amplification is performed on the initial latent code z0 to obtain the high-resolution latent code zh. In the DDPM denoising loop, U-Net is used to progressively predict and remove latent code noise, conditioned on text cue words, to generate the final latent code. ; The specific formula for predicting and removing latent code noise is as follows: ； in, Indicates by parameters Defined noise estimation, This represents the latent representation at time step t in the diffusion process. Indicates text prompt words, Indicates channel-level splicing; Using VAE-Decoder Mapping to pixel space yields the diffusion generation privileged information; The bounding box privilege information, segmentation label privilege information, text privilege information, high resolution privilege information, and diffusion generation privilege information constitute a three-channel privilege information map that is pixel-aligned with the RGB images in the UAV target detection dataset.
3. The UAV target detection optimization method based on privileged information knowledge transfer according to claim 1, characterized in that, In step S2, the privileged information fusion layer is structurally configured as a dual-modal input interface, with a multi-scale feature aggregation module and an adaptive weight generation module set in parallel; the multi-scale feature aggregation module includes parallel average pooling units and max pooling units, which are used to capture global and local feature descriptions, respectively; the adaptive weight generation module includes a cascaded multilayer perceptron and a sigmoid activation unit; the output of the privileged information fusion layer is set with a residual structure; The weights of the privileged information fusion layer are initialized using He, with the specific formula as follows: ； ； in, Indicates weight, This indicates the number of units in the input tensor. Indicates the number of input channels. Indicates the height of the convolution kernel. Indicates the width of the convolution kernel; The privileged information fusion layer uses a six-channel stitching structure to physically concatenate the RGB images and their three-channel privileged information maps in the UAV target detection dataset along the channel dimension. The specific formula is as follows: ； in, Represents an RGB image. This represents the concatenation function. This represents a three-channel privileged information diagram. This represents a six-channel image.
4. The UAV target detection optimization method based on privileged information knowledge transfer according to claim 1, characterized in that, In step S3, the trained teacher model includes the following: The six-channel images are input into the teacher model according to a fixed batch size, and forward propagation is performed to output the target detection results. Based on the real labels in the annotation file, the classification loss and regression loss are calculated, and the two losses are weighted and summed according to predetermined weights to form the total loss function. The gradient is calculated through the backpropagation algorithm, and the parameters of the teacher model are updated using a stochastic gradient descent class optimizer. After each update, the performance of the teacher model is evaluated; training is stopped when the performance no longer improves or the set number of training rounds is reached; the optimal teacher model parameters are saved as the final weights of the trained teacher model.
5. The UAV target detection optimization method based on privileged information knowledge transfer according to claim 1, characterized in that, In step S4, the trained student model includes the following: S401. Load the trained teacher model, fix all weights, and disable gradient backpropagation; S402. In the knowledge distillation stage, the RGB images and six-channel images from the UAV target detection dataset are used as inputs to the student model and the trained teacher model, respectively, while forward propagation is performed. A softmax function with a temperature coefficient is used to smooth the classification score and bounding box prediction; the specific formula is as follows: ； ； in, This represents the classification score and bounding box prediction of the smoothed a-th six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image in the student model. This represents the classification score and bounding box prediction of the a-th six-channel image in the trained teacher model. Represents an exponential function. Let represent the classification score and bounding box prediction of the j-th six-channel image in the trained teacher model. Let represent the classification score and bounding box prediction of the a-th RGB image in the student model. Let represent the classification score and bounding box prediction of the j-th RGB image in the student model. Indicates the temperature coefficient; S403. Using KL divergence as the supervision signal, calculate the classification scores and bounding box prediction distillation loss functions of the trained teacher and student models. The specific formula is as follows: ； ； in, This represents the classification score and bounding box prediction of the smoothed six-channel image in the trained teacher model. This represents the classification score and bounding box prediction of the smoothed RGB image in the student model; Using cross-entropy loss function The specific formula for supervising the student model's ability to predict the true labels is as follows: ； in, This represents the true label of the a-th RGB image; Total loss Depend on and It is composed of weighted combinations, and the specific formula is as follows: ； in, The weight coefficients representing the cross-entropy loss; Weighting coefficients representing distillation losses; This indicates the actual label.
6. The UAV target detection optimization method based on privileged information knowledge transfer according to claim 1, characterized in that, In step S5, the output of the target category and bounding box location includes the following: The RGB image to be detected is input into the trained student model, and standardized preprocessing operations are performed to adjust the RGB image to a preset resolution, resulting in a preprocessed image tensor. This image tensor is then forward-propagated, and features are extracted using a convolutional network. The detection head outputs the classification confidence and bounding box regression parameters in parallel. These regression parameters are then post-processed and decoded into actual coordinates. Invalid parameters are filtered out using a confidence threshold, and overlapping redundant detection boxes are removed using a non-maximum suppression algorithm. This yields the target category and high-precision bounding box position, completing the automatic target detection.
7. A system applied to the UAV target detection optimization method based on privileged information knowledge transfer as described in claim 1, characterized in that, include: The privileged information map acquisition module is used to parse the annotation files of the UAV target detection dataset, extract the category, bounding box and instance segmentation label, and generate a three-channel privileged information map that is pixel-aligned with the RGB images in the UAV target detection dataset using privileged information processing methods. The model building module is used to obtain the teacher model by using the YOLOv11 model as the backbone network and adding a privileged information fusion layer at the input of the YOLOv11 model; the RGB image and its three-channel privileged information image in the privileged information image acquisition module are stitched together in the channel dimension to form a six-channel image. The model training module is used to train the teacher model in an end-to-end manner using a six-channel image as input to obtain the trained teacher model; the parameters of the trained teacher model are frozen, and the student model is trained using the knowledge distillation loss function based on the RGB image in the privileged information graph acquisition module to obtain the trained student model; The results output module is used to automatically detect objects in the RGB image to be detected using the trained student model, and output the object category and bounding box position.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the UAV target detection optimization method based on privileged information knowledge transfer as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the UAV target detection optimization method based on privileged information knowledge transfer as described in any one of claims 1 to 6.