Generalized small sample remote sensing image target detection method based on double-path knowledge distillation

Abstract

The invention relates to a generalized small sample remote sensing image target detection method based on double-path knowledge distillation. The method comprises a basic training stage and a fine tuning stage. In the base training stage, abundant base class samples are utilized to train the network, and initial weights of a backbone network, base class branches, output branches and new class branches are obtained; and a fine adjustment stage: on the basis of the weight obtained by the base training, performing fine adjustment on a partial region of the network by using a small amount of new-class samples, and simultaneously performing distillation of base-class knowledge and distillation of new-class knowledge. According to the method, two distillation strategies of response-based knowledge distillation and feature-based knowledge distillation are combined, so that the accuracy of the detection result is further improved. The method effectively maintains the base class detection performance while improving the adaptability of the model to the new class, and is suitable for various target detection application scenes of remote sensing images, automatic driving and medical treatment.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically relates to a generalized small-sample remote sensing image target detection method based on dual-path knowledge distillation within the field of target detection technology. This invention can be used for target detection in remote sensing images under small-sample conditions. Background Technology

[0002] Generalized few-shot object detection is an important subtask within the field of few-shot object detection, aiming to learn new knowledge while simultaneously retaining previously learned knowledge. Training for few-shot object detection typically employs a two-stage strategy: first, pre-training on a large amount of base class data, and then fine-tuning and expanding the model by introducing new class data. However, during fine-tuning, the introduction of new class knowledge may impact existing base class knowledge, leading to a decline in base class detection performance. This problem is known as catastrophic forgetting, where the model loses its ability to handle older tasks when learning new ones. Therefore, a core issue in generalized few-shot object detection is how to effectively mitigate the problem of base class knowledge forgetting.

[0003] Xin Wang et al. proposed a simple yet effective small-sample object detection method in their paper "Frustratingly Simple Few-ShotObject Detection" published at ICML 2020. The implementation steps of this method are: (1) Base class pre-training: First, the Faster R-CNN model is trained on a base class dataset containing a large number of labeled samples to learn general image representation and category discrimination capabilities; (2) Feature freezing: The backbone network parameters of the pre-trained model are frozen as fixed feature extractors, and only a small number of updates are made on the head detector; (3) New class fine-tuning: Rapid fine-tuning is performed on a novel class dataset containing very few samples, and only the detection head is adapted to improve the model's ability to detect new categories. Depending on the training strategy, the detection head or RPN module can also be partially unfrozen to introduce new category information while maintaining the stability of the backbone. The core idea of ​​this method is to maximize the reuse of existing general feature extraction capabilities and achieve efficient detection of new categories through the rapid adaptation of the detection head. This "freeze + fine-tuning" mechanism significantly reduces the difficulty of model training and achieves excellent few-shot detection performance on multiple standard datasets (such as PascalVOC and COCO). However, this method still has shortcomings, namely the catastrophic forgetting problem. Because its fine-tuning stage focuses entirely on the adaptation learning of new classes and does not explicitly introduce a mechanism to preserve the knowledge of the base class in the optimization objective, the model's ability to recognize the original base class drops significantly when adapting to the novel class. In addition, this method lacks feature consistency constraints and response behavior supervision between new and old classes, and cannot effectively alleviate the representation conflict between new and old tasks. Therefore, when facing generalized few-shot object detection scenarios, it is difficult to achieve unified modeling and collaborative representation of new and old classes.

[0004] In their paper “DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection” published at ICCV 2021, Limeng Qiao et al. proposed a decoupled few-shot object detection method. The implementation steps of this method are as follows: (1) Pre-training on the base class: First, standard Faster R-CNN is trained on a sufficiently labeled base class dataset to build a general image feature extractor and classification framework. At this stage, a unified backbone network is used to extract image features, and all region proposal features are mapped to a shared representation space to support subsequent new class adaptation; (2) Freezing the backbone and reconstructing the detection head: The pre-trained backbone feature extractor is fixed, and the classifier in the detection head is reconstructed. A cosine similarity-based metric classifier is introduced to replace the traditional linear classifier, thereby improving the ability to distinguish new classes and the feature normalization effect; (3) Fine-tuning on the new class: The new class is fine-tuned in an episodic-based manner. In this stage, a few-shot episode, analogous to the few-shot task setting, is constructed to train the classifier in the detection head using a very small number of support samples. To alleviate overfitting in few-shot scenarios, a gradient detachment strategy is employed during fine-tuning, ensuring that the classifier training process does not affect the backbone features, thus maintaining the stability of the original feature space. Although DeFRCN introduces decoupled training, feature normalization, and episode-level adaptation strategies in its structure, achieving excellent performance in few-shot detection scenarios, this method still suffers from a catastrophic forgetting problem. Its fine-tuning stage focuses entirely on the detection head adaptation of the novel class, lacking an explicit mechanism for preserving the response behavior of the base class; simultaneously, it does not introduce knowledge preservation strategies or distillation constraints for the old class, making it difficult to prevent the training process of the new class from eroding the recognition ability of the base class. Furthermore, although its feature space is normalized, no cross-task consistency model is established between the old and new classes, limiting its collaborative expression of old and new knowledge in generalized few-shot object detection scenarios. Therefore, this method still struggles to fundamentally solve the catastrophic forgetting problem when faced with complex scenarios where base / novel classes coexist in real-world tasks.

[0005] The University of Electronic Science and Technology of China proposed a small-sample target detection method based on feature space structure decoupling in its patent application document "A method, system and terminal for small-sample target detection based on feature space structure decoupling" (application number: CN202410442379.5, publication number: CN118172544A, publication date: 2024-06-11). The implementation steps of this method are: (1) pre-training on the base class: pre-training the base class model on a base class containing a large number of labeled samples; (2) fine-tuning preparation: copying the pre-trained model and training the copied model on the base class with a dataset of the same number as the new class; (3) adding the parameters obtained after training the copied model to the pre-trained base class model; wherein, the fine-tuning preparation specifically includes: supplementing the foreground information of the new class on the region proposal network; and performing high-level feature vector class correlation processing on the feature vectors output by the model prediction layer. Fine-tuning on the new class: Using the base class model obtained after fine-tuning as the teacher model and the new class model as the student model, knowledge distillation techniques are added to the backbone network and region proposal network of the new class model, respectively. Fine-tuning ends after the new class model converges. The shortcomings of this method are: it mainly introduces knowledge distillation in the backbone network and RPN layer, lacking collaborative constraints on the detector output response layer, making it difficult to guarantee the consistency of prediction behavior between the old and new classes. Furthermore, although it employs high-level feature class correlation processing, it does not establish an explicit feature-response collaborative mechanism, limiting the depth of knowledge retention for the old class. In addition, this method does not clearly distinguish the importance and role of knowledge in the feature layer and response layer, making it difficult to achieve comprehensive optimization from multiple perspectives when one distillation path dominates. Therefore, although this method has some innovation in feature decoupling and distillation mechanisms, it still lacks a dual-path distillation mechanism that can integrate multi-granularity knowledge and collaboratively optimize the expression of the old and new tasks in the face of catastrophic forgetting challenges in general scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technology by proposing a generalized small sample target detection method based on dual-path knowledge distillation, which aims to solve the catastrophic forgetting problem caused by the introduction of new sample types during the fine-tuning process in the generalized small sample target detection task.

[0007] The technical approach to achieve the purpose of this invention is to simultaneously perform distillation of base class knowledge and distillation of new class knowledge, and to combine two distillation strategies: response-based knowledge distillation and feature-based knowledge distillation. A two-stage training strategy is adopted, namely, firstly, the network is trained using rich base class samples, and then, based on the weights obtained from the base training, a small number of new class samples are used to fine-tune some regions of the network.

[0008] In the base training phase, the model utilizes the YOLOv11 backbone network to extract multi-level feature representations of the input data. These features are then fed into three identical YOLOv11 neck network branches (Neck_base, Neck_all, Neck_novel) and their corresponding head branches (Head_base, Head_all, Head_novel) for parameter training. During this phase, the network model fully leverages large-scale base class data to learn rich feature representations and base class knowledge, thereby constructing a network model with powerful feature extraction capabilities. The network weights obtained in this phase will serve as the basis for subsequent fine-tuning.

[0009] During the fine-tuning phase, the backbone network is frozen to maintain its good representation of base class features. At this point, the network contains three branches: base class branches (Neck_base, Head_base), output branches (Neck_all, Head_all), and new class branches (Neck_novel, Head_novel). The base class branch is frozen and dedicated solely to detecting base classes; the output and new class branches remain trainable to adapt to new class data. To effectively mitigate the interference of new class knowledge on base class knowledge and balance the model's detection capabilities for both base and new classes, a two-way distillation mechanism is introduced in this phase. The base class and new class branches act as teacher networks, passing learned knowledge to the output branch through knowledge distillation. The output branch is trained using data and labels from all classes and also acts as a student network, receiving guidance from the base and new class branches, thereby improving its detection capabilities for all classes. During the inference phase, only the output branches are used to output prediction results; the base and new class branches are no longer used, thus avoiding additional computational overhead.

[0010] The steps of this invention include the following:

[0011] Step 1: Construct an object detection network consisting of the YOLOv11 backbone network and three parallel branches: a base class branch, a new class branch, and an output branch;

[0012] Step 2: Use the base class data to perform base training on the object detection network to obtain the initial weights of the network;

[0013] Step 3: Freeze the trained backbone network and base class branches; fine-tune the training of the new class branches and output branches using new class data; use both the base class branches and new class branches as teacher networks to pass the learned knowledge to the output branches, and obtain the trained dual-path knowledge distillation target detection network.

[0014] Step 4: Input the image to be detected into the trained object detection network. After the backbone network extracts features, the output branch performs inference to obtain the final object detection result.

[0015] Furthermore, the base class branch, new class branch, and output branch are all composed of the YOLOv11 neck sub-network and the detection head connected in series.

[0016] Furthermore, the steps of the base training are as follows:

[0017] The first step is to input the base class data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data;

[0018] The second step involves feeding multi-level features into three parallel branches: the base class branch, the new class branch, and the output branch. The network parameters are then iteratively updated via the YOLOv11 neck network branch and the corresponding detection head branch until the loss function of the base-trained network converges, thus obtaining the base-trained target detection network.

[0019] The loss function L of the base training network v11 To maintain consistency with the original YOLOv11 settings, specifically:

[0020] L v11 =L Cls +L Reg +L DFL

[0021] Among them, L Cls L Reg and L DFL These represent the binary cross-entropy classification loss function, the CIOU loss function, and the DFL loss function, respectively.

[0022] Furthermore, the fine-tuning training of the new class branch and the output branch refers to training the response-based dual-path knowledge distillation generalized few-shot target network and the feature-based dual-path knowledge distillation generalized few-shot target detection network, respectively.

[0023] The steps for training the response-based dual-path knowledge distillation generalized few-shot target network are as follows:

[0024] The first step is to load the network weights obtained from base training;

[0025] The second step is to input the new type of data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data.

[0026] The third step involves feeding the multi-level features into the new class branch, the base class branch, and the output branch, respectively. The prediction results for each branch are obtained through the YOLOv11 neck network branch and the corresponding detection head branch. The data labels for each branch are different. The base class branch contains only the labels for the base class data, and the new class is treated as background. This branch is specifically responsible for detecting the base class. The new class branch contains only the labels for the new class data, and the base class is treated as background. This branch is specifically responsible for detecting the new class. The output branch contains the labels for all class data and is responsible for detecting all classes.

[0027] The fourth step involves fine-tuning the new class branch by substituting the prediction results of the new class branch into the YOLOv11 loss function to calculate the weight update value. For fine-tuning the output branch, as a student network, the prediction results of the output branch not only need to be substituted into the YOLOv11 loss function for calculation, but also need to be distilled with the prediction results of the new class branch and the base class branch to calculate the weight update value.

[0028] Fifth, iterate through step three until the response-based two-way knowledge distillation loss function converges.

[0029] The response-based dual-path knowledge distillation loss function is as follows:

[0030] L distillation =αL dis-base +βL dis-novel

[0031] Among them, L dis-base and L dis-novel α and β represent the losses incurred during the fine-tuning phase of distilling regression and classification results when using the teacher network to perform response-based knowledge distillation on the student network. α and β represent the weights of the base class distillation loss and the new class distillation loss, respectively. α and β are set to 1 and 5, respectively.

[0032] The steps for training the feature-based dual-path knowledge distillation generalized few-shot object detection network are as follows:

[0033] The first step is to load the network weights obtained from base training;

[0034] The second step is to input the new type of data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data.

[0035] The third step is to feed the multi-level features into the new class branch, the base class branch and the output branch, obtain the feature map of each branch through the YOLOv11 neck network branch, and then obtain the prediction result of each branch through the corresponding detection head branch.

[0036] The fourth step involves fine-tuning the new class branch by substituting its prediction result into the YOLOv11 loss function to obtain the weight update value. For fine-tuning the output branch, as a student network, the prediction result of the output branch needs to be substituted into the YOLOv11 loss function for calculation. At the same time, the feature map of the output branch also needs to be distilled with the feature maps of the base class branch and the purified new class branch to obtain the weight update value.

[0037] Fifth, iterate through step three until the feature-based dual-path knowledge distillation loss function converges.

[0038] The feature-based dual-path knowledge distillation loss function is as follows:

[0039] L disF =L disF1 +L disF2

[0040] Among them, L disF1 L disF2 These represent the distillation loss of base class features and the distillation loss of new class features, respectively.

[0041] Furthermore, the inference process of the output branch refers to the following: the image to be detected is input into the trained target detection network. First, the backbone network extracts initial features, and then the initial features are input into the neck network of the output branch to obtain features with a higher degree of abstraction. Finally, the target detection result is obtained through the detection head of the output branch.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] First, this invention effectively maintains base class detection performance by freezing the base class branches (Neck_base and Head_base) and using them to perform response distillation and feature distillation on the output branches (Neck_all and Head_all). During the fine-tuning phase, because Neck_base and Head_base are frozen, accurate base class prediction probabilities and rich base class features are provided. The output branches, acting as student networks, significantly mitigate the catastrophic forgetting problem and prevent the loss of base class knowledge during fine-tuning by aligning with and inheriting intermediate layer features from the prediction results of the base class branches.

[0044] Second, this invention guides the student network (output branch) to learn refined new class representations through new class branch distillation (Neck_novel and Head_novel) and an innovative new class feature purification mechanism, thereby improving new class detection accuracy and enhancing new class adaptability. In response-based knowledge distillation, the new class branch adjusts the prediction probability of the output branch for the new class through response distillation. In feature-based knowledge distillation, to address the issue that the new class branch may carry base class knowledge, this invention introduces a feature purification operation. This mechanism calculates the dot product projection of the new class branch features and the base class branch features, removing the parts similar to the base class, thus obtaining enhanced new class features. These purified new class features are then used as guiding knowledge to perform feature distillation on the output branch, ensuring that the output branch can effectively learn pure new class knowledge, thereby significantly improving new class detection performance.

[0045] Third, this invention employs a "three-way detection branch structure" (base class branch, new class branch, and output branch) and a "dual-path distillation mechanism," enabling the output branch to simultaneously receive guidance from both the base class branch and the new class branch. During the fine-tuning phase, the base class branch and the new class branch are trained using their respective data and labels, and the learned knowledge is transferred to the output branch through response distillation and feature distillation. This parallel distillation strategy effectively balances the model's learning of the base class and the new class, mitigating the catastrophic forgetting of base class knowledge while enhancing the model's detection performance for the new class. Simulation results demonstrate that this mechanism avoids inter-class conflicts and improves the detection capability of the output branch.

[0046] Fourth, although this invention introduces a multi-branch structure and a complex two-way distillation mechanism during the training phase, it only utilizes the output branch to output the prediction result during the inference phase. The base class branch and the new class branch are only used during the training phase to guide the learning of the output branch by the teacher network, and are no longer used during inference. Therefore, this invention does not increase the number of model parameters or inference latency, maintains the lightweight model structure and has no added burden during the inference phase, and ensures practicality.

[0047] Fifth, this invention uses the backbone and neck network of YOLOv11 as its basic architecture and successfully implements a knowledge distillation strategy on top of it. This invention is adaptable to various mainstream one-stage target detectors, such as YOLOv5 and YOLOv11, and has good scalability and practical value, with strong method versatility. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the response-based dual-path knowledge distillation process of the present invention;

[0049] Figure 2 This is a schematic diagram of the feature-based dual-path knowledge distillation process of the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0051] The algorithm network of this embodiment is an improvement on YOLOv11, and its architecture includes the following main parts: (1) Backbone: responsible for extracting multi-scale features from the input image. The YOLOv11 backbone network has been optimized to extract richer and more discriminative features. (2) Neck: connects the backbone network and the detection head, and is used to fuse feature maps from different scales to enhance the model's ability to detect objects of different sizes. (3) Head: the final part of the network, which predicts the coordinates, confidence, and class probability of the bounding box based on the fused feature map.

[0052] The backbone of this invention uses the YOLOv11 backbone network to extract multi-level feature representations from the input data. The neck network consists of three identical neck branches: Neck_base, Neck_all, and Neck_novel, all of which are YOLOv11 neck networks. The detection head also consists of three identical detection head branches: Head_base, Head_all, and Head_novel, which are also YOLOv11 detection heads. All three neck branches share the same backbone network. In short, unlike the single-path detection branch of the original YOLO, this method employs a three-path detection branch structure. Only the output branches are used to output prediction results; the base class branch and the new class branch are no longer used, but only serve as a teacher network to guide the learning of the output branches during the training phase.

[0053] Response-based dual-path knowledge distillation and feature-based dual-path knowledge distillation use the same network structure and have identical base training phases. However, during the fine-tuning phase, the loss function is adjusted accordingly because the distillation positions for response-based and feature-based methods differ.

[0054] In this invention, both the response-based method and the feature-based method can independently complete the generalized small-sample target detection task. However, to obtain more accurate detection results, both methods can be used simultaneously. The two methods are described in detail below in Examples 1 and 2.

[0055] Example 1 proposes a response-based dual-path knowledge distillation generalized small-sample target detection method, including:

[0056] Base training phase: The network model fully utilizes large-scale base class data to learn rich feature representations and base class knowledge, thereby building a model with powerful feature extraction capabilities. The network weights obtained in this phase will serve as the basis for subsequent fine-tuning.

[0057] Fine-tuning phase: Freeze the backbone network and base class branches obtained from base training to maintain the detection performance of the base class branches on the base class. The output branches and new class branches remain trainable to adapt to the new class data.

[0058] Inference phase: During the inference phase, only the output branches are used to output the prediction results. The base class branch and the new class branch will no longer be used. They are only used during the training phase as the teacher network to guide the learning of the output branches.

[0059] The following is combined with Figure 1 The response-based dual-path knowledge distillation process of Embodiment 1 of the present invention will be further described.

[0060] Figure 1 The Backbone is the YOLOv11 backbone network, responsible for obtaining multi-level feature representations from the input data. The neck network consists of three identical neck branches: Neck_base, Neck_all, and Neck_novel, all of which are YOLOv11 neck networks and share the same backbone network. The head network also contains three identical head branches: Head_base, Head_all, and Head_novel, which are also the same YOLOv11 head.

[0061] The base training phase specifically includes: using the traditional strategy of transfer learning and two-stage training, training the Backbone and all three branches with base class data to obtain initial weights.

[0062] The fine-tuning phase includes: First, freezing the Backbone and the base class branch to maintain the base class branch's detection performance for the base class. The output branch and the new class branch remain trainable to adapt to the new class data. During fine-tuning training, all three branches receive the same input data, but the data labels for each branch are different. The base class branch contains only the labels for the base class data, and the new class is treated as background; this branch is specifically responsible for detecting the base class. The new class branch contains only the labels for the new class data, and the base class is treated as background; this branch is specifically responsible for detecting the new class. The output branch contains the labels for all class data and is responsible for detecting all classes. The base class branch and the new class branch are trained using their respective data and labels, and the knowledge they learn is passed to the output branch through knowledge distillation. The output branch is trained using data and labels for all classes and, as a student network, receives guidance from the base class branch and the new class branch, thereby improving its detection ability for all classes. In the inference phase, only the output branch is used to output prediction results; the base class branch and the new class branch are no longer used, but only serve as teacher networks to guide the learning of the output branch during the training phase.

[0063] The loss function in the fine-tuning phase includes:

[0064] In YOLOv11, the model output consists of regression and classification results. The regression output has 4 × reg_max, where 4 represents the offset of the bounding box (top, bottom, left, right) relative to the center point, and reg_max is the number of discrete set points used to calculate the DFL loss. If reg_max is 16, the regression output has 64 channels. The classification output has the same number of channels as the number of categories, nc. The total number of output channels for object detection is the sum of the regression and classification output channels, 64 + nc. When using the teacher network to perform response-based knowledge distillation on the student network, the regression and classification results need to be distilled separately.

[0065] The loss function for distillation uses the following mean square error (MSE) loss:

[0066]

[0067] Where N represents the number of samples, x i and y i These represent the results of the teacher network and the results of the student network, respectively.

[0068] Two distillation losses L during the fine-tuning stage dis-base and L dis-novel They are represented as follows:

[0069] L dis-base =MSE(H_base[:64],H_all[:64])

[0070] +MSE(H_base[64:],H_all[64:64+nc base ])

[0071] L dis-novel =MSE(H_novel[:64],H_all[:64])

[0072] +MSE(H_novel[64+nc base :64+nc],H_all[64+nc base :64+nc])

[0073] Among them, L dis-base H_base represents the base class distillation loss, and H_base represents the output of the base class detection head, whose output dimension is 64+nc. base H_all indicates the output of the detection head, with a dimension of 64 + nc. Ldis-base The first term in equation (3) is the distillation loss for calculating the regression results, where H_base[:64] and H_all[:64] represent the first 64 terms of H_base and H_all, respectively. The second term is the distillation loss for calculating the classification results. In equation (3), the first nc base Each category represents the base class, followed by nc. novel Each category represents a new class.

[0074] The total loss from response-based knowledge distillation is as follows:

[0075] L distillation =αL dis-base +βL dis-novel

[0076] Wherein, α and β represent the weights of the base class distillation loss and the new class distillation loss, respectively. In the embodiments of the present invention, α and β are taken as 1 and 5, respectively.

[0077] During the fine-tuning phase, the total loss function is expressed as follows:

[0078] L disL =L distillation +L Cls +L Reg +L DFL .

[0079] The present invention proposes a feature-based dual-path knowledge distillation generalized small sample target detection method in Embodiment 2.

[0080] In terms of model structure, both feature-based two-way knowledge distillation and feature-based two-way knowledge distillation use the same model structure, such as... Figure 2 As shown. However, in object detection, response-based knowledge distillation distills the regression and classification results of the model, while feature-based knowledge distillation distills the feature maps of the intermediate layers. Therefore, in the fine-tuning stage, the specific location of distillation and the loss function in Example 2 are different from those in Example 1.

[0081] The loss function for base class feature distillation still adopts the MSE loss function; therefore, the base class feature distillation loss L disF1 It can be represented as:

[0082] L disF1 =MSE(F base ,F all )

[0083] Among them, F base and F all These represent the characteristics of the base class branch and the output branch, respectively.

[0084] The purification process for the new class of features is as follows:

[0085] F res =F novel -F proj +F base

[0086]

[0087] Among them, F base and F novel F represents the feature maps output by the base class branch and the new class branch, respectively. res This represents the features of the new class after purification. <F1,F2> indicates the calculation of the dot product of F1 and F2. proj F represents novel In F base The projection on the surface.

[0088] The purification steps for the new class of features are as follows: first, calculate F... novel In F base The projection onto F yields F base With F novel The similar part, namely F proj Then, using the feature F of the new class branch. novel Subtract F proj , to obtain F base With F novel For the dissimilar parts, add these dissimilar parts back to the feature F of the new class branch. novel In this case, we obtain the feature map after enhancing the features of the new class. Further understanding, F... base With F novel The similarities are more in base class features and background features, while F base With F novel Dissimilar parts generally represent features of a new class, F novel The base class features and background features in the model result in enhancing the feature representation of the new class while weakening the feature representation of the base class.

[0089] After obtaining the enhanced new class branch features, the feature distillation of the new class branch can be performed according to the following formula:

[0090] L disF2 =MSE(F res ,F all )

[0091] Final total loss function:

[0092] L disF =L disF1 +L disF2 +L Cls +L Reg +L DFL

[0093] The effects of the present invention will be further described below with reference to simulation experiments.

[0094] 1. Simulation conditions.

[0095] The simulation environment for this invention is as follows: the operating system is Linux Ubuntu 20.04.1, the CPU is an AMD EPYC7302, and the GPU used is an NVIDIA GeForce RTX 3090. The code and framework used are Python and PyTorch, with PyTorch version 2.0.0.

[0096] The datasets and settings used in the simulation of this invention are as follows: This invention uses two remote sensing datasets, NWPUv2 and DIOR, for simulation. For the NWPUv2 dataset, the base classes contain seven categories: basketball court, bridge, track and field, port, ship, oil tank, and vehicle; the new classes contain three categories: airplane, baseball field, and tennis court. In the base training phase, a total of 100 epochs were trained, with a constant batch size of 8. The initial learning rate was set to 0.01, and the learning rate was gradually reduced during training, eventually reaching 0.1 times the initial learning rate. SGD was used as the optimizer. Simultaneously, a mosaic data augmentation strategy was enabled during training. In the fine-tuning phase, a total of 20,000 epochs were trained, with the batch size, optimizer, and data augmentation strategy consistent with the base training phase. The initial learning rate was 0.001, and the final learning rate was reduced to 0.001 times the initial learning rate. For the DIOR dataset, the base classes include 15 categories: airplanes, airports, highway service areas, highway toll booths, dams, golf courses, athletic fields, ports, overpasses, stadiums, oil storage tanks, tennis courts, train stations, vehicles, and windmills. The remaining categories—baseball fields, basketball courts, bridges, chimneys, and ships—are new classes. The base training phase consists of 200 epochs, with all other settings identical to the NWPUv2 dataset.

[0097] 2. Simulation content and result analysis.

[0098] To verify the effectiveness of the response-based knowledge distillation and feature-based knowledge distillation strategies proposed in this invention for mitigating base class knowledge forgetting, two simulation experiments were conducted. The method of this invention and five existing algorithms—Meta-YOLO, P-CNN, TFA, G-FSDet, and ST-FSOD—were used to conduct simulation experiments on the remote sensing datasets NWPUv2 and DIOR, respectively.

[0099] Simulation Experiment 1 of this invention is a simulation experiment conducted on the remote sensing dataset NWPUv2, and the comparison results are shown in Table 1. Meta-YOLO is a classic few-shot target detection algorithm. However, due to its poor baseline performance and early proposal, its performance advantage compared to newer algorithms is not significant, especially in terms of new class detection accuracy. P-CNN provides a benchmark for few-shot target detection in remote sensing images, and subsequent few-shot class classifications are mostly based on this benchmark. P-CNN has significantly improved in new class detection accuracy, but its overall performance is still not good. TFA sacrifices new class detection performance for improved base class detection accuracy, and its performance in detecting base classes is extremely excellent. However, because only the classifier and regressor are fine-tuned during the fine-tuning stage, its performance in detecting new classes is very poor. G-FSDet improves TFA, which can alleviate the problem of reduced base class detection accuracy while improving new class detection accuracy. ST-FSOD uses a semi-supervised approach to learn unlabeled target features, which improves the detection effect of new classes to a certain extent.

[0100] Table 1 Comparison of simulation results of different algorithms on the NWPUv2 dataset

[0101]

[0102]

[0103] The method employed in this invention significantly optimizes the mitigation of base class knowledge forgetting. In the 3-shot scenario, the base class detection performance decreases by only 4.28% compared to the base training phase; in the 10-shot scenario, the performance decreases by only 0.10%; and in the 20-shot scenario, knowledge forgetting is almost nonexistent. The results demonstrate that this mechanism can avoid inter-class conflicts and improve the detection capability of output branches.

[0104] As shown in Table 1, with the gradual increase of training samples, the three indicators—base class detection accuracy, new class detection accuracy, and overall detection accuracy—all show a gradual increasing trend. The dual-path distillation mechanism proposed in this invention employs a three-path detection branch structure combined with a dual-path knowledge distillation strategy. Both the base class branch and the new class branch can fully learn in their respective data domains, avoiding the situation where base class and new class features are similar. Furthermore, using the base class branch and the new class branch to distill the output branch separately can effectively alleviate base class knowledge forgetting while reducing inter-class conflicts.

[0105] Simulation Experiment 2 of this invention was performed on the DIOR dataset, and the results are compared in Table 2. Similar to the NWPUv2 dataset, Meta-YOLO performs poorly in new class detection on the DIOR dataset. While P-CNN improves new class detection performance compared to Meta-YOLO, its overall performance in base class, new class, and global class detection remains poor. Meta-YOLO and P-CNN are benchmark algorithms in their respective fields, but compared to some recent algorithms, their base class and new class detection accuracy is relatively low. The TFA algorithm significantly improved base class detection accuracy, but new class detection still falls short of practical requirements. The G-FSDet algorithm balances the issue of high base class performance but poor new class performance in TFA, sacrificing some base class detection performance for a significant improvement in new class detection performance. It also employs knowledge distillation to alleviate the problem of base class knowledge forgetting. ST-FSOD elevates the performance of few-shot object detection to a new level, fully utilizing unlabeled targets in training images. Compared to ordinary few-shot object detection algorithms, it learns more comprehensive features. However, the aforementioned algorithms either use a poorly performing one-stage object detection algorithm as a baseline or a two-stage object detection algorithm with poor real-time performance, resulting in poor detection accuracy or poor real-time performance. Meanwhile, current one-stage object detection algorithms are already quite excellent, comparable to two-stage detectors in terms of detection accuracy. Thanks to the powerful performance of YOLOv11, this invention achieves a significant performance advantage over previous methods.

[0106] Table 2 Comparison of Simulation Results of Different Algorithms on the DIOR Dataset

[0107]

[0108]

[0109] As can be seen from Table 2, the detection accuracy of the base class, new class, and overall class gradually improves with the increase of the number of training samples. However, when the number of training samples reaches a certain level, the improvement in the detection accuracy of the base class will no longer be significant. For example, the detection accuracy of the base class is comparable under 10-shot and 20-shot conditions. This is because as the number of samples increases, the output branch can replay more of the knowledge of the base class, and the problem of forgetting the knowledge of the base class will no longer be significant.

[0110] Compared to the latest few-shot object detection algorithm ST-FSOD, the method proposed in this invention outperforms it in various metrics. During the base training phase, the detection accuracy of this invention on the base class test set is 82.10%. After fine-tuning, the detection accuracy of this invention on the base class under the various shots mentioned above only decreases by 3.41%, 3.05%, 2.44%, and 2.44% respectively compared to the base training phase. This demonstrates the powerful effect of this invention in mitigating the catastrophic forgetting problem of base class knowledge. Furthermore, due to the dual-path distillation mechanism, the new class branch can effectively distill the output branch, enabling the output branch to strengthen the detection of new classes. Therefore, the method proposed in this invention improves both base class detection performance and new class detection performance.

Claims

1. A generalized small-sample remote sensing image target detection method based on dual-path knowledge distillation is proposed, characterized in that... Simultaneously, distillation of base class knowledge and distillation of new class knowledge are performed; the steps of this detection method include the following: Step 1: Construct an object detection network consisting of a backbone network and three parallel branches: a base class branch, a new class branch, and an output branch; Step 2: Use the base class data to perform base training on the object detection network to obtain the initial weights of the network; Step 3: Freeze the trained backbone network and base class branches; The new class data is used to fine-tune the training of the new class branch and the output branch respectively; the base class branch and the new class branch are used as teacher networks at the same time to pass the learned knowledge to the output branch, so as to obtain the trained dual-path knowledge distillation target detection network. Step 4: Input the image to be detected into the trained object detection network. After the backbone network extracts features, the output branch performs inference to obtain the final object detection result.

2. The target detection method according to claim 1, characterized in that, The base class branch, new class branch, and output branch mentioned in step 1 are all composed of the YOLOv11 neck sub-network and the detection head connected in series.

3. The target detection method according to claim 2, characterized in that, The base training steps described in step 2 are as follows: The first step is to input the base class data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data; The second step is to feed the multi-level features into three parallel base class branches, new class branches, and output branches, and then iteratively update the network parameters through the YOLOv11 neck network branch and the corresponding detection head branch until the loss function of the base-trained network converges. A well-trained target detection network is obtained.

4. The target detection method according to claim 3, characterized in that, The loss function L of the base training network v11 To maintain consistency with the original YOLOv11 settings, specifically: L v11 =L Cls +L Reg +L DFL Among them, L Cls L Reg and L DFL These represent the binary cross-entropy classification loss function, the CIOU loss function, and the DFL loss function, respectively.

5. The target detection method according to claim 1, characterized in that, The fine-tuning training of the new class branch and the output branch mentioned in step 3 refers to training the response-based dual-path knowledge distillation generalized few-shot target network and the feature-based dual-path knowledge distillation generalized few-shot target detection network, respectively.

6. The target detection method according to claim 5, characterized in that, The steps for training the response-based dual-path knowledge distillation generalized few-shot target network are as follows: The first step is to load the network weights obtained from base training; The second step is to input the new type of data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data. The third step involves feeding the multi-level features into the new class branch, the base class branch, and the output branch, respectively. The prediction results for each branch are obtained through the YOLOv11 neck network branch and the corresponding detection head branch. The data labels for each branch are different. The base class branch contains only the labels for the base class data, and the new class is treated as background. This branch is specifically responsible for detecting the base class. The new class branch contains only the labels for the new class data, and the base class is treated as background. This branch is specifically responsible for detecting the new class. The output branch contains the labels for all class data and is responsible for detecting all classes. The fourth step involves fine-tuning the new class branch by substituting the prediction results of the new class branch into the YOLOv11 loss function to calculate the weight update value. For fine-tuning the output branch, as a student network, the prediction results of the output branch not only need to be substituted into the YOLOv11 loss function for calculation, but also need to be distilled with the prediction results of the new class branch and the base class branch to calculate the weight update value. Fifth, iterate through step three until the response-based two-way knowledge distillation loss function converges.

7. The target detection method according to claim 6, characterized in that, The response-based dual-path knowledge distillation loss function is as follows: L distillation =αL dis-base +βL dis-novel Among them, L dis-base and L dis-novel α and β respectively represent the losses of distilling regression and classification results during the fine-tuning stage when using the teacher network to perform response-based knowledge distillation on the student network. α and β respectively represent the weights of the base class distillation loss and the new class distillation loss. In this invention, α and β are 1 and 5 respectively.

8. The target detection method according to claim 5, characterized in that, The steps for training the feature-based dual-path knowledge distillation generalized few-shot object detection network are as follows: The first step is to load the network weights obtained from base training; The second step is to input the new type of data into the object detection network and use the YOLOv11 backbone network to extract multi-level feature representations of the input data. The third step is to feed the multi-level features into the new class branch, the base class branch and the output branch, obtain the feature map of each branch through the YOLOv11 neck network branch, and then obtain the prediction result of each branch through the corresponding detection head branch. The fourth step involves fine-tuning the new class branch by substituting its prediction result into the YOLOv11 loss function to obtain the weight update value. For fine-tuning the output branch, as a student network, the prediction result of the output branch needs to be substituted into the YOLOv11 loss function for calculation. At the same time, the feature map of the output branch also needs to be distilled with the feature maps of the base class branch and the purified new class branch to obtain the weight update value. Fifth, iterate through step three until the feature-based dual-path knowledge distillation loss function converges.

9. The target detection method according to claim 8, characterized in that, The feature-based dual-path knowledge distillation loss function is as follows: L disF =L disF1 +L disF2 Among them, L disF1 L disF2 These represent the distillation loss of base class features and the distillation loss of new class features, respectively.

10. A terminal, comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that, When the processor executes computer instructions, it performs the relevant steps in the generalized small-sample remote sensing image target detection method according to any one of claims 1-9.

Images