A DDETR small sample target detection method based on transfer learning fine tuning

Abstract

The application provides a DDETR small sample target detection method based on transfer learning fine tuning. The method proposes a small sample target detection basic framework FS_DDETR, adds a category filtering module, a category remapping module, a classification result mapping module and a DDETR detection network with an adjustable classification head to the original DDETR network, and supports different stages and different evaluation methods of the small sample target detector; and a transfer learning fine tuning strategy based on the DDETR network is designed under the framework, so that the FS_DDETR reaches a quite competitive performance in the small sample target detection task on the related data set.

Description

Technical Field

[0001] This invention relates to a DDETR few-shot object detection method based on transfer learning fine-tuning, belonging to the field of computer vision technology. Background Technology

[0002] Object detection is fundamental to many visual tasks, such as instance segmentation, object tracking, image captioning, and behavior recognition, and has also been applied in real-world scenarios such as video surveillance, autonomous driving, and robot visual navigation. As one of the most fundamental and challenging areas of computer vision, object detection has received widespread attention and achieved significant progress in recent years. Thanks to the rapid development of big data and computer hardware, deep learning-based object detection algorithms have become dominant. For example, Fast R-CNN, proposed in 2015, uses deep convolutional networks to detect target regions (RossGirshick.Fast R-CNN.CVPR 2015). DEtection TRansformer, an end-to-end object detection network based on Transformer, was proposed in 2020 (DETR, Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers. European conference on computer vision, 2020:213-229). The subsequently proposed Deformable DETR network, abbreviated as DDETR (Zhu X, Su W, Lu L, et al. Deformable DETR: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,2020), solves the attention problem in standard Transformers through deformable attention, accelerating model convergence and reducing algorithm complexity. It also introduces multi-scale features to address the performance limitations of small object detection.

[0003] Generally, deep neural networks require a large amount of labeled data for training to enable the model to fully grasp the data distribution and converge to satisfactory results. Therefore, in object detection tasks based on deep neural networks, sufficient, structured labeled samples are often needed for supervised training of the detector. These structured labels include the location and category of the object in the image. However, in practical applications, obtaining labeled samples requires significant manual annotation, which is time-consuming and costly. While self-supervised object detection algorithms exist that only require sufficient samples without structured labels, at present, fully self-supervised object detection algorithms cannot compare with supervised object detection algorithms. Furthermore, aside from the required annotation resources, simply obtaining sufficient images is difficult in some scenarios, such as applications involving endangered species or medical imaging. The limited sample size and annotation difficulties significantly restrict the performance improvement of object detectors, and the differences between different scenarios also make object detector transfer difficult. This means that when performing object detection tasks in new application scenarios, a large amount of new labeled data must be obtained to train a new model, which undoubtedly increases costs.

[0004] However, even young children can quickly distinguish new objects from a small number of samples. Similarly, few-sample object detection tasks attempt to detect new classes using only a very small number of labeled samples. Therefore, few-sample object detection can alleviate the dependence of traditional object detectors on large amounts of labeled data and has important application value, attracting widespread attention from researchers in recent years.

[0005] However, training high-performance object detectors using only a very small number of labeled samples presents numerous challenges. First, as a multi-task learning model, object detection requires simultaneously regressing the object's location and predicting its category, making the task inherently difficult. Second, with extremely limited sample sizes, the model is prone to severe overfitting, poor generalization performance, high variance, and difficulty in fully understanding the data distribution, leading to a decline in model performance. Furthermore, existing small-sample object detection algorithms often employ a two-stage training strategy, with each stage involving different network structures and evaluation methods. Existing network model training frameworks cannot simultaneously meet these specific training requirements. Summary of the Invention

[0006] The purpose of this invention is to provide a transfer learning-based fine-tuning method for DDETR (Deformable DETR) few-shot object detection, hereinafter referred to as FS_DDETR (Few-Shot DDETR). This method is based on the DDETR object detection network framework, and designs a network framework specifically for few-shot object detection. It adds a class filter module, a class remapping module, a classification result mapping module (ClassResReID), and an adjustable DDETR network module to the original DDETR network, integrating model training and inference at different stages and under different evaluation metrics into this framework. Furthermore, a transfer learning fine-tuning strategy based on the DDETR network is designed within this framework, enabling FS_DDETR to achieve highly competitive performance in few-shot object detection tasks.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A DDETR small-sample target detection method based on transfer learning fine-tuning, characterized in that:

[0009] Using the DDETR object detection network as the basic network framework, we implement few-shot object detection based on transfer learning fine-tuning. The training process is divided into two stages: a basic training stage and a fine-tuning stage. In the basic training stage, we use base class data for training. In the fine-tuning training stage, we select different M-way-K-shot few-shot datasets depending on the evaluation metric: when using FSOD as the evaluation metric, we only use a few-shot dataset composed of new classes for training; when using G-FSOD as the evaluation metric, we use a few-shot dataset containing both base classes and new classes for training.

[0010] The method described above is characterized by using the DDETR target detection network as the basic network framework to achieve few-sample target detection based on transfer learning fine-tuning, and includes the following steps:

[0011] (1) Filtering target categories: For input data, the ClassFilter module is used to filter out labels that do not belong to the target category under the current settings. That is, in the basic training stage, the model only needs to use the labels of the base class and does not need additional labels belonging to the new class. Therefore, the labels of the new class are filtered out. In the fine-tuning stage, the evaluation index is used to determine whether to filter out the labels belonging to the base class to prevent the model from "seeing" labels that do not conform to the current settings during the training process.

[0012] (2) Class remapping: For the dataset after filtering the classes, the Class ReID module is used to map discontinuous class labels to continuous class labels for model training and evaluation, and to save the mapping table for use by subsequent modules;

[0013] (3) Adjustable Classification Head DDETR Detection Network: During training, different classification heads are used for the DDETR detection network according to different stages and evaluation metrics. The proposed adjustable classification head DDETR detection network includes three classification heads: base, novel FSOD, and novel G-FSOD. The base classification head is used in the basic training stage, which needs to classify all categories of the base class. In the fine-tuning stage, there are two optional classification heads, novel FSOD and novel G-FSOD, depending on different evaluation metrics: if FSOD is used as the evaluation metric, the novel FSOD classification head is used, which only needs to classify all categories of the new class; if G-FSOD is used as the evaluation metric, the novel G-FSOD classification head is used, which needs to classify all categories of both the base class and the new class.

[0014] (4) Transfer learning fine-tuning strategy based on DDETR network: The training steps in the basic training stage are the same as those in the traditional DDETR detection network model. The fine-tuning stage uses the transfer learning fine-tuning strategy based on DDETR network. When the model is inferenced, the detection results are directly output for the next step.

[0015] (5) Mapping classification results: During model inference, the classification result mapping module (ClassResReID) remaps the inference results output by the model back to the original labels, thereby supporting the official evaluation API of MS COCO.

[0016] The method described above is characterized by the following steps in training the DDETR detection network model in step (4):

[0017] The input image first enters the feature extraction module CNN, which uses ResNet50 as the backbone network and combines it with FPN (Feature Pyramid Networks) to complete the extraction of multi-level features and obtain multi-level feature maps.

[0018] The multi-level feature maps pass through the channel alignment module (CAM). The CAM contains a 1×1 convolutional layer, which is used to adjust the number of channels in the feature maps so that they can be uniformly input into the encoder. After passing through the channel alignment module (CAM), the size of the feature maps remains unchanged, but the number of channels becomes a given value.

[0019] The length and width dimensions of the multi-level feature map processed by the channel alignment module CAM are flattened to one dimension, added to the position embedding, and fed into the encoder.

[0020] The encoder processes the input multi-level feature maps using deformable self-attention.

[0021] The image features output by the encoder are input into the decoder. At the same time, the object query is also input into the decoder. In the decoder, a deformable cross-attention module is used to complete the interaction between the object query and the image features output by the encoder, thereby enhancing the features of the object query.

[0022] The enhanced object query is passed through the FFN layer to predict the target location and category. The FFN layer contains a bounding box regression subnetwork and a classification subnetwork, which are used to output bounding box regression and classification prediction results, respectively.

[0023] The loss of the output bounding box regression and classification prediction results is calculated once and used as the cost matrix of the bipartite graph matching algorithm. Then, the bipartite graph matching algorithm is used to match the bounding box regression results, classification prediction results and label information. The loss used to update the parameters is calculated based on the matching results, and the network parameters are updated through gradient backpropagation to complete one iteration.

[0024] After obtaining the bounding box regression results, classification prediction results, and label information matching results from the model output, the loss function is calculated using GIoU Loss and Focal Loss, respectively.

[0025] The method described above is characterized in that the transfer learning fine-tuning strategy based on the DDETR network in step (4) includes:

[0026] The parameter updates of the feature extraction module CNN, encoder, and decoder in the model are frozen, while the parameter updates of the FFN layer and channel alignment module CAM are opened. A single-stage DDETR detector is used, and the target box iterative refinement strategy is not used. GIoU Loss and Focal Loss are used as bounding box regression and classification loss functions, respectively, to constrain the parameter updates of the model.

[0027] Compared with existing technologies, the proposed FS_DDETR few-shot object detection method based on transfer learning fine-tuning of DDETR in this invention designs a few-shot object detection network based on the Transformer framework, integrating model training and inference under different stages and evaluation metrics into a unified framework. Furthermore, it proposes a transfer learning fine-tuning strategy to solve the problem of DDETR's performance degradation with a small number of training samples, thereby improving the model's detection capability. Experimental results show that FS_DDETR outperforms DDETR and TFA based on Faster R-CNN on 10-shot and 30-shot few-shot object detection tasks on the MS COCO dataset, demonstrating highly competitive performance compared to other recent improvements based on TFA. Attached Figure Description

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 This is a diagram illustrating the overall framework of the small-sample target detection method FS_DDETR based on DDETR described in this invention.

[0030] Figure 2 This is a schematic diagram of a transfer learning fine-tuning strategy based on a DDETR network.

[0031] Figure 3 This is a schematic diagram illustrating the effect of the category filtering module;

[0032] Figure 4 Comparison of some feature maps before and after updating the CAM open parameters for the channel alignment module. Detailed Implementation

[0033] As previously mentioned, this invention proposes a DDETR small sample target detection method based on transfer learning fine-tuning. The specific implementation of this invention is described below with reference to the accompanying drawings.

[0034] (1) Overall framework

[0035] Figure 1 The overall framework of this invention is described, integrating the training and inference of the model under different stages and evaluation metrics into a unified framework. FS_DDETR consists of five processing steps: (1) filtering target categories; (2) category remapping; (3) DDETR detection with adjustable classification head; (4) fine-tuning based on transfer learning of the DDETR network; and (5) mapping classification results.

[0036] Small sample datasets consist of two main categories: base classes with B categories and sufficient samples in each category, and new classes with N categories and a finite number of K samples in each category. In the MS COCO dataset, out of all 80 categories, the 20 categories overlapping with the VOC dataset are typically used as new classes, and the remaining 60 categories are used as base classes. This is to validate the model's transfer performance from the COCO dataset to the VOC dataset. The base classes use all samples and labels for the corresponding categories in the COCO dataset, while the new classes are obtained by sampling from the COCO dataset. During sampling, for each category, K samples and their corresponding labels are randomly sampled from the corresponding category in the dataset each time, forming an M-way-K-shot small sample dataset.

[0037] The base class dataset is used in the basic training phase, while the new class dataset is used in the transfer learning fine-tuning phase. Taking the COCO dataset as an example, the base class data containing 60 classes is used for training in the basic training phase; in the fine-tuning training phase, different M-way-K-shot few-shot datasets can be selected depending on the evaluation metric. There are two evaluation metrics: FSOD (Few-Shot Object Detection) and G-FSOD (Generalized Few-Shot Object Detection). FSOD only focuses on the detection performance of the detection model on new classes, while G-FSOD focuses not only on the detection performance of the detection model on new classes, but also on the detection performance on the base class and all classes. When using FSOD as the evaluation metric, M=20, and training is performed using a 20-way-K-shot few-shot dataset consisting of only 20 new classes; when using G-FSOD as the evaluation metric, M=80, and training is performed using an 80-way-K-shot few-shot dataset containing both base classes and new classes.

[0038] For the input dataset, the class filter module is first used to filter out labels that do not belong to the target category under the current settings. This prevents the model from "seeing" labels that do not conform to the current settings during training, which would cause the final model evaluation performance to be higher than the actual performance. For the dataset after class filtering, the class remapping module (ClassReID) is used to map discontinuous class labels to continuous class labels for model training and evaluation.

[0039] The categories the model needs to classify vary depending on the training stage and evaluation metrics. Therefore, different DDETR detection networks with different classification heads are required for different stages and evaluation metrics. The model training within the framework includes a basic training stage and a fine-tuning stage. The training steps in the basic training stage are the same as those of traditional DDETR detection models. The fine-tuning stage uses the transfer learning fine-tuning strategy based on the DDETR network proposed in this invention, as illustrated in the diagram below. Figure 2 As shown, this strategy freezes the parameter updates of the feature extraction module (CNN), encoder, and decoder in the model, while opening the parameter updates of the FFN layer and the channel alignment module (CAM). Although the two-stage DDETR detector and the iterative refinement strategy of the bounding box have been shown in related papers to improve the detection performance of the model, overly complex modules can lead to overfitting and decreased model accuracy when there are few training samples. Therefore, this invention uses a single-stage DDETR detector and does not use the iterative refinement strategy of the bounding box. In addition, GIoU Loss and Focal Loss are used as bounding box regression and classification loss functions, respectively, to constrain the parameter updates of the model.

[0040] During model inference, the ClassRes ReID module is used to remap the inference results output by the model back to the original labels, thereby supporting the official MS COCO evaluation API.

[0041] (2) Filter target categories

[0042] The effect diagram of the Class Filter module is shown below. Figure 3 As shown, an image may contain both base class and new class labels. The purpose of the Class Filter is to prevent the model from "seeing" labels that do not conform to the current settings during training. It filters out labels that do not belong to the target class under the current settings, based on the categories used during training. Specifically, during base class training and evaluation, only the base class categories are used, not the new class categories, so the new class categories are filtered out. Similarly, during new class training and evaluation, it is necessary to determine whether to filter out labels belonging to the base class based on the evaluation metric: if G-FSOD is chosen as the evaluation metric, no filtering is needed; however, if FSOD is chosen as the evaluation metric, labels belonging to the base class need to be filtered out.

[0043] (3) Category remapping

[0044] After category filtering, the input dataset becomes discontinuous in its class labels, making it unsuitable for direct model training. Therefore, Class ReID is needed to map these discontinuous class labels to continuous ones, and this mapping relationship is recorded in a mapping table. A Python dictionary structure is used as the mapping table for subsequent modules to use.

[0045] (4) DDETR detection with adjustable classification head

[0046] Based on different training stages and evaluation metrics, the classification heads used are determined. The proposed DDETR detection network with adjustable classification heads includes three types of classification heads: base, novel FSOD, and novel G-FSOD. During the basic training stage, the base classification head is used, which needs to classify all categories of the base class. During the fine-tuning stage, depending on the evaluation metric, there are two optional classification heads: novel FSOD and novel G-FSOD. If FSOD is used as the evaluation metric, the novel FSOD classification head is used, and only all categories of the new class need to be classified. If G-FSOD is used as the evaluation metric, the novel G-FSOD classification head is used, and in this case, both the base class and the new class need to be classified.

[0047] (5) Transfer learning fine-tuning strategy based on DDETR network

[0048] After determining the class leader, the model is fine-tuned using the remapped dataset and a transfer learning fine-tuning strategy based on the DDTER network.

[0049] (5.1) The input image first enters the feature extraction module CNN, which uses ResNet50 as the backbone network and combines it with FPN (Feature Pyramid Networks) to extract multi-level features, resulting in multi-level feature maps. In the multi-level feature maps, shallow feature maps usually contain rich image appearance information, such as edges, textures, and contours, while deep feature maps contain rich semantic information, such as information related to the object category in the image. In object detection tasks, both shallow and deep features play important roles in object localization and regression. In addition, in object detection tasks, the same image usually contains multiple objects of different sizes, and deep feature maps, due to multiple downsampling, often struggle to detect small objects. Therefore, current object detection networks typically perform multi-level feature fusion, detecting small objects on large feature maps and large objects on small feature maps.

[0050] (5.2) The multi-level feature maps pass through the channel alignment module (CAM), encoder, decoder, and FFN layer to finally predict the target location and target category. Here, the transfer learning fine-tuning strategy freezes the parameter updates of the feature extraction module (CNN), encoder, and decoder, while opening the parameter updates of the FFN layer and channel alignment module (CAM). We compared the differences between the extracted multi-level feature maps before and after freezing the channel alignment module (CAM), and the visualization results of some feature maps are shown below. Figure 4 As shown in the figure, the first column is the original input image, the second column is a partial feature map output by the frozen channel alignment module (CAM) when updating its parameters, and the third column is the corresponding feature map output by the open channel alignment module (CAM) after updating its parameters. The comparative visualization results show that updating the parameters of the open channel alignment module (CAM) enriches the feature information and makes the target contour clearer.

[0051] Single-stage detectors directly detect the location of the target, while two-stage detectors pre-extract candidate boxes and then perform regression and classification based on these candidate boxes to obtain more accurate target box locations. In existing work, two-stage detectors generally outperform single-stage detectors. However, with limited training samples, the quality of candidate boxes output by two-stage detectors is extremely poor, and low-quality candidate boxes negatively impact the final detection results. Iterative refinement of the target box refers to the output of the current decoder layer being an offset relative to the output of the previous decoder layer, improving the regression accuracy of the target box through this layer-by-layer refinement. However, with limited training samples, more layers in the regression network mean a larger number of parameters and an expression space, making the model more prone to overfitting and leading to performance degradation. Therefore, the transfer learning fine-tuning strategy proposed in this invention uses a single-stage DDETR detector and does not employ an iterative refinement strategy for the target box.

[0052] (5.3) The Channel Alignment Module (CAM) is used to unify the number of channels in multi-level features. The CAM module contains a 1×1 convolutional layer to adjust the number of channels in the feature map so that it can be uniformly input into the encoder later. After passing through the CAM module, the size of the feature map remains unchanged, but the number of channels becomes a given value c1′.

[0053] (5.4) The length and width dimensions of the multi-level feature map processed by the channel alignment module (CAM) are flattened to one dimension, added to the position embedding, and fed into the encoder. This is because the input to the encoder is a one-dimensional token, rather than a feature map containing both length and width dimensions.

[0054] (5.5) The encoder processes the input multi-level feature map according to formula (1) of the deformable self-attention module. The deformable self-attention module borrows the idea of ​​deformable convolution. Each pixel on the feature map only needs to interact with a portion of the sampled pixels, without needing to calculate the interaction with all points. The positions of these sampled pixels are learnable, thus realizing a local rather than global sparse attention mechanism, which greatly reduces the model complexity and computational cost. Among them, for the feature point z with index q on a specific feature map... q The enhanced features are located at position on each feature map. The features are linearly weighted. M is the number of attention heads, L is the number of feature maps, and K is the number of sampling points for each feature map. It is z q The reference point location, It is the reference point corresponding to each feature map layer, Δp mlqk and A mlqk These are the position offset and the level of attention, respectively.

[0055]

[0056] (5.6) The input to the decoder consists of two parts: the image features enhanced by the encoder and the object query. In the decoder, a deformable cross-attention module is used to facilitate the interaction between the object query and the image features output by the encoder. Similar to the deformable self-attention module, the deformable cross-attention module enhances the features of the object query, gradually leading it to converge towards a specific target feature.

[0057] (5.7) The enhanced object query is passed through the FFN layer to predict the target location and category. The FFN layer contains a bounding box regression subnetwork and a classification subnetwork, which are used to output the bounding box regression and classification prediction results, respectively.

[0058] (5.8) Calculate the loss of the output bounding box regression and classification prediction results as the cost matrix of the bipartite graph matching algorithm. Then, use the bipartite graph matching algorithm to match the bounding box regression results, classification prediction results and label information. Calculate the loss used to update the parameters based on the matching results, and update the network parameters through gradient backpropagation to complete one iteration.

[0059] (5.9) After obtaining the bounding box regression results, classification prediction results, and label information matching results from the model output, the loss functions are calculated using GIoU Loss and Focal Loss, respectively. For the predicted bounding box A and the ground truth bounding box B in the label output, IoU Loss calculates the ratio of the intersection and union of the two positions. GIoU Loss is an improvement on IoU Loss. The IoU Loss formula is shown in Equation (2), and the GIoU Loss formula is shown in Equation (3). Where C represents the area of ​​the minimum bounding rectangle of the ground truth bounding box and the predicted bounding box, and D represents the difference between the minimum bounding rectangle of the ground truth bounding box and the predicted bounding box and their union. Unlike IoU, which only focuses on the overlapping area, GIoU not only focuses on the overlapping area but also on other non-overlapping areas, which can better reflect the degree of overlap between the two. For the classification confidence p and the class information y in the label output, Focal Loss solves the problem of class imbalance and imbalance of easy and difficult samples in the dataset to a certain extent. Its formula is shown in Equation (4). Here, α and γ are hyperparameters representing the degree of attention given to imbalanced and difficult samples. Focal Loss can assign different training weights to different classes by statistically analyzing the number of targets in each class. Simultaneously, it limits the gradient contribution of easy samples, ensuring that the gradient information of difficult samples is not overwhelmed by the gradient information of easy samples.

[0060]

[0061]

[0062]

[0063] (6) During inference, the model output needs to be remapped. ClassResReID is used to remap the inference results output by the model back to the original labels using the mapping table.

[0064] The above-disclosed examples are merely specific instances of the present invention. Any variations that can be conceived by those skilled in the art based on the ideas provided by the present invention should fall within the protection scope of the present invention.

Claims

1. A DDETR small-sample target detection method based on transfer learning fine-tuning, characterized in that: Using the DDETR object detection network as the basic network framework, few-shot object detection based on transfer learning fine-tuning is implemented. The training process is divided into two stages: the basic training stage and the fine-tuning stage, including the following steps: (1.1) Filtering target categories: For input data, the class filter module is used to filter out labels that do not belong to the target category under the current settings. That is, in the basic training stage, the model only needs to use the labels of the base class and train using the base class data, without needing additional labels belonging to the new class. Therefore, the labels of the new class are filtered out. In the fine-tuning stage, the evaluation metric determines whether to filter out the labels belonging to the base class to prevent the model from "seeing" labels that do not conform to the current settings during training. That is, depending on the evaluation metric, different M-way-K-shot small sample datasets are selected: when using FSOD as the evaluation metric, only the small sample dataset composed of the new class is used for training; when using G-FSOD as the evaluation metric, the small sample dataset containing both the base class and the new class is used for training. (1.2) Class remapping: For the dataset after filtering the classes, the Class ReID module is used to map discontinuous class labels to continuous class labels for model training and evaluation, and to save the mapping table for use by subsequent modules; (1.3) DDETR detection network with adjustable classification head: During training, different classification heads are used for the DDETR detection network according to different stages and different evaluation metrics. The proposed DDETR detection network with adjustable classification head contains three classification heads: base, novel FSOD and novel G-FSOD. The base classification head is used in the basic training stage, which needs to classify all categories of the base class. In the fine-tuning stage, there are two optional classification heads, novel FSOD and novel G-FSOD, depending on different evaluation metrics: if FSOD is used as the evaluation metric, the novel FSOD classification head is used, which only needs to classify all categories of the new class; if G-FSOD is used as the evaluation metric, the novel G-FSOD classification head is used, which needs to classify all categories of both the base class and the new class. (1.4) Transfer learning fine-tuning strategy based on DDETR network: The training steps in the basic training stage are the same as those in the traditional DDETR detection network model. The fine-tuning stage uses a transfer learning fine-tuning strategy based on DDETR network. During model inference, the detection results are directly output for the next step. The training steps of the DDETR detection network model are as follows: The input image first enters the feature extraction module CNN, which uses ResNet50 as the backbone network and combines it with FPN (Feature Pyramid Networks) to complete the extraction of multi-level features and obtain multi-level feature maps; The multi-level feature maps pass through the channel alignment module (CAM). The CAM contains a 1×1 convolutional layer, which is used to adjust the number of channels in the feature maps so that they can be uniformly input into the encoder. After passing through the channel alignment module (CAM), the size of the feature maps remains unchanged, but the number of channels becomes a given value. The length and width dimensions of the multi-level feature map after processing by the channel alignment module CAM are flattened to one dimension, added to the position embedding, and fed into the encoder. The encoder processes the input multi-level feature maps using deformable self-attention. The image features output by the encoder are input into the decoder. At the same time, the object query is also input into the decoder. In the decoder, a deformable cross-attention module is used to complete the interaction between the object query and the image features output by the encoder, thereby enhancing the features of the object query. The enhanced object query is passed through the FFN (Feed-Forward-Network) layer to predict the target location and category. The FFN layer contains a bounding box regression subnetwork and a classification subnetwork, which are used to output bounding box regression and classification prediction results, respectively. The loss of the output bounding box regression and classification prediction results is calculated once and used as the cost matrix of the bipartite graph matching algorithm. Then, the bipartite graph matching algorithm is used to match the bounding box regression results, classification prediction results and label information. The loss used to update the parameters is calculated based on the matching results, and the network parameters are updated through gradient backpropagation to complete one iteration. After obtaining the bounding box regression results, classification prediction results, and label information matching results from the model output, the loss function is calculated using GIoU Loss and Focal Loss, respectively. (1.5) Mapping classification results: During model inference, the classification result mapping module (ClassResReID) remaps the inference results output by the model back to the original labels, thereby supporting the official evaluation API of MS COCO.

2. The DDETR small-sample target detection method based on transfer learning fine-tuning as described in claim 1, wherein the transfer learning fine-tuning strategy based on the DDETR network in step (1.4) includes: The parameter updates of the feature extraction module CNN, encoder, and decoder in the model are frozen, while the parameter updates of the FFN layer and channel alignment module CAM are opened. A single-stage DDETR detector is used, and the target box iterative refinement strategy is not used. GIoU Loss and Focal Loss are used as bounding box regression and classification loss functions, respectively, to constrain the parameter updates of the model.

Images