A small model object detection distillation method based on dual cues

Abstract

The present invention relates to the field of target detection technology, and discloses a small model target detection distillation method based on dual prompts. The external prompt distillation operation initializes the teacher external prompt through the teacher feature, and interacts through the multi-head self-attention mechanism to avoid information duplication; the teacher external prompt is updated in a momentum update manner; after the student feature is initialized, multi-head cross-attention interaction is performed with the teacher external prompt, and then added back to the student feature in the form of residual; the internal prompt distillation operation initializes the student internal prompt, multiplies it with the student feature, and then updates the student feature; the updated student feature is subjected to the ConvLoRA operation to obtain the final student feature in the backbone network stage; the parameters of the student model are updated through the feature distillation operation; the present invention can achieve better performance in small model distillation.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small model target detection distillation method based on double prompts. Background Art

[0002] Object detection, a classic task in computer vision, aims to identify objects in images or videos, determine their categories, and accurately calibrate their locations using minimal bounding boxes. This technology has broad applications in fields such as autonomous driving, surveillance systems, robotic vision, medical image analysis, and aerospace. However, applying these large and computationally intensive models to the real world presents a series of challenges, including model size, inference speed, and limited resources. These challenges limit the deployment and application of these powerful models in real-world scenarios.

[0003] Knowledge distillation technology is an important approach to addressing these issues. Knowledge distillation involves using a large, trained model as the teacher model and a small, untrained model as the student model. During training, the teacher model's test results supervise the student model's test results, allowing "knowledge" to be distilled from the large model to the small model, thereby improving the performance of the small model. However, existing knowledge distillation methods mostly focus on scenarios where the teacher and student models are of similar size. A few methods explore the case of a large teacher model, but distilling knowledge into smaller, faster, and lightweight models is rarely discussed. Existing distillation methods also offer limited performance improvements for lightweight models. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a small model target detection distillation method based on dual prompts, which can effectively transfer the knowledge of the large model to the small model, thereby achieving more obvious performance improvement of the small model.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A dual-cue-based small-model object detection distillation method transfers knowledge from a teacher model to a student model through a knowledge distillation process. During testing, an image is input into the student model that has completed the knowledge distillation process to detect objects and their categories. Both the teacher model and the student model include a backbone network and a neck network, and the knowledge distillation process includes a backbone network stage and a neck network stage. The object detection distillation method specifically includes the following steps:

[0007] Step 1: The backbone network stage includes external and internal cue distillation operations. Specifically, it includes the following steps:

[0008] S11, the external prompt distillation operation includes: the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompt E, and avoid information duplication through multi-head self-attention mechanism interaction; update the teacher's external prompt E in the form of momentum update; at the same time, the student feature F output by the backbone network of the student model S Perform the initialization operation, then perform multi-head cross attention interaction with the updated teacher external prompt, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S middle;

[0009] S12, the internal prompt distillation operation includes: initializing the learnable student internal prompt, and the student feature F output by the backbone network of the student model S Multiply, then perform Sigmoid operation to obtain the mask of student features, which is then multiplied by the student features F S Do the dot product and add it back to the student features in the form of residuals to update the student features; update the updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network stage are obtained

[0010] Step 2, the neck network stage includes feature distillation operations;

[0011] The feature distillation operation includes: distilling the final student features of the backbone network stage After a layer of projection, the student features The number of channels and teacher features F T The number of channels is the same as that of and teacher characteristics F T The Pearson similarity is used as the feature distillation supervision loss to update the parameters of the student model;

[0012] Step 3: During the testing phase, the image is input into the student model that has completed the knowledge distillation process to detect the objects and their categories in the image. During the testing phase, the internal prompt distillation operation and the external teacher prompts obtained during the knowledge distillation process are retained. During the testing process, the external teacher prompts are used to directly interact with the student features through a multi-head cross-attention mechanism.

[0013] Furthermore, in step S11, the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompts and avoid information duplication through multi-head self-attention mechanism interaction, specifically including:

[0014] Introducing a set of learnable external teacher hints E∈RT×C , used to store the feature region predicted by the teacher model; T represents the length of the external teacher prompt, and C represents the teacher feature F T The external teacher prompt E is randomly initialized before knowledge distillation and will be updated during the knowledge distillation process. To prevent the external teacher prompt E from storing duplicate information, different external teacher prompts are passed through the self-attention layer:

[0015]

[0016] Among them, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively.

[0017] Furthermore, in step S11, the updating of the teacher's external prompts in a momentum updating manner specifically includes:

[0018] By initializing the teacher features, we can obtain the keys and values ​​required to update the teacher external prompts: T Normalize the number of channels and select the first N pixels as candidate features. In addition, since the target features of the object to be detected are related to the category of the object to be detected, these candidate features are combined with the category-aware embedding of the object to be detected. This category-aware embedding is encoded by the one-bit effective category vector of the target feature. The process of initializing the teacher features is denoted as Init.

[0019] The updating process of the teacher's external prompts during knowledge distillation is as follows:

[0020]

[0021] Where β is the momentum update coefficient, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively.

[0022] Furthermore, in step S11, the student feature F output by the backbone network of the student model S Perform the initialization operation, then perform multi-head cross attention interaction with the updated teacher external prompt, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S Specifically, it includes:

[0023]

[0024] W mand W′ m Indicates the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively; Init(F S ) represents the student characteristics F S Perform initialization operations.

[0025] Furthermore, step S12 specifically includes:

[0026] Introduce learnable student internal hints as the internal knowledge basis of the student model; the student internal hint e describes the dependency of N custom features. By calculating the similarity between the student internal hint e and the student features in the spatial channel and performing Sigmoid operation, the mask of the student features is obtained. There are N channels in total:

[0027]

[0028] Among them, σ represents the Sigmoid function; based on the mask Use the Dice coefficient to guide students' learning of internal cues:

[0029]

[0030] in, and Represent the masks on the i-th channel and the j-th channel respectively, M is the number of attention heads, is the Dice loss supervision function, is the penalty term for the student’s internal prompt, and ρ dice represents the Dice coefficient;

[0031] The student characteristics F S Key information and masks in Match, update the student characteristics, and get the updated student characteristics F S1 :

[0032]

[0033] The updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network stage are obtained

[0034] Furthermore, step 2 specifically includes:

[0035] The neck network adopts feature pyramid; the feature distillation operation enhances the student features at the pixel level and teacher characteristics F T Feature similarity, feature distillation supervision loss for:

[0036]

[0037] Among them, H, W and C represent the teacher characteristics F T The height, width and number of channels, student characteristics The height, width and number of channels of the teacher feature F T Same; f represents a projection layer, which is used to transform student features The number of channels is adjusted to match the teacher feature F T Same; Mask It is a filtering mechanism that selects meaningful regions for knowledge distillation by specifying different masks.

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] 1. The dual-cue-based small-model object detection distillation method proposed in this paper achieves superior performance on small-model distillation. When GFL-Res101 is used as the teacher model and GFL-GhostNet as the student model, the resulting student model achieves a 3.6% improvement in mAP (mean average prediction accuracy), a level unmatched by other mainstream methods (FGD decreases by 1.2, PKD increases by 0.9, DIST increases by 0.4, and MaskD increases by 1.4).

[0040] 2. This invention has excellent generalization capabilities and can be applied to various lightweight models and detectors. On the lightweight model backbones GhostNet, FastNet, and MobileNetV2, the method of this invention outperformed other mainstream methods; it also achieved excellent performance on the various detectors GFL, RetinaNet, RepPoint, and Faster RCNN.

[0041] 3. The method of the present invention can be applied to heterogeneous distillation models. For example, using Mask RCNN-Res101 as the teacher model and RetinaNet-GhostNet as the student model, the student model obtained by distillation using the method of the present invention can achieve an improvement of 2.3mAP. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a basic block diagram of the method proposed in the present invention;

[0043] Figure 2 Performance comparison of the GFL-GhostNet student model distilled by the method proposed in this invention and other methods (w / o KD, FGD, PKD, DIST, MasKD) when GFL-Res101 is used as the teacher model. DETAILED DESCRIPTION

[0044] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] The present invention provides a small model target detection distillation method based on dual prompts, and the technical solution thereof is introduced in detail as follows.

[0046] First, this method follows the mainstream paradigm of current object detection distillation methods, preserving feature distillation at the neck network stage. Specifically, most detectors utilize feature pyramid networks (FPNs) to aggregate multi-scale information, enabling knowledge to be transferred from the teacher model to the student model via feature maps following the neck mesh. Feature distillation enhances the similarity between the features output by the teacher and student models at the pixel level, enabling the student model to obtain richer information and additional supervision. Its specific expression is as follows:

[0047]

[0048] in, represents the final student features of the backbone network stage, F T Represents the teacher features output by the backbone network of the teacher model, H, W and C represent the teacher features F T The height, width and number of channels of the student feature are as follows: f represents a projection layer, which is used to adjust the number of channels of the student feature to the same as the teacher feature. It is a filtering mechanism that selects meaningful regions for knowledge distillation by specifying different masks. In the method of the present invention, the strategy in PKD (Knowledge Distillation via Pearson Correlation Coefficient) is referred to, and the mask is filled with a scalar value of 1. This is the feature distillation supervision loss.

[0049] Secondly, the present invention designs an external hint distillation strategy. Due to the complex and precise structure of the teacher model, it has significant advantages in feature extraction and can effectively enhance the information of the foreground area of ​​the image and suppress background noise. However, relying solely on the teacher features output by the teacher model as a supervision signal is not an optimal solution. In order to more effectively utilize the knowledge of the teacher model, the present invention introduces a set of learnable external teacher hints, denoted as E∈R T×C, which is used to sparsely store the teacher feature regions output by the teacher model. These hints are randomly initialized before training and will be dynamically updated during the distillation process. To prevent the storage of duplicate information, different teacher external hints are passed through the self-attention layer so that they can "see" each other:

[0050]

[0051] Among them, W m and W′ m Denotes the learnable weights, M is the number of attention heads, Attn represents the attention mechanism, and Q, K, and V are the query, key, and value in the attention mechanism, respectively. This attention mechanism establishes a soft connection between each cue and its corresponding feature set, avoiding artificial hard connections between different input images.

[0052] However, the teacher features output by the teacher model contain a large amount of noisy background information, and interacting with dense feature pixels may reduce the efficiency of the attention mechanism and increase unnecessary computational burden. In order to pre-acquire keys and values ​​containing useful information, the present invention proposes a feature initialization strategy: the teacher features are normalized in terms of the number of channels, and the top N pixels are selected as candidates; in addition, since the target features are usually highly correlated with the category, the present invention combines these features with category-aware embeddings, which are encoded by a one-bit effective category vector of the target. The initialization process is denoted as Init. During the training process (i.e., the knowledge distillation process), the update process of the external teacher prompts of the teacher model is shown as follows:

[0053]

[0054] Where β is the momentum update coefficient, which is set to 0.8.

[0055] The interaction between the teacher's external prompts and the teacher model occurs only during the training phase; during the testing phase, the trained teacher's external prompts only participate in subsequent interactions with the student model. Therefore, the momentum update strategy gives the teacher's external prompts a larger weight, which helps ensure that the teacher's external prompts remain relatively consistent as input to the student model during training and testing. The student model uses the generated teacher's external prompts as input knowledge and similarly uses the cross-attention mechanism to search for effective information to enhance its representational capabilities.

[0056] The present invention also initializes the student features output by the student model and uses them as queries to prompt the generation of keys and values. The fused query is interpolated and converted into a residual term:

[0057]

[0058] Finally, the present invention proposes an internal hint distillation strategy. Considering the significant differences in structure and number of parameters between the teacher model and the student model, as well as the limitations of blindly injecting knowledge from the teacher model, it is also very important to ensure that the student model retains its own effective internal information. To this end, the present invention introduces learnable student internal hints as the internal knowledge basis of the student model. Because it does not involve cross-model interactions, the present invention adopts the hard association method of dot product. The student internal hint e describes the dependency of N custom features. By calculating the similarity between the student internal hints and the student features in the spatial channels, rich pixel information can be obtained:

[0059]

[0060] Where σ represents the Sigmoid function, Indicates a mask.

[0061] Through masking, the present invention uses the Dice coefficient to guide the learning of internal cues, as shown in the following formula:

[0062]

[0063] in, is the Dice loss supervision function, which is regarded as a penalty term for the prompt, and its purpose is to prevent the system from falling into a local optimal solution.

[0064] ρ dice Represents the Dice coefficient, and the specific calculation method is as follows:

[0065]

[0066] Subsequently, the key information in the original student features is matched with the mask of the student internal prompts to enhance the student features and obtain the updated student features F S1 :

[0067]

[0068] In addition, the present invention also introduces an additional ConvLoRA technology, which is embedded into the student model as a residual term. Through the compression and decompression process, the effective features are retained and the noise signal is suppressed to obtain the final student feature of the backbone network stage.

[0069] Example

[0070] The dual-cue-based small model target detection distillation method of the present invention can be applied to the distillation of lightweight small target detection models in various situations. A specific embodiment is introduced below.

[0071] like Figure 1As shown in the figure, given a lightweight student model to be trained, we first need to select a suitable teacher model with large parameters. The teacher model has been pre-trained, and the model parameters of the teacher model will not change during this training process.

[0072] During the training phase, the student model's detection results are supervised by the teacher model's detection results. In the backbone network stage, external and internal cue distillation operations are included, while in the neck network, feature distillation operations are included.

[0073] 1. External prompt distillation operation: The teacher's external prompts are initialized through the teacher features output by the backbone network of the teacher model. The multi-head attention mechanism is used to interact to avoid information duplication, and then the prompts are updated in a momentum update manner. At the same time, the student features output by the backbone network of the student model are also initialized. Then, multi-head cross-attention interaction is performed with the updated teacher prompts. The obtained interaction features are added back to the original student model backbone network features in a residual manner to realize the knowledge distillation of the teacher model to the student model.

[0074] 2. Internal Hint Distillation: Initialize a learnable internal student hint and multiply it with the student features output by the backbone network of the student model. Then perform a Sigmoid operation to obtain a mask of the student features. The mask is then dot-producted with the student features and added back to the original student features as a residual. The updated student features are then passed through a layer of ConvLoRA to improve feature validity.

[0075] 3. Feature distillation operation: In the neck network stage, the student features after the internal prompt distillation operation are passed through a projection layer to make the number of channels of the student model features the same as the number of feature channels of the teacher model. Then, the Pearson similarity between the student features and the teacher features is calculated, and the calculated result is used as the loss function to update the parameters of the student model.

[0076] During the testing phase, the teacher model is no longer used, the feature distillation operation is no longer performed, and the internal prompt distillation operation remains unchanged. However, for external prompt distillation, the external teacher prompts obtained during training are retained and directly used to interact with the multi-head attention mechanism of the student features during testing to enrich the feature information of the student model.

[0077] The present invention uses a batch size of 16 training samples to train the student model, and the training cycle is 24 rounds (epochs). For the single-stage detector, the initial learning rate is set to 0.01; for the dual-stage detector, the initial learning rate is set to 0.02. During the training process, at the 16th and 22nd epochs, the present invention reduces the learning rate to 0.1 times the original value, respectively. The optimizer uses the stochastic gradient descent (SGD) optimizer, and the momentum and weight decay parameters are designed to be 0.9 and 0.0001, respectively. All training is performed on 8 GPUs based on the PyTorch framework using the mmdetection toolbox and the mmrazor toolbox.

[0078] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0079] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A small model object detection distillation method based on dual prompts, characterized in that: Through the knowledge distillation process, the knowledge in the teacher model is transferred to the student model. During the test process, the image is input into the student model that has completed the knowledge distillation process to detect objects and their categories in the image. Both the teacher model and the student model include a backbone network and a neck network. The knowledge distillation process includes the backbone network stage and the neck network stage. The object detection distillation method specifically includes the following steps: Step 1: The backbone network stage includes external and internal cue distillation operations. Specifically, it includes the following steps: S11, the external prompt distillation operation includes: the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompt E, and avoid information duplication through multi-head self-attention mechanism interaction; update the teacher's external prompt E in the form of momentum update; at the same time, the student feature F output by the backbone network of the student model S Perform the initialization operation, then perform multi-head cross attention interaction with the updated teacher external prompt, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S middle; S12, the internal prompt distillation operation includes: initializing the learnable student internal prompt, and the student feature F output by the backbone network of the student model S Multiply, then perform Sigmoid operation to obtain the mask of student features, which is then multiplied by the student features F S Do the dot product and add it back to the student features in the form of residuals to update the student features; update the updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network stage are obtained Step 2, the neck network stage includes feature distillation operations; The feature distillation operation includes: distilling the final student features of the backbone network stage After a layer of projection, the student features The number of channels and teacher features F T The number of channels is the same as that of and teacher characteristics F T The Pearson similarity is used as the feature distillation supervision loss to update the parameters of the student model; Step 3: During the testing phase, the image is input into the student model that has completed the knowledge distillation process to detect the objects and their categories in the image. During the testing phase, the internal prompt distillation operation and the external teacher prompts obtained during the knowledge distillation process are retained. During the testing process, the external teacher prompts are used to directly interact with the student features through a multi-head cross-attention mechanism.

2. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompts and avoid information duplication through multi-head self-attention mechanism interaction, specifically including: Introducing a set of learnable external teacher hints E∈R T×C , used to store the feature region predicted by the teacher model; T represents the length of the external teacher prompt, and C represents the teacher feature F T The external teacher prompt E is randomly initialized before knowledge distillation and will be updated during the knowledge distillation process. To prevent the external teacher prompt E from storing duplicate information, different external teacher prompts are passed through the self-attention layer: Among them, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively.

3. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the teacher's external prompt is updated in a momentum updating manner, specifically including: By initializing the teacher features, we can obtain the keys and values ​​required to update the teacher external prompts: T Normalize the number of channels and select the first N pixels as candidate features. In addition, since the target features of the object to be detected are related to the category of the object to be detected, these candidate features are combined with the category-aware embedding of the object to be detected. This category-aware embedding is encoded by the one-bit effective category vector of the target feature. The process of initializing the teacher features is denoted as Init. The updating process of the teacher's external prompts during knowledge distillation is as follows: Where β is the momentum update coefficient, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively.

4. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the student features F output by the backbone network of the student model are S Perform the initialization operation, then perform multi-head cross attention interaction with the updated teacher external prompt, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S Specifically, it includes: W m and W′ m Indicates the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively; Init(F S ) represents the student characteristics F S Perform initialization operations.

5. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: Step S12 specifically includes: Introduce learnable student internal hints as the internal knowledge basis of the student model; the student internal hint e describes the dependency of N custom features. By calculating the similarity between the student internal hint e and the student features in the spatial channel and performing Sigmoid operation, the mask of the student features is obtained. There are N channels in total: Among them, σ represents the Sigmoid function; based on the mask Use the Dice coefficient to guide students' learning of internal cues: in, and Represent the masks on the i-th channel and the j-th channel respectively, M is the number of attention heads, is the Dice loss supervision function, is the penalty term for the student’s internal prompt, and ρ dice represents the Dice coefficient; The student characteristics F S Key information and masks in Match, update the student characteristics, and get the updated student characteristics F S1 : The updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network stage are obtained 6. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: Step 2 specifically includes: The neck network adopts feature pyramid; the feature distillation operation enhances the student features at the pixel level and teacher characteristics F T Feature similarity, feature distillation supervision loss for: Among them, H, W and C represent the teacher characteristics F T The height, width and number of channels, student characteristics The height, width and number of channels of the teacher feature F T Same; f represents a projection layer, which is used to transform student features The number of channels is adjusted to match the teacher feature F T Same; Mask It is a filtering mechanism that selects meaningful regions for knowledge distillation by specifying different masks.

Images