Small model target detection distillation method based on double prompts

Abstract

The invention relates to the technical field of target detection, and discloses a small model target detection distillation method based on double prompts, external prompt distillation operation initializes teacher external prompts through teacher features, and interaction is performed through a multi-head self-attention mechanism to avoid information repetition; external prompts of the teacher are updated in a momentum updating mode; after the student features are initialized, multi-head cross attention interaction with external prompts of a teacher is carried out, and then the interaction is added back to the student features in a residual error mode; the internal prompt distillation operation initializes student internal prompts, multiplies the student internal prompts with student characteristics, and then updates the student characteristics; performing ConvLoRA operation on the updated student features to obtain final student features in the backbone network stage; updating parameters of the student model through characteristic distillation operation; according to the invention, more excellent performance can be obtained on small model distillation.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small model target detection distillation method based on double prompts. Background Art

[0002] As a classic task in the field of computer vision, object detection has the core goal of identifying objects in images or videos, determining their categories, and accurately calibrating the location of objects in the form of a minimum bounding box. This technology has a wide range of applications in many fields, such as autonomous driving, surveillance systems, robotic vision, medical image analysis, aerospace, etc. However, when applying these large and computationally intensive models to the real world, there are a series of challenges, including the size of the model, the speed of inference, and the limited resources. These challenges limit the deployment and application of these powerful models in real-world scenarios.

[0003] Knowledge distillation technology is an important method to solve the above problems. Knowledge distillation is to use a trained large model as the teacher model and a small model to be trained as the student model. During the training process, the detection results of the teacher model supervise the detection results of the student model, so that the "knowledge" is distilled from the large model to the small model, thereby improving the performance of the small model. However, most of the existing knowledge distillation methods remain in the scenario where the teacher model and the student model are of similar size. A few methods have studied the situation where the teacher model is very large. However, how to distill knowledge into a smaller, faster lightweight model is rarely discussed, and the existing distillation methods have very limited performance improvements on lightweight models. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a small model target detection distillation method based on double prompts, which can effectively transfer the knowledge of the large model to the small model, thereby achieving more obvious performance improvement of the small model.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A small model object detection distillation method based on double prompts, through the knowledge distillation process, the knowledge in the teacher model is transferred to the student model, during the test process, the image is input to the student model that has completed the knowledge distillation process, and the objects and categories of the objects in the image are detected; the teacher model and the student model both include a backbone network and a neck network, and the knowledge distillation process includes a backbone network stage and a neck network stage; the object detection distillation method specifically includes the following steps:

[0007] Step 1: The backbone network stage includes external prompt distillation operations and internal prompt distillation operations; specifically, the following steps are included:

[0008] S11, the external prompt distillation operation includes: the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompt E, and avoid information duplication through multi-head self-attention mechanism interaction; update the teacher's external prompt E in the form of momentum update; at the same time, the student feature F output by the backbone network of the student model S Perform the initialization operation, and then perform multi-head cross-attention interaction with the updated teacher external prompts, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S middle;

[0009] S12, the internal prompt distillation operation includes: initializing the learnable student internal prompts and the student features F output by the backbone network of the student model S Multiply, then perform Sigmoid operation to obtain the mask of student features, which is then multiplied by the student features F S Do the dot product and add it back to the student features in the form of residuals to update the student features; update the updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network are obtained.

[0010] Step 2, the neck network stage includes feature distillation operations;

[0011] The feature distillation operation includes: distilling the final student features of the backbone network stage After a layer of projection, the student features The number of channels and teacher features F T The number of channels is the same as that of and teacher characteristics F T The Pearson similarity is used as the feature distillation supervision loss to update the parameters of the student model;

[0012] Step 3: During the test phase, the image is input into the student model that has completed the knowledge distillation process to detect the objects and categories in the image. During the test phase, the internal prompt distillation operation and the external teacher prompts obtained during the knowledge distillation process are retained. During the test process, the external teacher prompts are used to directly interact with the student features through a multi-head cross-attention mechanism.

[0013] Furthermore, in step S11, the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompts, and avoid information duplication through multi-head self-attention mechanism interaction, including:

[0014] Introduce a set of learnable external teacher hints E∈RT×C , used to store the feature region predicted by the teacher model; T represents the length of the external teacher prompt, and C represents the teacher feature F T The number of channels; the external teacher prompt E is randomly initialized before knowledge distillation and will be updated during the knowledge distillation process; in order to prevent the external teacher prompt E from storing duplicate information, different external teacher prompts are passed through the self-attention layer:

[0015]

[0016] Among them, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K, and V represent the query, key, and value in the attention mechanism, respectively.

[0017] Furthermore, in step S11, the updating of the teacher's external prompt in a momentum updating manner specifically includes:

[0018] By initializing the teacher features, we can obtain the keys and values ​​required to update the teacher external prompts: T Normalize on the number of channels and select the first N pixels as candidate features. In addition, since the target features of the object to be detected are related to the category of the object to be detected, these candidate features are combined with the category-aware embedding of the object to be detected. This category-aware embedding is encoded by a single-bit effective category vector of the target feature. The process of initializing the teacher features is denoted as Init.

[0019] The updating process of the teacher's external prompts during knowledge distillation is as follows:

[0020]

[0021] Where β is the momentum update coefficient, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K, and V represent the query, key, and value in the attention mechanism, respectively.

[0022] Further, in step S11, the student feature F output by the backbone network of the student model S Perform the initialization operation, and then perform multi-head cross-attention interaction with the updated teacher external prompts, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S Specifically, it includes:

[0023]

[0024] W mand W′ m Init(F S ) represents the student characteristics F S Perform initialization operation.

[0025] Furthermore, step S12 specifically includes:

[0026] Introduce learnable internal student hints as the internal knowledge basis of the student model; the internal student hint e describes the dependency of N custom features. By calculating the similarity between the internal student hint e and the student features in the spatial channel and performing a Sigmoid operation, the mask of the student features is obtained. There are N channels in total:

[0027]

[0028] Among them, σ represents the Sigmoid function; based on the mask Use the Dice coefficient to guide students' internal cues in learning:

[0029]

[0030] in, and They represent the masks on the i-th channel and the j-th channel respectively, M is the number of attention heads, is the Dice loss supervision function, is the penalty term for the student’s internal prompt, and ρ dice represents the Dice coefficient;

[0031] The student characteristics F S Key information and masks in Match, update the student characteristics, and get the updated student characteristics F S1 :

[0032]

[0033] The updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network are obtained.

[0034] Furthermore, step 2 specifically includes:

[0035] The neck network adopts feature pyramid; the feature distillation operation enhances the student features at the pixel level and teacher characteristics F T Feature similarity, feature distillation supervision loss for:

[0036]

[0037] Among them, H, W and C represent the teacher characteristics F T The height, width and number of channels, student characteristics The height, width and number of channels of the teacher feature F T Same; f represents a projection layer, which is used to transform the student features The number of channels is adjusted to match the teacher feature F T Same; Mask It is a filtering mechanism that selects meaningful regions for knowledge distillation by specifying different masks.

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] 1. The small model object detection distillation method based on double prompts proposed in the present invention can achieve better performance in small model distillation. When GFL-Res101 is selected as the teacher model and GFL-GhostNet is selected as the student model, the performance index mAP of the student model obtained by the present invention is improved by 3.6, and other mainstream methods cannot reach this level (among which, FGD decreased by 1.2, PKD increased by 0.9, DIST increased by 0.4, and MaskD increased by 1.4).

[0040] 2. The present invention has good generalization ability and can be applied to different lightweight models and different detectors. On the lightweight model backbones GhostNet, FastNet, and MobileNetV2, the method of the present invention outperforms other mainstream methods; and on different detectors GFL, RetinaNet, RepPoint, and Faster RCNN, excellent performance is also achieved.

[0041] 3. The method of the present invention can be applied to heterogeneous distillation models. For example, Mask RCNN-Res101 is used as the teacher model and RetinaNet-GhostNet is used as the student model. The student model obtained by distillation using the method of the present invention can achieve an improvement of 2.3mAP. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a basic block diagram of the method proposed in the present invention;

[0043] Figure 2 Performance comparison of the GFL-GhostNet student model distilled by the method proposed in this invention and other methods (w / o KD, FGD, PKD, DIST, MasKD) when GFL-Res101 is used as the teacher model. DETAILED DESCRIPTION

[0044] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0045] The present invention provides a small model target detection distillation method based on double prompts, and the contents of its technical solution are introduced in detail as follows.

[0046] First, this method follows the mainstream paradigm of current object detection distillation methods and retains the feature distillation at the neck network stage. Specifically, most detectors use feature pyramid (FPN) networks to aggregate multi-scale information, so that knowledge can be transferred from the teacher model to the student model through the feature map after the neck grid. Feature distillation enhances the similarity between the features output by the teacher model and the student model at the pixel level, enabling the student model to obtain richer information and additional supervision. Its specific expression is as follows:

[0047]

[0048] in, represents the final student features of the backbone network stage, F T represents the teacher features output by the backbone network of the teacher model, H, W and C represent the teacher features F respectively. T The height, width and number of channels of the feature vector are as follows: f represents a projection layer, which is used to adjust the number of channels of the student feature to the same as the teacher feature. It is a filtering mechanism that selects meaningful areas for knowledge distillation by specifying different masks. In the method of the present invention, the strategy in PKD (Knowledge Distillation via Pearson Correlation Coefficient) is referred to, and the mask is filled with a scalar value 1. This is the feature distillation supervision loss.

[0049] Secondly, the present invention designs an external prompt distillation strategy. Due to the complex and precise structure of the teacher model, it has significant advantages in feature extraction and can effectively enhance the information of the foreground area of ​​the image and suppress background noise. However, relying solely on the teacher features output by the teacher model as a supervision signal is not an optimal solution. In order to more effectively utilize the knowledge of the teacher model, the present invention introduces a set of learnable external teacher prompts, denoted as E∈R T×C, which is used to sparsely store the teacher feature regions output by the teacher model. These prompts are randomly initialized before training and will be dynamically updated during the distillation process. To prevent the storage of duplicate information, different teacher external prompts are passed through the self-attention layer so that they can "see" each other:

[0050]

[0051] Among them, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K and V represent the query, key and value in the attention mechanism respectively. This attention mechanism establishes a soft connection between each prompt and its corresponding feature set, avoiding artificial hard connections between different input images.

[0052] However, the teacher features output by the teacher model contain a large amount of noisy background information, and interacting with dense feature pixels may reduce the efficiency of the attention mechanism and increase unnecessary computational burden. In order to pre-acquire keys and values ​​containing useful information, the present invention proposes a feature initialization strategy: the teacher features are normalized in terms of the number of channels, and the first N pixels are selected as candidates; in addition, since the target features are usually highly correlated with the category, the present invention combines these features with category-aware embeddings, which are encoded by a one-bit effective category vector of the target. The initialization process is denoted as Init. During the training process (i.e., the knowledge distillation process), the update process of the external teacher prompt of the teacher model is shown as follows:

[0053]

[0054] Where β is the momentum update coefficient, which is set to 0.8.

[0055] The interaction between the teacher external prompts and the teacher model only occurs during the training phase; in the test phase, the trained teacher external prompts only participate in the subsequent interaction with the student model. Therefore, the momentum update strategy gives the teacher external prompts themselves a large weight, which helps to ensure that the teacher external prompts remain relatively consistent as input to the student model during training and testing. The student model uses the generated teacher external prompts as input knowledge and also uses the cross-attention mechanism to find effective information to enhance its representation ability.

[0056] The present invention also initializes the student features output by the student model and uses them as queries to prompt the generation of keys and values. The fused query is interpolated and converted into a residual term:

[0057]

[0058] Finally, the present invention proposes an internal hint distillation strategy. Considering the significant differences in structure and number of parameters between the teacher model and the student model, as well as the limitations of blindly injecting knowledge from the teacher model, it is also very important to ensure that the student model retains its own valid internal information. To this end, the present invention introduces learnable student internal hints as the internal knowledge basis of the student model. Because it does not involve cross-model interactions, the present invention adopts the hard association method of dot products. The student internal hint e describes the dependency of N custom features, and rich pixel information can be obtained by calculating the similarity between the student internal hints and the student features in the spatial channels:

[0059]

[0060] Where σ represents the Sigmoid function, Indicates a mask.

[0061] Through masking, the present invention uses the Dice coefficient to guide the learning of internal cues, as shown in the following formula:

[0062]

[0063] in, is the Dice loss supervision function, which is regarded as a penalty term for the prompt, aiming to prevent the system from falling into a local optimal solution.

[0064] ρ dice Represents the Dice coefficient. The specific calculation method is as follows:

[0065]

[0066] Subsequently, the key information in the original student features is matched with the mask of the student internal prompts to enhance the student features and obtain the updated student features F S1 :

[0067]

[0068] In addition, the present invention also introduces an additional ConvLoRA technology, which is embedded into the student model as a residual term. Through the compression and decompression process, the effective features are retained and the noise signal is suppressed to obtain the final student features of the backbone network stage.

[0069] Example

[0070] The dual-cue-based small model target detection distillation method of the present invention can be applied to the distillation of lightweight small target detection models in various occasions. A specific embodiment is introduced below.

[0071] like Figure 1As shown in the figure, given a lightweight student model to be trained, we first need to select a suitable teacher model with large parameters. The teacher model has been pre-trained, and the model parameters of the teacher model will not change during this training process.

[0072] During the training phase, the detection results of the student model are supervised by the detection results of the teacher model. In the backbone network stage, external cue distillation and internal cue distillation operations are included; while in the neck network, feature distillation operations are included.

[0073] 1. External prompt distillation operation: Initialize the teacher external prompts through the teacher features output by the backbone network of the teacher model, interact through the multi-head attention mechanism to avoid information duplication, and then update the prompts in the form of momentum update; at the same time, the student features output by the backbone network of the student model are also initialized, and then multi-head cross-attention interactions are performed with the updated teacher prompts. The obtained interactive features are added back to the original student model backbone network features in the form of residuals to realize the knowledge distillation of the teacher model to the student model.

[0074] 2. Internal hint distillation operation: Initialize a learnable student internal hint, multiply it with the student feature output by the backbone network of the student model, and then perform a Sigmoid operation to obtain the mask of the student feature. The mask is then dot-producted with the student feature and added back to the original student feature in the form of a residual. The updated student feature is passed through a layer of ConvLoRA operation to improve the effectiveness of the feature.

[0075] 3. Feature distillation operation: In the neck network stage, the student features after the internal prompt distillation operation are passed through a projection layer to make the number of channels of the student model features the same as the number of feature channels of the teacher model. Then, the Pearson similarity between the student features and the teacher features is calculated, and the calculated result is used as the loss function to update the parameters of the student model.

[0076] During the test phase, the teacher model will no longer be used, the feature distillation operation will no longer exist, and the internal prompt distillation operation will remain unchanged. For the external prompt distillation, the external teacher prompts obtained during the training process will be retained and directly used to interact with the multi-head attention mechanism of the student features during the test process to enrich the feature information of the student model.

[0077] The present invention uses a batch size of 16 training samples to train the student model, and the training cycle is 24 rounds (epoch). For the single-stage detector, the initial learning rate is set to 0.01; and for the dual-stage detector, the initial learning rate is set to 0.02. During the training process, at the 16th and 22nd epochs, the present invention reduces the learning rate to 0.1 times the original, respectively. The optimizer adopts the stochastic gradient descent (SGD) optimizer, and the momentum and weight decay parameters are designed to be 0.9 and 0.0001, respectively. All training is performed on 8 GPUs, based on the PyTorch framework, using the mmdetection toolbox and the mmrazor toolbox.

[0078] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0079] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A small model object detection distillation method based on double prompts, characterized in that: Through the knowledge distillation process, the knowledge in the teacher model is transferred to the student model. During the test process, the image is input into the student model that has completed the knowledge distillation process to detect the objects and categories of the objects in the image. Both the teacher model and the student model include a backbone network and a neck network. The knowledge distillation process includes a backbone network stage and a neck network stage. The target detection distillation method specifically includes the following steps: Step 1: The backbone network stage includes external prompt distillation operations and internal prompt distillation operations; specifically, the following steps are included: S11, the external prompt distillation operation includes: the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompt E, and avoid information duplication through multi-head self-attention mechanism interaction; update the teacher's external prompt E in the form of momentum update; at the same time, the student feature F output by the backbone network of the student model S Perform the initialization operation, and then perform multi-head cross-attention interaction with the updated teacher external prompts, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S middle; S12, the internal prompt distillation operation includes: initializing the learnable student internal prompts and the student features F output by the backbone network of the student model S Multiply, then perform Sigmoid operation to obtain the mask of student features, which is then multiplied by the student features F S Do the dot product and add it back to the student features in the form of residuals to update the student features; update the updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network are obtained. Step 2, the neck network stage includes feature distillation operations; The feature distillation operation includes: distilling the final student features of the backbone network stage After a layer of projection, the student features The number of channels and teacher features F T The number of channels is the same as that of and teacher characteristics F T The Pearson similarity is used as the feature distillation supervision loss to update the parameters of the student model; Step 3: During the test phase, the image is input into the student model that has completed the knowledge distillation process to detect the objects and categories in the image. During the test phase, the internal prompt distillation operation and the external teacher prompts obtained during the knowledge distillation process are retained. During the test process, the external teacher prompts are used to directly interact with the student features through a multi-head cross-attention mechanism.

2. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the teacher feature F output by the backbone network of the teacher model T , to initialize the teacher's external prompts, and avoid information duplication through multi-head self-attention mechanism interaction, including: Introduce a set of learnable external teacher hints E∈R T×C , used to store the feature region predicted by the teacher model; T represents the length of the external teacher prompt, and C represents the teacher feature F T The number of channels; the external teacher prompt E is randomly initialized before knowledge distillation and will be updated during the knowledge distillation process; in order to prevent the external teacher prompt E from storing duplicate information, different external teacher prompts are passed through the self-attention layer: Among them, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K, and V represent the query, key, and value in the attention mechanism, respectively.

3. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the teacher external prompt is updated in a momentum updating manner, specifically including: By initializing the teacher features, we can obtain the keys and values ​​required to update the teacher external prompts: T Normalize on the number of channels and select the first N pixels as candidate features. In addition, since the target features of the object to be detected are related to the category of the object to be detected, these candidate features are combined with the category-aware embedding of the object to be detected. This category-aware embedding is encoded by a single-bit effective category vector of the target feature. The process of initializing the teacher features is denoted as Init. The updating process of the teacher's external prompts during knowledge distillation is as follows: Where β is the momentum update coefficient, W m and W′ m represents the learnable weight, M is the number of attention heads, Attn represents the attention mechanism, Q, K, and V represent the query, key, and value in the attention mechanism, respectively.

4. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: In step S11, the student features F output by the backbone network of the student model are S Perform the initialization operation, and then perform multi-head cross-attention interaction with the updated teacher external prompts, and add the obtained interaction features back to the student features F output by the student model backbone network in the form of residuals S Specifically, it includes: W m and W′ m Init(F S ) represents the student characteristics F S Perform initialization operation.

5. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: Step S12 specifically includes: Introduce learnable internal student hints as the internal knowledge basis of the student model; the internal student hint e describes the dependency of N custom features. By calculating the similarity between the internal student hint e and the student features in the spatial channel and performing a Sigmoid operation, the mask of the student features is obtained. There are N channels in total: Among them, σ represents the Sigmoid function; based on the mask Use the Dice coefficient to guide students' internal cues in learning: in, and They represent the masks on the i-th channel and the j-th channel respectively, M is the number of attention heads, is the Dice loss supervision function, is the penalty term for the student’s internal prompt, and ρ dice represents the Dice coefficient; The student characteristics F S Key information and masks in Match, update the student characteristics, and get the updated student characteristics F S1 : The updated student features F S1 After another layer of ConvLoRA operation, the final student features of the backbone network stage are obtained 6. The double-cue-based small model object detection distillation method according to claim 1, characterized in that: Step 2 specifically includes: The neck network adopts feature pyramid; the feature distillation operation enhances the student features at the pixel level and teacher characteristics F T Feature similarity, feature distillation supervision loss for: Among them, H, W and C represent the teacher characteristics F T The height, width and number of channels, student characteristics The height, width and number of channels of the teacher feature F T Same; f represents a projection layer, which is used to transform the student features The number of channels is adjusted to match the teacher feature F T Same; Mask It is a filtering mechanism that selects meaningful regions for knowledge distillation by specifying different masks.

Images