A network structure-oriented mapping distillation method and a training method thereof

Abstract

The present application relates to the technical field of model compression and acceleration in computer vision, solves the technical problems that the relationship-based distillation cannot improve the relationship representation, the number of input picture batches is small, and the expression ability of feature representation and relationship representation is well combined with mutual advantages, made up and enhanced, and especially relates to a mapping type distillation method for network structure, the mapping type distillation method comprises the following process: a teacher model and a student model are formed according to a pre-trained target detection model; the feature maps of each layer module of the teacher model and the student model are extracted respectively.The present application has excellent performance in classification and detection tasks, and well combines the advantages of relationship information and attention information, makes up and enhances the expression ability of feature representation and relationship representation, and also provides two optional module mapping methods, which can accept larger image input and more batch quantity, and improve the applicability of the method.

Description

Technical Field

[0001] This invention relates to the field of model compression and acceleration technology in computer vision, and in particular to a mapping distillation method for network structures and its training method. Background Technology

[0002] Over the past few decades, with the continuous iteration and updates of deep learning technology, new neural network models have emerged in the field of computer vision. Although computing resources and memory have been greatly improved and are sufficient to meet daily needs, they remain obstacles to the performance of deep neural networks. Therefore, model compression has gained widespread research interest in computer vision and machine learning. Quantization and pruning are the most commonly used methods by previous researchers, focusing on network architecture design, reduction of computational cost and model size, and removal of redundant connections in large models.

[0003] However, the aforementioned work has limitations because it requires custom-designed hardware and software models for acceleration, restricting its practical application. To address this challenge, knowledge distillation has been proposed as an end-to-end solution to bridge model differences and minimize the computational burden on deep neural networks. Knowledge distillation specifically refers to the process of extracting knowledge from the teacher network and transferring it to the student network, with the aim of encouraging students to learn from the teacher, thereby improving the generalization ability of the student network.

[0004] In the field of knowledge distillation, researchers aim to find more efficient solutions for transferring knowledge from teacher networks to student networks. Generally, there are three types of knowledge: response-based knowledge, feature-based knowledge, and relation-based knowledge. In addition, other distillation strategies exist. Recent research on knowledge distillation has focused primarily on two aspects: first, uncovering more important distilled knowledge; and second, exploring more effective methods for transferring knowledge to student networks.

[0005] Most knowledge distillation methods typically extract knowledge from teacher networks in two forms: feature-based or relation-based. These works inevitably introduce some knowledge information, such as attention information and inter-instance relation information. However, these are not inherently complementary, leading to the following main problems in existing knowledge distillation methods: (1) In the absence of attention mechanisms, relation-based distillation cannot improve relation representation, while feature-based distillation ignores higher-level semantic inter-instance relationships, thus resulting in poor feature representation. Therefore, most existing distillation methods cannot achieve a good balance between detection and classification tasks.

[0006] (2) Existing distillation methods that take into account the relationship information between network modules have a series of drawbacks, such as large memory consumption, small batch size of input images, and inability to be accelerated by GPU.

[0007] (3) Existing methods that consider the combination of attention information and relational information simply add the loss functions of feature-based distillation and relation-based distillation, without effectively combining their respective advantages, compensating for and enhancing the expressive power of feature representation and relational representation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a mapping-based distillation method and its training method oriented towards network structure. This method solves the technical problems of relation-based distillation failing to improve relation representation, having a small batch size of input images, and effectively combining the advantages of each other to compensate for and enhance the expressive power of feature representation and relation representation.

[0009] This invention exhibits excellent performance in both classification and detection tasks, and effectively combines the advantages of relational and attentional information to compensate for and enhance the expressive power of feature and relational representations. It also provides two optional module mapping methods that can accept larger image inputs and more batches, thereby improving the applicability of the method.

[0010] To address the aforementioned technical problems, this invention provides the following technical solution: a mapping distillation method oriented towards network structure, based on a pre-trained target detection model as the basic data scalar, the mapping distillation method comprising the following processes: Teacher and student models are constructed based on the pre-trained target detection model; Extract feature maps from each layer of the teacher model and student model respectively; The extracted feature maps are then subjected to attention mapping to obtain a feature information representation map with attention information; The feature information representation map is mapped by modules to obtain a mapped representation map with pixel-level relationship information; The mapping representation graph, which contains attention information and pixel-level relationship information, is distilled from the teacher model to the student model.

[0011] Furthermore, the attention mapping includes global attention mapping and local attention mapping, and the module mapping includes global module mapping and local module mapping.

[0012] Furthermore, the attention mapping of the feature map includes the following process: Global attention mapping is performed on the feature information representation maps of the student model and the teacher model to obtain global feature information, resulting in global feature information representation maps of the student model and the teacher model respectively. Local attention mapping is performed on the feature information representation maps of the student model and the teacher model to obtain local feature information, thus obtaining the local feature information representation maps of the student model and the teacher model respectively.

[0013] Furthermore, the module mapping of the feature information representation map includes the following process: Global module mapping is performed on the global feature information representation maps of the student model and the teacher model to obtain global feature information and pixel-level relationship information, thus obtaining the global mapping representation maps of the student model and the teacher model respectively. Local module mapping is performed on the local feature information representation maps of the student model and the teacher model to obtain local feature information and pixel-level relationship information, resulting in local mapping representation maps of the student model and the teacher model, respectively.

[0014] Furthermore, the module mapping of the feature information representation map also includes the following process: Global module mapping is performed on the global feature information representation maps of the student model and the teacher model to obtain global feature information and pixel-level relationship information, thus obtaining the global mapping representation maps of the student model and the teacher model respectively. Local module mapping is performed on the local feature information representation maps of the student model and the teacher model to obtain local feature information and pixel-level relationship information, thus obtaining the local mapping representation maps of the student model and the teacher model respectively. Dimensional compression is performed on the global and local mapping representation graphs of the student and teacher models.

[0015] Furthermore, the module mapping process in the feature information representation graph also includes: The module mapping obtains global and local mapping representation maps of the student model and the teacher model. Combined with the loss function, the student model is trained by constraints and penalties. The mapping distillation is completed by the student model learning by imitation from the teacher model.

[0016] Furthermore, the module mapping process in the feature information representation graph also includes: The mapping representation graph of the teacher model is used as a label to supervise and constrain the training of the student model. The loss function of mapping distillation is added to the original loss function of the student model to jointly balance the training error and enhance the robustness of the model.

[0017] Furthermore, the target detection model can also be an image classification model.

[0018] This invention also proposes a training method for the above-mentioned mapping distillation method, the specific technical solution of which is as follows: A training method for a mapping-based distillation method oriented towards network structure includes the following steps: Pre-trained object detection models or image classification models are used to construct teacher and student models. Feature maps of each feature layer module are extracted from the teacher model and the student model, and then fed into the above-mentioned mapping distillation method and the student model for training.

[0019] By employing the above technical solution, the present invention provides a mapping distillation method and its training method oriented towards network structure, which has at least the following beneficial effects: 1. The mapping distillation method and its training method proposed in this invention embed pixel-level relationship information between adjacent modules based on an attention mechanism into a mapping representation graph, which is then passed from the teacher model to the student model to constrain and supervise its training and learning. This method exhibits excellent performance in both classification and detection tasks, and effectively combines the advantages of relational and attentional information, compensating for and enhancing the expressive power of feature and relational representations.

[0020] 2. The mapping distillation method proposed in this invention takes into account both image classification and object detection tasks, and exhibits excellent performance in both. Compared with existing distillation methods, this method achieves better performance and significantly reduces the computational workload. It also provides two optional module mapping methods, which can accept larger image inputs and more batches, thus improving the applicability of the method. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the mapping distillation method in Embodiment 1 of the present invention; Figure 2 This is a flowchart of attention mapping of feature maps in Embodiment 1 of the present invention; Figure 3 This is a flowchart of module mapping for the feature information representation map in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the training method for the mapping distillation method in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram illustrating the underlying principle of the mapping distillation method in Embodiment 1 of the present invention. Figure 6 This is a flowchart of the mapping distillation method in Embodiment 2 of the present invention; Figure 7 This is a flowchart illustrating the module mapping of the feature information representation map in Embodiment 3 of the present invention; Figure 8 This is a schematic diagram of an example distillation of the mapping distillation method and its training method in Embodiment 3 of the present invention; Figure 9 This is a graph showing the comparative experimental results of the mapping distillation method on the datasets Cars196 and CUB-200-2011 in the experimental examples of this invention; Figure 10This is a graph showing the accuracy curves of the comparative experiments on the Cars196 dataset in the experimental examples of this invention; Figure 11 This is a graph showing the accuracy curves of a comparative experiment on the CUB-200-2011 dataset in the experimental examples of this invention. Figure 12 This is a graph showing the comparative experimental results of the mapping distillation method on the Wideface dataset in the experimental examples of this invention; Figure 13 This is the probability distribution diagram of the mapping representation matrix of the teacher and student models in the experimental example of this invention; Figure 14 The figure shows the ablation results of the mapping distillation method on the datasets Cars196 and CUB-200-2011 in the experimental examples of this invention. Figure 15 This is a graph showing the ablation results of the mapping distillation method on the Wideface dataset in the experimental examples of this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0023] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0024] Example 1 Please refer to Figures 1-5 The present invention illustrates one embodiment, the specific implementation details of which are as follows: A mapping distillation method oriented towards network structure, based on a pre-trained object detection model as the base data scalar, includes the following process: S11. Construct teacher and student models based on the pre-trained target detection model.

[0025] S12. Extract the feature maps of each module of the teacher model and student model respectively.

[0026] S13. Perform attention mapping on the extracted feature maps to obtain feature information representation maps with attention information.

[0027] Attention mapping includes global attention mapping and local attention mapping.

[0028] The attention mapping process for feature maps includes the following steps: S131. Perform global attention mapping on the feature information representation maps of the student model and the teacher model to obtain global feature information, and obtain the global feature information representation maps of the student model and the teacher model respectively.

[0029] S132. Perform local attention mapping on the feature information representation maps of the student model and the teacher model to obtain local feature information, and obtain the local feature information representation maps of the student model and the teacher model respectively.

[0030] The global attention mapping function is: The specific manifestations are as follows:

[0031] The local attention mapping function is: The specific manifestations are as follows:

[0032] in, L represents the number of feature layers in the teacher and student models, N, C, H, and W represent the batch size, channels, width, and height of the feature map, respectively, and i, j, and k represent the spatial and channel dimensions, respectively. i th j th k th Vector fragment.

[0033] Attention mapping is used to obtain global and local feature information representation maps of the student and teacher models for subsequent module mapping.

[0034] S14. Perform module mapping on the feature information representation map to obtain a mapping representation map with pixel-level relationship information.

[0035] Module mapping includes global module mapping and local module mapping.

[0036] The module mapping of the feature information representation map includes the following process: S141. Perform global module mapping on the global feature information representation maps of the student model and the teacher model to obtain global feature information and pixel-level relationship information, and obtain the global mapping representation maps of the student model and the teacher model respectively. S142. Perform local module mapping on the local feature information representation maps of the student model and the teacher model to obtain local feature information and pixel-level relationship information, and obtain the local mapping representation maps of the student model and the teacher model respectively.

[0037] The global module mapping function is The specific manifestations are as follows:

[0038] The local module mapping function is The specific manifestations are as follows:

[0039] in, They represent Feature representation map of the layer, They represent A pixel-level relational mapping representation of the layer instances.

[0040] The above describes the module mapping method for module mapping mode one.

[0041] Based on the obtained mapping representation maps of the teacher model and the student model, the mapping representation map of the teacher model is used as a label to supervise and constrain the training of the student model. The loss function of mapping distillation is added to the original loss function of the student model to jointly balance the training error and enhance the robustness of the model.

[0042] Based on module mapping, global and local mapping representations of teachers and students are obtained. Combined with loss function, student models are trained by constraints and penalties, enabling student models to learn by imitation from teacher models, thus completing mapping distillation.

[0043] S15. Distill the mapping representation graph containing attention information and pixel-level relationship information from the teacher model to the student model.

[0044] This embodiment also proposes a training method for the above-mentioned mapping distillation method, the specific implementation of which is as follows: A training method for a mapping-based distillation method oriented towards network structure includes the following steps: S10. Obtain pre-trained target detection models or image classification models to form teacher and student models; S20. Extract feature maps from each feature layer module of the teacher model and student model, and input them into the above-mentioned mapping distillation method and student model for training.

[0045] Through this embodiment, the mapping distillation method and its training method provided in this embodiment embed pixel-level relationship information between adjacent modules based on the attention mechanism into the mapping representation graph, which is then passed from the teacher model to the student model to constrain and supervise its training and learning.

[0046] This mapping-based distillation method demonstrates superior performance in both classification and detection tasks, effectively combining the strengths of relational and attention information to compensate for and enhance the expressive power of feature and relation representations. Compared to previous distillation methods, this approach achieves better performance while significantly reducing computational workload.

[0047] Example 2 Please refer to Figure 6 The present invention illustrates one embodiment, the specific implementation details of which are as follows: A mapping distillation method oriented towards network structure, based on a pre-trained image classification model as the basic data scalar, comprises the following steps: S21. Construct teacher and student models based on the pre-trained image classification model; S22. Extract the feature maps of each layer module of the teacher model and student model respectively; S23. Perform attention mapping on the feature maps extracted above to obtain feature information representation maps with attention information; S24. Perform module mapping on the feature information representation map to obtain a mapped representation map with pixel-level relationship information; S25. Distill the mapping representation graph containing attention information and pixel-level relationship information from the teacher model to the student model.

[0048] The mapping distillation method for network structures proposed in this embodiment is based on Embodiment 1 and has the beneficial effects of the corresponding method embodiment. The same or similar technical contents covered in Embodiment 1 will not be described in detail here.

[0049] In this embodiment, the method takes into account both image classification and object detection tasks, both of which exhibit excellent performance, while significantly reducing the computational workload.

[0050] Example 3 Please refer to Figures 7-8 The present invention illustrates one embodiment, the specific implementation details of which are as follows: like Figure 8 As shown, the teacher model and student model extract feature representation maps of the feature layer, where circles, triangles, squares and rhombuses represent output instances of the feature representation maps.

[0051] Adjacent feature layers form a mapping representation map matrix, which is obtained by further module mapping of the feature representation maps through global and local attention mapping.

[0052] Here, the square represents the mapping matrix for each instance in each layer. Specifically, it can represent... , where L represents the total number of layers in the feature layer module, M and N represent the rows and columns of the mapped feature map matrix, and * indicates optional, representing the second mapping mode of the module mapping.

[0053] The module mapping of the feature information representation map includes the following process: S1411. Perform global module mapping on the global feature information representation maps of the student model and the teacher model to obtain global feature information and pixel-level relationship information, and obtain the global mapping representation maps of the student model and the teacher model respectively.

[0054] S1412. Perform local module mapping on the local feature information representation maps of the student model and the teacher model to obtain local feature information and pixel-level relationship information, and obtain the local mapping representation maps of the student model and the teacher model respectively.

[0055] S1413. Perform dimensionality compression on the global and local mapping representation graphs of the student model and the teacher model.

[0056] The above describes the module mapping method for Mode 2. Mode 2 performs dimensional compression based on Mode 1, ultimately resulting in a mapping representation diagram:

[0057] Based on the obtained mapping representation maps of the teacher model and the student model, the mapping representation map of the teacher model is used as a label to supervise and constrain the training of the student model. A mapping loss function is designed and added to the original loss function of the student model to jointly balance the training error and enhance the robustness of the model.

[0058] The loss functions for mapped distillation in Mode 1 and Mode 2 are as follows:

[0059] Where T and S represent the teacher model and student model, respectively.

[0060] In addition, the mapping distillation loss of Mode 2 provides an additional loss function, which calculates the loss function by taking into account the difference in conditional probability distributions of the mapping representation graphs of the teacher and student models. The conditional probability functions for samples i and j are as follows:

[0061] Where t represents the teacher model, N represents the number of samples, and K is the bandwidth. The kernel function is used to measure the similarity of samples. The specific kernel function is the cosine kernel function (to avoid calculating the kernel bandwidth), and its specific behavior is as follows:

[0062] in, Let each represent a vector representing one of the two sets of samples. Representing vectors Norm; The final loss function for the teacher and student model mapping representation graph is as follows:

[0063] Where t and s represent the teacher model and student model, respectively; Finally, the total distillation loss function is expressed as:

[0064] Among them, two hyperparameters To balance different distillation losses.

[0065] Based on the obtained mapping representation maps of the teacher model and the student model, the mapping representation map of the teacher model is used as a label to supervise and constrain the training of the student model. The loss function of mapping distillation is added to the original loss function of the student model to jointly balance the training error and enhance the robustness of the model.

[0066] The feature information representation map proposed in this embodiment for module mapping is based on Embodiment 1 or 2, and has the beneficial effects of the corresponding method embodiment. The same or similar technical content covered in Embodiment 1 or 2 will not be described in detail here.

[0067] This embodiment solves the problems of high memory consumption, small batch size of input images, and inability to be accelerated by GPU. At the same time, the module mapping method of mode 2 performs dimensional compression on the basis of mode 1 to obtain the final mapping representation map. Therefore, the module mapping of mode 2 can accept larger size and more batches of image input, thus improving its universality.

[0068] Experimental Example Please see Figures 9-15 This experimental example verifies a network-structure-oriented mapping distillation method and its training method as described in Examples 1, 2, and 3 above. Specific verification includes image classification experiments, object detection experiments, ablation experiments, and detection task ablation experiments. The specific scheme is as follows: 1. Image classification experiment 1.1 Experimental Datasets: The fine-grained image classification datasets Cars196 and CUB-200-2011 were selected.

[0069] 1.2 The experimental setup is as follows: The experiments used ResNet50 as the teacher network and ResNet18 as the student network. The same experimental settings were used for both datasets. The initial learning rate was set to 0.1, the batch size to 128, the total time to 80, and the ratio of the learning rate decrease to 0.1. All models used L2 normalization, and the loss function was calculated using the optimal combination of L2 norm and conditional probability distribution. The evaluation metric was Recall@K. Once all test images were embedded into the model, each test image was used as a query, and the top K nearest neighbor images not containing the query were retrieved from the test set. If a retrieved image contained the same category as the query, the recall for that query was considered 1. Recall@K was calculated by taking the average recall across the entire test set.

[0070] 1.3 Experimental results are as follows Figure 9 As shown, the experiment is summarized as follows: The ResNet18 extracted by AOMD in this invention achieves accuracies of 76.03% and 55.26% on the Cars196 and CUB-200-2011 datasets, respectively, representing improvements of 8.2% and 3.6% compared to the baseline. Compared to Fitnet (response-based), it improves accuracy by 4.2% and 3.4%; compared to feature-based attention, it improves accuracy by 21.4% and 11.2%; compared to SP (relationship-based), it improves accuracy by 5.6% and 3.0%; and compared to ICKD, it improves accuracy by 2.8% and 5.0%.

[0071] 1.4 The following results can be analyzed: (1) Attention distillation is not suitable for fine-grained classification tasks and the distillation effect is poor; relation distillation is not superior to traditional response-based distillation and performs poorly.

[0072] (2) Mapping distillation can mutually compensate for and enhance the representation of features and relationships, and is superior to feature-based and response-based distillation alone.

[0073] (3) Mapping distillation is adapted to image classification tasks.

[0074] Furthermore, such as Figure 10 and Figure 11 The accuracy curves shown in the comparison experiment on the Cars196 and CUB-200-2011 datasets can be analyzed as follows.

[0075] The AOMD of this invention achieves optimal accuracy within 10 to 20 epochs. Combined with... Figure 9 AOMD demonstrates advantages in fine-grained classification tasks, namely fast convergence and significant positive returns, which verifies the superiority and high performance of the AOMD method of this invention.

[0076] 2. Target Detection Experiment 2.1 Experimental Dataset: The benchmark dataset for face detection, Wideface, was selected.

[0077] 2.2 The experimental setup is as follows: The experiment was conducted on a wide-format platform, performing multi-branch tasks including face classification, face bounding box regression, and face five-point regression. To accommodate differences in network architecture, ResNet50 was used as the teacher network, and MobileNetV1x0.25 as the student network. The initial learning rate was set to 1e-3, with a total duration of 250 epochs. The learning rate decreased at 190 and 220 epochs, with a decreasing coefficient of 0.1. For evaluation, Deng's detection metrics were used. Based on the EdgeBox detection rate, it defines three difficulty levels (easy, medium, and hard) by progressively increasing the complexity of the samples.

[0078] 2.3 Experimental results are as follows Figure 12 As shown, the experimental results are summarized below: Experiments were conducted using distillations based on response (FT), features (Fine-grained, Zhang), and relations (CC, FSP, RKD, PKT). Compared to the response-based FT model, the accuracy was improved by 2% compared to the original student network, approximately 0.4%; compared to fine-grained and Zhang's attentional distillations, improvements were approximately 0.4% and 3.7%, respectively; and compared to relation-based distillations (CC, FSP, RKD, and PKT), improvements ranged from approximately 1.5% to 4.2%.

[0079] 2.4 The results are analyzed as follows: (1) By incorporating the attention mechanism into the relation matrix, the mapping representation is enhanced, which makes up for the shortcomings of relation-based extraction in object detection.

[0080] (2) The AOMD of the present invention can effectively compensate for feature representation and relation representation, and is suitable for target detection tasks.

[0081] Furthermore, such as Figure 13 The diagram shows the probability distribution of the mapping representation matrix of the teacher and student models. The trained student model exhibits a similar probability distribution to the teacher model.

[0082] 3. Ablation test The ablation experiments investigated the effects of two modes (module mapping modes one and two) and hyperparameters (α and β) on image classification tasks (Cars196, CUB-200-2011) and object detection tasks (WIDERFACE).

[0083] 3.1 The classification task ablation experiment is as follows: The experimental datasets selected were the fine-grained image classification datasets Cars196 and CUB-200-2011.

[0084] 3.2 The experimental setup is as follows: The experiments used ResNet50 as the teacher network and ResNet18 as the student network. The same experimental settings were used for both datasets. The initial learning rate was set to 0.1, the batch size to 128, the total time to 80, and the ratio of the learning rate decrease to 0.1. All settings used L2 normalization, and to ensure fairness, the loss function was calculated using the L2 norm. The evaluation metric was Recall@K. Once all test images were embedded into the model, each test image was used as a query, and the top K nearest neighbor images not containing the query were retrieved from the test set. If a retrieved image contained the same category as the query, the recall for that query was considered 1. Recall@K was calculated by taking the average recall across the entire test set.

[0085] 3.3 Experimental results are as follows Figure 14 As shown, the results are as follows: In terms of modes, Mode 1 on Cars196 is slightly higher than Mode 2 by about 0.1%, while Mode 1 on CUB-200-2011 is slightly lower than Mode 2 by about 0.8%. On Cars196 and CUB-200-2011, for the values ​​of hyperparameters α and β, when α=1 and β=0, the AOMD of Mode 1 (Mode 2) increases by about 7.4% (7.4%) and 0% (2.4%), respectively, showing a positive gain for distillation effect; when α=0 and β=1, the AOMD of Mode 1 (Mode 2) decreases by about 14.2% (14.5%) and 8.1% (7.5%), respectively, indicating a negative gain for distillation effect; when α=1 and β=1, the AOMD of Mode 1 (Mode 2) increases by about 8.2% (8.1%) and 1.9% (2.7%), respectively, which is the most suitable for distillation.

[0086] 3.4. From the above results, we can analyze: (1) Both AOMD modes are effective for fine-grained classification datasets, with mode 2 performing the best.

[0087] (2) AOMD performs best when both hyperparameters α and β are specified (e.g., α=1, β=1).

[0088] 4. Ablation experiment for detection task The experimental dataset was selected from the benchmark dataset for face detection, WideRFACE.

[0089] 4.1 The experimental setup is as follows: The experiment was conducted on a wide-format platform, performing multi-branch tasks including face classification, face bounding box regression, and face five-point regression. To accommodate differences in network architecture, ResNet50 was used as the teacher network, and MobileNetV1x0.25 as the student network. The initial learning rate was set to 1e-3, with a total duration of 250 epochs. The learning rate decreased at 190 and 220 epochs, with a decreasing coefficient of 0.1. For evaluation, Deng's detection metrics were used. Based on the EdgeBox detection rate, it defines three difficulty levels (easy, medium, and hard) by progressively increasing the complexity of the samples.

[0090] 4.2 Experimental results are as follows Figure 15 As shown, the results are as follows: For the modes, Mode 2 is slightly better than Mode 1 in terms of width (0.2% higher accuracy only for hardness level). Regarding hyperparameters α and β, when α=1 and β=0, Mode 1 (Mode 2) shows an approximately 1.4% (1.4%) improvement in AOMD, exhibiting a positive distillation effect; when α=0 and β=1, Mode 1 (Mode 2) shows an approximately 1.6% (1.4%) decrease in AOMD for hardness level accuracy, indicating a negative gain from the distillation effect; when α=1 and β=1, Mode 1 (Mode 2) shows an approximately 2.0% (1.8%) increase in AOMD for hardness level accuracy, which is optimal for distillation. In terms of memory consumption, Mode 2 allows for adding 20 640×640 images to the same memory environment, 12 more batch sizes than Mode 1.

[0091] 4.3. From the above results, we can analyze: (1) Both AOMD modes are effective for object detection datasets, with mode 2 performing the best.

[0092] (2) AOMD performs best when two hyperparameters α and β are assigned (e.g., α=1, β=1).

[0093] (3) On the target detection dataset, compared with AOMD mode 1 which has the same computing resources, AMOD mode 2 can train more batches and has better distillation effect.

[0094] The proposed mapping distillation method and its training method embed pixel-level relationship information between adjacent modules based on an attention mechanism into a mapping representation graph, which is then passed from the teacher model to the student model to constrain and supervise their training and learning. This method exhibits excellent performance in both classification and detection tasks, and effectively combines the advantages of relational and attentional information, compensating for and enhancing the expressive power of feature and relational representations.

[0095] The mapping distillation method proposed in this invention takes into account both image classification and object detection tasks, and exhibits excellent performance in both. Compared with existing distillation methods, this method achieves better performance and significantly reduces the computational workload. It also provides two optional module mapping methods, which can accept larger image inputs and more batches, thus improving the applicability of the method.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are substantially similar to the method embodiments, their descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0097] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A mapping-based distillation method for network structures, characterized in that, Based on pre-trained object detection models or image classification models as the basic data scalar, it is applied to the image classification tasks of the fine-grained image classification datasets Cars196 and CUB-200-2011, as well as the object detection task of the benchmark dataset WideRFACE for face detection. The face detection includes face classification, face bounding box regression, and face five-point regression. The mapping distillation method Includes the following processes: Teacher and student models are constructed based on pre-trained object detection or image classification models; Extract feature maps from each layer of the teacher model and student model respectively; The extracted feature maps are then subjected to attention mapping to obtain a feature information representation map with attention information; The attention mapping includes global attention mapping and local attention mapping; The feature map is subjected to attention mapping. The process includes the following steps: performing global attention mapping on the feature information representation maps of the student model and the teacher model to obtain global feature information, thus obtaining the global feature information representation maps of the student model and the teacher model respectively; the global attention mapping function is: Specifically: ; Furthermore, local attention mapping is performed on the feature information representation maps of the student model and the teacher model to obtain local feature information, resulting in local feature information representation maps of the student model and the teacher model, respectively; the local attention mapping function is: Specifically: ;in, L represents the number of feature layers in the teacher and student models, N, C, H, and W represent the batch size, channels, width, and height of the feature map, respectively, and i, j, and k represent the spatial and channel dimensions, respectively. i th j th k th Vector fragments; The feature information representation map is mapped by modules to obtain a mapped representation map with pixel-level relationship information; The module mapping includes global module mapping and local module mapping; the module mapping of the feature information representation map includes the following process: performing global module mapping on the global feature information representation maps of the student model and the teacher model to obtain global feature information and pixel-level relationship information, and obtaining the global mapping representation maps of the student model and the teacher model respectively. The global module mapping function is: Specifically: ； and ; Furthermore, local module mapping is performed on the local feature information representation maps of the student model and the teacher model to obtain local feature information and pixel-level relationship information, thereby obtaining the local mapping representation maps of the student model and the teacher model respectively. The local module mapping function is: Specifically: ； and ;in, They represent Layer feature representation diagram; The module mapping of the feature information representation graph also includes the following process: dimensionality compression of the global and local mapping representation graphs of the student model and the teacher model; The mapping representation graph, which contains attention information and pixel-level relationship information, is distilled from the teacher model to the student model.

2. The mapping distillation method according to claim 1, characterized in that: The module mapping process for feature information representation maps also includes: The mapping representation graph of the teacher model is used as a label to supervise and constrain the training of the student model. The loss function of mapping distillation is added to the original loss function of the student model to jointly balance the training error and enhance the robustness of the model.

3. A training method for a mapping-based distillation method oriented towards network structures, characterized in that, Includes the following processes: Pre-trained object detection models or image classification models are used to construct teacher and student models. Feature maps of each feature layer module are extracted from the teacher model and the student model, and then fed into the mapping distillation method of any one of claims 1-2 above for training together with the student model.

Images