Temperature normalization knowledge distillation method and system based on multi-scale decoupling

Abstract

The invention discloses a temperature normalization knowledge distillation method and system based on multi-scale decoupling, and the method comprises the following steps: S1, inputting an image into a teacher model and a student model at the same time, respectively outputting logit feature maps by the teacher model and the student model, decomposing the global logit output of the teacher model into a plurality of local logit outputs, each local logit output corresponding to a specific region of the input image, extracting local logit information of the region from the feature map of the teacher model; the student model takes the local logit output by the teacher model as a learning result, and the learning result corresponds to the local logit of the teacher model on the same scale and the same area; s2, inputting the image into a pre-trained teacher model and a pre-trained student model respectively, calculating a standard deviation, performing local logit calculation on the teacher model and the student model according to an adaptive temperature mechanism to obtain probability distribution of the teacher model and probability distribution of the student model respectively, introducing a temperature scaling factor, and controlling the softening degree; and S3, combining the teacher and student model probability distribution with the normalized distillation loss, the semantic decoupling distillation loss and the cross entropy loss to calculate the total loss.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image classification model optimization, and in particular relates to a knowledge distillation method and system combining scale decoupling and adaptive temperature, which is used to improve the classification performance of a student model in a complex scene. BACKGROUND

[0002] With the wide application of deep learning models in the field of computer vision, image classification tasks have made significant progress in the field of computer vision. From the early LeNet to the current ResNet, EfficientNet and other complex architectures, the classification performance of the model has been continuously improved, and it can handle complex tasks including fine-grained classification, multi-label classification, etc. However, in the process of continuously improving the accuracy of the model, its memory usage also gradually increases. In order to cope with this challenge, knowledge distillation is introduced to reduce the capacity of the model. As an effective model compression technology, knowledge distillation migrates the knowledge of a large teacher model to a small student model, significantly reducing the computational cost while maintaining the performance of the model.

[0003] Existing knowledge distillation methods can be mainly divided into two categories: feature matching-based and output logit-based. Among them, the logit-based distillation method is of great concern due to its simple structure and high efficiency, but it still has the following major defects in practical application: first, existing methods usually directly use the global logit output of the teacher model for knowledge transfer, ignoring the differentiated semantic information that different scale regions in the image may contain, resulting in insufficient fine-grained knowledge transfer; second, existing methods use a fixed temperature parameter to soften the logit distribution, which is difficult to adapt to the differences in logit distribution of different samples, resulting in low efficiency of knowledge transfer.

[0004] In view of the above problems, there is an urgent need in the field of image classification technology for a new distillation method to comprehensively improve the representation learning ability of the student model. SUMMARY

[0005] In view of the above status of the prior art, the present application proposes a temperature normalization knowledge distillation method based on multi-scale decoupling and a system. The innovative solution proposed by the present application realizes more efficient and more accurate knowledge transfer by combining multi-scale logit decoupling technology and temperature normalization method. The present application not only retains the computational efficiency advantage of traditional logit distillation, but also significantly improves the performance on complex tasks such as fine-grained classification, providing a new technical path for the deployment of lightweight models.

[0006] The present application adopts the following technical solutions:

[0007] A temperature normalization knowledge distillation method based on multi-scale decoupling, comprising the following steps:

[0008] S1: input the image into the teacher model and the student model at the same time, and the two respectively output logit feature maps retaining spatial dimensions, perform multi-scale pooling operation on the global logit output of the teacher model, decompose the global logit output into a plurality of local logit outputs, each local logit output corresponds to a specific region of the input image, and the local logit information of the region is extracted from the feature map of the teacher model through an average pooling operation; the student model takes the local logit output by the teacher model as a learning result, and corresponds to the local logit of the teacher model in the same scale and the same region;

[0009] S2: for the local logit outputs of the teacher model and the student model, calculate the standard deviation; according to the adaptive temperature mechanism, the local logits of the teacher model and the student model are calculated by softmax to obtain the teacher model probability distribution and the student model probability distribution respectively, and a temperature scaling factor T norm is introduced to adaptively control the softening degree;

[0010] S3: combine the teacher model probability distribution and the student model probability distribution to calculate the total loss of the normalized distillation loss, the semantic decoupling distillation loss and the cross-entropy loss.

[0011] The present application fully considers the semantic expression difference of the teacher model in different spatial regions, decouples the global logit into a plurality of local logit outputs, expands the semantic level of knowledge distillation, solves the ambiguity transmission problem caused by the global knowledge mixing in the traditional method, effectively reduces the distillation error caused by the inconsistency of sample regions, and improves the capturing ability of the student model to fine-grained features.

[0012] As a preferred, in step S1, the global logit output of the teacher model is decomposed into a plurality of local logit outputs, and each local logit output corresponds to a specific region of the input image. In this way, the student model can learn more fine-grained semantic information, thereby improving its processing capability for complex samples and ambiguous samples, and the formula is represented as:

[0013]

[0014] Wherein, C(m,n) represents a spatial position set of the nth region under the mth scale, L T is the logit feature map of the teacher model, L S is the logit feature map of the student model. Here, L T (j,k) is the feature map L TThe logit value at (j, k) is aggregated by an average pooling operation to obtain the local logit output V m,n ; L S (j, k) is the feature map L S The logit value at (j, k) is aggregated by an average pooling operation to obtain the local logit output Z m,n .

[0015] As preferred, in step S1, the feature map F T of the teacher model is subjected to a multi-scale pooling operation to segment it into regions of different scales m; define a scale set M = {1, 2, 4, …, w}, where w is the width or height of the feature map; for each scale m, segment the feature map into m x m regions, each region has a size of

[0016] As preferred, in step S1, a semantic decoupling distillation mechanism is introduced to decouple the local logit of the teacher model into consistency knowledge and complementary knowledge to improve the student model's ability to distinguish ambiguous samples.

[0017] As preferred, in step S1, each local logit output contains semantic information of the corresponding region, which may belong to the same class as the global logit output or belong to different classes. Consistency knowledge refers to the local logit output that belongs to the same class as the global logit output, which is used to convey semantic information at multiple scales. Complementary knowledge refers to the local logit output that belongs to a different class from the global logit output, which is used to retain the ambiguity information of the sample. Specifically, when the global prediction is correct but the local prediction is wrong, the inconsistent local knowledge encourages the student to maintain the ambiguity of the sample and avoid overfitting to ambiguous samples. On the other hand, when the global prediction is wrong but the local prediction is correct, the inconsistent local knowledge can encourage the student to learn from the similar components between different classes, alleviating the bias caused by the teacher, which is expressed in the formula as:

[0018] D(m, n) = LD(σ(V m,n ), σ(Z m,n ))

[0019] L SCD = D con + βD com

[0020] where σ(·) denotes the softmax function to convert logits to probability distribution, LD(·,·) denotes the base logit distillation loss function, which is KL divergence in this case, and D(m, n) denotes the distillation loss to transfer teacher's logit knowledge to student. L SCD denotes the semantic decoupling distillation loss, D con denotes the consistency logit knowledge distillation loss, D com denotes the complementary logit knowledge distillation loss, and a hyperparameter β is introduced to control the degree of regularization. As preferred, in step S1, after obtaining the local logit vectors of each scale and each region through the multi-scale pooling operation, the semantic decoupling distillation mechanism is introduced to compare the predicted classes of the obtained local logit and the global logit. The same class is consistent, and the different class is complementary. The local logit is decoupled and output as consistent knowledge and complementary knowledge.

[0021] As preferred, in step S2, an adaptive temperature mechanism is introduced to dynamically adjust the temperature parameter according to the characteristics of each local logit output. Specifically, the adaptive temperature mechanism uses the standard deviation of the local logit output as the temperature adjustment factor, thereby achieving adaptive adjustment for different samples and different local regions. The standard deviation reflects the dispersion degree of the logit distribution. A large standard deviation indicates that the logit distribution is dispersed, and a higher temperature is needed for softening. A small standard deviation indicates that the logit distribution is concentrated, and a lower temperature is needed to retain details. The formula is as follows:

[0022]

[0023] where σ m,n denotes the standard deviation, m denotes the scale, n denotes the region index at the scale, μ m,n is the mean of the local logit output, C is the number of classes, and V m,n,k is the logit value of the kth class of the local logit output of the teacher model.

[0024] As preferred, in step S2, the adaptive temperature mechanism performs softmax on the local logit of the teacher model and the student model, and introduces a temperature scaling factor T norm to control the overall scaling degree of the temperature. The temperature is no longer fixed, but is dynamically adjusted according to the logit distribution of each local region. The probability distribution of the teacher model and the student model is as follows:

[0025]

[0026] where the local output of the teacher model is denoted as V m,n , and the local output logit of the student model is denoted as Zm,n .

[0027] As preferred, in step S3, the normalized distillation loss formula is expressed as:

[0028]

[0029] wherein m represents a scale, n represents a region index at the scale, μ m,n is the mean of the local logit output, C is the number of classes, N represents the number of samples in the current training batch, V m,n,k is the logit value of the kth class of the local logit output of the teacher model, and the local output of the teacher model is denoted as V m,n , and the logit of the student model output is denoted as Z m,n . In order to further control the adjustment range of the temperature, a hyperparameter T norm , is introduced, which represents the standard deviation of the logit of the ith sample of the teacher model, and the multiplication of the two parameters constitutes a temperature compensation term, which forces the student to imitate the probability distribution of the teacher in the normalized distribution space.

[0030] As preferred, the cross-entropy loss function introduces uncertainty into the training process of the student model, and by calculating the difference between the probability distribution output by the student model and the real label, the model is prompted to continuously adjust the parameters in the retraining process to narrow the gap between the student model and the corresponding target in the training, and the formula can be expressed as:

[0031]

[0032] wherein L CE represents the cross-entropy loss, C is the number of classes, Z n k represents the normalized output feature of the student, y k represents the kth component of the real label, P(Z n k ) represents the probability of the kth class predicted by the student model after softmax.

[0033] As preferred, in step S3, each type of loss obtained above is combined with the cross-entropy function, and a new distillation function is reprogrammed using the loss function, and the formula is expressed as:

[0034] L total = L CE + L NKD + aL SCD

[0035] Wherein, alpha represents the corresponding weight coefficient. The importance of each distillation loss in knowledge distillation is reflected by different weights. The main idea of this strategy is to make the output performance of the teacher model and the student model as close as possible, and let the student model imitate various output characteristics of the teacher model.

[0036] The application further discloses a temperature normalization knowledge distillation system based on multi-scale decoupling, which is used for executing the method and comprises the following modules.

[0037] The multi-scale decoupling module: the image is input into the teacher model and the student model at the same time, and the two output logit feature maps that reserve spatial dimensions, respectively, the global logit output of the teacher model is subjected to a multi-scale pooling operation and is decomposed into a plurality of local logit outputs, each of which corresponds to a specific region of the input image, and the local logit information of the region is extracted from the feature map of the teacher model through an average pooling operation; the student model takes the local logit output by the teacher model as a learning result and corresponds to the local logit of the teacher model on the same scale and in the same region.

[0038] The adaptive temperature distillation module: the image is input into the pre-trained teacher model and the student model, respectively, the teacher model and the student model each extract features through a convolutional layer, and then output local logits through a fully connected layer; the standard deviation is calculated for the local logit outputs of the teacher model and the student model; the teacher model probability distribution and the student model probability distribution are obtained through softmax calculation on the local logits of the teacher model and the student model according to the adaptive temperature mechanism, and a temperature scaling factor T is introduced. norm The softening degree is adaptively controlled.

[0039] The total loss calculation module: the normalized distillation loss, the semantic decoupling distillation loss and the cross-entropy loss are combined to calculate the total loss.

[0040] Compared with the prior art, the application has the following beneficial effects:

[0041] (1) The temperature normalization knowledge distillation method and system based on multi-scale decoupling of the application, on the basis of the existing image classification knowledge distillation framework, according to the characteristics of the feature map space dimension of the convolutional neural network, the global logit output of the teacher model is innovatively decoupled by three spatial resolutions, which correspond to the global, intermediate and fine-grained visual regions of the image, respectively, the key channel weight is mapped to the corresponding local region by means of the cross-scale attention alignment mechanism, and the multi-scale local logit decoupling mechanism is introduced, which refines the global knowledge into the knowledge representation of a plurality of local regions, significantly improves the fineness and accuracy of knowledge transmission, and effectively solves the semantic ambiguity problem caused by the mixed multi-class information of the global logit in the traditional distillation.

[0042] (2) The application discloses a temperature normalization knowledge distillation method and system based on multi-scale decoupling. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. The drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the premise of the drawings.

[0044] Figure 1 is a flow chart of a temperature normalization knowledge distillation method based on multi-scale decoupling according to a preferred embodiment of the present application.

[0045] Figure 2 is a flow chart of a teacher model and a student model outputting a probability distribution according to a preferred embodiment of the present application.

[0046] Figure 3 is a semantic decoupling distillation loss output diagram according to a preferred embodiment of the present application.

[0047] Figure 4 is a cross-entropy loss output diagram according to a preferred embodiment of the present application.

[0048] Figure 5 is a block diagram of a temperature normalization knowledge distillation system based on multi-scale decoupling according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application claimed, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0051] As shown in Figure 1 The preferred embodiment of the present application discloses a temperature normalization knowledge distillation method based on multi-scale decoupling, comprising the following steps:

[0052] Step 1: input the image into the teacher model and the student model at the same time, and the two output logit feature maps that retain the spatial dimension, perform multi-scale pooling operation on the global logit output of the teacher model, and decompose it into multiple local logit outputs, each local logit output corresponds to a specific region of the input image, and the local logit information of the region is extracted from the feature map of the teacher model through the average pooling operation; the student model takes the local logit output by the teacher model as the learning result, and corresponds to the local logit of the teacher model on the same scale and in the same region; introduce a semantic decoupling distillation mechanism, compare the predicted classes of the obtained local logit and global logit, and the same class is consistent, and the different class is mutual, so as to decouple the local logit output into consistent knowledge and complementary knowledge;

[0053] Step 2: calculate the standard deviation for the local logit output of the teacher model and the student model; according to the adaptive temperature mechanism, the local logit of the teacher model and the student model is calculated by softmax to obtain the teacher model probability distribution and the student model probability distribution respectively, and a temperature scaling factor T norm is introduced to adaptively control the softening degree;

[0054] Step 3: combine the teacher model probability distribution and the student model probability distribution to normalize the distillation loss, the semantic decoupling distillation loss and the cross entropy loss, and calculate the total loss.

[0055] The embodiment is based on the temperature normalization knowledge distillation method based on multi-scale decoupling to realize the performance enhancement of the image classification model, the student model fits the different smoothing degree characteristics of the teacher model through the adaptive temperature, and the classification ability of the student model is significantly improved by using the multi-scale local logit decoupling, and the knowledge distillation performance is enhanced.

[0056] Specifically, in step 1, in order to capture the response information of the teacher model and the student model to different regions of the input image more meticulously, multi-scale pooling operations are performed on the global logits output by the teacher model and the student model. Specifically, first, the global logits output by the teacher model and the student model are regarded as a whole feature map, and then the feature map is divided by different scale pooling windows. Each pooling window corresponds to a specific region in the input image, and by performing average pooling operations in these windows, the local logit information of the corresponding region is extracted from the feature maps of the teacher model and the student model. The local logit output of the teacher model and the student model is:

[0057]

[0058] Where C(m, n) represents the spatial position set of the nth region at the mth scale, L T is the logit feature map of the teacher model, and L S is the logit feature map of the student model.

[0059] In step 1, the feature map F T of the teacher model is subjected to multi-scale pooling operation and divided into regions of different scales m; define the scale set M = {1, 2, 4, …, w}, where w is the width or height of the feature map; for each scale m, the feature map is divided into m x m regions, and the size of each region is

[0060] In step 1, in order to more effectively utilize the local logit information provided by the teacher model, a semantic decoupling distillation mechanism is designed, the core idea of which is to divide the local logit output of the teacher model into two types of knowledge: consistency knowledge and complementary knowledge, as shown in Figure 3

[0061] Specifically, consistency knowledge refers to the logit information that the teacher model and the student model show similar prediction tendencies on the same image region. This type of knowledge reflects the common cognition between the teacher model and the student model. Complementary knowledge, on the other hand, corresponds to the logit information that the teacher model and the student model have large prediction differences on the same region. This type of knowledge usually appears in regions of the image that are more ambiguous, complex or ambiguous. The formula is expressed as:

[0062] D(m, n) = LD(σ(V m,n ), σ(Z m,n ))

[0063] L SCD = D con + βD com ​

[0064] where D(m, n) represents the distillation loss of transferring the teacher's logit knowledge to the student, σ(·) represents the softmax function that converts logits to probability distribution, LD(·, ·) represents the basic logit distillation loss function; L SCD represents the semantic decoupling distillation loss, D con represents the consistency logit knowledge distillation loss, D com represents the complementary logit knowledge distillation loss, and β is introduced to control the relative weight of the complementary loss in the total distillation loss, determining the attention degree of the student model to the "fuzzy area" or "ambiguous samples".

[0065] In step 2, the input image is input into a pre-trained teacher model and a student model respectively. First, the image is feature extracted through multiple convolutional layers, and then the extracted features are sent to the fully connected layer to generate logit output. On this basis, the standard deviation of the logit values generated by the teacher model and the student model at each local position (or each class) is calculated respectively.

[0066] The standard deviation of the local logit is used as the adjustment factor of the temperature to realize the temperature self-adaptation at the sample level. The standard deviation reflects the dispersion degree of the logit distribution, and the larger the standard deviation, the higher the temperature required for softening; the smaller the standard deviation, the lower the temperature required to retain details. The formula of the standard deviation is:

[0067]

[0068] where C is the number of classes, m represents the scale, n represents the region index at the scale, μ m,n is the mean of the local logit output, V m,n,k is the logit value of the kth class of the local logit output of the teacher model.

[0069] The adaptive temperature mechanism is used to perform softmax (normalization exponential function) on the local logit of the teacher model and the student model, and a temperature scaling factor T norm is introduced to control the overall scaling degree of the temperature. The temperature is no longer fixed, but dynamically adjusted according to the logit distribution of each local region. As Figure 2 shown, the probability distribution of the teacher model and the student model is:

[0070]

[0071] where N represents the number of samples in the current training batch, the local output of the teacher model is denoted as V m,n , and the logit output of the student model is denoted as Zm,n To further control the adjustment range of temperature, a hyperparameter T norm , σ i t , which represents the standard deviation of the logit of the i-th sample of the teacher model, the multiplication of the two parameters constitutes a temperature compensation term, forcing the student to imitate the probability distribution of the teacher in the normalized distribution space, L NKD is the normalized distillation loss.

[0072] In step 3, the total loss function of the method can be combined according to the above steps, including the normalized distillation loss, the semantic decoupling distillation loss and the cross-entropy loss, and the loss minimization is carried out in the training and distillation process, and the formula can be expressed as:

[0073] L total = L CE + L NKD + αL SCD

[0074] wherein, L NKD represents the normalized distillation loss, L SCD represents the semantic decoupling distillation loss, and α represents the corresponding weight coefficient, which reflects the importance of each distillation loss in knowledge distillation through different weights. The main idea of the method is to make the output performance of the teacher model and the student model as close as possible, and let the student model imitate various output characteristics of the teacher model.

[0075] In training the student model, a cross-entropy loss function is introduced, which calculates the difference between the probability distribution output by the student model and the real label, and promotes the model to continuously adjust the parameters in the training process to reduce the prediction error. As Figure 4 shown, the formula is expressed as:

[0076]

[0077] wherein, C is the number of categories, Z n k represents the normalized output feature of the student, y k represents the k-th component of the real label, P(Z n k ) represents the probability of the k-th category predicted by the student model after softmax.

[0078] The method of the present application will be compared with the prior art in the following experiments:

[0079] Table 1: Image classification accuracy results (%) on CIFAR-100 dataset

[0080]

[0081] From the above experimental results, it can be found that, as shown in Table 1, in the image classification task, the method SADD of the present application is superior to the prior art in the same architecture scenario. The multi-scale decoupling and adaptive temperature mechanism can mine deeper local semantics, thereby continuously outperforming the prior art, proving that the distillation effect between similar models of the present application is better. This result fully shows that the SADD of the present application can more effectively bridge the semantic gap between models under the synergistic action of multi-scale local decoupling and adaptive temperature, and realize robust knowledge transfer across channels and receptive fields.

[0082] As shown in Figure 5 , the embodiment discloses a temperature normalization knowledge distillation system based on multi-scale decoupling, which is used to execute the above method embodiment, and includes the following modules: The multi-scale decoupling module: the image is input into the teacher model and the student model at the same time, and the two respectively output logit feature maps that retain spatial dimensions. The global logit output of the teacher model is subjected to a multi-scale pooling operation to decompose it into a plurality of local logit outputs, each of which corresponds to a specific region of the input image. The local logit information of the region is extracted from the feature map of the teacher model through an average pooling operation. The student model takes the local logit output by the teacher model as a learning result, and corresponds to the local logit of the teacher model in the same scale and the same region. A semantic decoupling distillation mechanism is introduced to decouple the local logit into consistent knowledge and complementary knowledge.

[0083] The multi-scale decoupling module: the image is input into the teacher model and the student model at the same time, and the two respectively output logit feature maps that retain spatial dimensions. The global logit output of the teacher model is subjected to a multi-scale pooling operation to decompose it into a plurality of local logit outputs, each of which corresponds to a specific region of the input image. The local logit information of the region is extracted from the feature map of the teacher model through an average pooling operation. The student model takes the local logit output by the teacher model as a learning result, and corresponds to the local logit of the teacher model in the same scale and the same region. A semantic decoupling distillation mechanism is introduced to decouple the local logit into consistent knowledge and complementary knowledge.

[0084] The adaptive temperature distillation module: the image is input into the pre-trained teacher model and the student model. The teacher model and the student model each extract features through a plurality of convolutional layers, and then output local logits through a fully connected layer. For the local logit output of each teacher model and student model, the standard deviation is calculated, the local logits of the teacher model and the student model are subjected to softmax according to the adaptive temperature mechanism, and a temperature scaling factor T norm is introduced to adaptively control the softening degree.

[0085] The total loss calculation module combines the teacher model probability distribution with the student model probability distribution to calculate a total loss of the normalized distillation loss, the semantic decoupling distillation loss and the cross-entropy loss.

[0086] Other contents of the embodiment can refer to the above method embodiment.

[0087] To sum up, the application discloses a temperature normalization knowledge distillation method and system based on multi-scale decoupling, a knowledge distillation strategy based on multi-scale decoupling, division of global logit (logistic regression) output of a teacher model into multiple local logit regions, extraction of consistency knowledge and complementary knowledge respectively, improvement of understanding ability of a student on fine-grained semantics, introduction of an adaptive temperature mechanism, dynamic adjustment of distillation temperature according to a standard deviation of local logit, realization of more reasonable knowledge softening and transmission, calculation of classification loss by using a cross-entropy loss, adjustment of loss weight by using a hyperparameter, construction of a more stable and efficient knowledge distillation framework, and significant improvement of discriminant performance and generalization ability of the student model in a complex scene.

[0088] The above merely describes preferred embodiments of the application, but should not be used to limit the application, and the application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

[0089] It is obvious for those skilled in the art that the application is not limited to the details of the above exemplary embodiments, and the application can be implemented in other specific forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application. Any reference signs in the claims should not be regarded as limiting the involved claims.

Claims

1. A temperature normalization knowledge distillation method based on multi-scale decoupling, characterized in that, Comprising the following steps: S1: input the image into the teacher model and the student model at the same time, and the two output logit feature maps that retain the spatial dimension, perform multi-scale pooling operation on the global logit output of the teacher model, decompose it into multiple local logit outputs, each local logit output corresponds to a specific region of the input image, and the local logit information of the region is extracted from the feature map of the teacher model through the average pooling operation; the student model takes the local logit output by the teacher model as the learning result, and corresponds to the local logit of the teacher model in the same scale and the same region; S2: For the local logit outputs of the teacher model and the student model, the standard deviation is calculated; the local logits of the teacher model and the student model are calculated by softmax to obtain the teacher model probability distribution and the student model probability distribution respectively, and a temperature scaling factor T is introduced to adaptively control the softening degree norm ; S3: combine the teacher model probability distribution and the student model probability distribution to normalize the distillation loss, the semantic decoupling distillation loss and the cross entropy loss, and calculate the total loss.

2. The temperature normalization knowledge distillation method based on multi-scale decoupling according to claim 1, characterized in that, In step S1, the global logit output of the teacher model is decomposed into multiple local logit outputs, each local logit output corresponds to a specific region of the input image, and the formula is represented as: where C(m, n) represents a spatial position set of the nth region at the mth scale, L T is a logit feature map of the teacher model, L S is a logit feature map of the student model; L T (j, k) is a logit value at (j, k) of the feature map L T The logit values within the region are aggregated by an average pooling operation to obtain a local logit output V m,n of the region; L S (j, k) is a logit value at (j, k) of the feature map L S The logit values within the region are aggregated by an average pooling operation to obtain a local logit output Z m,n of the region.

3. The method of claim 2, wherein, In step S1, the feature map F of the teacher model is processed. T Perform multi-scale pooling to divide it into regions of different scales m; Define a set of scales M = {1, 2, 4, …, w}, where w is the width or height of the feature map; for each scale m, partition the feature map into m x m regions, each of size 4. The temperature normalization knowledge distillation method based on multi-scale decoupling according to claim 3, characterized in that, In step S1, the semantic decoupling distillation mechanism is introduced, and the obtained local logit is compared with the predicted class of the global logit, and the same class is consistent, and the different classes are mutual.

5. The method of claim 4, wherein, In step S1, the local logit output of the teacher model is divided into consistent knowledge and complementary knowledge, and the formula is represented as: D(m, n) = LD(σ(V m,n ), σ(Z m,n )) L SCD = D con + βD com where D(m, n) represents the distillation loss of transferring the teacher's logit knowledge to the student, σ(·) represents the softmax function that converts the logit to the probability distribution, and LD(·, ·) represents the basic logit distillation loss function; L SCD represents the semantic decoupling distillation loss, D con represents the consistency logit knowledge distillation loss, D com represents the complementarity logit knowledge distillation loss, and β is introduced to control the relative weight of the complementarity loss in the total distillation loss.

6. The method of claim 5, wherein, In step S2, the standard deviation of the local logit is used as the adjustment factor of the temperature, and the self-adaptive adjustment of different samples and different local regions is realized, and the formula is represented as: where σ m,n denotes the standard deviation, m denotes the scale, n denotes the region index at that scale, μ m,n is the mean of the local logit output, C is the number of classes, V m,n,k is the logit value of the k-th class of the teacher model local logit output.

7. The method of claim 6, wherein, In step S2, the adaptive temperature mechanism is specifically as follows: the local logits of the teacher model and the student model are subjected to softmax, and a temperature scaling factor T is introduced norm to control the softening degree of the temperature; the temperature is dynamically adjusted according to the logit distribution of each local region; and the probability distribution of the teacher model and the student model is as follows: where the local output of the teacher model is denoted as V m,n , and the local output logit of the student model is denoted as Z m,n . To further control the adjustment range of the temperature, a hyper-parameter T norm , denotes the standard deviation of the teacher model logit for the i-th sample, and the multiplication of the two parameters constitutes the temperature compensation term, forcing the student to imitate the probability distribution of the teacher in the normalized distribution space, L NKD is the normalized distillation loss.

8. The method of claim 7, wherein, In step S3, the normalized distillation loss, the semantic decoupling distillation loss and the cross entropy loss are combined to minimize the loss in the training and distillation process, and the formula can be represented as: L total = L CE + L NKD + aL SCD wherein L CE denotes the cross-entropy loss, L NKD denotes the normalized distillation loss, L SCD denotes the semantic decoupling distillation loss, and a denotes the corresponding weight coefficient.

9. The method of claim 8, wherein, The cross entropy loss formula is represented as: where C is the number of classes, Z n k denotes the normalized output features of the student, y k denotes the kth component of the true label, P(Z n k denotes the probability of the kth class predicted by the student model after softmax.

10. A multi-scale decoupling based temperature-normalized knowledge distillation system for performing the method of any one of claims 1-9, characterized in that, Comprising the following modules: Multi-scale decoupling module: input the image into the teacher model and the student model at the same time, and the two output logit feature maps that retain the spatial dimension, perform multi-scale pooling operation on the global logit output of the teacher model, decompose it into multiple local logit outputs, each local logit output corresponds to a specific region of the input image, and the local logit information of the region is extracted from the feature map of the teacher model through the average pooling operation; the student model takes the local logit output by the teacher model as the learning result, and corresponds to the local logit of the teacher model in the same scale and the same region; Adaptive temperature distillation module: the image is input into a pre-trained teacher model and a student model respectively; the teacher model and the student model respectively extract features through a plurality of convolutional layers, and then output local logits through a fully connected layer; for the local logit output of each teacher model and student model, the standard deviation is calculated, the local logits of the teacher model and the student model are subjected to softmax according to the adaptive temperature mechanism, and a temperature scaling factor T is introduced norm , to adaptively control the softening degree. Total loss calculation module: combine the teacher model probability distribution and the student model probability distribution to normalize the distillation loss, the semantic decoupling distillation loss and the cross entropy loss, and calculate the total loss.

Images