A deep neural network model compression method and device based on knowledge distillation

Abstract

The application discloses a kind of based on knowledge distillation's deep neural network model compression method and system, the method includes: sample image data is preprocessed;Teacher network model is pre-trained based on sample image data after preprocessing;The spatial attention features of the image to be classified are extracted using the pre-trained teacher network model, spatial correlation features are obtained using self-attention mechanism based on each channel basic feature, and the spatial attention features are obtained by combining the two;Student network spatial attention features are extracted using the same method;Using spatial attention features, according to fine-grained layer weight to guide the training of student network model, iterate and distill student network model until it converges, wherein λ j Indicates the weight of the jth layer in the selected teacher and student network n intermediate layers, and the fine-grained layer weight is set as λ1<λ2<…<λ n ; Save student neural network model for image classification prediction. The application has the advantages of faster and more accurate.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and specifically to a method and apparatus for compressing deep neural network models based on knowledge distillation. Background Technology

[0002] In recent years, deep learning-based artificial intelligence (AI) technology has not only become a focus of research in academia but has also gained widespread popularity in industry, leading to the widespread adoption of numerous advanced technologies. From pose recognition, facial recognition with masks on, dialogue systems, and GAN (Generative Adversarial Network)-generated facial images to autonomous driving, these AI technologies have begun to serve human society, all thanks to the tireless efforts of researchers in the field of deep learning. Convolutional neural networks (CNNs), as one of the foundations for realizing AI, are also widely used in various task scenarios. However, application scenarios such as smart city security, smart city transportation, autonomous driving, edge computing, and video surveillance all place extremely high demands on network models. While these application models leverage the powerful feature extraction and processing capabilities of deep CNNs, they also face the high storage and computational costs that deep CNNs bring to deployment platforms. This means that although these deep models have met the performance requirements for the tasks, they are still difficult to deploy on mobile edge devices or embedded platforms. The huge number of parameters and computational demands of the network models limit their further development.

[0003] Therefore, research on lightweight convolutional neural network models is particularly important, and how to maintain the balance between lightweight network models and accuracy loss has become a key challenge. Currently, in addition to manually designing lightweight networks, the mainstream technical routes for lightweighting deep convolutional neural networks based on existing networks are mainly divided into four compression algorithms: parameter quantization, neural network architecture search, model pruning, and knowledge distillation. Among these techniques, knowledge distillation involves training a smaller network (student) under the guidance of a larger network (teacher), improving network performance without changing the structure and parameters of the original network. Compared with other model compression algorithms, knowledge distillation has the following advantages for compressing models: (1) The accuracy of the distilled model is guaranteed. (2) The distilled model does not require the design of a dedicated hardware platform and system architecture, and is highly flexible. (3) The network structure of the distilled model is regular, which is conducive to secondary acceleration. In summary, lightweight convolutional neural network technology can not only promote the reduction of the deployment difficulty of high-performance neural networks on mobile computing platforms, lay the foundation for convolutional neural networks to be competent in high real-time tasks, and promote the development and construction of video Internet of Things and smart cities, but also help researchers enhance their understanding of neural networks and the learning process of neural networks. Knowledge distillation, as a novel "non-destructive" lightweight technology, has rich research value and significance for solving the above problems.

[0004] Since the introduction of soft-label distillation in the KD work, knowledge distillation has undergone significant development. Here, KD refers to Hinton's initial work on knowledge distillation (Distilling the Knowledge in a Neural Network), and later also to Knowledge Distillation. However, the soft labels generated by high-performance teacher networks often do not differ significantly from the true labels output in the dataset. Therefore, when using high-performance teachers, relying solely on soft labels to train student networks may limit the effectiveness of knowledge distillation. To more effectively utilize the information available in the teacher network, FitNets introduced the practice of using the teacher's feature maps to guide the student model. However, this method only extracts instance-level consistency information, ignoring the structured information in the teacher network's feature space. The OEKD (On the Efficacy of Knowledge Distillation) approach points out that large models may not always be better teachers, and small models may not be able to fully mimic large models. To address the structural differences between teachers and students, they suggest stopping the strategy early in the training process. However, this approach is only a general direction, without a definitive solution or a specific early termination time. In the AT work "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer," two-dimensional features are used to represent the attention map, and constrained feature representations are used to guide the student. However, the results are still limited. Furthermore, the methods described above all neglect the importance of spatially relevant information, which is crucial for effective knowledge distillation. Summary of the Invention

[0005] Objective: This invention proposes a deep neural network model compression method based on knowledge distillation. Utilizing channel averaging, self-attention mechanisms, and fine-grained layer weighting techniques, it acquires spatial global knowledge and spatial correlation knowledge, better highlighting the feature representation capabilities of different layers in the neural network. Without altering any original model structure or parameter count, it significantly improves the performance of the network model in processing downstream tasks, and is computationally faster than other methods based on intermediate feature distillation.

[0006] The present invention also provides a deep neural network model compression device, apparatus, and storage medium based on knowledge distillation.

[0007] Technical Solution: To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0008] A deep neural network model compression method based on knowledge distillation includes the following steps:

[0009] The sample image data is preprocessed, including removing irrelevant information, enhancing the detectability of relevant information, and labeling.

[0010] The teacher network model is pre-trained based on preprocessed sample image data;

[0011] Spatial attention features of images to be classified are extracted using a pre-trained teacher network model, including: obtaining the basic features of each channel output of the teacher network model, obtaining spatial correlation features based on the basic features of each channel using a self-attention mechanism, and combining the spatial correlation features with the basic features to obtain spatial attention features; spatial attention features of the student network are extracted using the same method.

[0012] Utilizing extracted spatial attention features, the training of the student network model is guided by fine-grained layer weights. The student network model is iteratively distilled until it converges, where λ is the finite element. j This represents the weight of the j-th layer out of the n intermediate layers of the selected teacher and student network, where the fine-grained layer weights are set to λ1 < λ2 < ... < λ n ;

[0013] The student neural network model is saved for image classification prediction.

[0014] The present invention also provides a deep neural network model compression device based on knowledge distillation, comprising:

[0015] The data preprocessing module is used to preprocess the sample image data, including removing irrelevant information, enhancing the detectability of relevant information, and labeling.

[0016] The teacher network pre-training module is used to pre-train the teacher network model based on preprocessed sample image data.

[0017] The spatial attention feature extraction module is used to extract spatial attention features of images to be classified using a pre-trained teacher network model. This includes: obtaining the basic features output by each channel of the teacher network model; obtaining spatial correlation features based on the basic features of each channel using a self-attention mechanism; and combining the spatial correlation features with the basic features to obtain the spatial attention features of the teacher network. The same method is used to extract the spatial attention features of the student network.

[0018] The student network training module utilizes extracted spatial attention features and fine-grained layer weights to guide the training of the student network model, iteratively distilling the student network model until it converges, where λ is the threshold value. jThis represents the weight of the j-th layer out of the n intermediate layers of the selected teacher and student network, where the fine-grained layer weights are set to λ1 < λ2 < ... < λ n ;

[0019] The model saving and application module is used to save the student neural network model for image classification prediction.

[0020] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the knowledge distillation-based deep neural network model compression method as described above.

[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the knowledge distillation-based deep neural network model compression method described above.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] (1) This invention proposes a novel knowledge distillation method combining spatial attention features to guide the training of student network models with very low overhead. This method utilizes two types of spatial attention features: spatial relevance features obtained through self-attention mechanisms and global spatial features obtained through channel averaging. Spatial attention features utilize a compact two-dimensional feature representation containing less irrelevant and redundant information to effectively guide the student network. In addition, by introducing fine-grained layer distillation weights and utilizing the characteristics of different layers of the teacher network model, the guidance provided by each layer can be better utilized. (2) This invention effectively addresses the problems of limited soft-label knowledge distillation, excessive structural differences between teacher and student network models, insufficient simplification of knowledge in the teacher's intermediate layers, and overly complex and cumbersome learning processes for teachers and students. It enables student models to better understand the spatial attention features of the teacher model while avoiding potential misleading features that may have a negative impact on student networks, and achieves significant performance improvements in image classification tasks. (3) The improvements in the efficiency and versatility of model compression in this invention open up new opportunities for deploying edge models. Attached Figure Description

[0024] Figure 1 This is a flowchart of the deep neural network model compression method based on knowledge distillation according to the present invention;

[0025] Figure 2 This is a spatial attention feature extraction map according to the present invention;

[0026] Figure 3 This is a schematic diagram of the attention mechanism calculation according to the present invention;

[0027] Figure 4 This is a schematic diagram of the model training process according to the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0029] This invention proposes a deep neural network model compression method based on knowledge distillation. Unlike pruning and quantization in model compression, knowledge distillation constructs a lightweight smaller model and trains it using supervision information from a larger, more powerful model, aiming to achieve better performance and accuracy. This larger model is generally referred to as the teacher (or teacher network model), and the smaller model as the student (or student network model). The supervision information output from the teacher model is called knowledge, and the process by which the student learns and transfers the supervision information from the teacher is called distillation. In this embodiment, the teacher model uses ResNet110, and the student model uses ResNet32. (Refer to...) Figure 1 The method of the present invention includes the following steps:

[0030] Step (1) involves processing the sample image data.

[0031] Training the model requires a clean and efficient image dataset. Therefore, the original image data must first be cleaned, preprocessed, and shuffled. Specifically, image data cleaning, filtering, and feature labeling are performed. Further, irrelevant information in the images is eliminated, useful real information is recovered, the detectability of relevant information is enhanced, and the data is simplified to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching, and recognition. The main tasks involved include: grayscale conversion of images, geometric operations such as horizontal flipping and random cropping, and image numerical normalization.

[0032] Finally, the image dataset is divided into a training set and a test set. The training set is used to train the model, and the model is tested on the test set. The error on the test set is then used to approximate the model's generalization error in the real-world scenario.

[0033] Step (2) Pre-train a large-scale, high-performance image classification teacher network model.

[0034] The image training set is fed into the network, and the parameters are adjusted based on the difference between the network's actual output and the expected output. The training steps are as follows:

[0035] a. Select an image sample (A, B) from the training set, where A is the image data and B is the label, i.e., the category to which the image belongs;

[0036] b. Input the image sample into the neural network (initially the parameters in the network are all random values), and calculate the actual output Y of the network, i.e. the predicted value;

[0037] c. Calculate the error between the predicted value and the actual value: D = BY;

[0038] d. Adjust the parameters W in the network based on the error D using backpropagation;

[0039] e. Repeat the above ad process for each image sample until the entire image sample set has been traversed;

[0040] Repeat all the above ae steps until the network model converges and the error value no longer decreases, i.e., iterative convergence.

[0041] Furthermore, the trained teacher model is tested by feeding the image test set into the network and judging the image classification performance of the current network model based on the actual output. The steps for testing the model are as follows:

[0042] (i) Select an image sample (E, F) from the test set, where E is the image data and F is the label, i.e., the category to which the image belongs;

[0043] (ii) Input the image sample into the neural network. At this time, the parameters in the network have been updated after training. Calculate the actual output Y of the network.

[0044] (iii) Determine whether the maximum value of the network output is equal to the true label (F = Y). MAX If the results are equal, increment the number of correctly classified items by 1.

[0045] (iv) Repeat steps (i)-(iii) above for each image sample until the entire image test set has been traversed;

[0046] (v) Repeat all steps (i)-(iv) above to obtain the accuracy of the entire image test set and record the accuracy changes of the network model.

[0047] After training a large, high-performance image classification teacher network model, the trained high-performance teacher model is saved and used to guide student models.

[0048] Step (3) Extract spatial attention features of the teacher network and the student network.

[0049] The following is combined with Figure 2A description of the teacher's cyberspace attention feature extraction process is provided. However, it should be noted that in the method proposed in this invention, the same operations are performed on both the teacher model (t) and the student model (s). The only difference is that the teacher model is pre-trained, while the student model is not yet trained.

[0050] The process of extracting teachers' cyberspace attention features includes the extraction of global spatial features and spatial relevance features.

[0051] According to an embodiment of the present invention, a large trained model is used to extract the original features of the image to be classified, and the feature set is represented as follows: Here, 'n' represents the number of features. The specific features are selected from the intermediate layers of the convolutional neural network. This invention utilizes the features after the activation function of each convolutional block. Specifically, each feature is 3-dimensional data (C*H*W), including channel dimension C, height dimension H, and length dimension W. The feature dimensions differ at different depths within the same network model, and even at the same depth, the feature dimensions differ between different network models.

[0052] (3-1) The process of extracting spatial global features is as follows:

[0053] Each teacher feature is averaged along the channel dimension to generate an attention map with two-dimensional feature (H*W) attributes (query q). t , key k t Sum of values ​​v t The global spatial features here (query q) t , key k t Sum of values ​​v t Also known as basic features, these are used to subsequently obtain spatial correlation features. In the averaged feature space, each pixel represents the information from all channels at that corresponding location after averaging. The channel processing of the teacher model is mathematically represented by the following formula:

[0054]

[0055]

[0056] in Let represent the features obtained from layer j in the teacher network. Furthermore, the student network s and the teacher network t perform the same channel averaging operation, denoted by . Specifically, p is an exponential weight parameter, which is set to 2 in this method, |h i | 2 This means that the absolute value of the original feature is taken and then squared. After applying the above averaging method, a two-dimensional matrix v(H*W) representing the global spatial features is obtained. Let C represent the query vector, key vector, and value vector of the attention graph generated by the channel averaging operation, respectively. Let C be the number of channels, H be the height, W be the length, and t represent the teacher network.

[0057] (3-2) The process of extracting spatial correlation features is as follows:

[0058] However, relying solely on global features is insufficient; capturing the correlation information between pixels in the feature space is crucial. To address this issue, this invention utilizes a self-attention mechanism to obtain key spatial correlation features. The specific self-attention mechanism calculation is as follows: Figure 3 As shown. The equation for obtaining spatially relevant information is:

[0059]

[0060] The process of obtaining spatial relevance features involves calculating attention values ​​for the relationships between internal pixels in the spatial feature map using query q and key k, and then applying the softmax(·) function. Finally, the resulting attention scoring table is applied. value Weighted (two-dimensional matrix of global spatial features), two-dimensional matrix Each pixel in (H*W) is multiplied by its corresponding value in the attention scoring table to obtain a feature map m that captures relevant knowledge within the capture space. Scaling factor The weights are used to control the spatially relevant features. The spatially relevant features *m* obtained through self-attention are combined with the global spatial features *v* to obtain the final feature map *M*, representing the spatial attention features of the teacher model, as shown in the following formula:

[0061]

[0062] Using the same extraction method as that for teacher cyberspace attention features, student cyberspace attention features are extracted from student networks.

[0063] Step (4) utilizes the extracted spatial attention features to guide the training of the small target student network based on the fine-grained layer weights.

[0064] According to an embodiment of the present invention, the teacher's spatial attention characteristics are obtained in step (3). Characteristics of students' spatial attention Establish the following distillation term:

[0065]

[0066] During feature knowledge transfer, the feature maps used are subjected to L2 normalization. This means that each vectorized feature map M is... Replacement. Furthermore, assigning higher weights to deeper features can better demonstrate the guiding ability of the teacher network. To achieve this goal, this invention proposes a finer-grained weight distribution for both deep and shallow features. The fine-grained weight distribution formula is as follows:

[0067]

[0068] l sad λ represents the sum of all feature classification losses among the n intermediate layers of the manually selected teacher and student network. j Let λj represent the weights of the j-th layer. Where λ1 < λ2 < ... < λj. n This highlights the importance of deeper features in the intermediate layers of the network. Mean(l j ) represents the distillation term of the spatial attention characteristics of teacher networks and student networks. j The summation is followed by averaging (the summation and averaging of two-dimensional matrices yields a single value). The intermediate layers of the teacher and student networks are manually selected before training the student model. Each intermediate layer of the student network selects the end output of each model network layer group. The teacher network model is deeper than the student model, and the intermediate layers of the teacher model are selected based on the final outputs of every two model layer groups. In this way, the number of intermediate layers n for students and teachers can remain consistent (for example, the number of n is 4 in the distillation setting from ResNet110 to ResNet32).

[0069] Based on the input image data, spatial attention feature extraction, and fine-grained layer weights, the student model is iteratively distilled using three different image classification losses until the student model converges.

[0070] Besides the loss sad The total loss function of the student model in this invention also includes the classification loss of the true label and the classification loss of the soft label. The total loss function is expressed as follows:

[0071] l student =l rls +αl sls +βl sad

[0072] Reference Figure 4 In this method, the total loss function for updating the student model is represented by several components. rls It is a classification loss based on true labels, l sls It is a classification loss based on the teacher model generating soft labels. sad This represents the feature loss after enhancing the feature maps of the intermediate layers of the network model. The tradeoff parameters α and β control different distillation losses. Using this loss function, the student model can learn the knowledge of the teacher network and the true label information of the dataset.

[0073] The specific process of knowledge transfer from real labels is represented by the following formula:

[0074]

[0075] In the formula and Let represent the outputs of class i in the teacher network (ResNet110) and student network (ResNet32) before softmax classification, respectively. For example, in the Cifar-100 dataset, i has 100 classes. The output before softmax classification is the output of the last linear layer (output dimension 100), where each value represents the predicted value for that class. After softmax classification, the outputs are the normalized probabilities of each class. N represents the total number of classes in the classification task, and η(·) represents the softmax function. The function D... CEL (·) Use cross-entropy loss to represent the similarity between student and teacher categories.

[0076] The specific process of knowledge transfer using soft tags is represented by the following formula:

[0077]

[0078] In the formula and Let represent the outputs of class i in the teacher network (ResNet110) and student network (ResNet32) before softmax classification, respectively. For example, in the Cifar-100 dataset, i has 100 classes. The output before softmax classification is the output of the last linear layer (output dimension 100), where each value represents the predicted value for that class. After ordinary softmax classification, the result is the normalized probability of each class. T represents the distillation temperature, N represents the total number of classes in the classification task, and η(·) represents the softmax function. After softmax classification with temperature T, the result is the normalized, softened probability of each class. Function D kL (·) Use KL divergence to represent the similarity between student and teacher categories. Softened labels contain more generalized information than true labels, allowing student models to learn from teacher models.

[0079] Step (5) uses the total loss function obtained from the input image data, spatial attention feature extraction, and fine-grained layer weights to guide the training of the student network, resulting in a student network trained based on the teacher network model. This student network is then used as an image classification model to predict the category distribution of the input image.

[0080] To verify the effectiveness of the compression method proposed in this invention, experiments were conducted on image classification tasks. First, for image classification tasks, distillation experiments were performed on Cifar-100 and the large-scale image dataset ImageNet. The results were then compared with some influential works in this field.

[0081] Table 1 shows a comparison of the Top1 accuracy (%) of the present invention and existing knowledge distillation algorithms on the CIFAR-100 image classification dataset for different network structures.

[0082] Table 1 shows the accuracy on the CIFAR-100 image classification dataset.

[0083]

[0084]

[0085] Table 2 shows a comparison of the Top1 and Top5 accuracies (%) of the present invention and existing knowledge distillation algorithms on the ImageNet image classification dataset for different network structures.

[0086] Table 2 shows the accuracy on the ImageNet image classification dataset.

[0087] Teacher Network Student Network KD CRD AFD Ours Top1 73.31 69.75 70.66 71.35 71.38 71.57 Top 5 91.42 89.07 89.88 90.49 90.36 90.64

[0088] By comparing the classification accuracy of this invention with other knowledge distillation algorithms, it can be observed that this method achieves optimal results across all network structures. Notably, this invention is the first to achieve a Top-1 accuracy exceeding 74.38% on the Cifar-100 classification task using a ResNet32 network.

[0089] Based on the same inventive concept as the method embodiments, the present invention also provides a deep neural network model compression device based on knowledge distillation, comprising:

[0090] The data preprocessing module is used to preprocess the sample image data, including removing irrelevant information, enhancing the detectability of relevant information, and labeling.

[0091] The teacher network pre-training module is used to pre-train the teacher network model based on preprocessed sample image data.

[0092] The spatial attention feature extraction module is used to extract spatial attention features of images to be classified using a pre-trained teacher network model. This includes: obtaining the basic features output by each channel of the teacher network model; obtaining spatial correlation features based on the basic features of each channel using a self-attention mechanism; and combining the spatial correlation features with the basic features to obtain the spatial attention features of the teacher network. The same method is used to extract the spatial attention features of the student network.

[0093] The student network training module utilizes extracted spatial attention features and fine-grained layer weights to guide the training of the student network model, iteratively distilling the student network model until it converges, where λ is the threshold value. j This represents the weight of the j-th layer out of the n intermediate layers of the selected teacher and student network, where the fine-grained layer weights are set to λ1 < λ2 < ... < λ n ;

[0094] The model saving and application module is used to save the student neural network model for image classification prediction.

[0095] It should be understood that the knowledge distillation-based deep neural network model compression device in the embodiments of the present invention can implement all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above embodiments, which will not be repeated here.

[0096] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the knowledge distillation-based deep neural network model compression method as described above.

[0097] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the knowledge distillation-based deep neural network model compression method described above.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, computer devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations of steps in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1A device for a function specified in one or more processes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

Claims

1. A method for compressing deep neural network models based on knowledge distillation, characterized in that, Includes the following steps: The sample image data is preprocessed, including removing irrelevant information, enhancing the detectability of relevant information, and labeling. The teacher network model is pre-trained based on preprocessed sample image data; Spatial attention features of images to be classified are extracted using a pre-trained teacher network model, including: obtaining the basic features of each channel output of the teacher network model, obtaining spatial correlation features based on the basic features of each channel using a self-attention mechanism, and combining the spatial correlation features with the basic features to obtain the spatial attention features of the teacher network; the same method is used to extract the spatial attention features of the student network. Utilizing extracted spatial attention features, the training of the student network model is guided by fine-grained layer weights. The student network model is iteratively distilled until it converges, where λ is the finite element. j This represents the weight of the j-th layer out of the n intermediate layers of the selected teacher and student network, where the fine-grained layer weights are set to λ1 < λ2 < ... < λ n The iterative distillation student network model includes: Teacher network spatial attention features extracted from teacher network models Student network spatial attention features extracted from student network models Establish distillation terms Constructing the feature classification loss function of the intermediate layer of the network Mean represents the distillation term l for the spatial attention features of the teacher network and the student network. j Sum and then average; Classification loss combined with true labels rls The classification loss of the teacher network model in generating soft labels sls The total loss function of the student network model is obtained as follows: student =l rls +αl sls +βl sad α and β are trade-off parameters; The student network model is trained using this total loss function; The student neural network model is saved for image classification prediction.

2. The method according to claim 1, characterized in that, The basic features of each channel output of the teacher network model include: Extract the original features h of each channel i in the teacher network model i The original features are averaged along the channel dimension to generate an attention map with two-dimensional feature attributes H*W. In the averaged feature space, each pixel represents the averaged information of all channels at the corresponding position. The processing of the teacher network model on each channel is represented as follows: in This represents the feature obtained from the j-th layer of the teacher network model. This indicates channel averaging operation. Let p represent the query vector, key vector, and value vector of the attention map generated by the channel averaging operation, respectively. p is the exponential weight, C is the number of channels, H is the height, and W is the length.

3. The method according to claim 2, characterized in that, Spatial correlation features obtained based on the fundamental characteristics of each channel using a self-attention mechanism include: Using query vectors and key vector Calculate the attention values ​​for the relationships between internal pixels in the spatial feature map, then apply the softmax(·) function to obtain an attention scoring table, and finally use the attention scoring table to evaluate the value vector. The weighted calculation process is as follows: in Scaling factor The spatial correlation features obtained at the j-th layer are denoted as .

4. According to the method described in claim 3, the spatial relevance features combined with the basic features to obtain the teacher's network spatial attention features are represented as follows:

5. The method according to claim 1, characterized in that, True label classification loss rls The calculation method is as follows: in and Let η(·) represent the output of category i in the teacher network and student network before softmax classification, respectively, and let D represent the softmax function. CEL (·) Cross-entropy loss is used to represent the similarity between student and teacher categories, where N represents the total number of categories in the classification task.

6. The method according to claim 1, characterized in that, The classification loss of the teacher network model in generating soft labels sls The calculation method is as follows: in and Let T represent the output of category i in the teacher network and student network before softmax classification, respectively. Let T represent the distillation temperature, N represent the total number of categories in the classification task, η(·) represent the softmax function, and D represent the function... KL (·) Use KL divergence to represent the similarity between student and teacher categories.

7. A deep neural network model compression device based on knowledge distillation, characterized in that, include: The data preprocessing module is used to preprocess the sample image data, including removing irrelevant information, enhancing the detectability of relevant information, and labeling. The teacher network pre-training module is used to pre-train the teacher network model based on preprocessed sample image data. The spatial attention feature extraction module is used to extract teacher spatial attention features of images to be classified using a pre-trained teacher network model. This includes: obtaining the basic features output by each channel of the teacher network model; obtaining spatial correlation features based on the basic features of each channel using a self-attention mechanism; and combining the spatial correlation features with the basic features to obtain the teacher network spatial attention features. The same method is used to extract student network spatial attention features. The student network training module utilizes extracted spatial attention features and fine-grained layer weights to guide the training of the student network model, iteratively distilling the student network model until it converges, where λ is the threshold value. j This represents the weight of the j-th layer out of the n intermediate layers of the selected teacher and student network, where the fine-grained layer weights are set to λ1 < λ2 < ... < λ n The iterative distillation student network model includes: Teacher network spatial attention features extracted from teacher network models Student network spatial attention features extracted from student network models Establish distillation terms Constructing the feature classification loss function of the intermediate layer of the network Mean represents the distillation term l for the spatial attention features of the teacher network and the student network. j Sum and then average; Classification loss combined with true labels rls The classification loss of the teacher network model in generating soft labels sls The total loss function of the student network model is obtained as follows: student =l rls +αl sls +βl sad α and β are trade-off parameters; The student network model is trained using this total loss function; The model saving and application module is used to save the student neural network model for image classification prediction.

8. A computer device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the knowledge distillation-based deep neural network model compression method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep neural network model compression method based on knowledge distillation as described in any one of claims 1-6.

Images