A knowledge distillation method and system based on structural feature knowledge

Abstract

The application discloses a kind of knowledge distillation method and system based on structural feature knowledge, belong to model compression technical field, including: S1, student network and teacher network are built;Wherein, teacher network is loaded with trained model, is full-precision neural network;Student network is binary neural network;S2, the second training sample set that is collected in advance is respectively input to student network and teacher network, by simultaneously minimizing the difference between the intermediate layer features of student network and teacher network, and the difference between the relationship features in student network and teacher network, student network and teacher network are trained, to realize the model compression of teacher network. Through the above process, the present application can reduce the huge difference between the intermediate structure of teacher and student network, to solve the problem of knowledge transfer limited due to distribution difference and the overfitting problem of student network, the accuracy of neural network model compression is higher.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of model compression, and more particularly to a knowledge distillation method and system based on structural feature knowledge. BACKGROUND

[0002] In recent years, with the advent of the big data era and the rapid development of hardware, the performance of deep neural networks has been greatly improved. In the fields of image recognition, speech recognition, and natural language processing, etc., deep neural networks have shown outstanding performance, far exceeding other traditional machine learning methods, and even in some tasks, their accuracy far exceeds that of humans. The reason why deep learning can achieve such amazing results is that on the one hand, the deeper layers and larger scale parameters enable neural networks to have stronger fitting ability, and on the other hand, the rapid development of hardware also provides the basic conditions for the rapid training of neural networks. In addition, large-scale data sets are also the key to the rapid development of deep learning, because they enable the model to have a larger learning space and stronger generalization ability.

[0003] With the continuous improvement of the performance of neural networks, people's demand for the use of deep neural network models in life is also increasing, involving transportation, home, entertainment, medical treatment, and military fields, such as autonomous driving, smart home, medical auxiliary diagnosis system, face recognition of camera, etc. image recognition software of smart phones, etc. Therefore, in many application engineering, it is necessary to deploy large-scale neural network models on embedded or mobile terminal devices. However, due to the limitations of storage and computing power of these edge devices and battery power, it is difficult for existing neural network models to be deployed and applied on them. The main problems faced by the deployment of neural network models on these resource-constrained edge devices are threefold. On the one hand, the parameter scale of mainstream neural network models is relatively large, and edge devices not only need to have enough storage space to store network models, but also need to have enough memory to store the intermediate results, parameters and weights generated during model inference. On the other hand, the large-scale neural network also means a very complex amount of calculation, and the existing neural network model needs to go through hundreds of millions of calculations each time to get the final result, and generally needs to use a graphics accelerator GPU for acceleration. However, it is impossible to build such a high-performance acceleration device on these small edge devices. This will result in very slow model operation speed, high latency and device cost, making it difficult for people to accept. Finally, the huge amount of calculation and frequent access to memory also brings huge power consumption, which is also a key factor restricting the deployment of neural network models on edge devices.

[0004] One focus of solving the above problems lies in the optimization of neural network model compression, compressing a large-scale network into a lightweight network that can be deployed on a resource-constrained device, so that the performance of the compressed network approximates the original neural network as much as possible. In existing model compression algorithms, knowledge distillation is one of the current research hotspots. However, the existing knowledge distillation method utilizes a single form of teacher network knowledge, which cannot fully transfer knowledge to the student network. Although researchers have proposed different forms of knowledge to improve the distillation effect, for teacher-student networks with large capacity differences, it is difficult to fully utilize the knowledge contained in the teacher network to improve the representation ability of the lightweight network, which limits the knowledge transfer effect and reduces the accuracy of neural network model compression. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a knowledge distillation method and system based on structural feature knowledge to solve the technical problem of low accuracy of neural network model compression caused by the difficulty of fully utilizing the knowledge contained in the teacher network to improve the representation ability of the lightweight network for teacher-student networks with large capacity differences.

[0006] To achieve the above purpose, the present application provides a knowledge distillation method based on structural feature knowledge, comprising the following steps:

[0007] S1, building a student network and a teacher network; wherein the teacher network loads a trained model, which is a full-precision neural network; the student network is a binary neural network;

[0008] S2, inputting the pre-collected second training sample set into the student network and the teacher network respectively, training the student network and the teacher network by simultaneously minimizing the difference between the intermediate layer features of the student network and the teacher network, and the difference between the relationship features of the student network and the teacher network, so as to realize model compression of the teacher network.

[0009] Further preferably, the loss function when minimizing the difference between the intermediate layer features of the student network and the teacher network is:

[0010]

[0011]

[0012]

[0013] Wherein, k belongs to {S, T}, when k is S, it represents the student network, denoted as network S; when k is T, it represents the teacher network, denoted as network T; u T is the intermediate layer feature output by the intermediate layer of the teacher network; u Sintermediate layer features output by the student network for the intermediate layer; 2 denotes the 2-norm; denotes the cross-entropy loss function of the network k; N is the number of the second training samples input in the training process of step S2; y (j) denotes the true label corresponding to the jth second training sample; denotes the output of the network k when the jth second training sample is input; denotes the logits output by the network k.

[0014] Further preferably, the difference between the relationship features in the student network and the teacher network comprises: the difference between the sample relationship features in the student network and the teacher network and / or the difference between the inter-layer relationship features in the student network and the teacher network;

[0015] The loss function when minimizing the difference between the sample relationship features is:

[0016]

[0017] denotes a similarity matrix composed of the similarity between the feature maps obtained after the second training samples in the same batch are input to the pth layer of the teacher network in the training process of step S2; denotes a similarity matrix composed of the similarity between the feature maps obtained after the second training samples in the same batch are input to the qth layer of the student network in the training process of step S2; denotes a set of (p, q) pairs to be learned; b 2 denotes the set denotes the number of (p, q) pairs in the set denotes the square of the F-norm;

[0018] The loss function when minimizing the difference between the inter-layer relationship features is:

[0019]

[0020] N is the number of the second training samples input in the training process of step S2; denotes the cth relationship matrix in the set of relationship matrices between the feature maps obtained by any two layers of the teacher network when the jth second training sample is input; denotes the cth relationship matrix in the set of relationship matrices between the feature maps obtained by any two layers of the student network when the jth second training sample is input; λ c denotes the weight coefficient corresponding to the cth relationship matrix; M is the number of relationship matrices in the set of relationship matrices between the feature maps obtained by any two layers of the teacher network when the jth second training sample is input;

[0021] wherein, The cross-entropy loss function for network k is as follows:

[0022]

[0023]

[0024] k∈{S, T}, when k is S, it represents the student network, denoted as network S; when k is T, it represents the teacher network, denoted as network T; y (j) Let be the true label corresponding to the j-th second training sample; The output of network k is given when the j-th second training sample is input; This refers to the logits output by network k.

[0025] More preferably, the method for obtaining the student network in step S1 includes:

[0026] S11. Initialize the binary neural network using a pre-trained full-precision neural network;

[0027] S12. Input the pre-collected first training sample set into the binary neural network to update its weights, thereby achieving binary quantization training of the binary neural network and obtaining the above-mentioned student network, specifically including:

[0028] During the forward propagation, the following operations are performed in each layer: calculate the binary weights and binary activation values ​​of the binary neural network, and perform a convolution operation on the binary weights and binary activation values ​​to obtain the 1-bit activation value of the next layer; calculate the information entropy of the binary activation values ​​of the two layers connected by the shortcut layer.

[0029] The binary activation values ​​of the two layers connected by the shortcut layer are weighted according to the magnitude of their corresponding information entropy and then propagated forward.

[0030] During backpropagation, different approximation functions of the quantization function are selected according to different training stages to calculate the gradient values ​​of the loss function with respect to the activation values ​​and weights under binary quantization training, so as to update the weights in the binary neural network.

[0031] More preferably, the approximate function of the above quantization function is:

[0032]

[0033] in, NUM represents the total number of iterations; epoch represents the current iteration number; x is the activation value or weight of each layer.

[0034] Further preferably, the knowledge distillation method based on structural feature knowledge further comprises a step S3 performed after the step S2;

[0035] The step S3 comprises: inputting the pre-collected third training sample set into the student network and the teacher network respectively, training the student network by simultaneously minimizing the cross-entropy loss of the output of the student network and the soft label of the output of the teacher network, and the cross-entropy loss of the output of the student network and the real label, so as to realize further compression of the teacher network model; wherein the third training sample set comprises a third training sample and a corresponding real label thereof.

[0036] Further preferably, in the step S3, the cross-entropy loss of the output of the student network and the soft label of the output of the teacher network, and the cross-entropy loss of the output of the student network and the real label are minimized by minimizing the weighted sum result L total , and the student network is trained.

[0037] Further preferably, the weighted sum result L total is:

[0038] L total = λL KD +(1-λ)·L S

[0039]

[0040]

[0041]

[0042] wherein λ is a weight coefficient; L KD is the cross-entropy loss of the soft label output by the student network and the teacher network; L S is the cross-entropy loss of the soft label output by the student network and the real label; K is the number of samples in the third training sample set; p i (u i ,T) is the soft label output by the teacher network when the i-th third training sample is input; u i is the logits output by the teacher network when the i-th third training sample is input; T is a temperature parameter; p i (u i ,T) is the output of the student network when the i-th third training sample is input; z i is the logits output by the student network when the i-th third training sample is input; y i is the real label corresponding to the i-th third training sample.

[0043] In a second aspect, the present application provides a knowledge distillation system based on structural feature knowledge, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the knowledge distillation method based on structural feature knowledge provided in the first aspect of the present application.

[0044] In a third aspect, the present application further provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device where the storage medium is located to perform the knowledge distillation method based on structural feature knowledge provided in the first aspect of the present application.

[0045] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0046] 1. The present application provides a knowledge distillation method based on structural feature knowledge, which uses intermediate feature knowledge and feature relationship knowledge instead of output feature knowledge for online knowledge distillation, thereby narrowing the huge difference between teacher and student networks in the intermediate structure to solve the problem of limited knowledge transfer and overfitting of student networks due to distribution differences. The present application can fully utilize the knowledge contained in the teacher network to improve the representation ability of the lightweight network, and the accuracy of neural network model compression is high.

[0047] 2. The knowledge distillation method based on structural feature knowledge provided by the present application can obtain feedback information from each other's learning state by mutual distillation of teacher and student networks, and adjust and optimize the entire training process to narrow the distance between teacher and student networks, so that the full-precision teacher network is more adaptable to the student network. At the same time, combined with the cross-entropy loss function, the teacher and student networks are guided to train in the correct direction, so that the teacher and student networks reach the optimal state.

[0048] 3. The knowledge distillation method based on structural feature knowledge provided by the present application proposes an information preservation mechanism based on residual units in the binary quantization forward propagation process, which preserves full-precision activation values through shortcut connections. Compared with traditional binary quantization algorithms, the computational capacity of binary networks is significantly increased with only a small amount of additional computation. In the backward propagation process, two gradient approximation functions are designed according to the different characteristics of different training stages of neural networks. Compared with the mainstream straight-through estimator, the present application can greatly reduce the gradient approximation error while preserving the parameter update ability.

[0049] 4. The knowledge distillation method based on structural feature knowledge provided by the application uses offline knowledge distillation based on output features, distills and fine-tunes the student network using real labels combined with the soft labels output by the teacher network, further improves the precision and generalization ability of the student network, stops the training of the teacher network in advance, uses the teacher network in this state as a static network to perform one-way knowledge migration to the student network, and greatly reduces the memory and computing cost. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The flowchart of the knowledge distillation method based on structural feature knowledge provided by the first aspect of the application;

[0051] Figure 2 The knowledge distillation process diagram based on sample relationship features provided by the first aspect of the application;

[0052] Figure 3 The knowledge distillation process diagram based on inter-layer relationship features provided by the first aspect of the application;

[0053] Figure 4 The convolution operation process diagram of the binary neural network provided by the application;

[0054] Figure 5 The internal information expression capability diagram of the binary network based on residual units provided by embodiment 1 of the application;

[0055] Figure 6 The error diagram of the Identity function and the Clip function and the Sign function provided by embodiment 1 of the application; wherein (a) is the function image of Identity and sign(x), and (b) is the function image of Clip and sign(x);

[0056] Figure 7 The error diagram of the two continuous approximation functions and tanh(βx) and the Sign function provided by embodiment 1 of the application; wherein (a) is the function image of and sign(x); (b) is the function image of tanh(βx) and sign(x);

[0057] Figure 8 The knowledge distillation and fine-tuning process diagram based on output features provided by embodiment 2 of the application;

[0058] Figure 9 The whole process flowchart of the knowledge distillation method based on structural feature knowledge provided by embodiment 3 of the application;

[0059] Figure 10The change trend of the KL divergence of the output distribution between the re-distillation stage teacher-student network provided by the present application is shown in the following table. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0061] In a first aspect, the present application provides a knowledge distillation method based on structural feature knowledge, which is a binary quantization method based on structural mutual distillation, as shown in Figure 1 The method comprises the following steps:

[0062] S1, a student network and a teacher network are built; the teacher network is loaded with a trained model, which is a full-precision neural network; the student network is a binary neural network;

[0063] S2, the pre-acquired second training sample set is input into the student network and the teacher network respectively, the student network and the teacher network are trained by simultaneously minimizing the difference between the intermediate layer features of the student network and the teacher network, and the difference between the relationship features of the student network and the teacher network, so as to realize model compression of the teacher network; wherein the second training sample set includes a second training sample and its corresponding true label.

[0064] The present application regards the input and output of the neural network as a problem solving process, then the knowledge distillation is to help the student network obtain better solution results through different forms of feature knowledge in the teacher network. In this process, the output feature knowledge of the teacher network is equivalent to the answer to the problem, the intermediate feature knowledge is equivalent to the intermediate result of the problem, and the relationship feature knowledge is equivalent to the method of the problem. The present application uses intermediate feature knowledge and feature relationship knowledge to replace output feature knowledge for online knowledge distillation. The teacher-student network can get feedback information from each other's learning state, and then adjust and optimize the whole training process.

[0065] The knowledge distillation based on the intermediate feature knowledge utilizes the intermediate feature knowledge of the teacher network to provide the problem solving process for the student network, so that the student network can fully learn the rich knowledge contained in the intermediate hidden layer of the teacher network, and avoid the overfitting problem caused by the capacity difference between the teacher network and the student network. However, in the present application, the teacher network adopts a full-precision complex network, and the student network adopts a binary network with relatively simple structure, and there is a great difference between the intermediate feature distributions of the two networks. Therefore, if the full-precision network without processing and the binary network are directly used as the teacher-student combination for distillation, the difference problem will not only limit the knowledge transfer, but also may cause the overfitting of the student network. In order to solve the problem, the teacher-student network is distilled to each other to narrow the distance between the teacher-student network, so that the full-precision teacher network is more adaptive to the student network; and because the teacher network and the student network are not in the optimal state, the cross-entropy loss function is combined at the same time to guide the teacher-student network to train in the correct direction.

[0066] Specifically, the loss function when minimizing the difference between the intermediate layer features of the student network and the teacher network is:

[0067]

[0068]

[0069]

[0070] Wherein, k belongs to {S, T}, when k is S, it represents the student network, denoted as network S; when k is T, it represents the teacher network, denoted as network T; u T is the intermediate layer feature output by the intermediate layer of the teacher network; u S is the intermediate layer feature output by the intermediate layer of the student network; ||·|| 2 represents the 2-norm; represents the cross-entropy loss function of the network k; N is the number of inputting the second training samples in the training process of step S2; y (j) is the true label corresponding to the jth second training sample; is the output of the network k when the jth second training sample is inputted; is the logits output by the network k.

[0071] Furthermore, knowledge distillation based on intermediate features provides intermediate results for the student network. However, there may be many intermediate results for solving a problem, and the teacher network only provides one of them. Moreover, due to the differences in the structural characteristics of the teacher and student networks, simply having the student network simulate the intermediate results of the teacher network can actually increase the deviation of the final result. Therefore, knowledge distillation based on intermediate features may impose too many restrictions on the student network. To eliminate this limitation, this invention further conducts knowledge distillation based on relational features. This method does not have the student network learn intermediate results, but rather learns the inter-layer relationships of the teacher network, essentially providing the student network with the problem-solving process. Experiments show that the additional knowledge brought by relational features effectively corrects some of the overfitting problems caused by knowledge distillation based on intermediate features.

[0072] Specifically, the differences in relational features between student networks and teacher networks include: differences in sample relational features between student networks and teacher networks and / or differences in inter-layer relational features between student networks and teacher networks;

[0073] Among them, such as Figure 2 As shown, sample relationship features refer to the relationships between different samples within the same batch in the teacher network. For example, if two samples obtain similar activation values ​​after being input into a certain layer of the teacher network, but their activation values ​​differ significantly after being input into other networks or layers due to differences in weight parameters, it is clear that if the performance of the final student network is to approximate that of the teacher network, then the student network should also possess features that produce highly similar activation values ​​for the input; these features are the sample relationship features inherent in the teacher network. Furthermore, similar to the knowledge distillation process based on intermediate feature knowledge, since the teacher network in this invention uses a full-precision complex network and the student network uses a relatively simple binary network, the distribution of intermediate features between the two networks differs significantly. Therefore, if the unprocessed full-precision network and binary network are directly used as the teacher-student combination for distillation, the difference problem will not only limit knowledge transfer, but may also lead to overfitting of the student network. In order to solve this problem, the teacher and student networks are distilled together to narrow the gap between them, so that the full-precision teacher network is more adaptable to the student network. Since neither the teacher network nor the student network is in the optimal state, the cross-entropy loss function is also combined to guide the teacher and student networks to train in the right direction.

[0074] Specifically, the loss function for minimizing the differences between sample relation features is:

[0075]

[0076] The similarity matrix is ​​formed by the pairwise similarity between the feature maps obtained after the second training sample in the same batch is input into the p-th layer of the teacher network during the training process in step S2. The similarity matrix is ​​formed by the pairwise similarity between the feature maps obtained after the second training sample in the same batch is input into the q-th layer of the student network during the training process in step S2. b represents the set of (p,q) pairs to be learned; 2 For set The number of (p,q) pairs; This represents the square of the F-norm.

[0077] Specifically, given an input sample with a batch size of b, using Let represent the activation feature map output by the teacher network T at layer p. Here, c represents the number of output channels, and w and h represent the width and height of the feature map, respectively. Similarly, using... Let S represent the activation feature map output by the student network S at layer q. Here, the number of channels, width, and height of the teacher network can differ from those of the student network. To guide the student network in learning the sample relationship features of the teacher network, this invention defines an L2 normalized loss function to penalize... and The difference in cosine similarity of activation values ​​among different samples. (Based on the teacher network) Taking the calculation process of the student network as an example ( The calculation process and the teacher network The calculation process is the same, so it will not be repeated here. First, use d xy The cosine similarity between samples x and y from the same batch in the teacher network is expressed as follows:

[0078]

[0079] in, and Let x and y represent the feature maps of the p-th sample, respectively. ij y ij Representing feature maps respectively and The value of the i-th element in the j-th channel; c represents the number of channels; n represents the number of pixels in the feature map; the cosine distance d between the two feature maps is calculated using the above formula. xy , then d xy The size of the matrix represents the similarity between the feature maps of samples x and y after they are input into the p-th layer of the teacher network. By performing the above calculation pairwise on all samples in the same batch B, a cosine similarity matrix of size b×b can be obtained. The matrix is ​​a symmetric matrix, and all elements on the diagonal are 1, because its value is the cosine similarity between the sample and itself.

[0080] Furthermore, knowledge distillation based on sample relationship features only focuses on the relationship knowledge between each sample in a certain layer of the network, ignoring the relationship knowledge between different layers of the network, which is also important for representing the spatial structure of the teacher network model. Therefore, preferably, knowledge distillation based on the relationship features between intermediate layers is also needed, achieving knowledge transfer by having the student network learn the FSP matrix generated by the feature maps of different layers in the teacher network. This method requires that the network layers generating the FSP matrix have feature maps of the same size. However, in many network structures, the network size tends to gradually decrease with increasing depth, so FSP imposes too high a size constraint on the teacher and student networks. This invention, based on FSP, re-encodes the feature maps, such as... Figure 3 As shown. Similarly, since the teacher network in this invention uses a full-precision complex network and the student network uses a relatively simple binary network, there is a significant difference in the distribution of intermediate features between the two networks. Therefore, if the unprocessed full-precision network and binary network are directly used as the teacher-student combination for distillation, the difference problem will not only limit knowledge transfer but may also lead to overfitting of the student network. To solve this problem, the teacher and student networks are distilled together to narrow the gap between them, making the full-precision teacher network more adaptable to the student network. Furthermore, since neither the teacher nor the student network is in an optimal state, a cross-entropy loss function is also incorporated to guide the teacher and student networks to train in the correct direction.

[0081] Specifically, the loss function for minimizing the differences between inter-layer relation features is:

[0082]

[0083] N is the number of second training samples input during the training process in step S2; The c-th relation matrix in the set of relation matrices between any two layers of the teacher network when the j-th second training sample is input; λ is the c-th relation matrix in the set of relation matrices between any two layers of the student network when the j-th second training sample is input; c Let be the weight coefficient corresponding to the c-th relation matrix; M is the number of relation matrices in the set of relation matrices between any two layers of the teacher network when the j-th second training sample is input. It should be noted that the size of the set of relation matrices depends only on the input sample set. Specifically, the size of the set of relation matrices between any two layers of the teacher network when the j-th second training sample is input is equal to the size of the set of relation matrices between any two layers of the student network when the j-th second training sample is input.

[0084] Specifically, the relationship matrix in the set of relationship matrices between the feature maps obtained by any two layers in the teacher network when inputting a certain second training sample is calculated in the following manner, where F represents a set of feature maps in a certain layer of the network, w and h represent the width and height of the feature map, respectively, and m and n represent the corresponding channel numbers. The calculation formula of the relationship matrix of the i-th and j-th layers in the figure is as follows:

[0085]

[0086] where W and x represent the weights of the network and the input data, respectively, represents the i-th feature map in the first set of feature maps. This process is to perform dot product operation between the feature maps of the i-th layer and the feature maps of the j-th layer two by two, and the channel number of the former set of feature maps is m and the channel number of the latter set of feature maps is n, so as to obtain an m x n relationship matrix. In addition, the size of the feature map of the general shallow layer network is relatively large, in order to solve the problem of inconsistent sizes, a convolution kernel is introduced here to re-encode the feature map. Assuming that the size of the i-th layer feature map is m i ×h i ×w i , and the size of the j-th layer feature map is m j ×h j ×w j , at this time, because the sizes of the two sets of feature maps are inconsistent, the relationship matrix cannot be calculated, so a convolution kernel of k i ×k j ×m i ×m j is introduced to re-encode the former set of feature maps. Where h i -k i +1=h j , w i -k i +1=w j , and after convolution operation with the convolution kernel, a feature map with the same size as the j-th layer can be obtained. For the right student network in the figure, the same method can be used to obtain the relationship matrix G i′,j′ (x,W) of the i'-th and j'-th layer feature maps, so as to form the set of relationship matrices between the feature maps obtained by any two layers in the student network when inputting a certain second training sample.

[0087] In the loss functions and , represents the cross-entropy loss function of the network k, and is specifically as follows:

[0088]

[0089]

[0090] k e {S, T}, when k is S, it represents a student network, denoted as network S; when k is T, it represents a teacher network, denoted as network T; y (j) is a real label corresponding to the jth second training sample; is the output of network k when the jth second training sample is input; is the logits output by network k.

[0091] Further, in the classic knowledge distillation, the soft label of the teacher network is equivalent to a regularizer for the student network. If binary quantization is directly combined with knowledge distillation, due to the limited expression ability of the binary network, the role will degenerate into noise, and too strong noise will cause the binary network to fall into a local minimum with poor effect. Therefore, directly combining binary quantization and knowledge distillation not only cannot fully play the positive role of knowledge distillation, but also easily leads to performance degradation. In order to avoid this problem and provide a good initial state for subsequent knowledge distillation, the present application first performs pre-quantization before knowledge distillation. First, pre-train the full-precision student network for a certain period of time, and then use the cross-entropy loss function to separately train the pre-trained full-precision student network for binary quantization.

[0092] Specifically, the binary network is a neural network with 1-bit weight and activation value, and the weight w r and the activation value a r can be obtained by quantization as follows:

[0093]

[0094] where a r and w r represent the full-precision activation value and weight respectively, and Sign(a r ) represents a binary quantization function.

[0095] Due to convolution operation and batch normalization, there are full-precision activation values in both training and inference stages. For example, the full-precision activation value of the input of the jth layer of the network is denoted as a Figure 4The convolution operation process is shown, an arbitrary dimension 1bit activation tensor and a 3x3 convolution kernel are convolved, the convolution kernel and the activation tensor at a specific position are XNORed and Bitcounted to replace the full-precision floating-point addition and multiplication operation, and finally an output matrix of a positive integer value in the range of -9 to +9 is obtained. As can be seen from the convolution operation process, compared with the 32bit weight full-precision network, the 1bit binary network can reduce the memory consumption to 1 / 32 of the original. On the other hand, because the activation value is also 1bit during training, the convolution operation process which occupies a large amount of calculation can be calculated using the above convolution operation process, which can save 64 times of operation amount compared with the full-precision floating-point addition and multiplication operation. However, experiments prove that the inference accuracy of the binary network on large-scale data sets is much lower than that of the full-precision network, and one of the important reasons is that the quantization process from 32bit to 1bit sharply reduces the expression capacity of the neural network.

[0096] To solve the above problems, further preferably, in an optional embodiment 1, the obtaining method of the student network obtained in step S1 comprises:

[0097] S11, initializing the binary neural network by using the pre-trained full-precision neural network;

[0098] S12, inputting the pre-acquired first training sample set into the binary neural network to update the weights thereof, so as to realize binary quantization training of the binary neural network, and obtain the student network, specifically comprising:

[0099] In the forward propagation process, the following operations are performed in each layer: calculating the binary weights and binary activation values of the binary neural network, and performing convolution operation on the binary weights and binary activation values to obtain the 1bit activation values of the next layer; calculating the information entropy of the binary activation values of two layers connected by the shortcut layer,

[0100] After weighting the binary activation values of two layers connected by the shortcut layer according to the corresponding information entropy, continue to forward propagate;

[0101] In the backward propagation process, different approximation functions of the quantization function are selected according to different training stages to calculate the gradient value of the loss function with respect to the activation value and the weight under binary quantization training, so as to update the weight in the binary neural network.

[0102] In the forward propagation process, the binary weight and activation value are calculated layer by layer first, then the next layer 1bit activation value is obtained through convolution operation, the information entropy of the activation value of two layers connected by shortcut is calculated, and the information entropy is used for weighted summation to maximize the expression ability of the model. In the back propagation process, the gradient of the loss function with respect to the activation value is calculated first, then the gradient of the loss function with respect to the weight is calculated according to the result, and finally the weight is updated. Here, different approximation functions are selected according to different training stages, which can minimize the gradient error and retain the updating ability of the model parameters in the training process.

[0103] It should be noted that the present application adds a shortcut connection in a convolution block of the binary network, but the shortcut here is not used to change the learning characteristics of the network, but to save the feature tensor output by the last layer of the network, as shown in Figure 5 represents the output tensor of the sign function, represents the output tensor of the 1bit convolution layer, represents the output tensor of the batch normalization, represents the output tensor of the addition of the activation value of the last layer. After the 1bit convolution operation of the last convolution block, the output tensor is obtained by adding the output tensor obtained by the 1bit convolution operation of the next convolution block through the shortcut connection Similar to the analysis in the last section, the information expression ability of is 289 2*12*12*32 ​The information capacity of the final output tensor of each convolutional block is regarded as the information carrying capacity of the convolutional block, and the introduction of the shortcut connection can greatly increase the information carrying capacity of the entire binary network, thereby effectively reducing the information loss caused by discretization. Because the information features extracted by each layer of the binary network are different, and the amount of information actually carried in the feature map is also different, if the full-precision activation tensor obtained by the upper layer is simply added to the result of the convolution operation of the current layer, it may cause the dilution of effective information or increase of error information. In order to solve this problem, the present application introduces information entropy to quantify the information carried by the output tensor of each layer of the shortcut connection, and then weights the two layers connected with each other according to the size of the information entropy. Information entropy is a measure of information size, and information theory states that the size of a piece of information is directly related to its uncertainty. If information is regarded as a change introduced in the process of eliminating the uncertainty of a random event, then the size of the information is closely related to the distribution of the random event, and the smaller the probability of the random event, the larger the amount of information. For the two output tensors of the shortcut connection mentioned above, according to the analysis process of the information expression capacity of the convolution operation, it can be obtained that the value range of the elements is an integer in [-288, 288], because the subsequent convolution operation needs to quantize it to 1 bit through the sign function, so it is more reasonable to use the information entropy of the quantized activation tensor instead of the information entropy of the shortcut direct connection. The information entropy of the activation tensor is:

[0104]

[0105] Where p is the probability of 1 appearing in . Then the activation tensor obtained by combining the shortcut is The calculation formula is:

[0106] Further, in the process of back propagation of the binary network, because the activation value and the weight in the binary network are discretized, the gradient of the discontinuous activation value quantized by the sign function cannot be calculated, so the approximation of the gradient is inevitable, as shown in the following formula:

[0107]

[0108] Where, is the loss function, is the full-precision weight, is the 1-bit weight quantized by the sign function, that is, the approximation function of In the process of back propagation, the gradient of the sign function with respect to the weight is always equal to 0, that is, the According to the chain rule in the above formula, the gradient in the previous layer will always remain 0, that is, the gradient vanishing phenomenon occurs, so that the parameters in the network cannot be updated. In order to solve the above problem, the most common gradient approximation method is to use a straight-through estimator, that is, to use a mapping function to replace the function whose gradient is difficult to calculate, and to use the gradient of the previous layer network back propagation as the parameter to obtain the gradient of the current layer. Some existing functions (such as Identity function, Clip function) can be used as an approximate function of the quantization function, but the gradient error is very large and will be accumulated layer by layer during the back propagation process. It is crucial to obtain correct gradient information during training to avoid instability during training, otherwise the binary network will not obtain the optimal solution. There are two main challenges to solve this problem, one is to minimize the error between the approximate function and the sign function as much as possible, and avoid updating the parameters in the wrong direction during the back propagation process; the other is to retain the parameter update ability during the training process. In order to solve the above problems, the present application proposes a multi-stage approximation function, which designs different approximation functions according to the different training characteristics of the binary network in different training stages to balance the requirements of the binary network for minimizing the gradient error and the parameter update ability.

[0109] Specifically, the approximation function of the quantization function is:

[0110]

[0111] Wherein, NUM represents the total number of iterations; epoch represents the current iteration number; x is the activation value or weight of each layer.

[0112] Specifically, as shown in Figure 6 Error diagram of Identity function and Clip function and Sign function; wherein, figure (a) is the function image of Identity and sign(x), wherein the solid line is the function image of Identity, and the dashed line is the function image of sign(x); figure (b) is the function image of Clip and sign(x), wherein the solid line is the function image of Identity, and the dashed line is the function image of sign(x). As shown in Figure 7 Error diagram of the two continuous approximation functions and tanh(βx) and Sign function; wherein, figure (a) is the function image of and sign(x), wherein the solid curve is the function Figure (a) shows the graphs of the functions tanh(kx) and sign(x), with the dashed line representing the graph of sign(x). Figure (b) shows the graphs of tanh(kx) and sign(x), with the solid curve representing the graph of tanh(kx) and the dashed line representing the graph of sign(x). It can be seen from the figures that within the cutoff range, i.e., between the two points where the sign and identity functions intersect, the error between the tanh(kx) and sign functions is much smaller than that between the identity functions, and it can largely simulate the graph of the sign function. The error is relatively large. Outside the cutoff range, tanh(βx) remains consistent with the Sign function and exhibits characteristics consistent with the Clip function. It can maintain a certain small gap, although this small gap may introduce some noise into the gradient update, it can largely guarantee the ability to update parameters outside the cutoff range. This is also one of the important characteristics required for binary quantization gradient approximation. According to Based on the properties of the tanh(βx) function within and outside its cutoff range, this invention proposes to effectively combine two approximation functions to design an approximation function possessing both properties. Generally, the entire training process of a neural network has the following characteristics: In the early stages of training, the network parameters are just initialized, and almost all parameters are not at their optimal solutions, resulting in poor network performance. Therefore, it is necessary to ensure that as many parameters as possible are updated during this stage. In the later stages of training, most of the network parameters have been sufficiently updated, and the main focus at this stage is on fine-tuning the parameters. Therefore, it is necessary to minimize gradient errors and ensure the correct direction of parameter updates. This invention is based on this characteristic and... An approximation function for the aforementioned quantization function was designed based on the properties of the tanh(βx) function. In the early stages of training, using... As an approximation function, tanh(βx) has a relatively large error compared to the Sign function, but it has corresponding derivative values ​​for parameters in different ranges, avoiding the gradient vanishing problem and ensuring the parameter update capability during this period. As the number of training iterations increases, the approximation function gradually converges to the Sign function, and the gradient error gradually decreases. In the later stages of training, tanh(βx) is used as the approximation function, and due to the presence of the coefficient β, the approximation function is very close to the Sign function, meaning the gradient error is very small. This ensures that the model parameters are stably updated in the correct direction in the later stages of training.

[0113] Specifically, the algorithm for the prequantization described above in Example 1 is shown in Table 1:

[0114] Table 1

[0115]

[0116] Further, due to the mutual migration of the intermediate feature knowledge and the network layer relationship feature knowledge, the binary student network improves the performance while its structural features are more similar to the full-precision teacher network. At this time, the teacher network is more adaptive to the student network than the previous stage, and the similar feature distribution makes the knowledge migration ability of the teacher network to the student network stronger. Because the expression ability of the teacher network is better than that of the student network, after a certain stage of mutual distillation in step S2, the accuracy of the teacher network begins to saturate. In order to further improve the accuracy and generalization ability of the student network, the application further distills and fine-tunes the student network using the real label combined with the soft label output by the teacher network after step S2, that is, the problem solving ability of the student network is further verified and improved through the final correct answer. By using the output feature knowledge to distill the knowledge of the student network, the knowledge contained in the output of the teacher network can be migrated to the student network to the greatest extent, thereby further improving the performance of the binary student network.

[0117] Specifically, in an optional embodiment 2, the knowledge distillation method based on structural feature knowledge described above further comprises a step S3 performed after step S2;

[0118] As shown in the step S3, the pre-acquired third training sample set is input into the student network and the teacher network respectively, and the student network is trained by minimizing the cross-entropy loss of the output of the student network and the soft label output by the teacher network, and the cross-entropy loss of the output of the student network and the real label, so as to realize further compression of the teacher network model; wherein the third training sample set comprises a third training sample and its corresponding real label. Figure 8 Specifically, by minimizing the cross-entropy loss of the output of the student network and the soft label output by the teacher network, and the cross-entropy loss of the output of the student network and the real label, the weighted sum result L total of the cross-entropy loss is obtained, and the student network is trained; wherein the weighted sum result L total of the cross-entropy loss is:

[0119] L total =λL KD +(1-λ)·L S

[0120]

[0121]

[0122]

[0123]

[0124] wherein λ is a weight coefficient, the decay strategy T min ​= 10 -1 , N represents the total number of training, ep represents the current iteration number; when the training is half, it is attenuated to 0, at this time the influence of the distillation loss function disappears completely, from this stage, the real label is used completely to fine-tune the model; L KD is the cross entropy loss of the soft label output by the student network and the teacher network; L S is the cross entropy loss of the soft label output by the student network and the real label; K is the number of samples in the third training sample set; p i (u i , T) is the soft label output by the teacher network when the i th third training sample is input; u i is the logits output by the teacher network when the i th third training sample is input; T is a temperature parameter; p i (u i , T) is the output of the student network when the i th third training sample is input; z i is the logits output by the student network when the i th third training sample is input; y i is the real label corresponding to the i th third training sample.

[0125] It should be noted that the re-distillation process first stops the training of the teacher network, and the state of the teacher network is used as a static network to perform one-way knowledge migration to the student network. In order to make the final characteristics of the student network be able to imitate the output characteristics of the teacher network and further improve its accuracy, first, a large network is trained on the training set as a teacher model, and the performance of the network is as good as possible, and the temperature parameter T is set to a higher temperature t, and the soft label is generated by using the trained teacher network. The second step uses the soft label generated by the teacher network and the real label to train the student network together. When training the student network, T is set to t and 1 in the Softmax layer respectively, wherein the soft label generated by T=t and the soft label generated by the teacher network form a distillation loss function, and the soft label generated by T=1 and the real label form a cross entropy loss function.

[0126] In an optional embodiment 3, the present application is a multi-stage progressive knowledge distillation method based on structural feature knowledge, as shown in Figure 9 L F + L S is a distillation training method based on intermediate feature knowledge and relationship feature knowledge in the teacher network, L Ois a knowledge distillation method based on output feature relationship. The first module is the pre-quantization process of the student network, which allows the initial state of the full-precision student network to be individually binary quantized using the cross-entropy loss function, thereby avoiding the problem that the binary network falls into a local minimum value due to direct use of knowledge distillation; the process uses the binary quantization method based on residual units and multi-stage gradient approximation provided in embodiment 1. The second module first pre-trains the full-precision teacher network, and then uses L F +L S distill each other by allowing the teacher and student networks to learn from each other intermediate features and feature relationships, so as to reduce the difference between the teacher and student networks, thereby making the teacher network more adaptive to the binary student network; the third module fixes the teacher network, and then uses L O loss function to distill the student network, further improving the similarity between the binary network and the teacher network through the migration of output feature knowledge. Specifically, the following steps are included:

[0127] (1) Pre-quantization: pre-train the full-precision student network for a certain stage, and then individually perform binary quantization training based on residual units and multi-stage gradient approximation using the cross-entropy loss function for the pre-trained full-precision student network;

[0128] (2) Online distillation: use intermediate feature knowledge and feature relationship knowledge instead of output feature knowledge for online knowledge distillation, and the teacher and student networks can get feedback information from each other's learning state, and then adjust and optimize the entire training process;

[0129] (3) Re-distillation: stop the training of the teacher network, and use the teacher network as a static network to perform one-way knowledge migration to the student network.

[0130] The entire process is shown in Table 2:

[0131] Table 2

[0132]

[0133]

[0134] In summary, the application provides a binary quantization and knowledge distillation fusion method based on structural mutual distillation. In the pre-quantization process, the full-precision student network is pre-trained, and based on the student network, a binary quantization algorithm based on residual units and multi-stage gradient approximation is used for pre-quantization in a certain stage, thereby avoiding the problem of binary network falling into local minimum caused by directly using knowledge distillation. In the online distillation process, the teacher and student networks learn from each other the intermediate features and feature relationships to reduce the difference between the features of the teacher and student networks, thereby making the teacher network more adaptive to the binary student network. In the re-distillation process, the similarity between the binary network and the teacher network is further improved through the migration of output feature knowledge. The application effectively solves the problems of limited knowledge migration and falling into local minimum when directly distilling the binary network, and further improves the precision of the binary network.

[0135] It should be noted that the training sample sets of the above stages can be the same or different, and are determined by the specific task of the trained model loaded in the teacher network, such as image classification tasks, and other visual tasks such as object detection and image segmentation.

[0136] Further, in the field of model compression, the research object is generally a basic neural network structure, so image classification tasks are used to test the model compression algorithm, and other visual tasks such as object detection and image segmentation are also selected. The test task type selected in the experiments involved in the application is taken as an example. In order to verify the effectiveness of the model compression framework based on binary quantization and knowledge distillation fusion proposed in the application, the application selects VGG and ResNet series network structures to perform experiments on two different sizes of data sets, CIFAR-10 and ImageNet. In the setting of neural network parameters, the application uses the stochastic gradient descent algorithm to update the network parameters, the momentum coefficient is set to 0.9, the weight decay coefficient is 10 -4 , the total number of iterations is 400 epochs, and the batch size is 128. The number of iterations in the pre-quantization stage is 60 epochs, the initial learning rate is η = 0.1, and the learning rate is reduced to 0.1xη every 20 epochs. The number of iterations in the online distillation stage is 200 epochs, the initial learning rate is η = 0.1, and the learning rate is reduced to 0.1 at the 80th and 150th epochs. The teacher network uses the same experimental settings as the student network in this stage. The number of iterations in the re-distillation stage is 140 epochs, the initial learning rate is η = 0.1, and the learning rate is reduced to 0.1 at the 50th and 100th epochs. The temperature parameter T based on the output feature knowledge is set to 3. The experiment is trained on RTX 3090*2 GPU, and the algorithm is implemented based on the pytorch framework.

[0137] In order to intuitively verify the effectiveness of the pre-quantization method (PQ, Pre-Quantization Stage) and the on-line distillation method (OD, On-line distillation stage) in the knowledge distillation method based on structural feature knowledge provided by the present application, the present application records the KL divergence of the output distribution between the full-precision teacher network and the binary network in the redistillation module (Rs, Redistillation stage) after being processed in different ways, wherein the data set is CIFAR-10, the teacher and student networks are ResNet-34 and ResNet-18 respectively, and the change trend of the KL divergence of the output distribution between the teacher and student networks in the redistillation stage is as shown in Figure 10 .

[0138] In the figure, the Baseline does not process the teacher and student networks, and directly performs redistillation; the PQ performs pre-quantization processing on the student network before distillation. The PQ+OD performs on-line distillation processing on the pre-quantized student network together with the full-precision teacher network. As can be seen from Figure 10 , compared with the Baseline, the PQ can significantly reduce the output distribution distance between the teacher and student networks in the whole training stage, that is, the similarity between the teacher and student networks is increased. This shows that the pre-quantization method proposed by the present application effectively solves the problem that the binary network falls into a local minimum value caused by knowledge distillation. The distribution distance of the PQ+OD is similar to that of the PQ at the beginning of the training, and with the progress of the training, the distribution distance of the PQ+OD begins to rapidly decrease, and at the end of the training, it is reduced to below 0.5. Compared with the Baseline and the PQ, the similarity between the teacher and student networks is greatly improved. This shows that the on-line knowledge distillation designed by the present application can reduce the structural distribution difference between the teacher and student networks by migrating the structural feature knowledge of the teacher network to the binary network, so that the teacher network is more adaptable to the binary network, and therefore the binary network can learn more output feature knowledge of the teacher network in the subsequent redistillation module.

[0139] In order to further verify the effectiveness of the PQ and the OD in the whole fusion algorithm, the present application performs multiple combination experiments on the PQ and the OD on the ImageNet data set, the student network uses ResNet-18, and the teacher network uses ResNet-34. The comparison algorithms are BL, AP, PQ+AP and PQ+OD+AP. Among them, BL refers to the basic binary network obtained by using only the binary quantization algorithm; AP directly uses the full-precision teacher network to distill the low-bit network; PQ+AP first performs pre-quantization for a certain stage, and then uses the full-precision teacher network to distill the binary network. PQ+OD+AP means that the pre-quantization module and the on-line distillation module are performed in sequence, and then the full-precision teacher network is used for the distillation process; FP is a full-precision network for reference. The specific experimental results are shown in Table 3.

[0140] Table 3

[0141]

[0142] In Table 3, AP obtains the worst experimental results on Top-1 and Top-5 accuracy; PQ+AP obtains 57.3% and 79.4% results on Top-1 and Top-5 distribution, which has a performance improvement of 0.3% and 0.2% compared with AP, which shows that the pre-quantization stage proposed in the application can effectively improve the performance improvement effect of knowledge distillation on the binary network. After the pre-quantization module, further adding the online distillation module, the binary network obtains better experimental results, and has a large accuracy improvement compared with AP. In the table, Ours uses the complete training process proposed in the application, and the difference between PQ+OD+AP is the last stage, Ours uses the re-distillation module proposed in the application, which increases the distillation loss attenuation strategy compared with the distillation method used by the latter. From the table, it can be seen that Ours obtains the optimal experimental results on Top-1 and Top-5, and has a performance improvement of up to 1% compared with AP on the two experimental results. The experimental results fully show that the binary quantization and knowledge distillation fusion algorithm proposed in the application can effectively solve the problems faced when combining knowledge distillation and binary quantization, and further improve the performance of the binary network.

[0143] In order to verify the overall effectiveness of the binary quantization and knowledge distillation fusion algorithm, this section performs comparative experiments on CIFAR-10, the student network uses ResNet-20, the quantization method uses the binary quantization algorithm based on residual unit and multi-stage gradient approximation proposed in the third chapter, and the teacher network uses ResNet-34. The comparison algorithms are BL, AP and LQ-Net. Among them, AP directly uses the full-precision teacher network to distill the low-bit network, and the student network uses an 8-bit and a 4-bit network, which uses a binary network for distillation training here; LQ-Net proposes to jointly train the network and its quantizer, which optimizes the network quantizer through network training, minimizes the quantization error while adapting to the training target, thereby improving the final accuracy. The specific experimental results are shown in Table 4.

[0144] Table 4

[0145]

[0146] In Table 4, the classification accuracy of the base binary network obtained by using only the binary quantization algorithm is 85.5%, and the AP obtains a classification accuracy of only 84.3%, which is 1.2% lower than BL. The result confirms the proposed hypothesis that due to the limited expression ability of the binary network, the direct combination of knowledge distillation and binary quantization, and the excessive regularization of knowledge distillation, the binary network is easily trapped in a local minimum. The Ours method obtains a performance improvement of 0.9%, which is 0.3% higher than the best result of LQ-Net. Although there is still a certain gap with the full-precision network, the result fully proves that the fusion algorithm of binary quantization and knowledge distillation proposed in the application can well improve the performance of the binary network based on binary quantization.

[0147] In a second aspect, the application provides a knowledge distillation system based on structural feature knowledge, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to execute the knowledge distillation method based on structural feature knowledge provided in the first aspect of the application.

[0148] The related technical solutions are the same as those in the first aspect of the application, and will not be repeated here.

[0149] In a third aspect, the application further provides a computer readable storage medium, which comprises a stored computer program, wherein when the computer program is run by a processor, the computer readable storage medium controls the device where the storage medium is located to execute the knowledge distillation method based on structural feature knowledge provided in the first aspect of the application.

[0150] The related technical solutions are the same as those in the first aspect of the application, and will not be repeated here.

[0151] Those skilled in the art will easily understand that the above description is only a preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A knowledge distillation method based on structural feature knowledge, applied to an image classification task, characterized in that, The method comprises the following steps: S1, a student network and a teacher network are built; the teacher network is loaded with a trained model for an image classification task, and is a full-precision neural network; the student network is a binary neural network; S2, a pre-acquired second training sample set is input into the student network and the teacher network respectively, and the student network and the teacher network are trained by simultaneously minimizing the difference between the intermediate layer features of the student network and the teacher network and the difference between the relationship features of the student network and the teacher network, so as to realize model compression of the teacher network, and to perform image classification according to the compressed model; the second training sample set is an image sample set for an image classification task; The loss function when the difference between the intermediate layer features of the student network and the teacher network is minimized is: in, {S, T}, when When S represents the student network, it is denoted as network S; when When T represents the teacher network, it is denoted as network T; The intermediate layer features output by the intermediate layer of the teacher network; The intermediate layer features output by the intermediate layer of the student network; 2 Represents the 2-norm; Represents network The cross-entropy loss function; The number of second training samples is input during the training process in step S2; For the first j The true label corresponding to each second training sample; For input number j The network during the second training sample The output; For the network The output logits; The difference between the relationship features of the student network and the teacher network includes the difference between the sample relationship features of the student network and the teacher network and the difference between the interlayer relationship features of the student network and the teacher network; The loss function when the difference between the sample relationship features is minimized is: a similarity matrix composed of the similarity degrees between the feature maps obtained after inputting the second training sample under the same batch in the training process of step S2 to the student network p layer to each other; a similarity matrix composed of the similarity degrees between the feature maps obtained after inputting the second training sample under the same batch in the training process of step S2 to the student network q layer to each other; denotes a set of pairs to be learned; denotes the number of pairs in the set denotes the number of pairs in the set denotes the number of pairs in the set denotes the number of pairs in the set denotes the square of the F-norm; The loss function when the difference between the interlayer relationship features is minimized is: For input number j The second training sample is the first in the set of relationship matrices between any two layers of the teacher network. c A relation matrix; For input number j The second training sample is the first in the set of relationship matrices between any two layers of the student network. c A relation matrix; For the first c The weight coefficients corresponding to each relation matrix; M For input number j The number of relation matrices in the set of relation matrices between any two layers of the teacher network when a second training sample is used.

2. The knowledge distillation method based on structural feature knowledge according to claim 1, wherein, The method for obtaining the student network in step S1 comprises: S11, a pre-trained full-precision neural network is used to initialize the binary neural network; S12, a pre-acquired first training sample set is input into the binary neural network to update the weights thereof, so as to realize binary quantization training of the binary neural network and obtain the student network, specifically comprising: In the forward propagation process, the following operations are performed in each layer: the binary weights and binary activation values of the binary neural network are calculated, and the binary weights and binary activation values are subjected to convolution operation to obtain 1bit activation values of the next layer; the information entropy of the binary activation values of two layers connected by a shortcut layer is calculated, and the binary activation values of the two layers connected by the shortcut layer are weighted according to the sizes of the corresponding information entropy and then continue to be propagated forward; In the backward propagation process, different approximation functions of quantization functions are selected according to different training stages to calculate the gradient values of the loss function of the binary quantization training with respect to the activation values and the weights, so as to update the weights in the binary neural network; The first training sample set is an image sample set for an image classification task.

3. The knowledge distillation method based on structural feature knowledge according to claim 2, characterized in that, The approximation function of the quantization function is: wherein, ; denotes the total number of iterations; denotes the current iteration number; are the activation values or weights of the layers. 4.The method of claim 1-3, wherein, Further comprising a step S3 performed after step S2; The step S3 comprises: a pre-acquired third training sample set is input into the student network and the teacher network respectively, and the student network is trained by simultaneously minimizing the cross-entropy loss of the output of the student network and the soft label of the output of the teacher network and the cross-entropy loss of the output of the student network and the real label, so as to realize further compression of the teacher network model; wherein the third training sample set comprises a third training sample and a corresponding real label thereof; The third training sample set is an image sample set for an image classification task.

5. The knowledge distillation method based on structural feature knowledge according to claim 4, characterized in that, In the step S3, a weighted sum result of a cross-entropy loss of a soft label of an output of the student network and an output of the teacher network and a cross-entropy loss of the output of the student network and a real label is minimized The student network is trained.

6. The knowledge distillation method based on structural feature knowledge according to claim 5, characterized in that, a weighted sum of the cross-entropy losses is: wherein, is a weight coefficient; is a cross-entropy loss of soft labels output by the student network and the teacher network; is a cross-entropy loss of soft labels output by the student network and the real label; K is the number of samples in the third training sample set; is a soft label output by the teacher network when the first i third training sample is input; is a soft label output by the teacher network when the second i third training sample is input; is a temperature parameter; is an output of the student network when the first i third training sample is input; is an output of the student network when the second i third training sample is input; is logits output by the student network when the first i third training sample is input; 7. A knowledge distillation system based on structural feature knowledge, characterized in that, Comprise: A memory storing a computer program and a processor, wherein the processor executes the computer program to perform the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls a device where the storage medium is located to perform the method of any one of claims 1-6.

Images