An optimization method for deep learning networks suitable for classification tasks

Abstract

The application designs an optimization method suitable for a deep learning network for a classification task. Existing deep network optimization methods mainly use a large network as a teacher network to migrate knowledge to a student network. However, the Soft target is the Softmax output of the teacher network, and the Hard target is the dataset label, which is not suitable for a distributed student network. The application retains the basic framework of Teacher-Student knowledge distillation, but changes the Soft target to the feature output of the last convolutional layer of the Teacher network, comprehensively evaluates the influence degree of feature points on the final result according to the intra-class and inter-class fluctuations, and then filters and samples the feature points according to the composition of the distributed network. Finally, the spatial fluctuations of the Teacher-Student are counted into the loss function, and the overall training of the network is performed. The application can distribute the deep learning model to an embedded terminal in a distributed manner, and lays a foundation for the landing of deep learning.

Description

Technical Field

[0001] The technical field involved in this invention is knowledge distillation method in deep neural network optimization. Background Technology

[0002] With the continuous development of deep learning networks, various deep learning models are emerging. While bringing breakthroughs to various industries and tasks, the width and depth of deep learning network models are also constantly increasing. Consequently, the number of model parameters and computational cost are constantly increasing. For example, the VGG network, with a depth of 19 layers, has 143.67 Mb of model parameters and 15.62 G of computational cost (FLOPs). Both the model space and computational requirements exceed the capabilities of edge devices. Optimization methods for deep learning networks mainly include compression, quantization, pruning, and knowledge distillation. The framework of knowledge distillation is to use the existing model as a Teacher network and the trained target network as a Student network. Without significantly sacrificing recognition accuracy, the original Teacher network is replaced with a Student network structure, which has lower parameter and computational costs, thereby enabling the deployment of deep learning networks on edge devices.

[0003] The shortcomings of existing technologies are that as networks become increasingly complex, the Student model also becomes more complex, and a single edge device cannot meet its requirements, necessitating a distributed deployment of multiple edge devices. Traditional knowledge distillation Teacher-Student networks only focus on constraints on network parameters, computational load, and accuracy, and are not suitable for distributed Student networks that require balancing computational tasks among devices and robustness when a single device fails. Summary of the Invention

[0004] This invention discloses a method for generating a distributed Student network suitable for edge devices. Existing network optimization methods for classification tasks use the final Softmax output of the Teacher network (whose length is equal to the number of object categories) as the Soft objective for the Student network, and the dataset labels as the Hard objective. The weighted sum of these two losses is then used as the loss function for the Student network. The aim is to obtain a smaller Student network than the Teacher network without significantly sacrificing accuracy, making it suitable for devices with limited computing power. However, existing Student networks have identical learning objectives, which is unsuitable for distributed devices. This invention overcomes the problems of inconsistent device execution time and severe accuracy drops when one or more devices in the distributed network fail, caused by the balancing of device computing power and network computational load in existing distributed Student networks.

[0005] To achieve the above objectives, an optimization method for deep learning networks suitable for classification tasks is adopted. First, this invention moves the soft objective of the Student network to the feature output of the last convolutional layer, which provides more features for the distributed Student network to learn. Second, the soft objective is sampled and allocated according to the device's computing power. Finally, the loss function of the distributed Student network is adjusted for overall network training. The technical solution of this invention retains the framework of traditional knowledge transfer from Teacher to Student networks, but makes changes to the learning objectives and forms.

[0006] An optimization method for deep learning networks suitable for classification tasks includes a feature point selection module for the last convolutional layer of the Teacher network, a feature point sampling module, and a training module for the Student network.

[0007] The feature point filtering module output from the final convolutional layer of the Teacher network aims to filter out feature points that contribute significantly to the final result while discarding a large number of redundant feature points. It consists of an inter-class fluctuation calculation unit, an intra-class fluctuation calculation unit, and a decision unit.

[0008] The inter-class fluctuation calculation unit refers to the process of calculating the final convolutional layer output features from images of different object classes in the classification task dataset using a Teacher network. One image is taken from each object class, and the final output features are arranged into a new array according to their output layer index positions. This two-dimensional array has the same number of rows as the final output layer features and the same number of columns as the object classes in the dataset. The row standard deviation of this array is then calculated, resulting in a one-dimensional array with the same number of rows as the final output layer features. This is the first-order inter-class fluctuation array. This process is repeated to obtain another first-order inter-class fluctuation array with the same number of columns as the number of images for each object class. Finally, the maximum value of the row standard deviation and the row expectation of this two-dimensional first-order inter-class fluctuation array is taken to obtain a two-dimensional array with the same number of rows as the final output layer features. This is the second-order inter-class fluctuation array, or the inter-class fluctuation array. The formula is as follows:

[0009]

[0010] Among them O inter σ represents the inter-class fluctuation array output, which is a column vector whose size is equal to the number of rows of the final output features; σ represents the row standard deviation; E represents the row expectation; μ1 is a hyperparameter used to balance the order of magnitude difference between expectation and variance; x represents the final output layer features, with values ​​from 1 to M; y represents the total number of object categories in the dataset, with values ​​from 1 to N; z represents the total number of images of each object in the dataset, with values ​​from 1 to K.

[0011] The intra-class fluctuation calculation unit refers to the calculation of the final convolutional layer output features of images of the same type of objects in the classification task dataset using the Teacher network. The final output layer features of all images of the same type of objects are then arranged into a new array according to the output layer index position. That is, the number of rows in the two-dimensional array is the same as the number of final output layer features, and the number of columns in the array is the same as the total number of images of the same type of objects in the dataset. Then, the row standard deviation of this array is calculated, resulting in a one-dimensional array with the same number of rows as the final output layer features, which is the first intra-class fluctuation array. The above steps are repeated to obtain a first intra-class fluctuation array with the same number of columns as the number of object types. Finally, the weighted sum of the row standard deviation and row expectation of the above two-dimensional first intra-class fluctuation array is taken to obtain a one-dimensional array with the same number of rows as the final output layer features, which is the second intra-class fluctuation array, or the intra-class fluctuation array.

[0012]

[0013] Among them O intra σ represents the output of the intra-class fluctuation array, and its size is a column vector equal to the number of rows of the final output features; σ represents the row standard deviation; E represents the row expectation; μ2 is a hyperparameter used to balance the difference in magnitude between expectation and variance; x represents the final output layer features, with a value of 1 to M; y represents the total number of object categories in the dataset, with a value of 1 to N; z represents the total number of images of each object in the dataset, with a value of 1 to K.

[0014] The decision unit refers to the weighted sum of the inter-class fluctuation array and intra-class fluctuation array obtained above. Its purpose is to determine the position and value of the output of the last convolutional layer in the Teacher network that has the greatest impact on image classification in the dataset. The calculation method is shown in the following formula:

[0015] O f =α*O inter -(1-α)*O intra

[0016] Among them O f For the final output, since the ultimate goal of this invention is to select the comprehensive candidate point with the largest inter-class fluctuation and the smallest intra-class fluctuation, unlike other weighted methods, subtraction is used instead of addition, where α is a hyperparameter.

[0017] The feature point sampling module refers to the module that samples the O points obtained above. f Extract the top T largest values ​​from the sequence and record their corresponding positions. When the Student network is deployed in a distributed manner, the learning target should be sampled according to the computing power of the terminals of different Student networks. When a distributed network is built using homogeneous sampling devices, the sampling method is as shown in the following formula:

[0018] γ s(i，j) =Of ((j-1)*R+i)i=1,2,…,R; j=1,2,…,T / R

[0019] Where γ represents the learning objective of each Student network, which is a one-dimensional sequence, and R is the total number of Student networks. i represents the sub-sequence number, which is the soft objective sequence number of the Student sub-network, and j represents the index of the corresponding position in the sequence. When using heterogeneous devices to build a distributed network, the method is to treat high-computing-power devices as equivalent to N low-computing-power devices, and calculate according to the above formula.

[0020] The Student network training module refers to using the feature map output of the last convolutional layer of the Teacher network as a soft label for training the Student network when using the Teacher-Student network for network parameter compression, resulting in a smaller Student network. The difference between this invention and traditional training methods lies in the network loss function, as shown in the following formula:

[0021]

[0022] The loss function is divided into three parts. For the cross-entropy of traditional deep learning training, l kL (γ s(i) O s(i) To calculate the KL divergence of the feature outputs of the last convolutional layers of two networks, which is used to characterize the correlation between two sequences, but it is insufficient to reflect the spatiotemporal characteristics of the sequences, this invention adds σ(γ) s(i) -O s(i) As a loss auxiliary representation of spatiotemporal characteristics, and adjusted by hyperparameter β, it more effectively reflects the similarity between two sequences and achieves better results. Attached Figure Description

[0023] Figure 1 This is a diagram of the training structure for a distributed Student network.

[0024] Figure 2 This is the process for calculating inter-class fluctuations.

[0025] Figure 3 This is the process for calculating intra-class fluctuations.

[0026] Figure 4 This is a flowchart of the feature point sampling module. Detailed Implementation

[0027] To more clearly illustrate the technical solutions in the implementation of this invention, the invention will be further described and explained below with reference to the accompanying drawings and embodiments, but the content of this invention is not limited to the scope described.

[0028] Example: Figure 1 As shown, an optimization method for deep learning networks suitable for classification tasks is presented, in which VGG19 is used as the Teacher network and VGG7 is used as the backbone of a distributed Student network, wherein the distributed Student network has two homogeneous sub-networks. When two heterogeneous sub-networks are used, assuming that the computing power of the high-performance terminal is three times that of the low-performance terminal, it can be processed as if three homogeneous sub-networks were used, and two of the homogeneous sub-networks are allocated to the high-performance terminal. For the sake of simplicity, this example only uses two homogeneous sub-networks, but the invention is not limited to this case.

[0029] Assumptions: The last convolutional layer of the VGG19Teacher network outputs a row vector with a feature dimension of 512, the last convolutional layer of the VGG7Student network outputs a row vector with a feature dimension of 32, and the dataset is CIFAR10.

[0030] Step 1: Obtain the trained VGG19 model. Input the training set of the CIFAR10 dataset into the network to obtain the features of the final output layer of VGG. Its two-dimensional array has a dimension of 512*50000. Alternatively, a portion of the dataset can be input, in which case the number may be less than 50000.

[0031] Step 2: Rearrange the 512*50000 two-dimensional feature output from Step 1 into a three-dimensional array with dimensions of 512*5000*10. These dimensions represent the final number of output features, the total number of objects in each class, and the number of object types. Calculate the inter-class fluctuation using the following formula: First, calculate the fluctuation value using the standard deviation, resulting in a 512*5000 dimension, representing the final number of output features and the total number of objects in each class. Then, calculate the second fluctuation value using the standard deviation and expectation, resulting in a 512-dimensional vector, representing the final number of output features.

[0032]

[0033] Step 3: Rearrange the 512*50000 two-dimensional feature output from Step 1 into a three-dimensional array with dimensions of 512*10*5000. These dimensions represent the final output feature count, object types, and the total number of objects in each type. Calculate the intra-class fluctuation using the following formula: First, calculate the fluctuation value using the standard deviation, resulting in a 512*10 dimension, representing the final output feature count and object types. Then, calculate the second fluctuation value using the standard deviation and expectation, resulting in a 512-dimensional vector, which represents the final output feature vector.

[0034]

[0035] Step 4: Calculate the results from Steps 2 and 3 using the following formula. Here, α is a hyperparameter that can be adjusted based on experimental results. After calculation, extract the first T maximum values ​​while preserving the spatial position of their feature outputs. In this example, T = 64.

[0036] O f =α*O inter -(1-α)*O intra

[0037] Step 5: Output the final convolutional layer features extracted in Step 4, and then... Figure 4 Sampling is performed as shown in the figure. That is, sampling sequence 1 corresponds to the final output features 1, 3, 5, 7, ..., T / 2-1 after filtering, and sampling sequence 2 corresponds to the final output features 2, 4, 6, 8, ..., T / 2 after filtering, as shown in the following formula. Each sampling sequence corresponds to the soft target of a sub-network in a distributed Student network.

[0038] γ s(i，j) =O f ((j-1)*R+i)i=1,2,…,R; j=1,2,…,T / R

[0039] Step 6: Train the VGG7 Student network using the dataset. The training parameters can be set the same as or different from those of the Teacher network. During the overall training, use the following loss function for backpropagation.

[0040]

Claims

1. An optimization method for deep learning networks suitable for classification tasks, characterized in that, Keeping the existing Teacher-Student network framework unchanged, the soft objective of the distributed Student sub-network is changed to the feature output of the last convolutional layer of the Teacher network. The distributed Student network is then trained by including the feature point selection module, feature point sampling module, and Student network training module of the last convolutional layer of the Teacher network. The Teacher network's final convolutional layer output feature point filtering module aims to filter out the feature point locations that contribute the most to the final classification result. It consists of an inter-class fluctuation calculation unit, an intra-class fluctuation calculation unit, and a decision unit. The inter-class fluctuation calculation unit refers to the unit that calculates the importance of each position of the output feature of the last convolutional layer of the Teacher network in classifying different objects in the dataset. The process is as follows: The Teacher network is used to obtain the final convolutional layer outputs of all training images in the dataset. A three-dimensional array is then constructed using the number of final output features, the total number of objects in each class, and the number of object types as dimensions. The row standard deviation is calculated to obtain a two-dimensional first-order fluctuation array with the number of final output features and the total number of objects in each class as dimensions. Finally, the maximum value between the row standard deviation and the row expectation is calculated to obtain a one-dimensional second-order fluctuation array with the number of final output features as dimensions. This is the inter-class fluctuation array, as shown in the following formula: Among them O inter The output of the inter-class fluctuation array is a column vector whose size is equal to the number of rows of the final output features; σ represents the row standard deviation; E represents the row expectation; μ1 is a hyperparameter used to balance the difference in magnitude between expectation and variance; x represents the final output layer features, with values ​​from 1 to M; y represents the total number of object categories in the dataset, with values ​​from 1 to N; z represents the total number of images of each object in the dataset, with values ​​from 1 to K. The intra-class fluctuation calculation unit refers to the calculation of the importance of each position of the output feature of the last convolutional layer of the Teacher network in classifying the same object in the dataset. The process is as follows: The Teacher network is used to obtain the final convolutional layer outputs of all training images in the dataset. These outputs are then arranged into a three-dimensional array with dimensions of the number of output features, object types, and the total number of objects in each type. The row standard deviation is calculated to obtain a two-dimensional first-order fluctuation array with dimensions of the final output feature count and object type. Finally, the weighted sum of the row standard deviation and the row expectation is calculated to obtain a one-dimensional second-order fluctuation array with dimension equal to the final output feature count. This is the intra-class fluctuation array, as shown in the following formula: Among them O intra σ represents the output of the intra-class fluctuation array, and its size is a column vector equal to the number of rows of the final output features; σ represents the row standard deviation; E represents the row expectation; μ2 is a hyperparameter used to balance the difference in magnitude between expectation and variance; x represents the final output layer features, with a value of 1 to M; y represents the total number of object categories in the dataset, with a value of 1 to N; z represents the total number of images of each object in the dataset, with a value of 1 to K. The decision unit refers to the weighted difference between inter-class volatility and intra-class volatility, as shown in the following formula: The f =α*O inter -(1-a)*O intra Where α is a hyperparameter; The feature point sampling module refers to the process of sampling the feature point sequence output from the last convolutional layer of the filtered Teacher network to generate the soft target sequence of the distributed Student sub-network. The feature point sampling module involves taking the top T largest values ​​from the feature output of the last convolutional layer of the Teacher network after passing them through the decision unit, and then generating the soft target sub-sequence based on the actual distributed Student sub-network, as shown in the following formula: γ s(i,j) =O f ((j-1)*R+i)i=1,2,…,R;j=1,2,…,T / R Where γ is the learning objective of each Student network, which is a one-dimensional sequence, R is the total number of Student networks, i represents the sub-sequence number, which is the soft target sequence number of the Student sub-network, and j represents the index of the corresponding position in the sequence. The result is that the soft target of the Student sub-network is the equally spaced or equally distributed density sampling of the features of the last convolutional layer output by the decision unit. The Student network training module refers to training the Student network while maintaining the same parameters as the Teacher network or changing some parameters. However, in the loss function, while keeping the existing Teacher-Student network loss function, a fluctuation metric for the soft target sequence and the feature output of the last convolutional layer of the Student network is added to help characterize the spatiotemporal loss of the soft target, as shown in the following formula: The loss function is divided into three parts. For the cross-entropy of traditional deep learning training, l KL (γ s(i) O s(i) To calculate the KL divergence of the feature outputs of the last convolutional layers of two networks, which is used to characterize the correlation between the two sequences, but it is insufficient to reflect the spatiotemporal characteristics of the sequences, σ(γ) is added. s(i) -O s(i) ), which serves as an auxiliary representation of the loss of spatiotemporal characteristics, and is adjusted by the hyperparameter β.

Images