A classification network knowledge distillation method based on auxiliary instance integration

Abstract

The application discloses a kind of classification network knowledge distillation methods based on auxiliary instance integration, comprising the following steps: utilizing deep learning framework to build deployment instance;Utilize multi-stage branch method to construct multi-stage instance group;Based on multi-stage instance group, construct many-to-one network distillation loss function;Model training and optimization are carried out to multi-stage instance group, after training is finished, fixed and saved as multi-stage instance group network model and network model parameter;The optimized multi-stage instance group network model is deployed, and the network model parameter is optimized and cut, to obtain image classification model, and the output of image classification model is image class prediction result.The method can effectively improve the performance of the deployment instance after distillation, reduce the training deployment time and hardware requirements, and is suitable for various time and space limited or higher precision required classification task application scenarios.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a classification network knowledge distillation method based on auxiliary instance integration. BACKGROUND

[0002] Image classification task is a basic task in computer vision, which classifies the input image by sending the image into a neural network for forward inference to obtain a probability prediction vector based on categories. In the case of a determined data set, the traditional training optimization process can achieve good classification performance. When more reliable classification performance is required in the application scenario, a feasible method is to use a larger and more complex network structure, but such a large model requires more time and space resources, such as longer training time, larger memory and display occupation, which will further cause problems such as high training and deployment cost. Therefore, a large amount of research is invested in the field of model optimization.

[0003] Knowledge distillation, as a model compression technology, has received widespread attention in the application and research of deep learning in recent years. Compared with single model training, knowledge distillation refers to a training process, that is, a high-performance large model is prepared in advance, and the learned knowledge is extracted into a small model based on the large model; during the training process, the large model only serves as a teacher for the small model to provide optimization guidance, and the large model itself is no longer trained and optimized; when deployed, only the small model is used. The small model obtained through knowledge distillation is often superior to the general training small model in terms of energy consumption ratio, that is, it achieves better performance under the condition of equal model capacity. At the same time, knowledge distillation can also be regarded as a compression of the large model itself, which reduces the parameter quantity to a smaller scale while maintaining the classification performance of the large model.

[0004] Traditional knowledge distillation tasks are often offline, that is, the training of a large model is needed first, and then the large model is used to distill a small model. The current advanced knowledge distillation method is an online mutual learning knowledge distillation algorithm, which weakens the concept of teacher and student, simultaneously introduces multiple student models during the training phase, and selects the model with the best test set performance as the deployment model after the training is completed. The online knowledge distillation method makes it possible to realize the distillation task in a single stage and end-to-end, but this will increase the burden of the training phase, because it needs to simultaneously carry the training of multiple independent networks, construct the interaction between the networks, calculate the additional interaction loss function, and update the gradient in each network through back propagation. Compared with single-network traditional training, the training time and space occupation are doubled.

[0005] Meanwhile, in the training process of the online knowledge distillation method, the information flow is not strong in the tendency, because the final optimization target is to improve the performance of multiple networks, that is, all networks are symmetrical in structure and formula, while actually the distillation task itself is asymmetrical, only one model will appear in the deployment stage, and the optimization target should be placed on improving the performance of the deployment network as much as possible, and this contradiction limits the final performance of the online distillation to a certain extent. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a classification network knowledge distillation method based on auxiliary instance integration, which uses a multi-stage branch method to construct a multi-stage instance group, and uses a shallow wide auxiliary branch as an auxiliary instance to supplement the structure, which can greatly improve the classification and teaching performance of the auxiliary instance, thereby effectively improving the performance of the deployment instance after distillation, while effectively reducing the training and deployment time and hardware requirements, and is suitable for various time and space limited or higher precision required classification task application scenarios.

[0007] To achieve the above application purpose, the technical scheme provided by the present application is as follows:

[0008] The classification network knowledge distillation method based on auxiliary instance integration provided by the embodiment of the present application comprises the following steps:

[0009] A deployment instance is built using a deep learning framework, and the deployment instance is a residual neural network containing a multi-stage serial network structure;

[0010] A multi-stage instance group is constructed using a multi-stage branch method, the multi-stage instance group includes multiple cascaded sub-instances obtained by deconstructing the serial structure of the deployment instance, and shallow wide auxiliary branches obtained by expanding the branch points between the sub-instances, and the sub-instances and the shallow wide auxiliary branches are combined to form auxiliary instances that only appear in the training period;

[0011] A many-to-one network distillation loss function is constructed based on the multi-stage instance group, and the many-to-one network distillation loss function includes a cross-entropy loss of the deployment instance image category prediction value and the label true value, a cross-entropy loss of the auxiliary instance image category prediction value and the label true value, and a KL divergence of the integrated image category prediction value of the auxiliary instance and the deployment instance image category prediction value;

[0012] The multi-stage instance group is trained and optimized, and based on the many-to-one network distillation loss function, the gradient descent algorithm is used to optimize the parameters on the training data set, and after the training is completed, the multi-stage instance group network model and the network model parameters are fixed and saved;

[0013] The optimized multi-stage instance group network model is deployed, the network model parameters are optimized and cut, and an image classification model is obtained, and the output of the image classification model is the image category prediction result.

[0014] Preferably, the multiple cascaded sub-instances obtained based on the serial structure deconstruction of the deployment instance include:

[0015] Based on the serial structure of deployment instance F, select N branch points and deconstruct them into N+1 independent cascaded sub-instances S. i Thus, the intermediate layer feature f is derived. i That is, for the input image x, we have:

[0016]

[0017] Among them, f i S is the intermediate layer feature output by the i-th branch point. i For the i-th independent child instance, S i Accept input f i-1 and convert it to f i Then f N+1 That is, the output F(x) of the deployment instance F with respect to x, where F(x) is the image category prediction value of the deployment instance.

[0018] Preferably, the shallow-wide auxiliary branch obtained by expanding the branch points between each sub-instance combines the sub-instance with the shallow-wide auxiliary branch to form an auxiliary instance that only appears during the training period, including:

[0019] Based on N branch points, the corresponding shallow and wide auxiliary branch B is designed. i , will the intermediate layer features f i via shallow and wide auxiliary branch B i Transform into auxiliary instance image category prediction value p i Its expression is:

[0020] p i =A i (x)=B i (f i )

[0021] Where, p i For auxiliary example A i Based on the image category prediction given by the input, B will output the prediction. i f i The corresponding shallow-wide auxiliary branch, A i For the auxiliary instance formed by abstraction, it consists of the first i independent sub-instances and B. i Composition, namely:

[0022] A i =S1*S2*…*S i *B i

[0023] wherein * is the cascade of nonlinear functions on the network structure.

[0024] Preferably, the structure of the shallow wide auxiliary branch comprises, in sequence, a local average pooling layer AP1, a bottleneck block BNB1, a local average pooling layer AP2, a bottleneck block BNB2, a local average pooling layer AP2, a bottleneck block BNB3, a global average pooling layer GAP, and a fully connected layer FC.

[0025] Preferably, each bottleneck block contains two branches, wherein one branch comprises a convolution layer CONV0 and a batch normalization layer BN0; the other branch comprises, in sequence, a convolution layer CONV1, a batch normalization layer BN1, an activation layer ReLU1, a convolution layer CONV2, a batch normalization layer BN2, an activation layer ReLU2, a convolution layer CONV3, a batch normalization layer BN3; finally, a fusion layer ADD merges the information of the two branches, followed by an activation layer ReLU3.

[0026] Preferably, the many-to-one network distillation loss function expression is:

[0027]

[0028] wherein x is an input image, y is a label true value in the form of a one-hot code corresponding to x, Φ is an instance ensemble function, L CE (a, b) is the cross-entropy loss of two distributions a and b, L KLD (a, b) is the KL divergence of two distributions a and b, and α and β are hyperparameters for controlling the loss proportion.

[0029] Preferably, the instance ensemble function Φ is:

[0030]

[0031] wherein ω i represents A i (x) is the corresponding weight coefficient in the ensemble, which is learnable.

[0032] Preferably, the model training and optimization on the multi-stage instance group are performed before training, and the learning optimization method of the classification network model is set to use an SGD optimizer, covering all parameters of the multi-stage instance group.

[0033] Preferably, the many-to-one network distillation loss function uses a gradient descent algorithm to optimize parameters, comprising: based on the calculated many-to-one network distillation loss function, using a gradient descent algorithm to obtain a gradient, and using back propagation to update and optimize the parameters of the multi-stage instance group.

[0034] Preferably, the optimization and pruning of the network model parameters comprises: deleting the shallow wide auxiliary branch and branch point in the network topology structure, deleting the parameter key-value pairs that are not needed by the deployment instance in the model parameter table, and re-aggregating the network model serialization result.

[0035] Compared with the prior art, the present application has at least the following beneficial effects:

[0036] (1) The method of the present application expands the shallow wide auxiliary branch through different branch points of the deployment instance, ensures the asymmetry and low coupling between the auxiliary instances, avoids the failure of distillation due to the excessive similarity of the auxiliary instances, and improves the performance of the auxiliary instances and the teaching potential.

[0037] (2) The method of the present application uses a many-to-one network distillation loss function, focuses on the teaching role between the auxiliary instances, makes the final deployment instance tend to learn and the other auxiliary instances tend to transfer their own implicit knowledge, and improves the final performance of the deployment instance through an asymmetric distillation method.

[0038] (3) The deployment instance and each auxiliary instance of the method of the present application share the shallow network parameters of the sub-instance to different extents, which not only alleviates the problem of long training time and large memory occupation of common multi-instance training, but also provides more robust gradient backpropagation for the shared shallow network through multi-branch supervision in the training phase.

[0039] (4) In the final deployment, the network topology structure of the auxiliary instance is removed, the extra key-value pairs in the parameter table are pruned, and the inference volume is compressed, so the present application is suitable for various training deployment time and space limited or higher precision required classification task application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 is a flowchart of the classification network knowledge distillation method based on auxiliary instance integration provided by the embodiments of the present application;

[0042] Figure 2 is a basic structure diagram of the shallow wide auxiliary branch provided by the embodiments of the present application;

[0043] Figure 3 is a flowchart of model training and optimization of the multi-stage instance group provided by the embodiments of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0045] The inventive concept of the present application is that, in order to solve the problems of large space-time burden in the training stage of the online knowledge distillation method in the prior art and the influence of network structure and formula symmetry on the final performance of online distillation, an embodiment of the present application provides a classification network knowledge distillation method based on auxiliary instance integration. By constructing a multi-stage instance group, a shallow wide auxiliary branch is designed by using the serial structure of the deployment instance, a many-to-one network distillation loss function is used for optimization, and finally an image classification model is obtained by compressing the model volume, thereby reducing the burden of the training stage and improving the classification accuracy of the image classification model by using an asymmetric distillation method.

[0046] Figure 1 is a flowchart of the classification network knowledge distillation method based on auxiliary instance integration provided by the embodiment of the present application. As shown in Figure 1 , the embodiment provides a classification network knowledge distillation method based on auxiliary instance integration, which includes the following steps:

[0047] S1, a deployment instance is built by using a deep learning framework. The deployment instance is a residual neural network with a multi-stage serial network structure.

[0048] In the embodiment, the deployment instance is built by using a deep learning framework PyTorch, and the deployment instance ResNet-20 is a network structure with a three-stage serial main body.

[0049] S2, a multi-stage instance group is constructed by using a multi-stage branch method.

[0050] Specifically, the multi-stage instance group includes a plurality of cascaded sub-instances obtained by deconstructing the serial structure of the deployment instance, and a shallow wide auxiliary branch obtained by expanding the branch points between the sub-instances. The sub-instances and the shallow wide auxiliary branch are combined to form auxiliary instances that only appear in the training period.

[0051] The plurality of cascaded sub-instances obtained by deconstructing the serial structure of the deployment instance includes: based on the serial structure of the deployment instance F, selecting N branch points, and deconstructing them into N+1 independent cascaded sub-instances S i , thereby deriving the intermediate layer features f i , that is, for an input image x, there is:

[0052]

[0053] , wherein f i is the intermediate layer feature output by the i-th branch point, and Si For the i-th independent child instance, S i Accept input f i-1 and convert it to f i Then f N+1 That is, the output F(x) of the deployment instance F with respect to x, where F(x) is the image category prediction value of the deployment instance.

[0054] The shallow-width auxiliary branches, derived from the branch points between each sub-instance, are combined with the sub-instances to form auxiliary instances that only appear during the training period. These include: shallow-width auxiliary branches B designed based on the expansion of N branch points. i , will the intermediate layer features f i via shallow and wide auxiliary branch B i Transform into auxiliary instance image category prediction value p i Its expression is:

[0055] p i =A i (x)=B i (f i )

[0056] Where, p i For auxiliary example A i Based on the image category prediction given by the input, B will output the prediction. i f i The corresponding shallow-wide auxiliary branch, A i For the auxiliary instance formed by abstraction, it consists of the first i independent sub-instances and B. i Composition, namely:

[0057] A i =S1*S2*…*S i *B i

[0058] Where * represents the cascade of nonlinear functions in the network structure.

[0059] The structure of the shallow-wide auxiliary branch includes a local average pooling layer AP1, a bottleneck block BNB1, a local average pooling layer AP2, a bottleneck block BNB2, a local average pooling layer AP2, a bottleneck block BNB3, a global average pooling layer GAP, and a fully connected layer FC, which are connected in sequence.

[0060] Each bottleneck block in the shallow-wide auxiliary branch contains two branches, one of which includes a convolution layer CONV0, a batch normalization layer BN0; the other branch includes a convolution layer CONV1, a batch normalization layer BN1, an activation layer ReLU1, a convolution layer CONV2, a batch normalization layer BN2, an activation layer ReLU2, a convolution layer CONV3, a batch normalization layer BN3 in turn; finally, a fusion layer ADD merges the information of the two branches, followed by an activation layer ReLU3.

[0061] In the embodiment, based on the three-stage serial structure of the deployment instance F (ResNet-20), it is decomposed into three independent cascaded sub-instances S1, S2 and S3, a first branch point is established between the sub-instance S1 and the sub-instance S2, a second branch point is established between the sub-instance S2 and the sub-instance S3, the intermediate layer features f1 and f2 are derived from the first branch point and the second branch point respectively, f1 and f2 are in the form of feature maps, wherein the size of f1 is 128×16×32×32, and the size of f2 is 128×32×16×16, a shallow-wide auxiliary branch B1 and a shallow-wide auxiliary branch B2 are added after f1 and f2 respectively, the sub-instance S1 and the shallow-wide auxiliary branch B1 constitute an auxiliary instance A1, the sub-instance S1, the sub-instance S2 and the shallow-wide auxiliary branch B2 constitute an auxiliary instance A2, thereby obtaining a multi-stage instance group containing the deployment instance F and the auxiliary instances A1 and A2, the deployment instance F and the auxiliary instance A1 share the shallow network of the sub-instance S1, and the deployment instance F and the auxiliary instance A2 share the shallow network of the sub-instances S1 and S2.

[0062] The basic structure of the shallow-wide auxiliary branch B1 and the shallow-wide auxiliary branch B2 is designed according to the channel width of f1 and f2 respectively as shown in the left side of Figure 2 wherein each local average pooling layer performs a 2×2 average pooling operation, the output channel width ratio of the intermediate layer feature input to the bottleneck block BNB1, the bottleneck block BNB2 and the bottleneck block BNB3 is 1:4:8:4, and the feature map size (area) ratio is 64:16:4:1. The structure of each bottleneck block is as shown in the right side of Figure 2 wherein the convolution layer CONV0, the convolution layer CONV1 and the convolution layer CONV3 are 1×1 convolution, the convolution layer CONV2 is 3×3 convolution, the output width ratio of the convolution layer CONV0, the convolution layer CONV1, the convolution layer CONV2 and the convolution layer CONV3 is 4:1:1:4, all batch normalization layers perform batch normalization operation, all activation layers perform ReLU activation, and the fusion layer ADD realizes element-wise addition operation of the same size feature maps. Figure 2 The feature map size after the operation of the specific layer is marked, and the unmarked indicates that the operation does not affect the feature map size.

[0063] S3, based on the multi-stage instance group, a many-to-one network distillation loss function is constructed.

[0064] Specifically, the many-to-one network distillation loss function includes a cross-entropy loss of the deployment instance image class prediction value and the label true value, a cross-entropy loss of the auxiliary instance image class prediction value and the label true value, and a KL divergence of the auxiliary instance integrated image class prediction value and the deployment instance image class prediction value.

[0065] The expression of the many-to-one network distillation loss function is:

[0066]

[0067] Wherein, x is an input image, y is a label true value in the form of one-hot code corresponding to x, Φ is an instance integration function, L CE (a, b) is a cross-entropy loss of two distributions a and b, L KLD (a, b) is a KL divergence of two distributions a and b, and α and β are hyperparameters for controlling the loss proportion.

[0068] The instance integration function Φ is:

[0069]

[0070] Wherein, ω i represents A i (x) corresponding weight coefficient in integration, which is learnable.

[0071] In the embodiment, cross-entropy with the true value label y is added to each auxiliary instance A1 and A2 in the multi-stage instance group to ensure the basic performance of each auxiliary instance prediction. At the same time, the two auxiliary instances A1 and A2 are integrated into an auxiliary instance The KL divergence is used to supervise F. According to the formula of the above instance integration function Φ, ω i should be a learnable parameter, so the initial values ω1 and ω2 are both set to 0.5, ω1 and ω2 are adaptively learned in the subsequent model training process, the learning rate is 0.001, at the same time, through the numerical truncation and smoothing method, the values of ω1 and ω2 are controlled to be located between 0.375 and 0.625, i.e. 75% to 125% of the initial value, to prevent the collapse of the instance integration function.

[0072] An optional implementation form of the instance integration function Φ is:

[0073]

[0074] The many-to-one network distillation loss function is:

[0075]

[0076] Wherein, the parameters α = 1 and β = 2.

[0077] S4, model training and optimization are performed on the multi-stage instance group.

[0078] Specifically, on the training data set, the parameters are optimized based on the many-to-one network distillation loss function using the gradient descent algorithm, and after the training is completed, they are fixed and saved as the multi-stage instance group network model and network model parameters.

[0079] In the embodiment, the model training and optimization process is as shown in Figure 3 Before training, the learning optimization method of the classification network model is set to use the SGD optimizer, covering all parameters of the multi-stage instance group. The initial learning rate of the SGD optimizer is set to 0.1, the momentum is 0.99, and the weight decay is 5x10 -4 The maximum number of full data set training rounds is 200, and the learning rate is decayed to one-tenth of the previous value at 100, 150, and 180 rounds, respectively. The data batch size is set to 128, and the single image size is 32x32x3.

[0080] During training, the CIFAR-100 data set is used, and the data is forwarded through the network to obtain the outputs of the auxiliary instances A1, A2, the integrated auxiliary instance and the output of the deployment instance F. Based on the calculated many-to-one network distillation loss function, the gradient is obtained using the gradient descent algorithm, and the parameters of the multi-stage instance group are updated using backpropagation. After each training round, the validation loss is calculated on the validation set, and the forward propagation and backpropagation are repeated until the model converges. After 200 rounds of training, all parameters of the multi-stage instance group are frozen, and the batch normalization layer is converted from training mode to test mode.

[0081] S5, the optimized multi-stage instance group network model is deployed, the network model parameters are optimized and cut, and the image classification model is obtained. The output of the image classification model is the image class prediction result.

[0082] Specifically, the optimization and cutting of the network model parameters include:

[0083] In the network topology structure, the shallow wide auxiliary branch and branch point are deleted, i.e. the network topology structure is reconstructed, all shallow wide auxiliary branches and branch points in the previous step are deleted, only the original information flow of the deployment instance is left, and the end-to-end nature of network inference is ensured;

[0084] In the model parameter table, the parameter key-value pairs that are not needed by the deployment instance are deleted, i.e. the parameter table is traversed, redundant parameter key-value pairs are deleted, only the parameter group needed by the deployment instance is left, and the parameters are rearranged in order;

[0085] The re-aggregated network model serialization result, that is, the re-combination of the cropped network topology and the parameter key-value pair, generates the network serialized model file, and ensures the feasibility of inference after deserialization loading and the correctness of the result.

[0086] The above detailed description of the specific embodiments of the present application has described the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A knowledge distillation method for classification networks based on auxiliary instance ensemble, characterized in that, Includes the following steps: A deployment instance is built using a deep learning framework, wherein the deployment instance is a residual neural network containing a multi-stage serial network structure; A multi-stage instance group is constructed using a multi-stage branching method. This multi-stage instance group includes multiple cascaded sub-instances obtained from the deconstruction of the serial structure of deployed instances, and shallow-wide auxiliary branches expanded from the branch points between each sub-instance. Sub-instances and shallow-wide auxiliary branches are combined to form auxiliary instances that appear only during the training period. The structure of the shallow-wide auxiliary branch includes a locally average pooling layer AP1, a bottleneck block BNB1, a locally average pooling layer AP2, a bottleneck block BNB2, and a locally average pooling layer AP2, and a bottleneck block BNB1 connected sequentially.

3. A global average pooling layer (GAP) and a fully connected layer (FC); each bottleneck block contains two branches. One branch includes a convolutional layer (CONV0) and a batch normalization layer (BN0). The other branch includes, in sequence, a convolutional layer (CONV1), a batch normalization layer (BN1), an activation layer (ReLU1), a convolutional layer (CONV2), a batch normalization layer (BN2), an activation layer (ReLU2), a convolutional layer (CONV3), and a batch normalization layer (BN3). Finally, a fusion layer (ADD) merges the information from the two branches and is followed by the activation layer (ReLU3). A many-to-one network distillation loss function is constructed based on a multi-stage instance group. The many-to-one network distillation loss function includes the cross-entropy loss between the image class prediction value and the label ground truth value of the deployment instance, the cross-entropy loss between the image class prediction value and the label ground truth value of the auxiliary instance, and the KL divergence between the image class prediction value of the auxiliary instance integration and the image class prediction value of the deployment instance. The model is trained and optimized for a multi-stage instance group. On the training dataset, the parameters are optimized using the gradient descent algorithm based on the many-to-one network distillation loss function. After training, the parameters are fixed and saved as a multi-stage instance group network model and network model parameters. The optimized multi-stage instance cluster network model is deployed, and the network model parameters are optimized and pruned to obtain the image classification model. The output of the image classification model is the image category prediction result.

2. The classification network knowledge distillation method based on auxiliary instance integration according to claim 1, characterized in that, The multiple cascaded sub-instances obtained from the serial structure deconstruction of the deployment instance include: Based on the serial structure of deployment instance F, select N branch points and deconstruct them into N+1 independent cascaded sub-instances S. i Thus, the intermediate layer features f are derived. i That is, for the input image x, we have: Among them, f i S is the intermediate layer feature output by the i-th branch point. i For the i-th independent child instance, S i Accept input f i-1 and convert it to f i Then f N+1 That is, the output F(x) of the deployment instance F with respect to x, where F(x) is the image category prediction value of the deployment instance.

3. The classification network knowledge distillation method based on auxiliary instance integration according to claim 2, characterized in that, The shallow-wide auxiliary branch, obtained by expanding the branch points between each sub-instance, combines the sub-instance with the shallow-wide auxiliary branch to form an auxiliary instance that appears only during the training period, including: Based on N branch points, the corresponding shallow and wide auxiliary branch B is designed. i , will the intermediate layer features f i via shallow and wide auxiliary branch B i Transform into auxiliary instance image category prediction value p i Its expression is: p i =A i (x)=B i (f i ) Where, p i For auxiliary example A i Based on the image category prediction given by the input, B will output the prediction. i f i The corresponding shallow-wide auxiliary branch, A i For the auxiliary instance formed by abstraction, it consists of the first i independent sub-instances and B. i Composition, namely: A i =S1*S2*···*S i *B i Where * represents the cascade of nonlinear functions in the network structure.

4. The classification network knowledge distillation method based on auxiliary instance integration according to claim 3, characterized in that, The expression for the many-to-one network distillation loss function is: Where x is the input image, y is the true value of the label in one-hot code form corresponding to x, Φ is the instance ensemble function, and L CE (a,b) represents the cross-entropy loss between two distributions a and b, L KLD (a,b) represents the KL divergence of two distributions a and b, where α and β are hyperparameters controlling the loss ratio, respectively.

5. The classification network knowledge distillation method based on auxiliary instance integration according to claim 4, characterized in that, The instance integration function Φ is: Where, ω i A represents i (x) represents the weight coefficients in the ensemble, which are learnable.

6. The classification network knowledge distillation method based on auxiliary instance ensemble according to claim 1, characterized in that, The model training and optimization for the multi-stage instance group involves setting the SGD optimizer as the learning optimization method for the classification network model before training, which covers all parameters of the multi-stage instance group.

7. The classification network knowledge distillation method based on auxiliary instance integration according to claim 4, characterized in that, The many-to-one network distillation loss function optimizes parameters using the gradient descent algorithm, including: calculating the many-to-one network distillation loss function, obtaining the gradient using the gradient descent algorithm, and updating and optimizing the parameters of the multi-stage instance group using backpropagation.

8. The classification network knowledge distillation method based on auxiliary instance ensemble according to claim 1, characterized in that, The optimization and pruning of network model parameters includes: removing shallow wide auxiliary branches and branch points from the network topology, removing parameter key-value pairs that are not needed for deployment instances from the model parameter table, and re-aggregating the network model serialization results.

Images