Self-distillation training method and scalable dynamic prediction method of convolutional neural network

Abstract

According to the self-distillation training method of the convolutional neural network, the performance of the convolutional neural network is significantly enhanced by reducing the size of the convolutional neural network rather than expanding the size of the network. When knowledge is distilled in the network, the network is firstly divided into several parts; knowledge in the deeper portion ofthe network is then squeezed into the shallow portion. Under the condition that the response time is not used as the cost, the performance of the convolutional neural network is greatly improved through self-distillation, and the average precision improvement of 2.65% is obtained; a precision increase of 0.61% in the dataset ResNeXt is used as a minimum value to a precision increase of 4.07% in the VGG19 is used as a maximum value. According to the method, a convolutional neural network with multiple outputs can be regarded as a plurality of convolutional neural networks, and the output of each shallow classifier is utilized according to different requirements by matching with intensified extraction of features of the shallow classifier by an attention layer, so that the precision of the shallow classifier is remarkably improved.

Description

technical field [0001] The invention relates to the training of a convolutional neural network, in particular to a self-distillation training method and a scalable dynamic prediction method for a convolutional neural network. Background technique [0002] Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the range of applications to some areas where accuracy is critical, researchers have been studying methods to improve accuracy through deeper or wider network structures, which will bring exponential growth in computing and storage costs, thus will delay the response time. [0003] Applications such as image classification, object detection, and semantic segmentation are currently developing at an unprecedented rate with the help of convolutional neural networks. However, in some applications that require fault tolerance, such as autonomous driving and medical image analysis, further improvements in prediction and ...

AI Technical Summary

Problems solved by technology

Another problem is how to design and train an appropriate teacher model. Existing distillation frameworks require a lot of effort and experimentation to find the best architecture for the teacher model, which takes a relatively long time.

The third problem is that the teacher model and the student model work in their own way, and the knowledge transfer flows between different models, which involves the establishment of multiple models, which is cumbersome and has low accuracy.

[0005] In the prior art, efficient training is carried out through the self-distillation training method proposed, but the accuracy of the classifier is low during the self-distillation process, and its own function cannot be automatically separated, which affects the function of the classifier, thereby reducing the accuracy of the training method

[0006] At the same time, the neural network has incomparable advantages in dealing with nonlinear problems, and the predictive control is very pertinent to the problem of card-edge operation with constraints. Their respective advantages provide a good solution to the control of nonlinear, time-varying, strong constraints, and large lag industrial processes. Therefore, convolutional neural networks are widely used in the field of prediction; in the prior art, convolutional neural networks are based on The prediction needs to consider its response speed and the confidence of the prediction results. Therefore, for the prediction requirements of different needs, the algorithms of multiple models will be stored at the same time. According to the requirements of different response speeds and accuracy rates, different models will be replaced. A vacuum period is formed during the switching process, which brings security risks to practical applications

Images