Knowledge distillation method and system based on multi-student discussion
A distillation method and student technology, applied in the field of knowledge distillation methods and systems based on multi-student discussion, can solve problems such as the gap of teacher-student network representation ability, and achieve the effect of improving prediction performance, college student diversity, and good generalization ability.
Pending Publication Date: 2022-02-15
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF0 Cites 14 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Due to the addition of the discussion stage, the students can complement each other, which better solves the problem of the performance gap between the single teacher-student network, and finally obtains more accurate classification results.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0053] The present invention will be further described in detail below by specific examples and drawings. The flow chart of the knowledge distillation method based on multi-student discussion is based. figure 1 As shown, it is mainly divided into two phases of the training phase and the test phase.
[0054] The training stage is divided into three phases, and its steps are as follows:
[0055] 1) The first stage pre-training teachers network, select the RESNet32 * 4 network as a teacher network, and the training data set uses the CIFAR100.
[0056] The processing of this step 1) is: use all the data contained in the training set using the CIFAR100 data centralized training set (100 image categories, color maps containing 500 sheets 32 × 32 size) to train the RESNet32 * 4 network to obtain well-trained Teacher network. The initialization of the above-mentioned teacher network uses a strategy of random initialization. Training has passed 240 EPOCH iterations, and the initial learnin...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a knowledge distillation method and system based on multi-student discussion. The method comprises the following steps: 1) selecting a complex network ResNet32x4 as a teacher model of knowledge distillation, and pre-training the teacher model; (2) performing knowledge distillation, using a single-teacher and multi-student distillation mode, initializing and training parameters of multiple small student networks independently, and learning knowledge from the teacher network; (3) enabling the student networks to mutually discuss by means of a discussion module, taking logits output of each student model as input, coupling output of each student network together by adopting a multi-layer convolutional neural network, and outputting final category prediction; and 4) inputting to-be-classified images into the student network, and obtaining a final image classification result through discussion among students. Image classification accuracy is greatly improved, and the situation that the teacher and student model expression ability difference is large in the knowledge distillation field is improved.
Description
Technical field [0001] The present invention belongs to the field of computer vision, and specific relates to a knowledge distillation method and system based on multi-student discussions. Background technique [0002] With the improvement of power and the extensive appearance of large-scale data sets, the depth model has achieved great success, especially on the tasks of image and speech recognition. However, most of the depth learning models contain a large number of parameters, and a wide and wide model need to consume a lot of computing resources during training, and there is still high storage and calculation requirements when deploying models. Therefore, in order to obtain a faster computational speed, the compression of the depth model has become a recent research hotspot. Among them, knowledge distillation is an effective method for model compression, which is intended to compress complex models or model sets to smaller models for deployment. When complex models are well ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06V10/764G06V10/82G06K9/62G06N3/04G06N3/08
CPCG06N3/08G06N3/047G06N3/045G06F18/2415
Inventor 王蕊刘俐君吕飞霄
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI