A method based on knowledge transfer compression network

Abstract

The present application relates to the technical field of deep learning, in particular to a method based on knowledge transfer compression network, comprising the following steps: step 1, artificially designing a lightweight model; step 2, initializing network parameters; step 3, measuring the sensitivity of each layer on the trained network; step 4, fine-tuning the gate parameters; step 5, repeating steps 3 and 4, and finally obtaining a compressed network containing most of the information; step 6, deploying the compressed network to an embedded device; the beneficial effect is that the lightweight network for detecting student emotion recognition is arranged in the embedded device, and the teaching quality can be improved in combination with a small camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a method for compressing a network based on knowledge transfer. BACKGROUND

[0002] With the development of artificial intelligence and Internet of Things technology, more and more emerging technologies are applied to traditional industries. In the field of education, in order to help students better learn and develop, teachers need to better understand the emotional state of students, so as to more accurately judge the learning state of students, which is helpful for the progress and development of students. The technology of student emotion recognition is to analyze the data such as language, behavior and physiological indicators of students through a deep learning network to judge their emotional state.

[0003] In the prior art, for the school teaching scene, the purpose of applying this student emotion recognition is usually to better understand the emotional state of students, so that teachers can better understand the needs and difficulties of students, provide more effective teaching support and guidance, and thus improve the learning performance of students. In this case, the main goal of student emotion recognition is to identify the following emotions: understanding: refers to the understanding degree of students on a certain topic or knowledge point, including the depth, breadth and accuracy of understanding, etc.; confusion: refers to the confusion or doubt of students on a certain topic or knowledge point, including unclear understanding, concept confusion, language barriers, etc.; frustration: refers to the setbacks, difficulties or failures encountered by students in learning, including slow learning progress, poor performance, self-doubt, etc.; interest: refers to the interest and curiosity of students on a certain topic or knowledge point, including learning initiative, enthusiasm and thinking ability, etc.; pleasure: refers to the excitement, joy and love of students in learning a certain topic or knowledge point.

[0004] However, due to the hardware limitations of storage size and computing cost of embedded devices, we need to develop a lightweight model architecture, which transfers knowledge from an old large network model to compress the model by designing redundant modules and migrating the weights and channels of the old network. SUMMARY

[0005] The present application aims to provide a method for compressing a network based on knowledge transfer, to solve the problems raised in the background.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solution: a method for compressing a network based on knowledge transfer, the method comprising the following steps:

[0007] Step 1: manually design a lightweight model;

[0008] Step 2: initialize network parameters;

[0009] Step 3: Measure the sensitivity of each layer on the trained network;

[0010] Step 4: Fine-tune the gate parameters;

[0011] Step 5: Repeat steps 3 and 4 to obtain a compressed network containing most of the information.

[0012] Step 6 deploys the compressed network to the embedded device.

[0013] Preferably, when manually designing a lightweight model, the MobileNet series of networks currently suitable for mobile networks are combined with the SSD network for object detection.

[0014] Preferably, when initializing network parameters, the collected student emotion dataset is used to train the network.

[0015] Preferably, when measuring the sensitivity of each layer on the trained network, new modules, gate parameters, and old channels are added to the least sensitive layer design.

[0016] Preferably, when fine-tuning the gate parameters, the channel is subjected to L1 penalty, the network is fine-tuned, and the network is compressed.

[0017] Compared with the prior art, the beneficial effects of the present invention are:

[0018] The method proposed in this invention, based on knowledge transfer compression networks, involves manually designing a lightweight model for student emotion recognition; initializing network parameters and training the network using a corresponding dataset; calculating the sensitivity of each layer; adding new modules, gate parameters, and old channels to the least sensitive layer; fine-tuning the gate parameters; first fixing the old channel parameters; applying L1 penalty after a certain number of rounds; fine-tuning the network; and then performing compression. The above two operations are repeated to obtain the final model. Attached Figure Description

[0019] Fig. 1 This is a diagram of the overall network structure designed manually for this invention;

[0020] Fig. 2 Add a schematic diagram to the new module of this invention;

[0021] Fig. 3 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1

[0024] Please see Figs. 1-3 This invention provides a technical solution: a method based on a knowledge transfer compression network, the method comprising the following steps:

[0025] Step 1: Manually design a lightweight model;

[0026] Step 2: Initialize network parameters;

[0027] Step 3: Measure the sensitivity of each layer on the trained network;

[0028] Step 4: Fine-tune the gate parameters;

[0029] Step 5: Repeat steps 3 and 4 to obtain a compressed network containing most of the information.

[0030] Step 6 deploys the compressed network to the embedded device.

[0031] Example 2

[0032] Based on Example 1, the specific implementation is as follows:

[0033] 1. Lightweight model designed manually: The network designed here is an example of combining MobileOne and SSD networks.

[0034] The SSD network primarily uses VGG16 as its backbone, plus four additional convolutional layers. Fig. 1 A).

[0035] The classification and regression prediction processes extract a total of 6 layers from the SSD network structure. Shallow feature maps are used to extract small targets, while deep feature maps are used to extract larger targets. First, we replace the entire backbone network VGG16 with the MobileOne-S0 structure. In MobileOne-S0, stage 7 is an average pooling layer, so we only extract up to stage 6. Additionally, the four extra convolutional layers are replaced with four depthwise separable convolutions. Furthermore, since our model is applied to edge-based object detection tasks, the objects detected by the camera are all large targets. Considering the specific characteristics of this network, in the 6 layers extracted for classification and regression prediction, we removed the 38*38 feature output map and replaced it with a 19*19 map as the first feature output layer. Then, we changed the number of channels in stage 6 of MobileOne-S0 from 512 to 1024. We then sequentially obtained the output feature maps of stage 5, stage 6, and the four extra layers of MobileOne-S0. Fig. 1 B) 2. Initialize network parameters. The network example in step 1 is an object detection network, and the dataset is a dataset for student emotion recognition, collecting five emotions: understanding, confusion, frustration, interest, and pleasure. Train the network using the dataset with the following parameters: batch size of 32, weight decay coefficient of 0.0001, and momentum of stochastic gradient descent set to 0.9. Set the initial learning rate to 0.01, and use cosine annealing to adjust the learning rate. The training iterations are 700 epochs.

[0036] 3. Measure the sensitivity of each layer on the trained network. The sensitivity is set by calculating the sum of the absolute values ​​of the parameters of each channel in a single layer, Xj, and then setting all the parameters of the smallest Xj channel to 0 (here, 10% of the total number of channels in each layer are set to 0), and testing the accuracy of each layer.

[0037]

[0038] Where j represents a channel in this convolutional layer, taking a 3*3 layer as an example.

[0039] After the sensitivity calculation is completed, add new modules, gate parameters, and old channels to the least sensitive layer design;

[0040] Here, CONV1 represents the layer with the lowest sensitivity. We need to merge these two layers into a new layer CONVx, where γ is the gate parameter. The network output is...

[0041] O=(1-γ)*O o +γ*O n

[0042] Where O represents the final output, Oo represents the output of the old 2-layer convolutional layer, and On represents the output of the new convolutional layer. The weights of the original CONV1 channels are then added to the new convolutional layer.

[0043] 4. First, fix the weights of the original CONV1 channel and gradually change the gate parameter γ. Since the γ parameter ultimately needs to be gradually changed from 0 to 1, the value of γ can be adjusted according to the number of fine-tuning rounds. After a certain number of fine-tuning rounds, apply L1 regularization penalty to the added channel parameters, gradually setting their parameters to 0, ultimately transforming it into a new layer.

[0044] 5. Repeat steps 3 and 4 to obtain a compressed network containing most of the information.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method of compressing a network based on knowledge transfer, characterized by: The method comprises the following steps: Step 1: Artificially design a lightweight model; the lightweight model is combined by MobileNet series network suitable for mobile network and SSD network for target detection, and is used for student emotion recognition; Step 2: Initialize network parameters and train by using student emotion data set; Training parameters: the batch size of data reading is 32, the weight attenuation coefficient is 0.0001, the momentum setting of random gradient descent is 0.9; The initial learning rate is set to 0.01, and the cosine annealing is used to adjust the learning rate; The training iteration round is 700 rounds; Step 3: Measure the sensitivity of each layer on the trained network; after the sensitivity calculation is completed, a new module, a gate parameter and an old channel are designed in the least sensitive layer; The gate parameter is used to dynamically balance the weight of the output of the old convolution layer and the output of the new convolution layer, and the output result is: ； Gamma is a gate parameter, O represents the final output, Oo represents the output of the old 2 layers, and On represents the output of the new convolution layer; Step 4: Fine-tune the gate parameter; fix the old channel parameter first, then perform L1 penalty after a certain number of rounds, fine-tune the network, and compress; Step 5: Repeat step 3 and step 4 to finally obtain a compressed network containing most of the information; Step 6: Deploy the compressed network to an embedded device.

Images