Teacher-student co-evolution method for multi-stage progressive knowledge extraction

Abstract

The embodiment of the invention relates to the technical field of deep learning, in particular to a teacher-student co-evolution method for multi-stage progressive knowledge extraction, which comprises the following steps: constructing a student model S0 based on a teacher model T0, connecting the T0 with a flowing knowledge fusion module psi0, inputting a training sample into the T0 and the S0, performing offline distillation training on the S0, and updating feature fusion parameters of the psi0, obtaining psi 1; s0 is pruned to obtain S1, S0 serves as a new teacher model T1, T1 is connected with psi 1, training samples are input into T1 and S1, T1 is trained, S1 is subjected to online distillation training, feature fusion parameters of psi 1 are updated, and psi 2 is obtained; judging whether the S1 meets a preset parameter requirement or not; if yes, S1 is determined as a target compressed student model; otherwise, carrying out the online distillation training of the next stage until the satisfied Sn is obtained. According to the method, wider stable and efficient transmission of teacher knowledge is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and in particular to a teacher-student collaborative evolution method for multi-stage progressive knowledge extraction. Background Art

[0002] Deep neural networks have achieved remarkable results in computer vision, data mining, and natural language processing. However, as the demand continues to increase, deep neural networks require deeper and wider network structures, which results in a very large number of parameters, limiting their practical deployment on resource-limited devices (such as IoT devices). Knowledge distillation is a popular model compression technique that can solve this problem by compressing the knowledge in a large teacher model into a small student model, thereby enabling the deployment of the network in a resource-constrained environment. However, traditional knowledge distillation techniques still have shortcomings when facing the challenges of dynamically changing student and teacher models.

[0003] Previous studies mainly used hard labels or soft labels as training guidance, usually in the last layer of the model. The latest research tends to use intermediate features as additional supervision, so that the student model can acquire richer knowledge by learning the characteristics of the teacher model. Although new methods have made some progress, these methods will also seriously reduce the distillation performance, resulting in a large gap between teachers and students. Based on this, a research team proposed the idea of ​​progressive distillation, which completes knowledge transfer through multi-stage medium-sized teacher models and gradually reduces the teacher model to achieve more stable knowledge transfer and dynamic compression. However, current research generally ignores the dynamic changes of student models and teacher models in model architecture, which is not in line with actual teaching scenarios. In real teaching scenarios, students and teachers are constantly developing. Not only are students gradually learning knowledge, but teachers are also constantly improving their teaching skills.

[0004] Specifically, the current mainstream distillation methods include the DIST method proposed by Hu et al. to learn softened teacher labels to reduce the difficulty of the student model learning from the strong teacher model, and use the internal and external category similarity to match the relevant knowledge of the student model and the teacher model, or use the SemCKD method to select the use of all teacher feature knowledge to transfer to the student model during the training process, and use the attention mechanism to allocate the weight of the student model to each layer of the teacher model. These methods often do not consider that the ability of the student model is constantly improving during the training process, and the teacher model suitable for learning also needs to be constantly changed, that is, dynamically adjust the learning process of the student model to the teacher model. Some progressive teacher model compression methods propose to use a multi-stage medium-sized teacher model to complete the transfer of knowledge from the teacher model to the student model with a large gap, gradually reduce the teacher model during training to complete a more stable knowledge transfer, dynamically compress the teacher model, and restore the performance in time through distillation after compression. However, these methods often only focus on transferring static teacher logits knowledge, without considering feature knowledge containing richer semantic information. On the other hand, few studies consider dynamically adjusting the knowledge transferred from the teacher model to the student model, that is, the number of feature knowledge that the teacher model should transfer to the student model each time under different stages of teacher-student knowledge transfer. Summary of the invention

[0005] In view of this, an embodiment of the present application proposes a multi-stage progressive knowledge refinement teacher-student co-evolution method, which can integrate a small but precise feature knowledge of the teacher model in the early stage of training, and gradually increase the feature knowledge used for fusion as the size of the teacher model decreases, thereby achieving a smooth and efficient transfer of a wider range of teacher knowledge.

[0006] On the one hand, the embodiment of the present application proposes a multi-stage progressive knowledge extraction teacher-student co-evolution method, including: based on the full-size teacher model T0, initializing and constructing a full-size student model S0, and connecting T0 with the pre-built flow knowledge fusion module Ψ0, inputting the training samples into T0 and S0, and obtaining the teacher fusion attention feature map output by Ψ0 And the student characteristic diagram of S0 output The training samples are marked with the true label y; the final output p based on y and S0 is s0 , and S0 is trained offline for distillation, and based on y and Update the feature fusion parameters of Ψ0 offline to obtain the offline updated flow knowledge fusion module Ψ1; perform convolutional layer channel pruning on S0 that has completed offline distillation training according to the preset pruning ratio to obtain the compressed student model S1, use S0 as the new teacher model T1, and connect T1 with Ψ1, input the training samples into T1 and S1, and obtain the teacher fusion attention feature map output by Ψ1 And the student characteristic map output by S1 The final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and Update the feature fusion parameters of Ψ1 online to obtain the online updated flow knowledge fusion module Ψ2; determine whether S1 that has completed the online distillation training meets the preset parameter requirements; if S1 that has completed the online distillation training meets the preset parameter requirements, then determine S1 that has completed the online distillation training as the student model of the target compression; if S1 that has completed the online distillation training does not meet the preset parameter requirements, then perform convolutional layer channel pruning on S1 that has completed the online distillation training according to the preset pruning ratio to obtain the compressed student model S2, use S1 as the new teacher model T2, and connect T2 with Ψ2 to perform the next stage of online distillation training until S1 that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the S that completes the online distillation training n Determine the student model for target compression; where n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

[0007] Optionally, based on y, the final output p of S0 s0 , and Offline distillation training of S0 is achieved through the following formula:

[0008]

[0009] Among them, Φ(·) is the channel alignment function, B(·) is the size alignment function, N(·) is the L2 regularization function, Dist(·) represents the distance loss after the channel alignment function, size alignment function and L2 regularization function are processed. is the L2 loss term in the offline distillation loss function, L CE (y,p s0 ) is the cross entropy loss term in the offline distillation loss function, α and β are preset offline hyperparameters, L stu-offline is the offline distillation loss function;

[0010] Based on y and The offline update of the feature fusion parameters of Ψ0 is achieved through the following formula:

[0011]

[0012] Among them, Θ(·) represents a fully connected mapping network, L kfk is the offline fusion loss function.

[0013] Optionally, performing convolution layer channel pruning on S0 that has completed offline distillation training according to a preset pruning ratio to obtain a compressed student model S1, including: determining the shape of the weight of the convolution layer of S0 that has completed offline distillation training, and the shape of the weight of the convolution layer of S1 that has completed offline distillation training is expressed as: Among them, W represents the weight of the convolutional layer of S1 after offline distillation training, C i Indicates the number of input channels, C o Indicates the number of output channels, K h and K w represents the kernel size; the weights of the convolutional layer of S1 that has completed offline distillation training are flattened and summed to obtain a reshaped weight vector; based on the reshaped weight vector, the weight values ​​of each output channel are sorted from small to large, and the channel index that needs to be pruned is determined based on the preset pruning ratio; the corresponding output channels of each convolutional layer are pruned according to the channel index that needs to be pruned, and the corresponding input channels of the next layer are pruned.

[0014] Optionally, T k and S k denote the classroom model and student model in the kth round of online distillation training, respectively, k is an integer greater than 0 and less than n, T0, S0, T k and S kThe number of convolutional layers of the teacher model is the same and all are m layers, where m is an integer greater than 2; the flow knowledge fusion module consists of m input units, m-1 fusion units and output units. The m convolutional layers of the teacher model are connected to the m input units of the flow knowledge fusion module respectively, and the m-1 fusion units are connected in sequence. The first two input units are connected to the first fusion unit, the m-1 fusion unit is connected to the output unit, and the j-th input unit is connected to the j-1 fusion unit, where j is an integer greater than 2 and less than m; the input unit and the output Each unit is composed of a 1×1 convolutional layer. The fusion unit is composed of a sequentially connected downsampling layer, a 1×1 convolutional layer, and an element-by-element addition layer. For the first fusion unit, the first input unit is connected to the downsampling layer of the first fusion unit, and the second input unit is connected to the element-by-element addition layer of the first fusion unit. For the j-1th fusion unit, the element-by-element addition layer of the j-2th fusion unit is connected to the downsampling layer of the j-1th fusion unit, and the jth input unit is connected to the element-by-element addition layer of the j-1th fusion unit.

[0015] Optionally, for the kth round of online distillation training, the teacher model T k Fusion module with flow knowledge k Connect, T k The first a layers of Ψ participate in this round of online distillation training, k Output teacher fusion attention feature map It is expressed by the following formula:

[0016]

[0017] in, Indicates T k The output of the i-th convolutional layer.

[0018] Optionally, for the student model S k Online distillation training is performed using the following formula:

[0019]

[0020] in, is the online distillation loss function for the kth round, α ′ and β ′ is the preset online hyperparameter, For S k The output student feature map, is the L2 loss term in the online distillation loss function of the kth round, p sk For S k The final output, L CE (y,p sk ) is the cross entropy loss term in the online distillation loss function of the kth round;

[0021] Online Update k The feature fusion parameters are implemented by the following formula:

[0022]

[0023] in, is the online fusion loss function of the kth round;

[0024] To T k Training is performed using the following formula:

[0025]

[0026] Among them, p tk T k The final output is is the teacher loss function of the kth round.

[0027] Optionally, after obtaining the student model S of the target compression, the method further includes: using fine-tuning training samples to perform preset rounds of fine-tuning optimization training on the student model of the target compression to obtain an optimized student model, and deploying the optimized student model in a preset application scenario.

[0028] Through the above method, the present application explores a multi-stage progressive knowledge extraction teacher-student co-evolution method, which has the following three design advantages: First, multi-stage teacher-student co-evolution, by gradually transitioning from a large teacher model and a student model to a smaller teacher model and a student model, online distillation training is performed, and the student model of the previous stage is used as the teacher model of the current stage until the student model reaches the size of the required compressed model. This co-evolution method fully considers the gradual changes of the student model and the teacher model at each stage, and can achieve smooth and efficient knowledge dissemination; second, the sequential distillation process, through different orders to integrate all teacher knowledge, to achieve more effective distillation, deep teacher features can be gradually transferred from back to front or from front to back to cope with the effectiveness changes of different feature layers caused by the gap between teachers and students; third, the progressive integration of teacher features, that is, in the multi-stage teacher knowledge transfer process, enables the student model to learn the most suitable knowledge more adaptively. This strategy integrates a small number of refined teacher features in the early stage of training, and gradually increases the feature knowledge used for integration as the size of the teacher model decreases, thereby achieving a wider range of teacher knowledge transfer.

[0029] On the second aspect, the embodiment of the present application proposes a multi-stage progressive knowledge distillation teacher-student co-evolution system, including: an initialization module, which is used to initialize and construct a full-size student model S0 based on the full-size teacher model T0, and connect T0 with the pre-built flow knowledge fusion module Ψ0; an offline distillation module, which is used to input training samples into T0 and S0 to obtain the teacher fusion attention feature map output by Ψ0 And the student characteristic diagram of S0 output The training samples are marked with the true label t, and the final output p based on y and S0 s0 , and S0 is trained offline for distillation, and based on y and The feature fusion parameters of Ψ0 are updated offline to obtain the flow knowledge fusion module Ψ1 after offline update; the pruning module is used to prune the convolutional layer channels of S0 that has completed offline distillation training according to the preset pruning ratio to obtain the compressed student model S1, and S0 is used as the new teacher model T1, and T1 is connected to Ψ1; the online distillation module is used to input the training samples into T1 and S1 to obtain the teacher fusion attention feature map output by ψ1 And the student characteristic map output by S1 The final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and The feature fusion parameters of Ψ1 are updated online to obtain the flow knowledge fusion module Ψ2 after online update; the judgment module is used to judge whether S1 that has completed the online distillation training meets the preset parameter requirements; the output module is used to determine S1 that has completed the online distillation training as the student model of the target compression when it is determined that S1 that has completed the online distillation training meets the preset parameter requirements; wherein, when it is determined that S1 that has completed the online distillation training does not meet the preset parameter requirements, the pruning module performs convolutional layer channel pruning on S1 that has completed the online distillation training according to a preset pruning ratio to obtain the compressed student model S2, and uses S1 as a new teacher model T2, and connects T2 with Ψ2, and the online distillation module performs the next stage of online distillation training until S1 that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the output module will complete the online distillation training S n Determine the student model for target compression, n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

[0030] In the third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a multi-stage progressive knowledge extraction teacher-student collaborative evolution method as described in the first aspect above.

[0031] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a multi-stage progressive knowledge extraction teacher-student collaborative evolution method as described in the first aspect above.

[0032] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is a flow chart of a teacher-student collaborative evolution method for multi-stage progressive knowledge extraction provided in one embodiment of the present application;

[0035] Figure 2 is a structural schematic diagram of a flow knowledge fusion module provided in an embodiment of the present application;

[0036] Figure 3 It is a schematic diagram of the full process of offline distillation and online distillation provided in one embodiment of the present application;

[0037] Figure 4 It is a structural schematic diagram of a teacher-student collaborative evolution system for multi-stage progressive knowledge extraction provided in another embodiment of the present application;

[0038] Figure 5 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0040] An embodiment of the present application proposes a multi-stage progressive knowledge extraction teacher-student collaborative evolution method, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the multi-stage progressive knowledge extraction teacher-student collaborative evolution method proposed in this embodiment are described in detail below. The following content is only the implementation details provided for easy understanding and is not necessary for the implementation of this solution. The specific process of the multi-stage progressive knowledge extraction teacher-student collaborative evolution method proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0041] Step 101: Based on the full-size teacher model T0, a full-size student model S0 is initialized and constructed, and T0 is connected to the pre-built flow knowledge fusion module Ψ0, and the training samples are input into T0 and S0 to obtain the teacher fusion attention feature map output by Ψ0. And the student characteristic diagram of S0 output

[0042] In the specific implementation, the unstable knowledge transfer process of the current direct distillation method leads to the instability of the performance improvement of the compressed student model. To solve this problem, this embodiment proposes a progressive compression distillation structure to achieve stable top-down knowledge propagation. At each stage, a certain proportion of the number of convolution channels is pruned to obtain a compression model, and then semi-online distillation is used to train this compression model and the teacher model compressed in the next stage. The method includes three stages: offline distillation, pruning compression and online distillation. The server first performs offline distillation and uses a full-size teacher model T0 to train the initialized full-size student model S0 to obtain the initial knowledge of progressive compression distillation. This stage needs to be implemented with the help of a pre-built flow knowledge fusion module Ψ0. A connection is established between T0 and Ψ0. The server inputs the training samples marked with the true label y into T0 and S0, and the teacher fusion attention feature map output by Ψ0 can be obtained. Student characteristic map of S0 output And the final output p of S0 s0 .

[0043] In one example, the structure of the flow knowledge fusion module can be as follows Figure 2 As shown in Figure 2, the whole process of offline distillation, pruning and compression, and online distillation can be described as follows: Figure 3 shown.

[0044] In an example, the student model and the teacher model may be an image classification model that performs an image classification task, a semantic segmentation model that performs a semantic segmentation task, or a face recognition model that performs a face recognition task.

[0045] Step 102, the final output p based on y and S0 s0 , and S0 is trained offline for distillation, and based on y and The feature fusion parameters of Ψ0 are updated offline to obtain the offline updated flow knowledge fusion module Ψ1.

[0046] In the specific implementation, in the offline distillation stage, the student model S0 and the flow knowledge fusion module Ψ0 need to be updated. The server is based on y and the final output p of S0. s0 , and S0 is trained offline for distillation, and based on y and The feature fusion parameters of Ψ0 are updated offline to obtain the offline updated flow knowledge fusion module Ψ1. The convergence standard of the offline distillation training is to reach the preset maximum number of offline distillation training times.

[0047] In one example, the server's final output p is based on y and S0. s0 , and Offline distillation training of S0 can be achieved by the following formula:

[0048]

[0049] Among them, Φ(·) is the channel alignment function, B(·) is the size alignment function, N(·) is the L2 regularization function, Dist(·) represents the distance loss after the channel alignment function, size alignment function and L2 regularization function are processed. is the L2 loss term in the offline distillation loss function, l CE (y,p s0 ) is the cross entropy loss term in the offline distillation loss function, α and β are preset offline hyperparameters, L stu-offline is the offline distillation loss function. The setting of L2 loss term ensures smoothness.

[0050] In one example, the server is based on y and The offline update of the feature fusion parameters of Ψ0 can be achieved by the following formula:

[0051]

[0052] Among them, Θ(·) represents a fully connected mapping network, L kfk is the offline fusion loss function.

[0053] Step 103, prune the convolutional layer channels of S0 that has completed offline distillation training according to the preset pruning ratio to obtain the compressed student model S1, use S0 as the new teacher model T1, connect T1 with Ψ1, input the training samples into T1 and S1, and obtain the teacher fusion attention feature map F output by Ψ1 t 1 And the student characteristic map output by S1

[0054] In the specific implementation, channel pruning is selected in the pruning and compression stage, which is implemented based on the weight of each channel of the convolutional layer. The server performs convolutional layer channel pruning on S0 that has completed offline distillation training according to the preset pruning ratio to obtain the compressed student model S1, and uses S0 as the new teacher model T1, and connects T1 with Ψ1, inputs the training samples into T1 and S1, and obtains the teacher fusion attention feature map output by Ψ1 And the student characteristic map output by S1

[0055] In an example, when the server performs convolution layer channel pruning on S0 that has completed offline distillation training, the server first determines the shape of the weight of the convolution layer of S0 that has completed offline distillation training. The shape of the weight of the convolution layer of S0 that has completed offline distillation training is expressed as: Among them, W represents the weight of the convolutional layer of S0 after offline distillation training, C i Indicates the number of input channels, C o Indicates the number of output channels, K h and K w Indicates the kernel size. Next, the weights of the convolutional layer of S1 that has completed offline distillation training are flattened and summed to obtain a reshaped weight vector. Based on the reshaped weight vector, the weight values ​​of each output channel are sorted from small to large, and the channel index to be pruned is determined based on the preset pruning ratio. Finally, the corresponding output channels of each convolutional layer are pruned according to the channel index to be pruned, and the corresponding input channels of the next layer are pruned. The preset pruning ratio can be set to 1 / 8, 1 / 4, 1 / 2, etc.

[0056] Step 104, the final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1, and Perform online distillation training on S1 and The feature fusion parameters of Ψ1 are updated online to obtain the online updated flow knowledge fusion module Ψ2.

[0057] In the specific implementation, the final compression model size and compression iteration rounds are determined based on experience. Different compression iteration rounds under the same compression size will also lead to changes in the order of feature fusion. Higher compression iteration rounds will make the compression model performance better, which represents a trade-off between time resource consumption and final compression model performance. Therefore, online distillation is multi-round. During the online distillation stage, the teacher model, student model and flow knowledge fusion module must be updated. For the first round of online distillation, the server is based on y and T1’s final output p t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and The feature fusion parameters of Ψ1 are updated online to obtain the online updated flow knowledge fusion module Ψ2.

[0058] In one example, T k and S k denote the classroom model and student model in the kth round of online distillation training, respectively. k is an integer greater than 0 and less than n. No matter how many rounds of compression and online distillation are performed, T0, S0, T k and S k The number of convolutional layers of the networks are the same, which is m (m is an integer greater than 2). The difference between them is that the number of channels is getting smaller and smaller.

[0059] like Figure 2 As shown in Figure 1, the flow knowledge fusion module consists of m input units, m-1 fusion units and an output unit. The m convolutional layers of the teacher model are connected to the m input units of the flow knowledge fusion module respectively. The m-1 fusion units are connected in sequence. The first two input units are connected to the first fusion unit, the m-1th fusion unit is connected to the output unit, and the jth input unit is connected to the j-1th fusion unit, where j is an integer greater than 2 and less than m.

[0060] Both the input unit and the output unit are composed of 1×1 convolutional layers. The fusion unit is composed of sequentially connected downsampling layers, 1×1 convolutional layers, and element-by-element addition layers. For the first fusion unit, the first input unit is connected to the downsampling layer of the first fusion unit, and the second input unit is connected to the element-by-element addition layer of the first fusion unit. For the j-1th fusion unit, the element-by-element addition layer of the j-2th fusion unit is connected to the downsampling layer of the j-1th fusion unit, and the jth input unit is connected to the element-by-element addition layer of the j-1th fusion unit.

[0061] For the kth round of online distillation training, the teacher model T k Fusion module with flow knowledge k Connect, T k The first a layers of Ψ participate in this round of online distillation training, k Output teacher fusion attention feature map It is expressed by the following formula:

[0062]

[0063] in, Indicates T k The output of the i-th convolutional layer, the parameter a can be set based on the total number of layers m.

[0064] It is understandable that if the teacher model that needs to be compressed initially is compared to the expert model, then the auxiliary expert with a similar number of parameters only needs to absorb the most appropriate knowledge to restore the performance of the original model. The feature knowledge that is most suitable for the first auxiliary expert to learn changes as the gap with the expert model increases. When the number of model compression iterations decreases, the appropriate fusion order is from front to back. However, as the multi-stage model compression process proceeds, the learning ability of the auxiliary teacher becomes weaker and weaker, that is, the number of parameters in the kth round of compression process is constantly decreasing, so the teacher's distillation knowledge needs to be continuously increased to ensure the effective transfer of knowledge.

[0065] Based on this, when performing online distillation, the teacher model, student model, and flow knowledge fusion module need to be trained separately. For the kth round of online distillation training, the server trains the student model S k Online distillation training can be achieved through the following formula:

[0066]

[0067] in, is the online distillation loss function for the kth round, α ′ and β ′ is the preset online hyperparameter, For S k The output student feature map, is the L2 loss term in the online distillation loss function of the kth round, p sk For S k The final output, L CE (y,p sk ) is the cross entropy loss term in the online distillation loss function of the kth round.

[0068] Server online update k The feature fusion parameters can be achieved by the following formula:

[0069]

[0070] in, is the online fusion loss function of the kth round.

[0071] Server to T k Training can be achieved through the following formula:

[0072]

[0073] Among them, p tk T k The final output is is the teacher loss function of the kth round.

[0074] Step 105 , determining whether S1 after the online distillation training meets the preset parameter requirements, if yes, executing step 106 , if no, executing step 107 .

[0075] Step 106: determine S1 that has completed the online distillation training as the student model for target compression.

[0076] Step 107, prune the convolutional layer channels of S1 that has completed the online distillation training according to the preset pruning ratio to obtain the compressed student model S2, use S1 as the new teacher model t2, and connect T2 with Ψ2 to perform the next stage of online distillation training until the S that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the S that completes the online distillation training n The student model determined as target compression.

[0077] In the specific implementation, the server needs to determine whether S1 that has completed the online distillation training meets the preset parameter requirements. If S1 that has completed the online distillation training meets the preset parameter requirements, S1 that has completed the online distillation training is determined as the student model for target compression; if S1 that has completed the online distillation training does not meet the preset parameter requirements, the convolutional layer channels of S1 that has completed the online distillation training are pruned according to the preset pruning ratio to obtain the compressed student model S2, and then S1 is used as the new teacher model T2, and T2 is connected to Ψ2 to perform the next stage of online distillation training until S1 that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the S that completes the online distillation training n Determine the student model for target compression. Where n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

[0078] In one example, after obtaining the student model of the target compression, the server needs to use the fine-tuning training samples to perform a preset round of fine-tuning optimization training on the student model of the target compression to obtain the optimized student model, and deploy the optimized student model in the preset application scenario. The fine-tuning optimization further improves the performance of the student model of the target compression.

[0079] This embodiment explores a multi-stage progressive knowledge extraction teacher-student co-evolution method, which has the following three design advantages: First, multi-stage teacher-student co-evolution, by gradually transitioning from a large teacher model and a student model to a smaller teacher model and a student model, offline distillation training and online distillation training are performed, and the student model of the previous stage is used as the teacher model of the current stage until the student model reaches the size of the required compressed model. This co-evolution method fully considers the gradual changes of the student model and the teacher model at each stage, and can achieve smooth and efficient knowledge dissemination; second, the sequential distillation process, through different orders to integrate all teacher knowledge, to achieve more effective distillation, deep teacher features can be gradually transferred from back to front or from front to back to cope with the effectiveness changes of different feature layers caused by the gap between teachers and students; third, the progressive integration of teacher features, that is, in the multi-stage teacher knowledge transfer process, the student model is more adaptive to learn the most suitable knowledge. This strategy integrates a small number of refined teacher features in the early stage of training, and gradually increases the feature knowledge used for integration as the size of the teacher model decreases, thereby achieving a wider range of teacher knowledge transfer.

[0080] Another embodiment of the present application proposes a teacher-student collaborative evolution system for multi-stage progressive knowledge extraction. The implementation details of the teacher-student collaborative evolution system for multi-stage progressive knowledge extraction proposed in this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this example. Figure 4 It is a structural diagram of a teacher-student collaborative evolution system for multi-stage progressive knowledge extraction proposed in this embodiment, including: an initialization module 201, an offline distillation module 202, a pruning module 203, an online distillation module 204, a judgment module 205 and an output module 206.

[0081] The initialization module 201 is used to initialize and construct a full-size student model S0 based on the full-size teacher model T0, and connect T0 with the pre-built flow knowledge fusion module Ψ0.

[0082] Offline distillation module 202, used to input training samples into T0 and S0 to obtain the teacher fusion attention feature map output by Ψ0 And the student characteristic diagram of S0 output The training samples are marked with the true label y, and the final output p based on y and S0 s0 , and S0 is trained offline for distillation, and based on y and The feature fusion parameters of Ψ0 are updated offline to obtain the offline updated flow knowledge fusion module Ψ1.

[0083] The pruning module 203 is used to prune the convolutional layer channels of S0 that has completed offline distillation training according to a preset pruning ratio to obtain a compressed student model S1, use S0 as a new teacher model T1, and connect T1 with Ψ1.

[0084] Online distillation module 204, used to input training samples into T1 and S1 to obtain the teacher fusion attention feature map output by Ψ1 And the student characteristic map output by S1 The final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and The feature fusion parameters of Ψ1 are updated online to obtain the online updated flow knowledge fusion module Ψ2.

[0085] The judgment module 205 is used to judge whether S1 after completing the online distillation training meets the preset parameter requirements.

[0086] The output module 206 is used to determine S1 that has completed the online distillation training as the student model of the target compression when it is determined that S1 that has completed the online distillation training meets the preset parameter requirements.

[0087] Among them, when it is determined that S1 that has completed the online distillation training does not meet the preset parameter requirements, the pruning module prunes the convolutional layer channels of S1 that has completed the online distillation training according to the preset pruning ratio to obtain the compressed student model S2, and uses S1 as the new teacher model T2, and connects T2 with Ψ2, and the online distillation module performs the next stage of online distillation training until S that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the output module will complete the online distillation training S n Determine the student model for target compression, n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

[0088] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0089] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0090] In another embodiment, in order to verify the superiority of the teacher-student collaborative evolution method and system for multi-stage progressive knowledge extraction (hereinafter referred to as Ours) proposed in this application, we conducted relevant simulation experiments, and the final experimental results of different training methods are compared as shown in Table 1.

[0091] Table 1: Comparison of final experimental results of different training methods

[0092]

[0093]

[0094] As can be seen from Table 1, Ours can achieve a smooth transfer of knowledge when the gap between the teacher model and the student model is large. It is 3% higher than the second best result in mIoU and 0.6% higher in Pixel Acc, and its performance is excellent.

[0095] Another embodiment of the present application provides an electronic device, whose specific structure can be as follows: Figure 5As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a multi-stage progressive knowledge extraction teacher-student collaborative evolution method described in the above-mentioned method embodiments.

[0096] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators and power management circuits together, which is well known in the art and will not be further described in this article. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0097] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0098] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a multi-stage progressive knowledge extraction teacher-student collaborative evolution method described in the above method embodiments.

[0099] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0100] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A multi-stage progressive knowledge extraction teacher-student co-evolution method, characterized by: include: Based on the full-size teacher model T0, a full-size student model S0 is initialized and constructed, and T0 is connected to the pre-built flow knowledge fusion module Ψ0. The training samples are input into T0 and S0 to obtain the teacher fusion attention feature map output by Ψ0. And the student characteristic diagram of S0 output Among them, the training samples are marked with the real label y; The final output p based on y and S0 s0 , and S0 is trained offline for distillation, and based on y and Offline update of feature fusion parameters of Ψ0 to obtain offline updated flow knowledge fusion module Ψ1; According to the preset pruning ratio, the convolutional layer channels of S0 that has completed offline distillation training are pruned to obtain the compressed student model S1. S0 is used as the new teacher model T1, and T1 is connected to Ψ1. The training samples are input into T1 and S1 to obtain the teacher fusion attention feature map output by Ψ1. And the student characteristic map output by S1 The final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and Update the feature fusion parameters of Ψ1 online to obtain the online updated flow knowledge fusion module Ψ2; Determine whether S1 that has completed online distillation training meets the preset parameter requirements; If S1 that has completed the online distillation training meets the preset parameter requirements, S1 that has completed the online distillation training is determined as the student model for target compression; If S1 that has completed the online distillation training does not meet the preset parameter requirements, the convolutional layer channels of S1 that has completed the online distillation training are pruned according to the preset pruning ratio to obtain the compressed student model S2, and S1 is used as the new teacher model T2, and T2 is connected to Ψ2 to perform the next stage of online distillation training until S1 that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the S that completes the online distillation training n Determine the student model for target compression; where n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

2. According to claim 1, a multi-stage progressive knowledge extraction teacher-student collaborative evolution method is characterized by: The final output p based on y and S0 s0 , and Offline distillation training of S0 is achieved through the following formula: Among them, Φ(·) is the channel alignment function, B(·) is the size alignment function, N(·) is the L2 regularization function, Dist(·) represents the distance loss after the channel alignment function, size alignment function and L2 regularization function are processed. is the L2 loss term in the offline distillation loss function, L CE (y,p s0 ) is the cross entropy loss term in the offline distillation loss function, α and β are preset offline hyperparameters, L stu-offline is the offline distillation loss function; Based on y and The offline update of the feature fusion parameters of Ψ0 is achieved through the following formula: Among them, Θ(·) represents a fully connected mapping network, L kfk is the offline fusion loss function.

3. According to the multi-stage progressive knowledge extraction teacher-student collaborative evolution method of claim 1, it is characterized by: According to the preset pruning ratio, the convolutional layer channels of S0 that has completed offline distillation training are pruned to obtain the compressed student model S1, including: Determine the shape of the weight of the convolutional layer of S0 that completes the offline distillation training. The shape of the weight of the convolutional layer of S0 that completes the offline distillation training is expressed as: Among them, W represents the weight of the convolutional layer of S1 after offline distillation training, C i Indicates the number of input channels, C o Indicates the number of output channels, K h and K w represents the kernel size; Flatten and sum the weights of the convolutional layer of S0 that has completed offline distillation training to obtain a reshaped weight vector; Based on the reshaped weight vector, the weight values ​​of each output channel are sorted from small to large, and the channel index that needs to be pruned is determined based on the preset pruning ratio; Prune the corresponding output channels of each convolutional layer according to the channel index that needs to be pruned, and prune the corresponding input channels of the next layer.

4. According to claim 1, a multi-stage progressive knowledge extraction teacher-student collaborative evolution method is characterized by: T k and S k denote the classroom model and student model in the kth round of online distillation training, respectively, k is an integer greater than 0 and less than n, T0, S0, T k and S k The number of convolutional layers of are the same and are all m layers, where m is an integer greater than 2; The flow knowledge fusion module consists of m input units, m-1 fusion units and output units. The m convolutional layers of the teacher model are connected to the m input units of the flow knowledge fusion module respectively. The m-1 fusion units are connected in sequence. The first two input units are connected to the first fusion unit. The m-1 fusion unit is connected to the output unit. The j-th input unit is connected to the j-1 fusion unit. j is an integer greater than 2 and less than m. Both the input unit and the output unit are composed of 1×1 convolutional layers. The fusion unit is composed of sequentially connected downsampling layers, 1×1 convolutional layers, and element-by-element addition layers. For the first fusion unit, the first input unit is connected to the downsampling layer of the first fusion unit, and the second input unit is connected to the element-by-element addition layer of the first fusion unit. For the j-1th fusion unit, the element-by-element addition layer of the j-2th fusion unit is connected to the downsampling layer of the j-1th fusion unit, and the jth input unit is connected to the element-by-element addition layer of the j-1th fusion unit.

5. According to claim 4, a multi-stage progressive knowledge extraction teacher-student collaborative evolution method is characterized in that: For the kth round of online distillation training, the teacher model T k Fusion module with flow knowledge k Connect, T k The first a layers of Ψ participate in this round of online distillation training, k Output teacher fusion attention feature map It is expressed by the following formula: in, Indicates T k The output of the i-th convolutional layer.

6. The teacher-student collaborative evolution method for multi-stage progressive knowledge extraction according to claim 5 is characterized in that: For student models S k Online distillation training is performed using the following formula: in, is the online distillation loss function for the kth round, α ′ and β ′ is the preset online hyperparameter, For S k The output student feature map, is the L2 loss term in the online distillation loss function of the kth round, p sk For S k The final output, L CE (y,p sk ) is the cross entropy loss term in the online distillation loss function of the kth round; Online Update k The feature fusion parameters are implemented by the following formula: in, is the online fusion loss function of the kth round; To T k Training is performed using the following formula: Among them, p tk T k The final output is is the teacher loss function of the kth round.

7. A multi-stage progressive knowledge extraction teacher-student collaborative evolution method according to any one of claims 1 to 6, characterized in that: After obtaining the student model of target compression, the method further includes: The target compressed student model is fine-tuned and optimized for a preset number of rounds using fine-tuning training samples to obtain an optimized student model, which is then deployed in a preset application scenario.

8. A multi-stage progressive knowledge extraction teacher-student collaborative evolution system, characterized by: include: An initialization module is used to initialize and construct a full-size student model S0 based on the full-size teacher model T0, and connect T0 with the pre-built flow knowledge fusion module Ψ0; Offline distillation module, used to input training samples into T0 and S0 to obtain the teacher fusion attention feature map output by Ψ0 And the student characteristic diagram of S0 output The training samples are marked with the true label y, and the final output p based on y and S0 s0 , and S0 is trained offline for distillation, and based on y and Offline update of feature fusion parameters of Ψ0 to obtain offline updated flow knowledge fusion module Ψ1; The pruning module is used to prune the convolutional layer channels of S0 that has completed offline distillation training according to a preset pruning ratio to obtain a compressed student model S1, use S0 as a new teacher model T1, and connect T1 with Ψ1; Online distillation module, used to input training samples into T1 and S1 to obtain the teacher fusion attention feature map output by Ψ1 And the student characteristic map output by S1 The final output p based on y and T1 t1 , train T1, based on y, the final output p of S1 s1 , and Perform online distillation training on S1 and Update the feature fusion parameters of Ψ1 online to obtain the online updated flow knowledge fusion module Ψ2; A judgment module is used to judge whether S1 that has completed the online distillation training meets the preset parameter requirements; An output module, used for determining S1 that has completed the online distillation training as a student model for target compression when it is determined that S1 that has completed the online distillation training meets the preset parameter requirements; Among them, when it is determined that S1 that has completed the online distillation training does not meet the preset parameter requirements, the pruning module prunes the convolutional layer channels of S1 that has completed the online distillation training according to the preset pruning ratio to obtain the compressed student model S2, and uses S1 as the new teacher model T2, and connects T2 with Ψ2, and the online distillation module performs the next stage of online distillation training until S that has completed the online distillation training and meets the preset parameter requirements is obtained. n , the output module will complete the online distillation training s n Determine the student model for target compression, n represents S n The number of rounds of online distillation training, where n is an integer greater than 1.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a multi-stage progressive knowledge extraction teacher-student collaborative evolution method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a multi-stage progressive knowledge extraction teacher-student collaborative evolution method as described in any one of claims 1 to 7.

Images