Lightweight method, device and equipment for progressive knowledge distillation based on pruning network

Abstract

The present application relates to a progressive knowledge distillation lightweight method, device and equipment based on a pruned network. The method includes: obtaining an original model, a preset number of pruned layers and training samples in a target task scenario; pruning the first layer of the original model according to a preset pruning ratio, using the original model as a teacher model, using the pruned model as a student model, training the teacher model and the student model using a distillation training strategy according to the training samples, determining the first lightweight model, and pruning the next layer based on the lightweight model obtained after pruning, and repeating the above training steps until the number of pruned layers meets the set value. This method adopts the concept of progressive pruning, gradually pruning from the shallow layer of the model, and learning based on the proposed intermediate feature distillation loss and output distillation loss, which can better transfer the knowledge in the original model, thereby avoiding the loss of accuracy of the reasoning results caused by model lightweighting to the greatest extent.

Description

Technical Field

[0001] The present application relates to the technical field of model lightweighting, and in particular to a progressive knowledge distillation lightweighting method, device and equipment based on pruning networks. Background Art

[0002] With the continuous development and application of deep learning models, especially in resource-constrained environments such as edge computing, the Internet of Things (IoT), and mobile devices, how to efficiently deploy large deep learning models on these devices has become a key issue. Deep learning models usually have a large number of parameters and computational complexity, which makes it difficult to achieve real-time and efficient reasoning on resource-constrained devices. To this end, model lightweight deployment technology has emerged, aiming to enable deep learning models to run efficiently on edge devices and embedded systems by reducing resource consumption such as model computation, storage space, and energy consumption. Model lightweighting can not only improve the operating efficiency and response speed of the device, but also reduce hardware costs, and enable artificial intelligence technology to be more widely used in practical scenarios, such as autonomous driving, smart homes, smart healthcare, and other fields.

[0003] Traditional lightweight deployment methods mainly reduce the size of the model and improve computing efficiency by optimizing the model structure and reducing redundant calculations. Common traditional methods include parameter pruning, quantization, and low-rank decomposition. Parameter pruning methods reduce the storage requirements and computational complexity of the model by removing redundant parameters in the neural network that have little impact on model performance; quantization reduces storage space and increases inference speed by compressing model weights and activation values ​​into lower-precision data types (such as 8 or 16 bits); low-rank decomposition reduces the amount of computation by decomposing the weight matrix and representing it as the product of multiple low-rank matrices. Although these methods effectively reduce the complexity of the model, they usually face the problem of reduced accuracy, and for complex network structures and large-scale deep models, it is still difficult to meet the dual requirements of speed and accuracy in practical applications.

[0004] In recent years, with the increasing complexity of deep neural networks and the diversification of application scenarios, researchers have proposed many new lightweight methods. These methods not only focus on how to reduce model parameters and computational complexity, but also emphasize how to maintain the accuracy and performance of the model. For example, the knowledge distillation method transfers the knowledge of a large teacher model to a small student model to achieve performance compression while retaining most of its original predictive capabilities. In addition, the deep pruning method further optimizes the model structure, compresses its volume and improves the inference efficiency by combining knowledge distillation and pruning techniques. However, the existing lightweight methods still have some defects. For example, in the process of knowledge distillation, the knowledge transfer of the teacher model may be limited by the capacity and network structure, resulting in poor distillation effect; and in technologies such as pruning and quantization, excessive compression will lead to a significant decrease in model performance. Therefore, how to ensure efficient knowledge transfer and high accuracy while reducing the model size is still a major challenge in the current model lightweight research. Summary of the invention

[0005] Based on this, it is necessary to provide a lightweight method, device and equipment for progressive knowledge distillation based on pruning networks to address the above technical problems.

[0006] A progressive knowledge distillation lightweight method based on pruning network, the method comprising:

[0007] Get the original model, the preset number of pruned layers, and the text sequence in the text classification task scenario.

[0008] The first layer of the original model is pruned according to the preset pruning ratio, the original model is used as the teacher model, and the pruned model is used as the student model. The teacher model and the student model are trained using the distillation training strategy according to the text sequence to determine the first lightweight model.

[0009] Based on the first lightweight model, the second layer of the first lightweight model is pruned, the first lightweight model is used as a new teacher model, and the pruned first lightweight model is used as a new student model. The new teacher model and the new student model are trained using a distillation training strategy according to the text sequence to obtain a second lightweight model. The third layer of the second lightweight model is pruned, and so on, until the number of pruned layers meets the preset number of pruning layers to obtain a pruned lightweight model.

[0010] The pruned lightweight model is used to infer the test text sequence to obtain the text classification result.

[0011] The first layer of the original model is pruned according to a preset pruning ratio, the original model is used as a teacher model, the pruned model is used as a student model, the teacher model and the student model are trained using a distillation training strategy according to the text sequence, and the first lightweight model is determined, including:

[0012] The first layer of the original model is pruned according to the preset pruning ratio to obtain a pruned model.

[0013] The original model is used as the teacher model and the pruned model is used as the student model.

[0014] The text sequence is input into the teacher model and the student model respectively to obtain the first-layer output features and logical output of the teacher model and the student model.

[0015] The first layer output features of the student model are processed by the adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer.

[0016] According to the adaptive features and the first-layer output features of the teacher model, the first intermediate feature distillation loss is calculated; wherein the expression of the first intermediate feature distillation loss is:

[0017] ;

[0018] in, is the first intermediate characteristic distillation loss, and Represent the feature maps of the original model and the lightweight model respectively, represents the mapping regression operation after the adaptive layer, Denotes L2 norm.

[0019] Calculate the cross entropy loss based on the logistic output of the student model and the corresponding true label.

[0020] The logistic outputs of the teacher model and the student model are soft-normalized respectively, and the output distillation loss is calculated based on the obtained processing results.

[0021] The total training loss function is determined based on the first intermediate feature distillation loss, the cross entropy loss, and the output distillation loss.

[0022] The parameters of the student model are optimized and updated according to the total training loss function and the text sequence to obtain the first lightweight model.

[0023] In one embodiment, the output distillation loss is:

[0024] ;

[0025] ;

[0026] in, is the output distillation loss, Represents the logical output of the lightweight model regarding the i-th sample in the j-th category. and Represent the soft normalized outputs of the lightweight model and the original model respectively, Represents the temperature coefficient.

[0027] In one embodiment, the total training loss function is:

[0028] ;

[0029] in, is the total training loss, is the output distillation loss, is the intermediate feature distillation loss, is the cross entropy loss, and is a trade-off parameter.

[0030] In one of the embodiments, the pruning process adopts structured pruning, and each module is pruned in turn starting from the first layer, so that each pruned block becomes a lightweight model.

[0031] In one of the embodiments, the original model includes: a BERT encoder and a classification head.

[0032] A progressive knowledge distillation lightweight device based on a pruning network, the device comprising:

[0033] The original model and pruning parameter acquisition module is used to obtain the original model, the preset number of pruning layers, and the text sequence in the text classification task scenario.

[0034] The progressive pruning module is used to prune the first layer of the original model according to a preset pruning ratio, use the original model as the teacher model, use the pruned model as the student model, and train the teacher model and the student model using a distillation training strategy according to a text sequence to determine a first lightweight model; based on the first lightweight model, prune the second layer of the first lightweight model, use the first lightweight model as a new teacher model, use the pruned first lightweight model as a new student model, and train the new teacher model and the new student model using a distillation training strategy according to a text sequence to obtain a second lightweight model, continue to prune the third layer of the second lightweight model, and so on, until the number of pruned layers meets the preset number of pruning layers to obtain a pruned lightweight model.

[0035] The text classification module is used to use the pruned lightweight model to infer the text sequence to be tested and obtain the text classification result.

[0036] Among them, the progressive pruning module is also used to prune the first layer of the original model according to a preset pruning ratio to obtain a pruned model; the original model is used as a teacher model, and the pruned model is used as a student model; the text sequence is input into the teacher model and the student model respectively to obtain the first layer output features and logical outputs of the teacher model and the student model; the first layer output features of the student model are processed by the adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer; according to the adaptive features and the first layer output features of the teacher model, the first intermediate feature distillation loss is calculated using the expression of the first intermediate feature distillation loss; wherein the expression of the first intermediate feature distillation loss is:

[0037] ;

[0038] in, is the first intermediate characteristic distillation loss, and Represent the feature maps of the original model and the lightweight model respectively, represents the mapping regression operation after the adaptive layer, Denotes L2 norm.

[0039] According to the logical output of the student model and the corresponding true label, the cross entropy loss is calculated; the logical outputs of the teacher model and the student model are soft-normalized respectively, and the output distillation loss is calculated according to the obtained processing results; the total training loss function is determined according to the first intermediate feature distillation loss, the cross entropy loss and the output distillation loss; the parameters of the student model are optimized and updated according to the total training loss function and the text sequence to obtain the first lightweight model.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] The above-mentioned progressive knowledge distillation lightweight method, device and equipment based on pruning network, the method includes: obtaining the original model, the preset number of pruning layers and the training samples in the target task scenario; pruning the first layer of the original model according to the preset pruning ratio, using the original model as the teacher model, using the pruned model as the student model, training the teacher model and the student model using the distillation training strategy according to the training samples, determining the first lightweight model, pruning the next layer based on the lightweight model obtained after pruning, and repeating the above training steps until the number of pruning layers meets the set value. This method adopts the concept of progressive pruning, gradually pruning from the shallow layer of the model, and learning based on the proposed intermediate feature distillation loss and output distillation loss, which can better transfer the knowledge in the original model, thereby avoiding the accuracy loss of the reasoning result caused by the lightweight model to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a process flow of a progressive knowledge distillation lightweight method based on a pruning network in one embodiment;

[0043] Figure 2 A flowchart of a lightweight method for progressive knowledge distillation based on a pruning network in another embodiment;

[0044] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] In one embodiment, Figure 1 , Figure 2 As shown, a progressive knowledge distillation lightweight method based on pruning network is provided, which includes the following steps:

[0047] Step 100: Obtain the original model, the preset number of pruning layers, and the text sequence in the text classification task scenario.

[0048] Specifically, this method can be any task other than text classification task scenario. Preferably, the architecture of the original model is BERT+classification head; wherein the classification head is a two-layer fully connected network, and the final logical output is a probability matrix of size N. The number of pruning layers can be set according to actual requirements. The more the number, the greater the degree of model compression. In this embodiment, it is set to 5.

[0049] For example, in text classification tasks, the BERT model is usually used to stack classification linear layers for learning. However, the BERT model has a large number of parameters and consists of 12 encoding layers. This method can gradually prune the parameters of the encoding layer from shallow to deep and perform knowledge distillation, thereby achieving the purpose of lightweight model.

[0050] Step 102: Prune the first layer of the original model according to a preset pruning ratio, use the original model as the teacher model, use the pruned model as the student model, train the teacher model and the student model using a distillation training strategy according to the text sequence, and determine the first lightweight model.

[0051] Step 102 specifically includes: pruning the first layer of the original model according to a preset pruning ratio to obtain a pruned model; using the original model as a teacher model and the pruned model as a student model; inputting the text sequence into the teacher model and the student model respectively to obtain the first layer output features and logical outputs of the teacher model and the student model; processing the first layer output features of the student model through an adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer; calculating the first intermediate feature distillation loss according to the adaptive features and the first layer output features of the teacher model; the expression of the first intermediate feature distillation loss is:

[0052] ;

[0053] in, is the first intermediate characteristic distillation loss, and Represent the feature maps of the original model and the lightweight model respectively, represents the mapping regression operation after the adaptive layer, Denotes L2 norm.

[0054] According to the logical output of the student model and the corresponding true label, the cross entropy loss is calculated; the logical outputs of the teacher model and the student model are soft-normalized respectively, and the output distillation loss is calculated according to the obtained processing results; the total training loss function is determined according to the first intermediate feature distillation loss, the cross entropy loss and the output distillation loss; the parameters of the student model are optimized and updated according to the total training loss function and the text sequence to obtain the first lightweight model.

[0055] Specifically, a lightweight model is initialized and pruning is started from the shallow layer of the model. This can retain the high-level feature extraction capability of the model while reducing the computing resources occupied by low-level redundant parameters.

[0056] Pruning starts from the shallow layer of the model. Here we use structured pruning to prune each module in turn starting from the first layer, so that each pruned block becomes a lightweight model, and the corresponding unpruned block of the original model becomes the teacher. In this embodiment, the pruning ratio of each layer is set to 60%.

[0057] Step 104: Based on the first lightweight model, prune the second layer of the first lightweight model, use the first lightweight model as a new teacher model, use the pruned first lightweight model as a new student model, train the new teacher model and the new student model using a distillation training strategy according to the text sequence to obtain a second lightweight model, continue to prune the third layer of the second lightweight model, and so on, until the number of pruned layers meets the preset number of pruning layers, to obtain a pruned lightweight model.

[0058] Specifically, the next layer is pruned based on the first lightweight model determined in step 102, and the above training steps are repeated until the number of pruned layers meets the set value.

[0059] The lightweight effect and performance of the model are continuously improved in the process of layer-by-layer pruning. Specifically, the first layer of the original model is pruned first, and a lightweight model with functions close to the original model is obtained through the distillation training strategy. Subsequently, based on the lightweight model, the next layer of modules is pruned, and the distillation training method is used for optimization again to ensure that the performance of the pruned model is not significantly affected. This process is carried out recursively, and the pruning operation gradually penetrates into each layer of the model structure. The above training steps are repeated in each iteration until the number of pruned layers reaches the preset number and stops. Through this layer-by-layer pruning and distillation strategy, the progressive optimization of the model is effectively achieved, taking into account both computational efficiency and prediction accuracy.

[0060] This method does not use the traditional method of directly pruning the original model once. Although the method of pruning the original model once can reduce the number of model parameters, it will seriously lose the accuracy performance of the lightweight model.

[0061] Step 106: Use the pruned lightweight model to infer the text sequence to be tested to obtain a text classification result.

[0062] In the above-mentioned progressive knowledge distillation lightweight method based on pruned network, the method includes: obtaining the original model, the preset number of pruned layers and the training samples in the target task scenario; pruning the first layer of the original model according to the preset pruning ratio, using the original model as the teacher model, using the pruned model as the student model, training the teacher model and the student model using the distillation training strategy according to the training samples, determining the first lightweight model, and pruning the next layer based on the lightweight model obtained after pruning, and repeating the above training steps until the number of pruned layers meets the set value. This method adopts the concept of progressive pruning, gradually pruning from the shallow layer of the model, and learning based on the proposed intermediate feature distillation loss and output distillation loss, which can better transfer the knowledge in the original model, thereby avoiding the accuracy loss of the reasoning result caused by the lightweight model to the greatest extent.

[0063] In one embodiment, the expression for outputting the distillation loss is:

[0064] ;

[0065] ;

[0066] in, is the output distillation loss, Represents the lightweight model about i The sample in j logical output on each class. and Represent the soft normalized outputs of the lightweight model and the original model respectively, Represents the temperature coefficient.

[0067] In one embodiment, the expression of the total training loss function is:

[0068] ;

[0069] in, is the total training loss, is the output distillation loss, is the intermediate feature distillation loss, is the cross entropy loss, and These are all trade-off parameters. As the preferred choice, and Set them to 0.50 and 0.25 respectively.

[0070] Specifically, first, it is necessary to reduce the feature map difference between the lightweight model and the original model at the intermediate feature level to ensure the performance of the model after pruning. During the pruning process of the lightweight model, structured pruning will delete those unimportant channels, which usually leads to a significant difference in the number of channels between the student model and the teacher model. If not handled, this channel mismatch will significantly affect the efficiency and effect of knowledge distillation. To solve this problem, this method introduces an adaptive layer, which consists of point-by-point convolution (1×1 convolution kernel) and batch normalization layer. Point-by-point convolution can efficiently adjust the number of channels and map the output features of the student network to the channel dimension matching the teacher model, while batch normalization further improves the stability and consistency of the feature distribution. Through the introduction of this adaptive layer, efficient alignment of feature maps is achieved, and the feature map difference between the pruned model and the original model is effectively reduced, thereby improving the knowledge extraction ability and performance of the lightweight model. Therefore, the intermediate feature distillation loss shown in the expression of the first intermediate feature distillation loss above is proposed. The intermediate feature distillation loss enables the lightweight model to effectively learn the features of the original model during the structural pruning process.

[0071] Compared with the single-layer distillation method, multi-level distillation learning has higher training efficiency and better distillation performance. It can more comprehensively capture the knowledge of the teacher model and improve the performance of the student model. Therefore, this application extends this concept to the pruning process to achieve a deep combination of pruning and distillation. At each stage of pruning, not only is knowledge distilled through intermediate feature representations to learn implicit feature structures from the teacher model, but the output logic of the teacher model is further imitated to extract richer decision-making knowledge. This double distillation strategy can maintain model performance while pruning, avoiding the loss of accuracy caused by feature loss or reduced model capacity. By alternating layers of pruning and distillation, the model can better inherit the excellent characteristics of the original model while maintaining lightweight, thereby achieving better pruning and distillation effects. Here, the Kullback-Leibler divergence (KL divergence) loss between the output of the original model and the lightweight model is used as the output distillation loss, and the output distillation loss is shown in the expression of the above output distillation loss.

[0072] The output distillation loss enables the lightweight model to more efficiently learn the predictions of the high-performance original model, thereby significantly reducing the classification error rate.

[0073] In addition to the aforementioned intermediate feature distillation loss and output distillation loss, each lightweight model is also trained using a classic cross-entropy loss, which combines the true label of the data with the output logic generated by the student model. Specifically, this training method can further help the lightweight model fully learn the essential characteristics of the dataset while retaining the advantages of knowledge distillation. In this way, the model can not only obtain effective guidance information from the original model, but also enhance its generalization ability by directly optimizing the match with the true label. The use of cross-entropy loss can effectively reduce the difference between the output of the lightweight model and the true distribution, thereby improving the overall performance. The process can be formalized as:

[0074] ;

[0075] ;

[0076] in Represents the normalized logical output of the lightweight model, Represents the true label of the sample. This method of combining the true label and the logical output significantly improves the adaptability of the student model to complex data sets, while giving the model stronger discrimination ability and robustness.

[0077] Finally, the total training loss is obtained as shown in the expression of the above total training loss function.

[0078] In one of the embodiments, the pruning process adopts structured pruning, and each module is pruned in turn starting from the first layer, so that each pruned block becomes a lightweight model.

[0079] In one embodiment, the original model in step 100 includes: a BERT encoder and a classification head.

[0080] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0081] In one embodiment, a progressive knowledge distillation lightweight device based on a pruning network is provided, comprising: an original model and pruning parameter acquisition module, a progressive pruning module and a text classification module, wherein:

[0082] The original model and pruning parameter acquisition module is used to obtain the original model, the preset number of pruning layers, and the text sequence in the text classification task scenario;

[0083] A progressive pruning module is used to prune the first layer of the original model according to a preset pruning ratio, use the original model as the teacher model, use the pruned model as the student model, and train the teacher model and the student model using a distillation training strategy according to a text sequence to determine a first lightweight model; based on the first lightweight model, prune the second layer of the first lightweight model, use the first lightweight model as a new teacher model, use the pruned first lightweight model as a new student model, and use a distillation training strategy according to a text sequence to train the new teacher model and the new student model to obtain a second lightweight model, continue to prune the third layer of the second lightweight model, and so on, until the number of pruned layers meets the preset number of pruning layers to obtain a pruned lightweight model;

[0084] The text classification module is used to use the pruned lightweight model to infer the text sequence to be tested and obtain the text classification result.

[0085] Among them, the progressive pruning module is also used to prune the first layer of the original model according to a preset pruning ratio to obtain a pruned model; the original model is used as the teacher model, and the pruned model is used as the student model; the text sequence is input into the teacher model and the student model respectively to obtain the first layer output features and logical output of the teacher model and the student model; the first layer output features of the student model are processed by the adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer; according to the adaptive features and the first layer output features of the teacher model, the first intermediate feature distillation loss is calculated using the expression of the above-mentioned first intermediate feature distillation loss; according to the logical output of the student model and the corresponding true label, the cross entropy loss is calculated; the logical outputs of the teacher model and the student model are soft-normalized respectively, and the output distillation loss is calculated according to the obtained processing results; according to the first intermediate feature distillation loss, the cross entropy loss and the output distillation loss, the total training loss function is determined; according to the total training loss function and the text sequence, the parameters of the student model are optimized and updated to obtain the first lightweight model.

[0086] In one of the embodiments, the output distillation loss in the progressive pruning module is as shown in the above expression of the output distillation loss.

[0087] In one embodiment, the total training loss function in the progressive pruning module is as shown in the expression of the total training loss function mentioned above.

[0088] In one of the embodiments, the pruning process adopts structured pruning, and each module is pruned in turn starting from the first layer, so that each pruned block becomes a lightweight model.

[0089] In one of the embodiments, the original model and the pruning parameter acquisition module include: a BERT encoder and a classification head.

[0090] For the specific limitations of the progressive knowledge distillation lightweight device based on pruned networks, please refer to the limitations of the progressive knowledge distillation lightweight method based on pruned networks above, which will not be repeated here. Each module in the above-mentioned progressive knowledge distillation lightweight device based on pruned networks can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0091] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a progressive knowledge distillation lightweight method based on a pruning network is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0092] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0094] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A lightweight method for progressive knowledge distillation based on pruning networks, characterized in that: The method comprises: Get the original model, the preset number of pruning layers, and the text sequence in the text classification task scenario; Pruning the first layer of the original model according to a preset pruning ratio, using the original model as a teacher model, using the pruned model as a student model, and training the teacher model and the student model using a distillation training strategy according to the text sequence to determine a first lightweight model; Based on the first lightweight model, the second layer of the first lightweight model is pruned, the first lightweight model is used as a new teacher model, the pruned first lightweight model is used as a new student model, the new teacher model and the new student model are trained according to the text sequence using a distillation training strategy to obtain a second lightweight model, and the third layer of the second lightweight model is pruned continuously, and so on, until the number of pruned layers meets the preset number of pruned layers, thereby obtaining a pruned lightweight model; Use the pruned lightweight model to infer the text sequence to obtain the text classification result; The first layer of the original model is pruned according to a preset pruning ratio, the original model is used as a teacher model, the pruned model is used as a student model, the teacher model and the student model are trained using a distillation training strategy according to the text sequence, and a first lightweight model is determined, including: Pruning the first layer of the original model according to a preset pruning ratio to obtain a pruned model; The original model is used as a teacher model, and the pruned model is used as a student model; Inputting the text sequence into the teacher model and the student model respectively to obtain the first layer output features and logical output of the teacher model and the student model; Processing the first layer output features of the student model through an adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer; According to the adaptive features and the first layer output features of the teacher model, a first intermediate feature distillation loss is calculated; wherein the expression of the first intermediate feature distillation loss is: ； in, is the first intermediate characteristic distillation loss, and Represent the feature maps of the original model and the lightweight model respectively, represents the mapping regression operation after the adaptive layer, represents the L2 paradigm; Calculate the cross entropy loss based on the logical output of the student model and the corresponding true label; Performing soft normalization processing on the logical outputs of the teacher model and the student model respectively, and calculating the output distillation loss according to the obtained processing results; Determining a total training loss function according to the first intermediate feature distillation loss, the cross entropy loss, and the output distillation loss; The parameters of the student model are optimized and updated according to the total training loss function and the text sequence to obtain a first lightweight model.

2. The progressive knowledge distillation lightweight method based on pruning network according to claim 1 is characterized in that: The output distillation loss is: in, is the output distillation loss, represents the logical output of the lightweight model for the i-th sample in the j-th category, and Represent the soft normalized outputs of the lightweight model and the original model respectively, Represents the temperature coefficient.

3. The progressive knowledge distillation lightweight method based on pruning network according to claim 1 is characterized in that: The total training loss function is: in, is the total training loss, is the output distillation loss, is the intermediate feature distillation loss, is the cross entropy loss, and is a trade-off parameter.

4. The progressive knowledge distillation lightweight method based on pruning network according to claim 1 is characterized in that: The pruning process adopts structured pruning, starting from the first layer, and pruning each module in turn, so that each pruned block becomes a lightweight model.

5. The progressive knowledge distillation lightweight method based on pruning network according to claim 1 is characterized in that: The original model includes: a BERT encoder and a classification head.

6. A lightweight device for progressive knowledge distillation based on pruning network, characterized in that: The device comprises: The original model and pruning parameter acquisition module is used to obtain the original model, the preset number of pruning layers, and the text sequence in the text classification task scenario; A progressive pruning module is used to prune the first layer of the original model according to a preset pruning ratio, use the original model as a teacher model, use the pruned model as a student model, and train the teacher model and the student model using a distillation training strategy according to the text sequence to determine a first lightweight model; based on the first lightweight model, prune the second layer of the first lightweight model, use the first lightweight model as a new teacher model, use the pruned first lightweight model as a new student model, and train the new teacher model and the new student model using a distillation training strategy according to the text sequence to obtain a second lightweight model, continue to prune the third layer of the second lightweight model, and so on, until the number of pruned layers meets the preset number of pruning layers to obtain a pruned lightweight model; The text classification module is used to use the pruned lightweight model to infer the text sequence to be tested and obtain the text classification result; The progressive pruning module is further used to prune the first layer of the original model according to a preset pruning ratio to obtain a pruned model; the original model is used as a teacher model and the pruned model is used as a student model; the text sequence is input into the teacher model and the student model respectively to obtain the first layer output features and logical outputs of the teacher model and the student model; the first layer output features of the student model are processed by an adaptive layer to obtain adaptive features; the adaptive layer includes a point convolution layer and a batch normalization layer; the first intermediate feature distillation loss is calculated according to the adaptive features and the first layer output features of the teacher model; wherein the expression of the first intermediate feature distillation loss is: ； in, is the first intermediate characteristic distillation loss, and Represent the feature maps of the original model and the lightweight model respectively, represents the mapping regression operation after the adaptive layer, represents the L2 paradigm; According to the logical output of the student model and the corresponding true label, the cross entropy loss is calculated; the logical outputs of the teacher model and the student model are soft-normalized respectively, and the output distillation loss is calculated according to the obtained processing results; the total training loss function is determined according to the first intermediate feature distillation loss, the cross entropy loss and the output distillation loss; the parameters of the student model are optimized and updated according to the total training loss function and the text sequence to obtain a first lightweight model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the progressive knowledge distillation lightweight method based on pruning network described in any one of claims 1 to 5 is implemented.

Images