Network pruning method based on attention head and self-distillation

Abstract

The invention discloses a network pruning method based on an attention head and self-distillation, and belongs to the technical field of multi-modal large model pruning, and the method comprises the steps: obtaining a student model and a teacher model based on a token pruning device, an attention head pruning device and a basic model; the token pruning device is used for carrying out importance evaluation and pruning on tokens input into the basic model; the attention head pruning device is used for carrying out importance evaluation and pruning on the attention head of the basic model; training the student model and the teacher model based on the training sample and the self-distillation objective function to obtain an optimized student model; a token pruning device and an attention head pruning device in the student model are activated; a token pruning device and an attention head pruning device in the teacher model are frozen, and the problems that according to an existing multi-mode pruning method, dynamic pruning cannot be carried out, and the interaction relation between a visual mode and a language mode cannot be effectively considered are solved.

Description

Technical Field

[0001] The invention relates to the technical field of multimodal large model pruning, and in particular to a structured network pruning method based on minimum entropy. Background Art

[0002] Transformer model pruning methods can be roughly divided into token pruning and model structure pruning. Token pruning refers to a pruning method that reduces the computational complexity of the model by reducing the number of input tokens. Model structure pruning refers to a pruning method that reduces the computational complexity and number of parameters of the model by removing redundant model structures.

[0003] As for the pruning method of large multimodal models, since large multimodal models are usually based on the Transformer architecture, the classification of Transformer model pruning methods can also be used. In previous studies, Gan et al. directly applied the single-modal model pruning method to the model pruning of large multimodal models, verifying the existence of the lottery hypothesis in large multimodal models. Fang et al. proposed DistillVLM, proving that the knowledge distillation method can be used to imitate the attention distribution of large multimodal models. Shi et al. proposed UPop, which completes the pruning of large multimodal models through two stages: consistent search and aggressive pruning.

[0004] Most of the current multimodal large model pruning methods are static model pruning. That is, the pruning mask for the model structure is independent of the input token sequence and is determined once the pruning is completed. However, for different input token sequences, the importance of each attention head is different, and the importance of each token is also different.

[0005] Secondly, many pruning methods for large multimodal models do not consider the interaction between the visual and language modalities, but simply apply the pruning method of the single-modal Transformer model to the multimodal model. Such pruning methods may cause inaccuracy in the pruned model.

[0006] The method of model pruning using knowledge distillation requires fine-tuning the teacher network on the target dataset. The fine-tuning of the teacher network and the knowledge distillation process cannot be performed at the same time. This leads to low efficiency of model pruning.

[0007] The problem with the above pruning methods is that they cannot implement dynamic pruning based on the input. And they either only perform model structure pruning or only perform token pruning. Moreover, they do not effectively consider the interactive relationship between the visual modality and the language modality. Summary of the invention

[0008] The purpose of the present invention is to provide a network pruning method based on attention heads and self-distillation to solve the problems that current multimodal pruning methods cannot perform dynamic pruning and do not effectively consider the interactive relationship between visual modality and language modality.

[0009] To achieve the above object, the present invention adopts the following technical solution: On the one hand, the present specification provides a network pruning method based on attention head and self-distillation, including: Step 102, based on a token pruner, an attention head pruner and a basic model, a student model and a teacher model are obtained; the token pruner is used to evaluate and prune the importance of tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention heads of the basic model; Step 104, training the student model and the teacher model based on the training samples and the objective function of self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

[0010] On the other hand, the present specification provides a network pruning device based on attention head and self-distillation, including: A pruning model building module, used to obtain a student model and a teacher model based on a token pruner, an attention head pruner and a basic model; the token pruner is used to evaluate and prune the importance of tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention head of the basic model; The model self-distillation module is used to train the student model and the teacher model based on the training samples and the objective function of the self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

[0011] Based on the above technical solution, this specification can achieve the following technical effects: (1) Combining token pruning and model structure pruning. The pruning method in this application includes screening the input tokens, reducing the number of tokens based on a learnable lightweight module, pruning the attention heads, and pruning the model structure to remove unimportant attention heads. Based on the hybrid paradigm, the computational complexity and parameter quantity of the model are minimized.

[0012] (2) Self-distillation training process. In the process of training the parameters of adaptive pruning, we adopt the self-distillation training method based on curriculum learning. The pruned model and the pruning model share the same parameters. The only difference is that the pruned model will activate the model pruner during the forward propagation process. In this way, the separate fine-tuning process of the teacher network in traditional knowledge distillation methods is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of a flow chart of a network pruning method based on attention head and self-distillation in one embodiment of the present invention.

[0014] Figure 2 This is a framework diagram of a network pruning method based on attention heads and self-distillation in one embodiment of the present invention.

[0015] Figure 3 Schematic diagram of the structure of a network pruning device based on attention head and self-distillation in one embodiment of the present invention.

[0016] Figure 4 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0018] It should be noted that, in order to clearly explain the content of the present invention, the present invention specifically cites multiple embodiments to further illustrate different implementations of the present invention, wherein the multiple embodiments are enumerated rather than exhaustive. In addition, for the sake of brevity of explanation, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, the contents not mentioned in the subsequent embodiments can refer to the previous embodiments accordingly. Example 1

[0019] Please refer to Figure 1 , Figure 1 The figure shows a network pruning method based on attention head and self-distillation provided in this embodiment. In this embodiment, the method includes: Step 102, based on a token pruner, an attention head pruner and a basic model, a student model and a teacher model are obtained; the token pruner is used to evaluate and prune the importance of tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention heads of the basic model; In this embodiment, the token pruner is a lightweight module based on MLP; the lightweight module based on MLP includes a local policy network and a global policy network; the local policy network is used to evaluate the importance of a single token; and the global policy network is used to evaluate the importance of cross-modal tokens.

[0020] In this embodiment, the token pruner performs importance evaluation and pruning on the tokens of the input basic model, including: Step 202, input the target token into the local policy network, and calculate the local importance score of the target token; In this embodiment, step 202 includes: Step 2021, performing dimension reduction processing on the target token to obtain the target token after dimension reduction; Step 2022, obtaining learnable parameters of the local policy network through stochastic gradient descent learning; Step 2023, input the target token after dimension reduction into the local policy network, and calculate the local importance score of the target token.

[0021] Step 204, fusing and mapping the global representations of the visual modality and the textual modality to obtain a cross-modal global representation; In this embodiment, the global representation based on the visual modality and the textual modality is a corresponding category token.

[0022] Step 206, inputting the cross-modal global representation and the target token into the global policy network, and calculating the cross-modal global importance score of the target token; Step 208, obtaining a final importance score of the target token based on the sum of the local importance score and the global importance score of the target token; Step 210, pruning or retaining the target token based on the final importance score of the target token.

[0023] In this embodiment, one implementation of step 210 is: Step 2101, mapping the final importance score of the target token to between 0 and 1 based on the neural network threshold function to obtain a pruning mask of the target token; Step 2102, if the pruning mask of the target token is 1, the target token is retained, otherwise, the target token is removed.

[0024] In this embodiment, the attention head pruner performs importance evaluation and pruning on the attention heads of the base model, including: Step 302: Input the global representation of the current modality and another modality into the attention pruner, and calculate the importance score of the attention head of the current modality; Step 304, mapping the importance score of the attention head of the current modality to between 0 and 1 based on the neural network threshold function, and obtaining a pruning mask of the attention head of the current modality; Step 306: If the pruning mask of the attention head is 1, the attention head of the current modality is retained; otherwise, the attention head of the current modality is removed.

[0025] Step 104, training the student model and the teacher model based on the training samples and the objective function of self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

[0026] In this embodiment, before step 104, the following steps are also included: Step 103, based on the student model and teacher model parameters and the target loss function, obtain the target function of self-distillation.

[0027] In this embodiment, one implementation of step 104 is: Step 402, input the training sample into the teacher model to obtain the prediction confidence of the training sample in the teacher model; Step 404, sorting the training samples in descending order based on the prediction confidence of the training samples in the teacher model to obtain descending order samples; Step 406, extract target samples from the descending samples and input them into the teacher model and the student model, and update the model parameters by batch gradient descent method according to the objective function of self-distillation to obtain the student model and the teacher model after parameter update.

[0028] In this embodiment, the output result of the student model after the parameter update is aligned with the output result of the teacher model.

[0029] Specifically, refer to Figure 2 , a network pruning method based on attention head and self-distillation, which is implemented as follows: Step 1: Adaptive Token Pruning By evaluating the importance of the tokens input to the model, the less important tokens are removed, thereby reducing the computational complexity of the model. The evaluation of token importance is implemented through a learnable lightweight module. The calculated token importance combines the local importance and cross-modal global importance of the token.

[0030] In this step, the input tokens of each encoder block are pruned step by step, unimportant tokens are eliminated, and more important tokens are passed to the next block. To evaluate the importance of each token, a lightweight MLP-based module named XModal-aware trimmer is inserted before the encoder block of each multimodal model. Assume Tokens are represented as , then first Input the Local Policy Network to extract local importance. The process can be expressed as follows:

[0031] in, Indicates the importance of each token. Yes It is obtained by dimensionality reduction. When , each token is calculated independently without considering the contribution of other modal tokens in the multimodal input. a is a learnable parameter obtained by stochastic gradient descent.

[0032] In order to introduce the importance of multimodal tokens to the current token, cross-modal interaction is required. Specifically, the global representations of the visual and textual modalities (using the category token as the global representation of each modality) are fused, and then mapped to obtain the cross-modal global representation g, which is finally combined with Input to the global policy network to calculate token importance scores across modalities :

[0033] in, Represents the mapping layer, represents a hybrid paradigm. Ultimately, the importance of each token The calculation is as follows:

[0034] During inference, the pruning mask Directly from The sampling results are: 1 means retaining the corresponding token, otherwise it means removing the corresponding token. It is a neural network threshold function, which is used to map variables to between 0 and 1.

[0035] Step 2: Adaptive Attention Head Pruning This method designs a modality-adaptive attention head pruner to estimate the importance of attention heads and integrates them into the multi-head self-attention module and the multi-head cross-attention module. Using the importance estimated by the attention head pruner, attention heads with low importance can be removed, thereby reducing the computational complexity and parameter amount of the model.

[0036] Multimodal large models can capture the interactions between tokens within and across modalities through multi-head self-attention mechanisms and multi-head mutual attention mechanisms. However, the complexity of attention calculation is related to the input, that is, simple samples do not require too much attention from the modality, while complex samples require more attention from the model. To this end, this project designs a modality-adaptive attention head pruner and integrates it into the attention module. Specifically, the global representation of the input sequence is first input into the attention head pruner, which is expressed as follows:

[0037] in, Represent the category token representation of the current modality and the other modality respectively. Similar to the token pruner, at inference time, directly from get To decide whether to remove or keep the corresponding attention head.

[0038] Step 3: Self-distillation performance improvement The token pruner and attention head pruner are trained on the dataset at the same time as the base model. In this method, we call it self-distillation. The self-distillation process simultaneously trains a pruned student model and a pre-pruned teacher model, and aligns the output of the student model with the teacher model. The two models share parameters, but the difference is that the student model will enable the pruner during the forward calculation, while the teacher model will not enable the pruner in the forward calculation.

[0039] During model training, this project designs a self-distillation strategy to promote the pruned model Output prediction results and unpruned model Alignment. Among them, and are models with shared parameters. The difference between them is that in the forward process The adaptive pruner is activated, and The adaptive pruner of is frozen. During training, and The optimization and update are performed simultaneously. The objective function of self-distillation is expressed as follows:

[0040] in, Represent the input and output logits (fully connected layer), is the loss function for recognition. This approach can avoid additional fine-tuning of the teacher model by traditional knowledge distillation methods.

[0041] In the process of optimizing the objective function, this project proposes an optimization method based on curriculum learning. Curriculum learning simulates how humans learn the laws of things through courses, that is, starting with simple samples and gradually increasing the learning difficulty. The optimization method based on curriculum learning is as follows: (1) Use samples in the teacher model The prediction confidence is used as an indicator, and the samples are sorted in descending order to obtain . (2) Samples are extracted in the sorted order, passed through the teacher model and the student model, and the parameters are updated using the SGD method according to the objective function of 4.5.

[0042] (3) After the parameter update is completed, the pruner weights of the student network are fixed to evaluate the importance of input tokens and attention heads during the testing phase.

[0043] In the updating process of course learning, samples with high confidence are regarded as simple samples, while samples with low confidence are regarded as difficult samples. Through the strategy of selecting samples from easy to difficult, the weight of the student model is better optimized to make it close to the output distribution of the teacher model.

[0044] The above is the process of network pruning method based on attention head and self-distillation.

[0045] In summary, this method can achieve the following technical effects: (1) Combining token pruning and model structure pruning. The pruning method in this application includes screening the input tokens, reducing the number of tokens based on a learnable lightweight module, pruning the attention heads, and pruning the model structure to remove unimportant attention heads. Based on the hybrid paradigm, the computational complexity and parameter quantity of the model are minimized.

[0046] (2) Self-distillation training process. In the process of training the parameters of adaptive pruning, we adopt the self-distillation training method based on curriculum learning. The pruned model and the pruning model share the same parameters. The only difference is that the pruned model will activate the model pruner during the forward propagation process. In this way, the separate fine-tuning process of the teacher network in traditional knowledge distillation methods is avoided. Example 2

[0047] Please refer to Figure 3 , Figure 3 The figure shows a network pruning device based on attention head and self-distillation provided in this embodiment. In this embodiment, the device includes: A pruning model building module, used to obtain a student model and a teacher model based on a token pruner, an attention head pruner and a basic model; the token pruner is used to evaluate and prune the importance of tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention head of the basic model; The model self-distillation module is used to train the student model and the teacher model based on the training samples and the objective function of the self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

[0048] Optionally, the token pruner is a lightweight module based on MLP; the lightweight module based on MLP includes a local policy network and a global policy network; the local policy network is used to evaluate the importance of a single token; and the global policy network is used to evaluate the importance of cross-modal tokens.

[0049] Optionally, a token pruning module is also included to evaluate the importance and prune the tokens input into the base model.

[0050] Optionally, the token pruning module includes: A token local importance analysis unit, used for inputting a target token into a local strategy network and calculating a local importance score of the target token; A cross-modal global representation acquisition unit is used to fuse and map the global representations of the visual modality and the textual modality to obtain a cross-modal global representation; A token global importance analysis unit, used to input the cross-modal global representation and the target token into the global policy network, and calculate the cross-modal global importance score of the target token; A token importance score determination unit, configured to obtain a final importance score of a target token based on a sum of a local importance score and a global importance score of the target token; The token pruning unit is used to prune or retain the target token based on the final importance score of the target token.

[0051] Optionally, the token local importance analysis unit includes: The token dimension reduction processing subunit is used to perform dimension reduction processing on the target token to obtain the target token after dimension reduction; A learnable parameter determination subunit is used to obtain the learnable parameters of the local policy network through stochastic gradient descent learning; The local analysis subunit is used to input the target token after dimension reduction into the local strategy network and calculate the local importance score of the target token.

[0052] Optionally, the global representation based on the visual modality and the textual modality is a corresponding category token.

[0053] Optionally, the token pruning unit includes: A token pruning mask calculation subunit, used for mapping the final importance score of the target token to between 0 and 1 based on a neural network threshold function to obtain a pruning mask of the target token; The token pruning subunit is used to retain the target token if the pruning mask of the target token is 1, otherwise, remove the target token.

[0054] Optionally, it also includes an attention head pruning module, which is used to evaluate and prune the attention heads of the base model; Optionally, the attention head pruning module also includes: Input the global representation of the current modality and the other modality into the attention pruner and calculate the importance score of the attention head of the current modality; Based on the neural network threshold function, the importance score of the attention head of the current modality is mapped to between 0 and 1 to obtain the pruning mask of the attention head of the current modality; If the pruning mask of the attention head is 1, the attention head of the current modality is retained, otherwise, the attention head of the current modality is pruned.

[0055] Optionally, also include: The objective function determination module of self-distillation is used to obtain the objective function of self-distillation based on the student model and teacher model parameters and the target loss function.

[0056] Optionally, the model self-distillation module includes: A prediction confidence determination unit, used to input the training sample into the teacher model to obtain the prediction confidence of the training sample in the teacher model; A sample descending order sorting unit, used to sort the training samples in descending order based on the prediction confidence of the training samples in the teacher model to obtain descending samples; The model parameter updating unit is used to extract target samples from descending samples and input them into the teacher model and the student model, and update the model parameters by batch gradient descent method according to the objective function of self-distillation to obtain the student model and the teacher model after parameter update.

[0057] Optionally, the output result of the student model after the parameter update is aligned with the output result of the teacher model.

[0058] Based on this, the device can achieve the following technical effects: (1) Combining token pruning and model structure pruning. The pruning method in this application includes screening the input tokens, reducing the number of tokens based on a learnable lightweight module, pruning the attention heads, and pruning the model structure to remove unimportant attention heads. Based on the hybrid paradigm, the computational complexity and parameter quantity of the model are minimized.

[0059] (2) Self-distillation training process. In the process of training the parameters of adaptive pruning, we adopt the self-distillation training method based on curriculum learning. The pruned model and the pruning model share the same parameters. The only difference is that the pruned model will activate the model pruner during the forward propagation process. In this way, the separate fine-tuning process of the teacher network in traditional knowledge distillation methods is avoided. Example 3

[0060] Please refer to Figure 4 , this embodiment provides an electronic device, which includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a network pruning method based on attention head and self-distillation at the logical level. Of course, in addition to software implementation methods, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0061] The network interface, processor and memory can be connected to each other through a bus system. The above bus can be divided into an address bus, a data bus, a control bus, etc.

[0062] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor.

[0063] The processor is used to execute the program stored in the above memory, and specifically perform: Step 102, based on a token pruner, an attention head pruner and a basic model, a student model and a teacher model are obtained; the token pruner is used to evaluate and prune the importance of tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention heads of the basic model; Step 104, training the student model and the teacher model based on the training samples and the objective function of self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

[0064] The processor may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of the processor hardware or by instructions in the form of software.

[0065] Based on the same invention, the embodiment of this specification also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1-Figure 2 The corresponding embodiment provides a network pruning method based on attention head and self-distillation.

[0066] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code.

[0067] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.

[0068] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between it and other embodiments.

[0069] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the specific order or sequential order shown in the process depicted in the drawings is not necessarily required to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A network pruning method based on attention head and self-distillation, characterized in that: include: Based on the token pruner, the attention head pruner and the basic model, a student model and a teacher model are obtained; the token pruner is used to evaluate and prune the importance of the tokens input into the basic model; the attention head pruner is used to evaluate and prune the importance of the attention head of the basic model; The student model and the teacher model are trained based on the training samples and the objective function of self-distillation to obtain an optimized student model; the token pruner and the attention head pruner in the student model are activated; the token pruner and the attention head pruner in the teacher model are frozen.

2. The method according to claim 1, characterized in that The token pruner is a lightweight module based on MLP; the lightweight module based on MLP includes a local policy network and a global policy network; the local policy network is used to evaluate the importance of a single token; the global policy network is used to evaluate the importance of cross-modal tokens.

3. The method according to claim 2, characterized in that The token pruner performs importance evaluation and pruning on the tokens of the input base model, including: Input the target token into the local policy network and calculate the local importance score of the target token; The global representations of the visual modality and the textual modality are fused and mapped to obtain a cross-modal global representation; Input the cross-modal global representation and the target token into the global policy network and calculate the cross-modal global importance score of obtaining the target token; Obtaining a final importance score of the target token based on the sum of the local importance score and the global importance score of the target token; The target token is pruned or retained based on its final importance score.

4. The method according to claim 3, characterized in that The step of inputting the target token into the local policy network and calculating the local importance score of the target token comprises: Performing dimension reduction processing on the target token to obtain the target token after dimension reduction; Obtain learnable parameters of the local policy network through stochastic gradient descent learning; The reduced target token is input into the local policy network to calculate the local importance score of the target token.

5. The method according to claim 3, characterized in that: The global representations of the visual modality and the textual modality are corresponding category tokens.

6. The method according to claim 3, characterized in that: The pruning or retaining of the target token based on the final importance score of the target token includes: Map the final importance score of the target token to between 0 and 1 based on the neural network threshold function to obtain the pruning mask of the target token; If the pruning mask of the target token is 1, the target token is retained, otherwise, the target token is removed.

7. The method according to claim 1, characterized in that The attention head pruner performs importance evaluation and pruning on the attention head of the base model including: Input the global representation of the current modality and the other modality into the attention pruner and calculate the importance score of the attention head of the current modality; Based on the neural network threshold function, the importance score of the attention head of the current modality is mapped to between 0 and 1 to obtain the pruning mask of the attention head of the current modality; If the pruning mask of the attention head is 1, the attention head of the current modality is retained, otherwise, the attention head of the current modality is pruned.

8. The method according to claim 1, characterized in that Before the student model and the teacher model are trained based on the training samples and the objective function of self-distillation to obtain the optimized student model, the following steps are also included: Based on the student model and teacher model parameters and the target loss function, the objective function of self-distillation is obtained.

9. The method according to claim 8, characterized in that The student model and the teacher model are trained based on the training samples and the objective function of self-distillation to obtain an optimized student model including: Input the training samples into the teacher model to obtain the prediction confidence of the training samples in the teacher model; Sort the training samples in descending order based on the prediction confidence of the training samples in the teacher model to obtain descending samples; Target samples are extracted from descending samples and input into the teacher model and student model, and the model parameters are updated by batch gradient descent method according to the objective function of self-distillation to obtain the student model and teacher model after parameter update.

10. The method according to claim 9, characterized in that The output result of the student model after the parameter update is aligned with the output result of the teacher model.

Images