Model lightweight training method and device and medium

Abstract

The invention provides a model lightweight training method and device and a medium, and relates to the technical field of computers. The method comprises the following steps: acquiring sample data of a target scene; training an existing original teacher model and an original student model by using the sample data of the target scene to obtain a target teacher model and a first target student model of the target scene; and obtaining contribution degrees of channels, weights and / or neurons of the first target student model to the sample data of the target scene, and pruning the first target student model according to the contribution degrees to obtain a second target student model of the target scene. According to the invention, joint training is carried out based on the existing original teacher model and original student model, the target teacher model and the first target student model adapted to the target scene are obtained, then the contribution degree of each neuron and the like of the first target student model is evaluated, and pruning is carried out on the first target student model according to the contribution degree. And a second target student model with a lighter target scene is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to at least computer technology field, and particularly relates to a model lightening training method and device and medium. BACKGROUND

[0002] In actual application, in order to reduce the calculation resource consumption and deployment cost of model, there is a demand for model lightening. At present, there is a model distillation method, which uses a teacher model to obtain a lightened student model, but there is no technical solution for further lightening processing the student model for the target scene. SUMMARY

[0003] The present application provides a model lightening training method and device and medium to solve the technical problem of how to further lightening processing the student model for the target scene.

[0004] In the first aspect, the present application provides a model lightening training method, which comprises:

[0005] obtaining sample data of a target scene;

[0006] training an existing original teacher model and an original student model using the sample data of the target scene to obtain a target teacher model and a first target student model of the target scene;

[0007] obtaining the contribution of the channels, weights and / or neurons of the first target student model to the sample data of the target scene, and pruning the first target student model according to the contribution to obtain a second target student model of the target scene.

[0008] Further, the sample data of the target scene is obtained, specifically comprising:

[0009] obtaining business data of a business scene as input features in the sample data of the target scene, including at least one of an identity card image, a face image, device information, a business approach and a business time of a user handling a business;

[0010] obtaining a classification annotated according to a business scene task as a hard label in the sample data of the target scene, including at least one of an image quality label, a verification result label, an image category label and a risk level label.

[0011] Further, the sample data of the target scene is obtained, specifically comprising:

[0012] completing supervised fine-tuning training on the existing original teacher model using the sample data of the target scene to obtain a first target teacher model of the target scene;

[0013] The original student model is distilled and trained by combining the first target teacher model using sample data of the target scene to obtain a first target student model of the target scene.

[0014] Further, the original student model is distilled and trained by combining the first target teacher model using sample data of the target scene to obtain a first target student model of the target scene, specifically including in each round of distillation training process:

[0015] obtain the distillation training parameters of the current round, the distillation training parameters including: student model internal parameters, training control parameters, temperature coefficient t, t>1, and weight coefficients a, b and g of supervision loss, soft label distillation loss and intermediate layer feature distillation loss, a+b+g=1;

[0016] The softmax function of the output layer of the first target teacher model is added with the t of the current round to obtain the second target teacher model of the current round, and the softmax function of the output layer of the student model of the last round is divided into two kinds of adding the t of the current round and not adding the t to obtain the student model of the current round;

[0017] The input features in the sample data of the target scene are input into the student model and the second target teacher model of the current round to obtain the supervision loss L1 of the predicted output of the student model of the current round and the hard label in the sample data of the target scene, the soft label distillation loss L2 of the predicted output of the second target teacher model of the current round, and the intermediate layer feature distillation loss L3 of the intermediate layer features of the student model of the current round and the second target teacher model;

[0018] obtain the loss L of the current round L= aL1 + bL2 + gL3, if L is greater than a preset loss threshold, adjust the distillation training parameters of the next round and enter the next round of distillation training process, otherwise, the part of the student model of the current round whose softmax function of the output layer does not add t is the first target student model of the target scene.

[0019] Further, the supervision loss L1 of the predicted output of the student model of the current round and the hard label in the sample data of the target scene, and the soft label distillation loss L2 of the predicted output of the second target teacher model of the current round are obtained.

[0020] obtain the cross-entropy loss of the predicted output of the softmax function of the output layer of the student model of the current round and the hard label in the sample data of the target scene, which is the supervision loss L1;

[0021] The softmax function of the student model output layer of the current round is added to the prediction output of t of the current round, and the forward KL divergence or inverse KL divergence of the prediction output of the second target teacher model of the current round is obtained as a soft label distillation loss L2.

[0022] Further, the softmax function of the student model output layer of the current round is added to the prediction output of t of the current round, and the forward KL divergence or inverse KL divergence of the prediction output of the second target teacher model of the current round is obtained as a soft label distillation loss L2, specifically comprising:

[0023] When p(x) distribution does not tend to 0, the forward KL divergence of the prediction classification probability distribution of the softmax function of the student model output layer of the current round added to the t output of the current round is obtained according to the following formula:

[0024]

[0025] When p(x) distribution tends to 0, the inverse KL divergence of the prediction classification probability distribution of the softmax function of the student model output layer of the current round added to the t output of the current round is obtained according to the following formula:

[0026]

[0027] Wherein, p(x) is the input feature x in the sample data of the target scene in the prediction classification probability distribution output by the softmax function of the student model output layer of the current round added to the t output of the current round, is the prediction classification probability distribution output by the second target teacher model of the current round.

[0028] Further, the contribution of the neurons of the first target student model to the sample data of the target scene is obtained, and the first target student model is pruned according to the contribution to obtain the second target student model of the target scene, specifically comprising:

[0029] Set the current pruning mask for the neurons of the first target student model;

[0030] Obtain the current contribution of the neurons of the first target student model with the current pruning mask being 0 to the sample data of the target scene;

[0031] If the current contribution is less than the contribution threshold, delete the redundant neurons in the first target student model with the current pruning mask being 0;

[0032] Fine-tune the target student model after deleting the redundant neurons to obtain the second target student model of the target scene.

[0033] Further, the current contribution degree of the neurons of the first target student model with the current pruning mask of 0 to the sample data of the target scene is obtained, specifically comprising:

[0034] The current contribution degree of the neurons of the first target student model with the current pruning mask of 0 to the sample data of the target scene is obtained according to the following formula:

[0035]

[0036] Wherein: represents the sample data of the target scene; W={ }={w1,...,w d} represents all current neuron weights; i={1,...,d} represents the i th neuron of the first target student model; m={ }={m1,...,m d}∈{0,1} represents the pruning mask; represents the multiplication of each item; represents the loss of the prediction output of the first target student model after adding the current pruning mask to the hard label in the sample data of the target scene, which takes a positive value; respectively, three preset evaluation coefficients, which take positive values; represents the square of the L2 norm; represents the regularization term, which is used to punish the pruned neurons; x={ }={x1,...,x N} represents the input features in the sample data of the target scene; represents the activation value of the i th neuron on xj; represents the average activation value of the i th neuron on x; represents the prediction output of the first target teacher model, represents the prediction output of the first target student model after adding the current pruning mask.

[0037] In a second aspect, the present application provides a model lightening training device, the device comprising:

[0038] A sample module is configured to obtain sample data of a target scene;

[0039] A distillation module is connected to the sample module and is configured to train an existing original teacher model and an original student model using the sample data of the target scene to obtain a target teacher model and a first target student model of the target scene;

[0040] ​The pruning module is connected with the distillation module, and is configured to obtain a contribution degree of a channel, a weight and / or a neuron of the first target student model to sample data of the target scene, and perform pruning processing on the first target student model according to the contribution degree, to obtain a second target student model of the target scene.

[0041] In a third aspect, the present application provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the model lightening training method described above is implemented.

[0042] The present application provides a model lightening training method, device and medium. The method includes jointly training an existing original teacher model and an original student model to obtain a target teacher model and a first target student model suitable for a target scene, evaluating a contribution degree of each channel, weight and / or neuron of the first target student model, and pruning the first target student model according to the contribution degree to obtain a second target student model of the target scene, which is more lightened. The method can reduce the consumption of computing resources and the deployment cost of the model. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 FIG. 1 is a flowchart of a model lightening training method according to an embodiment of the present application;

[0044] Figure 2 FIG. 2 is a structural schematic diagram of a model lightening training device according to an embodiment of the present application;

[0045] Figure 3 FIG. 3 is a flowchart of another model lightening training method according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the skilled in the art better understand the technical solutions of the present application, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0047] It can be understood that the specific embodiments and drawings described herein are only used to explain the present application, but not to limit the present application.

[0048] It can be understood that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0049] It can be understood that, for the convenience of description, only parts related to the present application are shown in the drawings of the present application, and parts unrelated to the present application are not shown in the drawings.

[0050] It can be understood that each module and unit involved in the embodiments of the present application can only correspond to one entity structure, or can be composed of multiple entity structures, or multiple modules and units can be integrated into one entity structure.

[0051] It can be understood that the functions, steps marked in the flowcharts and block diagrams of the present application can occur in an order different from that marked in the drawings without conflict.

[0052] It can be understood that in the flowcharts and block diagrams of the present application, the architecture, functions and operations of possible implementations of the system, device, equipment and method according to the embodiments of the present application are shown. Each block in the flowchart or block diagram can represent a module, unit, program segment, code, which contains executable instructions for implementing the specified function. Moreover, each block or combination of blocks in the block diagram and flowchart can be implemented by a hardware-based device for implementing the specified function, or by a combination of hardware and computer instructions.

[0053] It can be understood that the modules and units involved in the embodiments of the present application can be implemented in software or hardware, for example, the modules and units can be located in a processor.

[0054] For the convenience of understanding, first, some technical terms that will appear in the embodiments are explained: the teacher model is usually a relatively mature or better performing model, which is used to generate pseudo labels or provide supervision signals; the student model is a model that is learning, which obtains knowledge from the teacher model; in the training process, the teacher model is first pre-trained using limited labeled data, and then uses this pre-trained knowledge to predict the pseudo labels of a large amount of unlabeled data; among them, the label in the labeled data is called a hard label, and the pseudo label output by the teacher model is called a soft label; the core idea of knowledge distillation is to convert the output (usually including class probability) of the teacher model into a softened probability distribution as the training target of the student model, and the student model not only learns the information in the original data, but also learns the knowledge in the teacher model; model pruning is a model compression technique, which aims to reduce the model size, computation and memory occupation, while maintaining the performance of the original model as much as possible; weight pruning removes individual weights (connections), usually based on the size of the weights (such as weights with smaller absolute values are pruned); channel pruning removes entire convolution channels (such as channels with smaller norm, smaller contribution to loss gradient, smaller sensitivity, etc.), maintaining the regularity of the model structure; neuron pruning removes entire neurons or nodes; layer pruning removes entire network layers (less common); channel pruning, neuron pruning and layer pruning all belong to structured pruning; hybrid pruning combines weight and structured pruning.

[0055] Embodiment 1:

[0056] As shown in Figure 1 The present application provides a model lightweight training method, which comprises:

[0057] S1, obtaining sample data of a target scene;

[0058] S2, training the existing original teacher model and the original student model by using sample data of the target scene to obtain a target teacher model and a first target student model of the target scene;

[0059] S3, obtaining a contribution degree of a channel, a weight and / or a neuron of the first target student model to sample data of the target scene, and pruning the first target student model according to the contribution degree to obtain a second target student model of the target scene.

[0060] In the embodiment, the method first performs joint training based on the existing original teacher model and the original student model to obtain a target teacher model and a first target student model adapted to the target scene, then evaluates the contribution degrees of the channels, the weights and / or the neurons of the first target student model, and prunes the first target student model according to the contribution degrees to obtain a second target student model of the target scene which is more lightweight, thereby reducing the calculation resource consumption and deployment cost of the model. Figure 1 As shown in the method, the method is applied to the device as shown in Figure 2 .

[0061] Specifically, the embodiment provides a lightweight training method for a multi-modal large model in an operator business scene by fusing distillation and pruning. With the wide application of large models in image recognition, behavior analysis and other tasks, their high performance is accompanied by huge calculation resource consumption and deployment cost, which is difficult to adapt to the demand for model lightweight and high efficiency in actual business. Especially in real scenes such as telecom operators, the model needs to process multi-modal data such as ID photos, face photos and user operation behaviors, and also needs to meet the requirements of privacy protection, real-time response and low-power deployment. Therefore, how to compress the model size and improve the inference efficiency while ensuring the accuracy of the model has become a key challenge for the application of current models.

[0062] The embodiment focuses on extracting the intermediate layer features, soft labels and implicit information of the teacher model by white-box knowledge distillation technology, and guiding the student model to learn by combining the multi-task loss function, and at the same time, optimizing the model structure by sparsification through pruning technology, so as to maintain relatively optimal performance while significantly reducing the computing power and resource consumption. The embodiment focuses on the training process of compressing and optimizing deep models, migrates knowledge by distilling large models and combining soft labels, hard labels and intermediate layer features, so that small models still have strong business modeling ability in a limited computing resource environment.

[0063] The embodiment overcomes the existing model training and deployment technology, and solves the following main problems in processing image and behavior data in the operator business system: 1) poor generalization ability of business scenarios, traditional models are mostly trained by relying on data collected in fixed scenarios, and it is difficult to effectively adapt to the complexity of user behavior in different business processes, especially in actual business environment with different image quality and various operation behaviors, the model performance fluctuates greatly, and the generalization ability is insufficient. 2) High deployment cost of large models, although large pre-training models have strong recognition ability, but there are problems of high inference delay and large calculation resource consumption in deploying on resource-limited devices, which are not suitable for edge devices or lightweight deployment scenarios. 3) Insufficient knowledge transfer and low training efficiency, in the existing model distillation practice, only the final classification output is used for knowledge transfer, and the intermediate layer feature expression is ignored, so that the student model cannot sufficiently understand and absorb the deep semantic understanding of the teacher model in the training process, which limits the performance of the compressed model. 4) The distillation process is unstable and prone to training anomalies, the existing technology usually uses forward KL divergence (Kullback-Leibler Divergence, KLD, also known as relative entropy) as the loss function in the distillation training, but when the teacher model output probability distribution is biased (such as some classification probability close to 0), it may cause gradient explosion problem, resulting in unstable training of student model and even divergence. 5) The model pruning method is rough, which affects the final accuracy, the existing pruning technology mostly uses static scoring method, lacks important evaluation mechanism for dynamic adjustment of different task characteristics, is easy to mis-prune key neurons, and then causes model accuracy to decline, and cannot ensure the performance integrity while keeping the structure lightweight.

[0064] In an embodiment, S1, sample data of a target scene is obtained, specifically including:

[0065] The business data of the business scenario is obtained as the input feature in the sample data of the target scene, including at least one of the identity card image, the face image, the device information, the handling way and the handling time of the user handling the business;

[0066] The classification annotated according to the business scenario task is obtained as the hard label in the sample data of the target scene, including at least one of the image quality label, the verification result label, the image category label and the risk level label.

[0067] In the embodiment, as Figure 3As shown, the method comprises: collecting original data in the process of user handling business from the operator business system in real time or offline manner, the original data including but not limited to: user-submitted identity card image, face image, device information, operation behavior log and timestamp data and other multi-modal data; performing field-level desensitization processing on the original data, including replacing personal sensitive information such as name, ID number, phone number, face image, desensitization methods including hash coding, occlusion processing or pseudo-data generation; encoding image data into tensor input format, using standard image preprocessing procedures, including scaling, normalization, alignment and edge enhancement, etc.; encoding behavior data into a multi-dimensional vector, constructing a unified input structure and storing it in a feature warehouse; building a label system, setting image quality labels (blur / clear), business verification labels (pass / fail), image categories (certificate / face), risk level labels (low / medium / high) according to business objectives, for downstream intelligent recognition tasks. In addition, the method provided in the embodiment can also be applied to other business scenarios such as finance and medical treatment, and the specific business scenarios do not affect the implementation of the method, as long as the original data of the corresponding scene can be obtained.

[0068] In an embodiment, S2, the existing original teacher model and the original student model are trained using sample data of the target scene to obtain a target teacher model and a first target student model of the target scene, specifically including:

[0069] The existing original teacher model is trained using sample data of the target scene to obtain a first target teacher model of the target scene;

[0070] The existing original student model is distilled trained using sample data of the target scene in combination with the first target teacher model to obtain a first target student model of the target scene.

[0071] In the embodiment, as Figure 3As shown, the method comprises: aligning the above-mentioned original data sample features, using a public Internet pre-training large model (original teacher model) for fine-tuning training (sft, supervised fine-tuning), to obtain a teacher model (first target teacher model) with target business scenario modeling capability; the business data includes but is not limited to certificate images, face images, business handling behavior data and the like collected when the user handles the package; a small parameter quantity large model (original student model) with moderate model parameters is selected; the model is loaded using default training parameters, wherein the default training parameters include but are not limited to learning rate, batch, temperature smoothing, alpha and beta and gamma, training batch and the like; in addition, the teacher model is used to infer the business data to generate a prediction output as a soft label, while retaining the original manual label corresponding to the business data as a hard label; the above-mentioned data is placed in the training process, and the outputs of the teacher model and the student model are temperature smoothed using a softmax function with a temperature coefficient, the temperature coefficient T is a positive real number greater than 1, and is used to improve the generalization ability of the soft label.

[0072] In an embodiment, the existing original student model is distilled and trained with the sample data of the target scene in combination with the first target teacher model to obtain a first target student model of the target scene, specifically comprising in each round of distillation training process:

[0073] obtaining the distillation training parameters of the current round, the distillation training parameters including: student model internal parameters, training control parameters, temperature coefficient t, t>1, and weight coefficients alpha, beta and gamma of supervision loss, soft label distillation loss and intermediate layer feature distillation loss, alpha + beta + gamma = 1;

[0074] adding the t of the current round to the softmax function of the output layer of the first target teacher model to obtain the second target teacher model of the current round, and adding the t of the current round to the softmax function of the output layer of the student model of the last round or not adding the t of the current round to the softmax function of the output layer of the student model of the last round to obtain the student model of the current round;

[0075] inputting the input features in the sample data of the target scene into the student model of the current round and the second target teacher model of the current round to obtain the supervision loss L1 of the prediction output of the student model of the current round and the hard label in the sample data of the target scene, the soft label distillation loss L2 of the prediction output of the second target teacher model of the current round, and the intermediate layer feature distillation loss L3 of the intermediate layer features of the student model of the current round and the intermediate layer features of the second target teacher model;

[0076] The loss L of the current round is obtained, and if the loss L is greater than a preset loss threshold, the distillation training parameters of the next round are adjusted and the next round of distillation training process is entered, otherwise, the part of the student model without the softmax function of the output layer of the current round is taken as the first target student model of the target scene.

[0077] In the embodiment, there is a distillation loss L of the student model and the teacher model in the training process, L2 is a soft label distillation loss of the student model and the teacher model, L1 is a real loss (supervision loss) of the student model and the hard label classification; L3 is an intermediate layer feature distillation loss; by jointly optimizing the intermediate layer feature distillation loss, the soft label distillation loss and the supervision loss, the student model is guided to learn the knowledge expression ability of the teacher model layer by layer, so as to maintain high generalization performance under the premise of small parameter quantity. The soft label distillation loss term is used to measure the difference between the output of the student model and the soft label generated by the teacher model; the supervision loss term is used to measure the difference between the output of the student model and the original hard label of the business data; the intermediate layer feature migration loss term is used to measure the difference between the student model and the teacher model in the intermediate layer semantic representation; the three loss terms are combined into a total loss function in a weighted manner, and the weight coefficients corresponding to each loss term are α, β and γ, respectively, satisfying α + β + γ = 1, and the weight can be dynamically adjusted through a dynamic threshold or a validation set performance. It can be understood that the training process also includes adjusting the parameters of the student model itself (student model internal parameters) and the like, which are conventional training methods.

[0078] In an embodiment, the prediction output of the student model of the current round and the supervision loss L1 of the hard label in the sample data of the target scene, and the soft label distillation loss L2 of the prediction output of the second target teacher model of the current round are obtained, specifically including:

[0079] The prediction output of the softmax function of the output layer of the student model of the current round without adding t is obtained, and the cross-entropy loss of the hard label in the sample data of the target scene is the supervision loss L1.

[0080] The prediction output of the softmax function of the output layer of the student model of the current round is obtained, and the forward KL divergence or inverse KL divergence of the prediction output of the second target teacher model of the current round is the soft label distillation loss L2.

[0081] In the embodiment, the soft label distillation loss L2 is obtained by , which represents the probability distribution of the teacher model under the t smoothing degree for task x (input feature in sample data) , and the soft label distillation loss L2 is obtained by , which represents the probability distribution of the student model under the t smoothing degree for task x ; wherein the parameter t represents the knowledge smoothness (temperature coefficient); because the user data has hard labels, and the teacher model will generate soft label information in the distillation training process, for example, in the hard label case, the data is strictly classified: cat: 1, dog: 0 state; but in the distillation process, the teacher model will label the data with soft labels, such as dark message knowledge: cat: 0.8, dog 0.2 classification state, the distillation process adopts the white box knowledge distillation method for distillation training; the parameter smooth value t will make a joint loss smooth calculation (soft label smoothing) of hard label knowledge and soft label knowledge; the dark knowledge is to adjust the Softmax temperature parameter T to improve the smoothness of the probability distribution between classes, and introduce the KL divergence or cross entropy metric in the distillation loss to strengthen the absorption and expression of the student model to the fine-grained output features of the teacher model.

[0082] In an embodiment, the softmax function of the student model output layer of the current round is added to the prediction output of t of the current round, and the forward KL divergence or inverse KL divergence of the prediction output of the second target teacher model of the current round is the soft label distillation loss L2, specifically including:

[0083] When p(x) does not tend to 0, the forward KL divergence of the prediction classification probability distribution of the softmax function of the student model output layer of the current round added to the t output of the current round, and the prediction classification probability distribution output by the second target teacher model of the current round is obtained according to the following formula:

[0084]

[0085] When p(x) tends to 0, the inverse KL divergence of the prediction classification probability distribution of the softmax function of the student model output layer of the current round added to the t output of the current round, and the prediction classification probability distribution output by the second target teacher model of the current round is obtained according to the following formula:

[0086]

[0087] Wherein p(x) is the prediction classification probability distribution of the input feature x in the sample data of the target scene in the softmax function of the student model output layer of the current round added to the t output of the current round, is the prediction classification probability distribution of the input feature x in the sample data of the target scene in the output of the second target teacher model of the current round.

[0088] In the embodiment, the forward KL divergence, i.e., KL(P||Q), is the similarity of the probability distribution of the student model and the teacher model for the task; from the formula, when the p(x) and q(x) distributions of the teacher model tend to be consistent, it indicates that the student model and the teacher model are well trained, and the student model is better; but in the extreme case, when the p(x) distribution tends to 0 (the teacher contribution is 0 for the task), the distribution of q(x) greater than 0 will cause the gradient explosion p(x)log (p(x)) / (q(x))----> positive infinity. The reverse KL divergence, i.e., KL(Q||P), is the similarity of the probability distribution of the student model and the teacher model for the task; but the reverse KL divergence pays more attention to the confidence region of the student model, and prompts the student model to be consistent with the teacher model output as much as possible in the high prediction region; specifically, when there are some extreme cases, for example, the teacher model output probability distribution is flat or has high uncertainty, the reverse kl divergence is more likely to converge.

[0089] In an embodiment, in S3, the contribution of the neurons of the first target student model to the sample data of the target scene is obtained, and the first target student model is pruned according to the contribution to obtain a second target student model for the target scene, specifically including:

[0090] The current pruning mask is set for the neurons of the first target student model;

[0091] The current contribution of the neurons of the first target student model with the current pruning mask being 0 to the sample data of the target scene is obtained;

[0092] If the current contribution is less than the contribution threshold, the redundant neurons in the first target student model with the current pruning mask being 0 are deleted;

[0093] The target student model after deleting the redundant neurons is fine-tuned to obtain a second target student model for the target scene.

[0094] In the embodiment, as Figure 3As shown, the method comprises: after the model distillation is completed, performing a structure pruning operation on the student model to realize the sparsification of the model structure and reduce the calculation resource consumption of the model in the inference process; the pruning operation comprises at least one of channel pruning, weight pruning or neuron pruning, and the pruning strategy can be based on a preset importance scoring mechanism or by dynamically evaluating the neuron activation contribution during the training process. The details of the above pruning can be implemented as follows: for a fully connected layer or a high-dimensional embedding layer, the average activation value or response variance of each neuron on the training data is calculated to measure its actual contribution to the model output. According to the contribution score, a threshold value θ is set, and the neurons below θ are marked as "redundant units". Remove all input and output connections corresponding to the neuron, and adjust the dimension parameters of the front and rear layers to ensure the consistency of the network structure. Fine-tune the model after the structure adjustment, and adjust the weight distribution of other neurons to make up for the information loss caused by the removed neurons.

[0095] In an embodiment, the current contribution of the neurons of the first target student model with the current pruning mask of 0 to the sample data of the target scene is obtained, specifically comprising:

[0096] The current contribution of the neurons of the first target student model with the current pruning mask of 0 to the sample data of the target scene is obtained according to the following formula:

[0097]

[0098] Wherein: represents the sample data of the target scene; W={ }={w1,...,w d} represents the current all neuron weights; i={1,...,d} represents the i-th neuron of the first target student model; m={ }={m1,...,m d}∈{0,1} represents the pruning mask; represents the term-by-term multiplication; represents the loss of the prediction output of the first target student model after adding the current pruning mask to the hard label in the sample data of the target scene, which takes a positive value; respectively, three preset evaluation coefficients, which take positive values; represents the square of the L2 norm; represents a regularization term for penalizing the pruned neurons; x={ }={x1,...,x N} represents the input features in the sample data of the target scene; represents the activation value of the i-th neuron on xj; ​denotes the average activation value of the i-th neuron on x; denotes the prediction output of the first target teacher model, denotes the prediction output of the first target student model with the current pruning mask.

[0099] In the above detailed formula, for pruning neurons that do not bring large real losses, for pruning neurons with weights close to 0, for pruning neurons with small changes in activation values for different samples, for pruning neurons that do not bring large differences from the teacher model, the lightweight model suitable for the target scene is effectively simplified by the combination of the four, and each term maintains the same sign, for example, all take positive values. It can be understood that each term takes a negative value, and the judgment basis is changed to greater than negative θ. Such changes are the same technical solutions as the present embodiment.

[0100] In summary, the present embodiment 1: based on the multi-modal data (such as ID card pictures, face pictures, and behavior logs) collected in the actual business scenarios of operators, combined with pre-training large models for fine-tuning training, a teacher model suitable for business needs is generated to improve the practicality and generalization ability of the model in real scenarios; by using the soft labels generated by the teacher model and the original manual labels (hard labels) for training at the same time, the student model can better learn the knowledge expression of the teacher model while being lightweight, and the model precision is improved; a method for dynamically evaluating the importance of neurons based on activation values and response changes is proposed, which effectively identifies and prunes redundant neurons, thereby reducing the model calculation cost while maintaining the model performance; a joint optimization method that combines soft label loss, supervision loss, and intermediate layer feature transfer loss is designed, and the weights of each loss term can be dynamically adjusted according to the performance of the validation set.

[0101] The present embodiment 1 can be implemented based on the current mainstream deep learning framework (PyTorch), and all steps including data desensitization, image / behavior encoding, label construction, teacher model fine-tuning, student model distillation, triple loss joint training, and pruning processing, etc. have a standardized implementation process; the technical solution is aimed at image recognition and risk control tasks of the intelligent business handling system of communication operators, especially suitable for high-frequency online application scenarios such as identity verification, face comparison, and device risk identification, and has been preliminarily verified in an internal test environment. By building a closed-loop process from raw data collection → desensitization → preprocessing → label design → teacher / student collaborative training → pruning → deployment → feedback, it is suitable for complex identity verification, risk identification, embedded AI (Artificial Intelligence, Artificial Intelligence), etc. scene.

[0102] Embodiment 2:

[0103] As shown in Figure 2 The present application provides a model lightweight training device, which comprises:

[0104] A sample module 1 is configured to obtain sample data of a target scene.

[0105] A distillation module 2 is connected to the sample module 1 and configured to train an existing original teacher model and an original student model by using the sample data of the target scene, so as to obtain a target teacher model and a first target student model of the target scene.

[0106] A pruning module 3 is connected to the distillation module 2 and configured to obtain a contribution degree of a channel, a weight and / or a neuron of the first target student model to the sample data of the target scene, and perform pruning processing on the first target student model according to the contribution degree, so as to obtain a second target student model of the target scene.

[0107] In an embodiment, the sample module 1 specifically comprises:

[0108] A business unit is configured to obtain business data of a business scene as input features in the sample data of the target scene, including at least one of an identity card image, a face image, device information, a business approach and a business time of a user.

[0109] A labeling unit is connected to the business unit and configured to obtain a classification of the business data labeled according to a business scene task as a hard label in the sample data of the target scene, including at least one of an image quality label, a verification result label, an image category label and a risk level label.

[0110] In an embodiment, the distillation module 2 specifically comprises:

[0111] A supervised fine-tuning training unit is configured to complete supervised fine-tuning training on the existing original teacher model by using the sample data of the target scene, so as to obtain a first target teacher model of the target scene.

[0112] A distillation training unit is connected to the supervised fine-tuning training unit and configured to perform distillation training on the existing original student model by using the sample data of the target scene in combination with the first target teacher model, so as to obtain a first target student model of the target scene.

[0113] In an embodiment, the distillation training unit specifically comprises, in each round of distillation training process:

[0114] a distillation parameter unit, configured to obtain distillation training parameters of a current round, the distillation training parameters comprising: student model internal parameters, training control parameters, a temperature coefficient t, t>1, and weight coefficients a, b and g of a supervision loss, a soft label distillation loss and an intermediate layer feature distillation loss, a+b+g=1;

[0115] a model updating unit, connected with the distillation parameter unit, configured to add the t of the current round to a softmax function of an output layer of a first target teacher model to obtain a second target teacher model of the current round, and divide a softmax function of an output layer of a student model of a previous round into two parts, one of which adds the t of the current round and the other of which does not add the t, to obtain a student model of the current round;

[0116] a loss calculation unit, connected with the model updating unit, configured to input input features in sample data of a target scene into the student model of the current round and the second target teacher model of the current round to obtain a supervision loss L1 of a predicted output of the student model of the current round and hard labels in the sample data of the target scene, a soft label distillation loss L2 of a predicted output of the second target teacher model of the current round, and an intermediate layer feature distillation loss L3 of intermediate layer features of the student model of the current round and intermediate layer features of the second target teacher model;

[0117] a judgment and selection unit, connected with the loss calculation unit, configured to obtain a loss L= aL1 + bL2 + gL3 of the current round, and if the L is greater than a preset loss threshold, adjust the distillation training parameters of a next round and enter a distillation training process of the next round, otherwise, obtain a part of the student model of the current round in which the softmax function of the output layer does not add the t as a first target student model of the target scene.

[0118] In an embodiment, the loss calculation unit specifically comprises:

[0119] a first loss unit, configured to obtain a cross entropy loss of a predicted output of the softmax function of the output layer of the student model of the current round which does not add the t and hard labels in the sample data of the target scene, as the supervision loss L1;

[0120] a second loss unit, configured to obtain a forward KL divergence or inverse KL divergence of a predicted output of the softmax function of the output layer of the student model of the current round which adds the t of the current round and a predicted output of the second target teacher model of the current round, as the soft label distillation loss L2.

[0121] In an embodiment, the second loss unit specifically comprises:

[0122] a forward divergence unit configured to obtain, when the p(x) distribution does not tend to 0, a forward KL divergence between a prediction classification probability distribution of the t output of the current round of the student model output layer of the softmax function and a prediction classification probability distribution output by the second target teacher model of the current round according to the following formula:

[0123]

[0124] a reverse divergence unit configured to obtain, when the p(x) distribution tends to 0, a reverse KL divergence between a prediction classification probability distribution of the t output of the student model output layer of the softmax function of the current round and a prediction classification probability distribution output by the second target teacher model of the current round according to the following formula:

[0125]

[0126] wherein p(x) is a prediction classification probability distribution of an input feature x in the sample data of the target scene in the student model output layer of the softmax function of the current round of the t output, is a prediction classification probability distribution of the input feature x in the sample data of the target scene in the output of the second target teacher model of the current round.

[0127] In an embodiment, the pruning module 3 specifically comprises:

[0128] a mask setting unit configured to set a current pruning mask for a neuron of the first target student model;

[0129] a contribution calculation unit connected with the mask setting unit and configured to obtain a current contribution of the neuron of the first target student model with the current pruning mask being 0 to the sample data of the target scene;

[0130] a redundancy judging unit connected with the contribution calculation unit and configured to delete a redundant neuron in the first target student model with the current pruning mask being 0 if the current contribution is less than a contribution threshold;

[0131] a fine-tuning training unit connected with the redundancy judging unit and configured to perform fine-tuning training on the target student model from which the redundant neuron is deleted to obtain the second target student model of the target scene.

[0132] In an embodiment, the contribution calculation unit is specifically configured to:

[0133] obtain the current contribution of the neuron of the first target student model with the current pruning mask being 0 to the sample data of the target scene according to the following formula:

[0134]

[0135] wherein: sample data representing a target scene; W={ }={w1,...,w d} represents all current neuron weights; i={1,...,d} represents the i-th neuron of the first target student model; m={ }={m1,...,m d}∈{0,1} represents a pruning mask; represents term-by-term multiplication; represents the loss of the prediction output of the first target student model after adding the current pruning mask on the hard label in the sample data of the target scene, taking a positive value; respectively, three preset evaluation coefficients, taking positive values; represents the square of the L2 norm; represents a regularization term, used to punish pruned neurons; x={ }={x1,...,x N} represents input features in the sample data of the target scene; represents the activation value of the i-th neuron on xj; = represents the average activation value of the i-th neuron on x; represents the prediction output of the first target teacher model, represents the prediction output of the first target student model after adding the current pruning mask.

[0136] Embodiment 3:

[0137] The application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is run by a processor, the model lightening training method as described in Embodiment 1 is implemented, or the model lightening training device as described in Embodiment 2 is implemented.

[0138] The computer readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, computer program elements or other data. The computer readable storage medium includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile discs (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

[0139] In addition, the present application can also provide a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the model lightweight training method as described in Embodiment 1. The computer device can be the model lightweight training device as described in Embodiment 2.

[0140] Among them, the memory is connected with the processor, and the memory can adopt flash memory or read-only memory or other storage, and the processor can adopt central processing unit or single-chip microcomputer.

[0141] Embodiments 1-3 of the present application provide a model lightweight training method, device and medium, first joint training based on the existing original teacher model and the original student model to obtain the target teacher model and the first target student model suitable for the target scene, then evaluate the contribution degree of each channel, weight and / or neuron of the first target student model, and prune the first target student model according to the contribution degree to obtain the second target student model which is more lightweight for the target scene, so as to reduce the calculation resource consumption and deployment cost of the model.

[0142] It can be understood that the above embodiments are only exemplary embodiments adopted to illustrate the principles of the present application, but the present application is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present application, and these modifications and improvements are also regarded as the protection scope of the present application.

Claims

1. A lightweight model training method, characterized in that, The method includes: Obtain sample data for the target scene; The existing original teacher model and original student model are trained using sample data from the target scenario to obtain the target teacher model and the first target student model for the target scenario. Obtain the contribution of the channels, weights, and / or neurons of the first target student model to the sample data of the target scene, and prune the first target student model according to the contribution to obtain the second target student model of the target scene.

2. The method according to claim 1, characterized in that, Obtain sample data for the target scene, specifically including: Business data from the business scenario is used as input features in the sample data of the target scenario, including at least one of the following: the user's ID card image, facial image, device information, processing method, and processing time. Obtain hard labels from the sample data of the target scenario, which are classified according to the business scenario task and labeled as business data. These labels include at least one of the following: image quality label, verification result label, image category label, and risk level label.

3. The method according to claim 1, characterized in that, The existing original teacher model and original student model are trained using sample data from the target scenario to obtain the target teacher model and the first target student model for the target scenario, specifically including: Supervised fine-tuning training of the existing original teacher model is performed using sample data from the target scenario to obtain the first target teacher model for the target scenario. The existing original student model is distilled and trained using sample data from the target scenario, combined with the first target teacher model, to obtain the first target student model for the target scenario.

4. The method according to claim 3, characterized in that, Using sample data from the target scenario, the existing original student model is combined with the first target teacher model for distillation training to obtain the first target student model for the target scenario. Specifically, this includes the following steps in each round of distillation training: Obtain the distillation training parameters for this round, which include: student model intrinsic parameters, training control parameters, temperature coefficient t (t > 1), and weight coefficients α, β, and γ for supervised loss, soft label distillation loss, and intermediate layer feature distillation loss (α + β + γ = 1). The softmax function of the output layer of the first target teacher model is given the value t from the current round to obtain the second target teacher model of the current round. The softmax function of the output layer of the previous student model is divided into two types: one with the value t from the current round and one without the value t, to obtain the student model of the current round. Input features from the sample data of the target scene into the student model and the second target teacher model in this round, to obtain the supervision loss L1 of the predicted output of the student model in this round and the hard label in the sample data of the target scene, the soft label distillation loss L2 of the predicted output of the second target teacher model in this round, and the intermediate layer feature distillation loss L3 of the intermediate layer features of the student model in this round and the intermediate layer features of the second target teacher model. Obtain the current round loss L = αL1 + βL2 + γL3. If L is greater than the preset loss threshold, adjust and obtain the distillation training parameters for the next round and enter the next round of distillation training. Otherwise, obtain the student model of the softmax function of the output layer of this round without adding t as the first target student model of the target scene.

5. The method according to claim 4, characterized in that, Obtain the supervised loss L1 of the student model's predicted output and the hard labels in the target scene's sample data, and the soft label distillation loss L2 of the teacher model's predicted output in the second target scenario. Specifically, this includes: The softmax function of the student model output layer in this round, without adding the predicted output of t, is obtained as the cross-entropy loss between the softmax function and the hard labels in the sample data of the target scene, which is the supervision loss L1. The forward or backward KL divergence between the softmax function of the student model output layer in this round and the predicted output of the second objective teacher model in this round is obtained and added to the predicted output of t in this round. This is the soft label distillation loss L2.

6. The method according to claim 5, characterized in that, The softmax function of the student model output layer in this round is added to the predicted output of t in this round, and the forward or backward KL divergence between it and the predicted output of the second objective teacher model in this round is used as the soft label distillation loss L2, which specifically includes: When the p(x) distribution does not tend to 0, the positive KL divergence between the predicted classification probability distribution of the softmax function of the student model output layer in this round and the predicted classification probability distribution of the second objective teacher model output in this round is obtained according to the following formula: When the p(x) distribution approaches 0, the inverse KL divergence between the predicted classification probability distribution of the softmax function added to the output layer of the student model in this round and the predicted classification probability distribution of the output of the second objective teacher model in this round is obtained according to the following formula: Where p(x) is the predicted classification probability distribution of the input feature x in the sample data of the target scene, which is added to the softmax function of the output layer of the student model in this round, and t is used in this round. It is the predicted classification probability distribution of the input feature x in the sample data of the target scene in this round of the second target teacher model output.

7. The method according to any one of claims 1-6, characterized in that, The contribution of neurons in the first target student model to the sample data of the target scene is obtained. Based on the contribution, the first target student model is pruned to obtain the second target student model for the target scene. Specifically, this includes: Set the current pruning mask for the neurons of the first target student model; Obtain the current contribution of the neurons of the first target student model with the current pruning mask of 0 to the sample data of the target scene; If the current contribution is less than the contribution threshold, delete redundant neurons in the first target student model with a current pruning mask of 0; Fine-tuning training is performed on the target student model with redundant neurons removed to obtain a second target student model for the target scenario.

8. The method according to claim 7, characterized in that, Obtain the current contribution of the neurons in the first target student model with a current pruned mask of 0 to the sample data of the target scene, specifically including: The current contribution of the neurons in the first target student model with a current pruned mask of 0 to the sample data of the target scene is obtained according to the following formula: in: Represents sample data for the target scene; W={ }={w1,...,w d } represents the weights of all neurons; i={1,...,d} represents the i-th neuron of the first target student model; m={ }={m1,...,m d }∈{0,1} represents a pruning mask; This indicates that the terms are multiplied one by one. This represents the loss of the first target student model's prediction output after adding the current pruning mask to the hard labels in the sample data of the target scene, and is taken as a positive value; These are three preset evaluation coefficients, all taking positive values; Represents the square of the L2 norm; This represents a regularization term used to penalize pruned neurons; x={ }={x1,...,x N } represents the input features in the sample data of the target scene; This represents the activation value of the i-th neuron on xj; = This represents the average activation value of the i-th neuron on x; This represents the predicted output of the first-objective teacher model. This represents the predicted output of the first target student model after adding the current pruning mask.

9. A lightweight model training device, characterized in that, The device includes: The sample module is used to acquire sample data of the target scene; The distillation module, connected to the sample module, is used to train the existing original teacher model and original student model using sample data from the target scenario, in order to obtain the target teacher model and the first target student model for the target scenario. The pruning module, connected to the distillation module, is used to obtain the contribution of the channels, weights, and / or neurons of the first target student model to the sample data of the target scene, and to prune the first target student model according to the contribution to obtain the second target student model of the target scene.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the lightweight model training method as described in any one of claims 1-8.

Images