A vector encoding method based on lifting method

Abstract

The present invention discloses a vector encoding method based on a lifting method, including a model training method and an encoding method. The model training method includes the following steps: S1, taking the trained original model as a teacher model T, and taking a plurality of lightweight models as student models; S2, performing distillation training on S1 according to the teacher-student mode; and generating a first mask; S3, training the Nth student model S N When , the Nth mask is generated, the sum of the results obtained by inputting the training data into the teacher model and the results obtained by inputting the same training data into the N-1 student models obtained by the previous N-1 trainings is subtracted, and the residual plus the Nth mask is used as the loss. Then, S is trained according to the teacher-student model. N Perform distillation training; S4, connect the trained student models in parallel to obtain the final model. The final model of this solution can be processed in parallel, which means it has a higher processing speed and reliable accuracy.

Description

Technical Field

[0001] The present invention relates to the acceleration of a deep vectorized model, and in particular to a vector encoding method based on a lifting method. Background Art

[0002] In the field of machine learning, data vectorization has always been a very important research direction. Both natural language vectorization and image vectorization play a pivotal role in various classification and recognition tasks. In addition to making vectors more representative, how to make the vectorization process faster and take up less resources is also an important topic. Existing vectorization methods are difficult to strike a balance between accuracy and speed, and in actual applications, they often have low efficiency or low accuracy. Summary of the invention

[0003] The present invention mainly solves the technical problem that it is difficult to strike a balance between accuracy and speed in the prior art, and provides a vector encoding method based on the lifting method that can be processed in parallel, that is, has a higher processing speed and reliable accuracy.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a vector encoding method based on a lifting method, including a model training method and an encoding method, wherein the model training method includes the following steps:

[0005] S1. Use the trained original model as the teacher model T and several lightweight models as student models.

[0006] S2, when training the first student model S1, S1 is distilled and trained according to the teacher-student mode, and the loss function is:

[0007]

[0008] The training goal is to minimize the loss and continue training without reducing it; B is a subset of the training data, L i is the i-th data in B. For example, in a natural language scenario, B is a set of natural language sentences, each of which is a piece of data; Li is the i-th data in B, that is, a sentence. S1(L i ) refers to L i The output after input to the student model S1, T(L i ) refers to L i The output after inputting into the teacher model T; and generating the first mask;

[0009] S3, train the Nth student model S NWhen , the Nth mask is generated, the result obtained by inputting the training data into the teacher model and the result obtained by inputting the same training data into the N-1 student models obtained by the previous N-1 trainings are added and subtracted, and then the residual plus the Nth mask is taken as the loss, and then S is trained according to the teacher-student model. N For distillation training, the loss function is:

[0010]

[0011] N>1,MASK N Add the Nth mask to the first N-1 residuals; the training goal is also to minimize the loss, and continue training without getting smaller;

[0012] S4, connect the trained student models in parallel to obtain the final model;

[0013] The encoding method is: input the original data into each student model in the final model to obtain K outputs, and process each output result using the corresponding mask MASK, that is, the output result of the p-th student model uses the p-th mask generated in the training, 1≤p≤K, to obtain K results, and add the K results to obtain the final encoded vector data, where K is the number of student models contained in the final model.

[0014] The number of student models contained in the final model is determined according to demand. The more student models there are, the better the vectorization effect will be, but the amount of calculation will be relatively larger, consuming more hardware resources.

[0015] The main purpose of the teacher-student model architecture is to compress deep learning models, which is a popular practice in the field of model distillation. In order to obtain better accuracy under deep learning, the trained network is often complex in structure. For some online prediction tasks, the complex model structure is not conducive to the rapid response requirements of online prediction tasks, so the demand for model compression arises. In this model framework, the teacher model structure is equivalent to the original complex deep neural network structure, and the student model is a lightweight network structure; therefore, the teacher model will have a higher prediction accuracy, and it will guide the student model to achieve the best model effect after simplifying the parameters.

[0016] Based on the above teacher-student architecture, we adopted a boosting method so that the student model is no longer a single one, but a group of weak models. During training, the student models are trained one by one in series. The Nth student model does not learn the output vector of the teacher model, but the residual between the output of the teacher model and the result of the model group composed of the first N-1 student models. The teacher model can be a large model with a complex structure and deep network that we have trained, or a large model with excellent results in the open source community. Such models usually perform well, but the network structure is complex, the inference speed is slow, and they are heavily dependent on graphics card resources. The method to measure the gap in output between the student model family and the teacher model can be the mean square error.

[0017] Preferably, the lightweight model is obtained by any one or more of the following three methods:

[0018] (1) Extracted from the teacher model, that is, randomly and non-repeatedly extract some deep network layers from the teacher model and retain their weights; (2) Select an open source pre-trained model, that is, select the BERT pre-trained model with a simple network structure and fast inference speed as the student model; (3) The language model trained from scratch, that is, select the BERT model initialized with random parameters as the student model.

[0019] Preferably, during encoding, each processor core runs a student model, and M student models are inferred simultaneously, where M is the smaller value of the number of processor cores and the number of student models included in the final model.

[0020] Preferably, the mask is generated by:

[0021] For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest of the positions to 0; for the Kth mask, set all positions to 1.

[0022] Preferably, the mask is generated by:

[0023] For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest to 0; for the Kth mask, first randomly set some positions of the mask to 1, and set the rest to 0, then add all the masks bitwise, and set all the positions that are 0 in the corresponding sum in the Kth mask to 1.

[0024] Preferably, the mask is generated by:

[0025] Each time, Q consecutive positions in the selection vector are set to 1 and the remaining positions are set to 0.

[0026] The positions of 1s in two adjacent masks can be continuous or discontinuous, with or without cross-overlap. For example, if the length of the vector is 10, we can generate H continuous masks. For example, if H=3, we can generate continuous masks without overlap like [1,1,1,0,0,0,0,0,0],[0,0,0,1,1,1,0,0,0,0],[0,0,0,0,0,0,1,1,1,1] or continuous masks with overlap like [1,1,1,1,1,0,0,0,0,0],[0,0,0,1,1,1,1,1,0,0],[0,0,0,0,0,1,1,1,1,1].

[0027] The mask can be understood as a vector with the same length as the model output vector. There are two ways to generate a mask, one is to generate the mask randomly (method 1 and method 2), and the other is to generate the mask non-randomly (method 3).

[0028] Preferably, after the mask is generated, a random number is replaced on the mask, specifically:

[0029] Select a digit in the mask. When this digit is 1, generate a random number greater than 0.9 and less than 1, and replace this digit with this random number. When this digit is 0, generate a random number greater than 0 and less than 0.1, and replace this digit with this random number. Replace the numbers in the mask one by one until all the numbers are replaced.

[0030] Generate a random number each time you replace it, so that the random numbers on each bit in the same mask will not be exactly the same. After replacing 1 and 0, each student model can learn different information as much as possible. You can also generate random numbers directly when generating the mask, instead of replacing them after generation.

[0031] The substantial effect of the present invention is that the teacher-student method is combined with the boosting method to perform multiple distillation training on the large model to obtain N weaker small models, and then these small models are combined into a model group close to the original large model effect in the inference stage to imitate the original large model effect. In this way, each small model is decoupled in the inference stage, and can be processed in parallel in engineering, achieving the effect of fast inference even on the CPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a model training method of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0034] Embodiment: A vector encoding method based on a lifting method in this embodiment includes a model training method and an encoding method. Figure 1 As shown, the model training method includes the following steps:

[0035] S1. Use the trained original model as the teacher model T and several lightweight models as student models.

[0036] S2, when training the first student model S1, S1 is distilled and trained according to the teacher-student mode, and the loss function is The training goal is to minimize the loss and continue training without reducing it; B is a subset of the training data, L i is the i-th data in B. For example, in a natural language scenario, B is a set of natural language sentences, each of which is a piece of data; Li is the i-th data in B, that is, a sentence. S1(L i ) refers to L i The output after input to the student model S1, T(L i ) refers to L i The output after inputting into the teacher model T; and generating the first mask;

[0037] S3, train the Nth student model S N When , the Nth mask is generated, the result obtained by inputting the training data into the teacher model and the result obtained by inputting the same training data into the N-1 student models obtained by the previous N-1 trainings are added and subtracted, and then the residual plus the Nth mask is taken as the loss, and then S is trained according to the teacher-student model. N For distillation training, the loss function is:

[0038]

[0039] N>1,MASK N Add the Nth mask to the first N-1 residuals; the training goal is also to minimize the loss, and continue training without getting smaller

[0040] S4, connect the trained student models in parallel to obtain the final model;

[0041] The encoding method is: input the original data into each student model in the final model to obtain K outputs, and process each output result using the corresponding mask MASK, that is, the output result of the p-th student model uses the p-th mask generated in the training, 1≤p≤K, to obtain K results, and add the K results to obtain the final encoded vector data, where K is the number of student models contained in the final model.

[0042] Take natural language processing as an example. Specifically, first, we need a set of corpus data (L0, L1, ..., L m ) to train the student model. This set of data does not need to be labeled, as long as it is a normal sentence in the required language. Each data L i That is, a sentence in natural language. In addition to training data, we also need a teacher model T that has been trained and has excellent results as a "benchmark" for the student model to learn. The teacher model can be a large model with complex structure and deep network that we have trained, or a large model with excellent results from the open source community. Such models usually perform well, but have complex network structures, slow inference speed, and are heavily dependent on graphics card resources.

[0043] The goal of our training is to continuously reduce the gap between the output of the student model family and the teacher model under the same input conditions, so that Here, T is the teacher model, S j is the student model, L is the input text, N is the number of student models, T() and S() represent the output vectors of the teacher model and the student model respectively. The method to measure the difference in output between the student model family and the teacher model can be the mean square error.

[0044] When training the first student model S1, we perform model distillation training according to the conventional teacher-student model. The method for calculating the error at this time can be expressed as: Among them, B is a small subset of the training data, that is, a small batch of training data.

[0045] When training the Nth student model S N When we have N-1 student models (S0, S1, ..., S N-1 ). The mean square error at this time can be expressed as:

[0046]

[0047] That is, the Nth student model S N Compared with the first N-1 models (S0, S1, ..., S N-1 ) is the mean square error of the residuals between the predictions of the model and the predictions of the teacher model. Each model learns the residuals of all the previous models, which is the idea of ​​the boosting method.

[0048] One of the key points of the improvement method is to make each weak model learn as much as possible different from other models. In order to make each student model learn as much different as possible, we add a random mask to the residuals of the teacher model and the first N-1 student models, and randomly change some residuals to 0. This can greatly enhance the learning ability of the student model after multiple rounds of learning. The above process can be expressed as:

[0049]

[0050] Among them, the function MASK() sets the number at the corresponding position to 0.

[0051] When selecting the base model for the student model, you can extract part of the teacher model, use an open source pre-trained model with a simpler structure, train a language model with a simpler structure from scratch, or use a mixture of the above methods.

[0052] Masks can be generated in a random or non-random way. Random methods include method 1:

[0053] For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest of the positions to 0; for the Kth mask, set all positions to 1.

[0054] Method 2:

[0055] For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest to 0; for the Kth mask, first randomly set some positions of the mask to 1, and set the rest to 0, then add all the masks bitwise, and set all the positions that are 0 in the corresponding sum in the Kth mask to 1.

[0056] Non-random methods include method 3:

[0057] Each time, Q consecutive positions in the selection vector are set to 1, and the remaining positions are set to 0. The positions set to 1 of two adjacent masks can be continuous or discontinuous, with or without cross-overlap.

[0058] In addition, in addition to directly selecting 0 or 1 as the value of the mask, you can also select a random number close to 0 or 1 as the value of each bit of the mask, or replace the mask with a random number after generating the mask, specifically:

[0059] Select a digit in the mask. When this digit is 1, generate a random number greater than 0.9 and less than 1, and replace this digit with this random number. When this digit is 0, generate a random number greater than 0 and less than 0.1, and replace this digit with this random number. Replace the numbers in the mask one by one until all the numbers are replaced.

[0060] After training, the student model family obtained is decoupled from each other, so it can be operated simultaneously during the reasoning process, and the final result is added before output. The reasoning process can be expressed as: or The number N of student models selected here can be determined according to the situation, as long as the selected student models start from S1 and are continuous during training.

[0061] Since the structure of each student model is simple, the inference speed can be very fast even on the CPU, so the parallel inference of N student models can also make full use of the computing power of multi-core CPUs.

[0062] The training and reasoning methods described in this article are a way of compressing and distilling large models, which can be applied but not limited to natural language vectorization models. The processing methods described in this article are also applicable to other vectorization models such as graphic vectorization.

[0063] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0064] Although the terminology such as teacher model, student model, and improvement method is used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A vector encoding method based on a lifting method, characterized in that: The method includes a model training method and an encoding method. The model training method includes the following steps: S1. Use the trained original model as the teacher model T and several lightweight models as student models. S2, when training the first student model S1, S1 is distilled and trained according to the teacher-student mode, and the loss function is: B is a set of natural language sentences, each sentence is a piece of data; L i is the i-th data in B, that is, a sentence; S1(L i ) refers to L i The output after input to the student model S1, T(L i ) refers to L i The output after inputting into the teacher model T; and generating the first mask; S3, train the Nth student model S N When , the Nth mask is generated, the result obtained by inputting the training data into the teacher model and the result obtained by inputting the same training data into the N-1 student models obtained by the previous N-1 trainings are added and subtracted, and then the residual plus the Nth mask is taken as the loss, and then S is trained according to the teacher-student model. N For distillation training, the loss function is: N>1,MASK N Add the Nth mask to the first N-1 residuals; S4, connect the trained student models in parallel to obtain the final model; The encoding method is: input the original data into each student model in the final model to obtain K outputs, process each output result with the corresponding mask to obtain K results, and add the K results to obtain the final encoded vector data, where K is the number of student models contained in the final model.

2. A vector encoding method based on a lifting method according to claim 1, characterized in that: The lightweight model is obtained by any one or more of the following three methods: (1) Extracted from the teacher model; (2) Selected open source pre-trained model; (3) Language model trained from scratch.

3. A vector encoding method based on a lifting method according to claim 1 or 2, characterized in that: During encoding, each processor core runs one student model, and M student models are inferred simultaneously, where M is the smaller of the number of processor cores and the number of student models included in the final model.

4. The vector encoding method based on the lifting method according to claim 1, characterized in that: The mask is generated by: For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest of the positions to 0; for the Kth mask, set all positions to 1.

5. The vector encoding method based on the lifting method according to claim 1, characterized in that: The mask is generated by: For the 1st to K-1th masks, randomly select some positions of the mask and set them to 1, and set the rest to 0; for the Kth mask, first randomly set some positions of the mask to 1, and set the rest to 0, then add all the masks bitwise, and set all the positions that are 0 in the corresponding sum in the Kth mask to 1.

6. The vector encoding method based on the lifting method according to claim 1, characterized in that: The mask is generated by: Each time, Q consecutive positions in the selection vector are set to 1 and the remaining positions are set to 0.

7. A vector encoding method based on a lifting method according to claim 4, 5 or 6, characterized in that: After the mask is generated, the mask is replaced with random numbers, specifically: Select a digit in the mask. When this digit is 1, generate a random number greater than 0.9 and less than 1, and replace this digit with this random number. When this digit is 0, generate a random number greater than 0 and less than 0.1, and replace this digit with this random number. Replace the numbers in the mask one by one until all the numbers are replaced.

Images