Training method of discrete variational autoencoder, autoencoder, device and medium
By employing a discrete variational autoencoder training method and utilizing the feature mapping and initialization strategy of the teacher model, the problems of information loss and resource consumption in the knowledge distillation process of the student model are solved, thus achieving efficient model training and deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-24
- Publication Date
- 2026-03-17
AI Technical Summary
In existing knowledge distillation methods, the student model cannot directly learn from the output layer of the teacher model, resulting in information loss. Furthermore, random initialization requires a large amount of data and computational resources, increasing training costs and latency.
The discrete variational autoencoder training method is adopted. The teacher model is used as the encoding layer of the variational autoencoder. The CodeBook and MLP layers are used for feature mapping and fitting. The parameters of the embedding layer and MLP layer of the student model are initialized to avoid information loss and improve the convergence speed.
It effectively avoids information loss during the distillation process, improves model convergence speed and computational resource utilization, and reduces training data and resource input.
Smart Images
Figure CN114861604B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, autoencoder, device, and medium for a discrete variational autoencoder. Background Technology
[0002] In recent years, artificial intelligence (AI) and deep learning technologies have developed rapidly, making significant progress in the two major areas of natural language processing (NLP) and computer vision (CV). With the popularization of AI technologies such as intelligent customer service and autonomous driving, artificial intelligence has penetrated into all aspects of our daily learning, work and life.
[0003] In the field of NLP, as natural language processing has entered the era of pre-trained models, the scale of these models is growing rapidly. The "Source" model, released by Inspur AI Research Institute in 2021, reached a scale of 245.7 billion, becoming the largest pre-trained Chinese language model to date. "Source" possesses powerful text generation capabilities, achieving a new record on the Clue-Zero-Shot leaderboard. Currently, the "Source" model demonstrates excellent performance in applications such as classical Chinese (poetry) generation, modern Chinese (novel continuation writing, etc.), dialogue, and translation. However, with the increasing size of NLP language models, deploying these massive models in real-world applications with limited resources presents a significant challenge.
[0004] Taking a language model with 13 billion characters as an example, during inference, the model parameters alone require at least 26GB of GPU memory, while actual inference requires around 40GB. Furthermore, even with the current optimized inference logic, generating 32 tokens on two NVIDIA A100 GPUs still requires at least 3 seconds of latency. These GPU memory limitations and inference latency severely restrict the deployment and application of downstream tasks.
[0005] In specific tasks, the current industry practice is knowledge distillation, which involves using a large model as the teacher model and designing a smaller student model (the "small model") for knowledge transfer training. Many experiments have shown that training the large model first followed by knowledge distillation typically results in higher accuracy compared to directly training a small model. The small model obtained through knowledge distillation significantly reduces memory requirements and inference latency, facilitating deployment in downstream tasks. However, ensuring that the distilled small model performs comparably to the large model in specific tasks, and designing the loss function and distillation strategy for the small model, remains a major research focus.
[0006] For language models, a common knowledge distillation principle diagram is as follows: Figure 1As shown in Figure 1, the core of knowledge distillation lies in the design of the loss function. Common losses fall into three main categories: one is learning the distribution of the teacher model's output (soft-target), with common calculation methods including KL divergence and MSE (mean squared error); another is the student's own loss; and the third is learning the output of certain layers or modules within the teacher model. Furthermore, in existing methods, the parameters of the small model (student) are often randomly initialized, and the teacher model learns from scratch. Meanwhile, the large model (teacher) has learned a vast amount of knowledge during pre-training on massive amounts of text. Therefore, the distillation process of the small model (student) requires a large amount of data for training to achieve good results. However, existing knowledge distillation methods have the following drawbacks:
[0007] I. Regarding the third type of loss design mentioned above, since the student model and teacher model have different dimensions, the student model cannot directly learn from the output (a certain layer) of the teacher model. For example... Figure 1 As shown, assume that the x-th layer of the student model learns the output of the y-th layer of the teacher model, where x < M and y < N.
[0008] The current mainstream loss design is as follows:
[0009] Loss hid =g(h S W,h T )
[0010] Among them, Loss hid The loss function designed for the student's x-th layer to learn from the teacher's y-th layer output; g(x) represents different loss functions, such as MSE (mean squared error), KL divergence, etc.; h S h represents the dimension of the student model. T h represents the dimension of the teacher model. S <h T .
[0011] Due to h S <h T Therefore, the output of layer M of the student model will pass through an MLP (fully connected layer with parameter matrix W), mapped to the same dimension as the teacher, before loss calculation. However, when the output of layer x of the student model is passed to the next layer, it is still passed to the dimension h before processing by the fully connected layer. S In other words, during training, in order to reduce the loss... hidAs the size decreases, the MLP layer learns some information from the teacher's output, but this information is not passed to the next layer of the student model and is not used in the inference process, thus resulting in information loss.
[0012] Second, since the parameters of the teacher model contain a large amount of pre-trained knowledge, if the student model uses random initialization for distillation, it will require a large amount of textual knowledge for training, which undoubtedly increases the training cost and slows down the convergence process of the student model. Summary of the Invention
[0013] In view of this, the present invention proposes a training method, autoencoder, device and medium for discrete variational autoencoders. By training the discrete variational autoencoder, fixed MLP layer weights and the embedding matrix of the student model can be obtained, which can effectively avoid information loss during the distillation process, improve the convergence speed of the model, reduce the input of training data and training resources, and greatly improve the utilization of computing resources.
[0014] To achieve the above objectives, one aspect of the present invention provides a training method for a discrete variational autoencoder, specifically including the following steps:
[0015] The teacher model is used as the encoding layer of the variational autoencoder;
[0016] In the encoding layer, teacher features corresponding to the vocabulary of the student model are obtained based on the teacher model;
[0017] In the CodeBook lookup layer of the discrete variational autoencoder, the feature representation of the student model dimension is obtained based on the CodeBook and the teacher features corresponding to the student model vocabulary.
[0018] In the decoding layer of the discrete variational autoencoder, the feature representation of the student model dimension is fitted based on the MLP layer, and the fitting result is inversely mapped based on the Emebedding layer to output the distribution probability on the vocabulary. The loss between the predicted value and the true value is calculated based on the distribution probability.
[0019] In some implementations, obtaining teacher features corresponding to the student model vocabulary based on the teacher model includes:
[0020] The sample is input into the teacher model to output teacher features, wherein the sample is a text sequence;
[0021] The teacher features are encoded twice with the vocabulary weight matrix of the MLP layer to obtain the feature representation of the teacher model dimension.
[0022] In some implementations, obtaining the feature representation of the student model dimension based on the CodeBook and the teacher features corresponding to the student model vocabulary includes:
[0023] Perform a matrix multiplication operation between the CodeBook and the teacher features corresponding to the student model vocabulary to obtain the feature representation of the student model dimension.
[0024] In some implementations, fitting the feature representation of the student model dimension based on the MLP layer includes:
[0025] The feature representation of the student model dimension is calculated by combining it with the dimension weight matrix of the MLP layer to obtain the feature representation of the teacher model dimension.
[0026] In some implementations, the method further includes:
[0027] The feature representation of the student model dimension obtained from the CodeBook lookup layer is used to initialize the parameters of the Embedding layer of the knowledge distillation student model; and / or
[0028] The feature representation of the teacher model dimension obtained in the decoding layer is used for parameter initialization of the knowledge distillation MLP layer.
[0029] In some implementations, inverse mapping of the fitting results based on the Embedding layer to output the distribution probability on the vocabulary includes:
[0030] Based on the weights of the Embedding layer of the teacher model, the feature representation of the teacher model dimension is inversely mapped to obtain the distribution probability of the output on the vocabulary.
[0031] In some implementations, calculating the loss between the predicted and actual values based on the probability distribution includes:
[0032] Based on the probability distribution of the output on the vocabulary, select the token with the highest probability.
[0033] The loss for this training is obtained by performing loss calculations on the token at the position with the highest probability and the corresponding token in the input sample.
[0034] Another aspect of this invention provides a discrete variational autoencoder, comprising:
[0035] The coding layer is configured to use the teacher model as the coding layer of the variational autoencoder;
[0036] The encoding layer is also configured to obtain teacher features corresponding to the vocabulary of the student model based on the teacher model;
[0037] The CodeBook lookup layer is configured to obtain a feature representation of the student model dimension based on the CodeBook and the teacher features corresponding to the student model vocabulary.
[0038] The decoding layer is configured to fit the feature representation of the student model dimension based on the MLP layer, and to perform inverse mapping on the fitting result based on the Emebedding layer to output the distribution probability on the vocabulary, and to calculate the loss between the predicted value and the true value based on the distribution probability.
[0039] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing a computer program executable on the processor, the computer program performing the steps of the method described above when executed by the processor.
[0040] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.
[0041] The present invention has at least the following beneficial technical effects: Firstly, the scheme of the present invention uses a discrete variational autoencoder to obtain the initialization strategy for the MLP layer parameters. During the distillation training process, these MLP layer parameters are fixed and used as a mapping from the student layer output features to the teacher layer output features, thus avoiding information loss and improving the convergence speed of the distillation process and the accuracy of the student model. Secondly, the discrete variational autoencoder is used to obtain the initialization strategy for the student model's embedding layer parameters. The teacher model is used as the encoding layer of the variational autoencoder, and the teacher model's embedding layer is used for token ID mapping, fully utilizing the already trained teacher model and achieving fast convergence of the discrete variational autoencoder. Thirdly, the codebook of the discrete variational autoencoder is used for initializing the student model's embedding layer parameters, effectively improving the convergence speed of the distillation process and greatly reducing resource consumption. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.
[0043] Figure 1 A diagram illustrating the principle of knowledge distillation;
[0044] Figure 2 This is a block diagram of an embodiment of the training method for the discrete variational autoencoder provided by the present invention;
[0045] Figure 3 A flowchart illustrating an embodiment of training based on a discrete independent variable encoder provided by the present invention;
[0046] Figure 4 A schematic diagram of an embodiment of the discrete variational autoencoder provided by the present invention;
[0047] Figure 5 A schematic diagram of the structure of an embodiment of the computer device provided by the present invention;
[0048] Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.
[0050] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.
[0051] In existing knowledge distillation methods, on the one hand, the layer outputs of the student model are passed through an MLP layer for dimensionality upscaling before being fitted to the layer outputs of the teacher model, and the parameters of the MLP are continuously updated during training. It is certain that because the MLP layer continuously learns and updates, it acquires knowledge from the teacher model, but the student model does not use this knowledge in subsequent training and inference processes.
[0052] On the other hand, the parameter initialization of the student model is either random initialization, or distillation training is performed from scratch; or a large amount of data and computing resources are used to pre-train a small model to achieve parameter initialization of the small model, which will result in high resource and cost consumption.
[0053] Based on the above objectives, a first aspect of the present invention proposes an embodiment of a training method for a discrete variational autoencoder. For example... Figure 2 As shown, it includes the following steps:
[0054] S10. Use the teacher model as the encoding layer of the variational autoencoder;
[0055] S20. In the encoding layer, teacher features corresponding to the vocabulary of the student model are obtained based on the teacher model.
[0056] S30. In the CodeBook lookup layer of the discrete variational autoencoder, the feature representation of the student model dimension is obtained based on the CodeBook and the teacher features corresponding to the student model vocabulary.
[0057] S40. In the decoding layer of the discrete variational autoencoder, the feature representation of the student model dimension is fitted based on the MLP layer, and the fitting result is reverse mapped based on the Embedding layer to output the distribution probability on the vocabulary, and the loss between the predicted value and the true value is calculated based on the distribution probability.
[0058] In steps S10 and S20, the teacher model is used as the Encoder part (encoding layer) of the variational autoencoder, and the Embedding layer of the teacher model is used to map the token ID. This can make full use of the trained teacher model and quickly achieve the convergence of the discrete variational autoencoder.
[0059] The specific process is as follows:
[0060] 1. The input is a text sequence. After passing through the teacher model, the probability distribution of the teacher model on the vocabulary is obtained, with a size of [bs, seq]. len ,vocab t ], where vocab t is the vocabulary size of the teacher model, bs is the batch size, and seq_len is the number of tokens after the input text is converted.
[0061] 2. The output of the teacher model is input into the MLP layer for secondary encoding, resulting in an output with dimensions [bs, seq].len ,vocab s ], where vocab s This refers to the vocabulary size of the student model. This step maps the probability distribution of the teacher in the vocabulary to the probability distribution of the student in the vocabulary. Generally, the vocabulary size is... t equals vocab s .
[0062] In step S30, CodeBook is the size [vocab] s hid s The matrix W codebook The output of the discrete variational autoencoder from the previous step is then multiplied by a matrix to obtain the Latent Code, a feature representation of the student model dimension, with a size of [bs, seq]. len hid s ], hid s This represents the dimension of the student model. In subsequent knowledge distillation, this dimension can be used to initialize the parameters of the student model's embedding layer. Using the codebook of the discrete variational autoencoder for initializing the student model's embedding layer parameters can effectively improve the convergence speed of the distillation process and significantly reduce resource consumption.
[0063] In step S40, a fixed MLP and the loss of the weighted discrete variational autoencoder are obtained through training the discrete variational autoencoder. The specific process is as follows:
[0064] 1. Input the Latent Code into the MLP layer for fitting, and obtain the size [bs, seq] len hid t ](hid t The output (where the dimensionality is the size of the teacher model) will extract the features from the hidden array. s Upgrade to HID t This allows us to obtain the fitted teacher features. Additionally, the dimension [hid] of the MLP layer weights W... s hid t This can be used as the initialization of MLP layer parameters for fitting the outputs of the teacher and student models during the distillation process.
[0065] 2. Perform a reverse mapping of the output from the previous step using the Embedding layer to obtain a size of [bs, seq]. len ,vocab tThe output of [] is processed by the SoftMax layer to obtain the probability distribution of the output on the vocabulary. The position with the highest probability is taken as its predicted token-id, and loss is calculated with the input, where loss is the cross-entropy.
[0066] Use the obtained MLP weights to perform Figure 1 The initialization of MLP layer parameters in loss3 can effectively avoid information loss during the distillation process, while also improving the convergence speed of the model, reducing the input of training data and training resources, and greatly improving the utilization of computing resources.
[0067] After training convergence, the weights of the CodeBook can be used as parameters to initialize the student model's embedding layer; the parameters of the Decoder part of the MLP can be used as parameters of the mapping MLP from the student model layer output to the teacher layer output, and these parameters can be frozen during distillation training without being updated, thus avoiding information loss.
[0068] This invention proposes a training method using a Discrete Variational Autoencoder (DVA). Firstly, the DVA is used to obtain the initialization strategy for the MLP layer parameters. These MLP layer parameters are fixed during the distillation training process and used as a mapping from the student layer output features to the teacher layer output features, thus avoiding information loss and improving the convergence speed and accuracy of the student model. Secondly, the DVA is used to obtain the initialization strategy for the student model's embedding layer parameters. The teacher model is used as the encoding layer of the DVA, and its embedding layer is used for token ID mapping, fully utilizing the pre-trained teacher model to achieve rapid convergence of the DVA. Thirdly, the DVA's codebook is used for initializing the student model's embedding layer parameters, effectively improving the convergence speed of the distillation process and significantly reducing resource consumption.
[0069] In some implementations, obtaining teacher features corresponding to the student model vocabulary based on the teacher model includes:
[0070] The sample is input into the teacher model to output teacher features, wherein the sample is a text sequence;
[0071] The teacher features are encoded twice with the vocabulary weight matrix of the MLP layer to obtain the feature representation of the teacher model dimension.
[0072] like Figure 3 The diagram shows a flowchart of training based on a discrete independent variable encoder.
[0073] At the coding layer ( Figure 3 The Encoder section in the code performs the following steps:
[0074] The input (i.e., the sample) is a text sequence with a size of [bs, seq_len], where bs is the batch size and seq_len is the number of tokens after the input text is converted.
[0075] The input, after passing through the teacher model, yields teacher features, which can be expressed by the following formula:
[0076] h T =Teacher(x)
[0077] Where x represents the input token, Teacher represents the teacher model, and h T This represents the output of the teacher network, also known as the teacher feature, with a size of [bs, seq]. len ,vocab t ], where vocab t This represents the vocabulary size of the teacher model's vocab.
[0078] The teacher features, after secondary encoding by an MLP layer, can be expressed as follows:
[0079] h E1 =MLP(W, h) T )
[0080] Where W is the vocabulary weight matrix of the MLP layer, and its size is [vocab t ,vocab s ], where vocab s The vocabulary size of the student network is denoted as . The teacher features are encoded twice through an MLP layer to map the probability distribution of the teacher in the vocabulary to the probability distribution of the student in the vocabulary.
[0081] Under normal circumstances, vocab t equals vocab s However, it is not limited to this.
[0082] After passing through this layer, the output h is obtained. E1 The dimensions are [bs, seq] len ,vocab s ].
[0083] In some implementations, obtaining the feature representation of the student model dimension based on the CodeBook and the teacher features corresponding to the student model vocabulary includes:
[0084] Perform a matrix multiplication operation between the CodeBook and the teacher features corresponding to the student model vocabulary to obtain the feature representation of the student model dimension.
[0085] like Figure 3 As shown, in the CodeBook table lookup layer ( Figure 3 (In the CodeBook table lookup section) Perform the following steps:
[0086] CodeBook is of size [vocab] s hid s The matrix W codebook hid s If the student outputs a feature dimension, then the Latent Code for the feature representation of the student model dimension is:
[0087] h ls =W codebook h E1
[0088] Among them, Latent Code h ls The size of the [bs, seq] is len hid s ].
[0089] Based on the above process, the dimension of the teacher output feature was changed from hid. t Dimensionality reduced to HID s .
[0090] In the above process, by using matrix W codebook From the dimension of hid t Compressed to a dimension of hid s It can be used in subsequent distillation processes, such as... Figure 1 The parameters of the embedding layer of the student model shown are initialized.
[0091] In some implementations, fitting the feature representation of the student model dimension based on the MLP layer includes:
[0092] The feature representation of the student model dimension is calculated by combining it with the dimension weight matrix of the MLP layer to obtain the feature representation of the teacher model dimension.
[0093] like Figure 3 As shown, in the decoding layer ( Figure 3 The Decoder section in the code performs the following steps:
[0094] Latent Code h ls After fitting through an MLP layer, the result is expressed as follows:
[0095] h D1 =MLP(W, h) ls )
[0096] The output h after fitting is... D1 The size is [bs, seq] len hid t This can be understood as the fitted teacher features.
[0097] At this point, the dimension of the MLP layer weight W is [hid] s hid t ], can be used to Figure 1 The loss3 calculation in the MLP layer parameter initialization.
[0098] In some implementations, the method further includes:
[0099] The feature representation of the student model dimension obtained from the CodeBook lookup layer is used to initialize the parameters of the Embedding layer of the knowledge distillation student model; and / or
[0100] The feature representation of the teacher model dimension obtained in the decoding layer is used for parameter initialization of the knowledge distillation MLP layer.
[0101] In some implementations, inverse mapping of the fitting results based on the Embedding layer to output the distribution probability on the vocabulary includes:
[0102] Based on the weights of the Embedding layer of the teacher model, the feature representation of the teacher model dimension is inversely mapped to obtain the distribution probability of the output on the vocabulary.
[0103] In some implementations, calculating the loss between the predicted and actual values based on the probability distribution includes:
[0104] Based on the probability distribution of the output on the vocabulary, select the token with the highest probability.
[0105] The loss for this training is obtained by performing loss calculations on the token at the position with the highest probability and the corresponding token in the input sample.
[0106] like Figure 3 As shown, in the decoding layer ( Figure 3 After obtaining the fitted teacher features from the Decoder part, the following steps are performed:
[0107] Fit teacher features h D1 The inverse mapping of the Embedding layer is expressed by the following formula:
[0108] h D2 =Emb(W t h D1 )
[0109] Among them W t The embedding layer weights of the teacher model have a size of [vocab] t hid t Output h D2 The size is [bs, seq] len ,vocab t After performing the SoftMax operation, the probability distribution of the output on the vocabulary is obtained. The position with the highest probability is taken as its predicted token-id, and the predicted token is obtained. The loss is calculated between the token and the corresponding input, where the loss is the cross-entropy.
[0110] The above describes the training process of the Discrete Variational Autoencoder proposed in this invention. After the Discrete Variational Autoencoder training converges, the weights of the CodeBook can be used as parameters to initialize the student model embedding layer. The parameters of the Decoder part of the MLP can be used as parameters to initialize the mapping MLP from the student model layer output to the teacher layer output. These parameters are frozen during the distillation training process and are not updated to avoid information loss.
[0111] In summary, this invention proposes a strategy for initializing student model parameters by training a discrete variational autoencoder. Before distillation, fixed MLP weights and the student model's embedding matrix are obtained through training the discrete variational autoencoder. During distillation, the trained embedding matrix is used to initialize the parameters of the student model's embedding layer, and the fixed MLP weights are used to initialize the MLP layer parameters in loss3. This approach effectively avoids information loss during distillation, improves the model's convergence speed, reduces the input of training data and resources, and significantly improves the utilization of computational resources.
[0112] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 4 As shown, an embodiment of the present invention also provides a discrete variational autoencoder 10, comprising:
[0113] Encoding layer 11, wherein the encoding layer 11 is configured to use the teacher model as the encoding layer of a variational autoencoder;
[0114] The encoding layer 11 is also configured to obtain teacher features corresponding to the vocabulary of the student model based on the teacher model;
[0115] CodeBook lookup layer 12 is configured to obtain a feature representation of the student model dimension based on the CodeBook and the teacher features corresponding to the student model vocabulary;
[0116] Decoding layer 13 is configured to fit the feature representation of the student model dimension based on the MLP layer, and to perform inverse mapping on the fitting result based on the Emebedding layer to output the distribution probability on the vocabulary, and to calculate the loss between the predicted value and the true value based on the distribution probability.
[0117] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 5 As shown, an embodiment of the present invention also provides a computer device 30, which includes a processor 310 and a memory 320. The memory 320 stores a computer program 321 that can be run on the processor. When the processor 310 executes the program, it performs the steps of the method described above.
[0118] The memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the discrete variational autoencoder training method described in the embodiments of this application. The processor executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in the memory, thereby implementing the discrete variational autoencoder training method of the above-described method embodiments.
[0119] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0120] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer-readable storage medium 40, which stores a computer program 410 that, when executed by a processor, performs the methods described above.
[0121] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.
[0122] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.
[0123] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. The sequence numbers of the disclosed embodiments of this invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.
[0124] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.
[0125] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.
Claims
1. A method for training a discrete variational autoencoder, the method comprising: The application relates to a method for training a student model based on a teacher model. The teacher model is used as an encoding layer of a variational autoencoder; In the encoding layer, teacher features corresponding to a vocabulary of the student model are obtained based on the teacher model; In a CodeBook lookup layer of the discrete variational autoencoder, feature representation of the student model dimension is obtained based on the CodeBook and the teacher features corresponding to the vocabulary of the student model; In a decoding layer of the discrete variational autoencoder, the feature representation of the student model dimension is fitted based on an MLP layer, and the fitted result is de-mapped based on an Emebedding layer to output distribution probability on the vocabulary, and a loss of a predicted value and an actual value is calculated based on the distribution probability; After the training of the discrete variational autoencoder converges, the weight of the CodeBook is used as the parameter initialization of the embedding layer of the student model, and the parameter of the MLP in the decoding layer is used as the parameter of the mapping MLP output by the output layer of the teacher model.
2. The method of claim 1, wherein, The teacher features corresponding to the vocabulary of the student model are obtained based on the teacher model, which comprises: The sample is input into the teacher model to output the teacher features, wherein the sample is a text sequence; The teacher features are twice encoded and calculated with the vocabulary weight matrix of the MLP layer to obtain the feature representation of the student model dimension.
3. The method of claim 1, wherein, The feature representation of the student model dimension is obtained based on the CodeBook and the teacher features corresponding to the vocabulary of the student model, which comprises: The CodeBook and the teacher features corresponding to the vocabulary of the student model are subjected to matrix multiplication to obtain the feature representation of the student model dimension.
4. The method of claim 1, wherein, The feature representation of the student model dimension is fitted based on the MLP layer, which comprises: The feature representation of the student model dimension is calculated with the dimension weight matrix of the MLP layer to obtain the feature representation of the teacher model dimension.
5. The method of claim 4, wherein, The application further comprises: The feature representation of the student model dimension obtained in the CodeBook lookup layer is used for parameter initialization of the Emebedding layer of the knowledge distillation student model; and / or The feature representation of the teacher model dimension obtained in the decoding layer is used for parameter initialization of the knowledge distillation MLP layer.
6. The method of claim 4, wherein, The fitted result is de-mapped based on the Emebedding layer to output the distribution probability on the vocabulary, which comprises: The feature representation of the teacher model dimension is de-mapped and calculated based on the weight of the Emebedding layer of the teacher model to obtain the distribution probability output on the vocabulary.
7. The method of claim 6, wherein, The loss of the predicted value and the actual value is calculated based on the distribution probability, which comprises: According to the distribution probability output on the vocabulary, a token with the maximum probability is selected. The token at the position of the maximum probability is loss calculated with the corresponding token in the input sample.
8. A discrete variational autoencoder, comprising: Comprising: an encoding layer configured to take the teacher model as an encoding layer of a variational autoencoder; the encoding layer is further configured to obtain teacher features corresponding to a vocabulary of a student model based on the teacher model; a CodeBook lookup layer configured to obtain feature representations of a student model dimension based on a CodeBook and the teacher features corresponding to the vocabulary of the student model; a decoding layer configured to fit the feature representations of the student model dimension based on an MLP layer, and to de-map the fitting result based on an Emebedding layer to output a distribution probability on the vocabulary, and to calculate a loss of a predicted value and a true value based on the distribution probability; after the training of the discrete variational autoencoder converges, the weights of the CodeBook are initialized as parameters of an embedding layer of the student model, and the parameters of the MLP in the decoding layer are output to the parameters of the mapping MLP output by the teacher layer.
9. A computer device comprising: at least one processor; and a memory storing a computer program executable on the processor, wherein the processor executes the program to perform the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to perform the steps of the method of any one of claims 1 to 7.