Language model construction method combining pre-training and energy model

By combining pre-trained language models with energy models, an energy-based language model training framework is constructed, which solves the exposure bias and label bias problems of pre-trained language models and achieves better natural language processing performance.

CN118940747APending Publication Date: 2024-11-12TSINGHUA UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411112502.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing pre-trained language models suffer from problems such as exposure bias and label bias, and energy models have not received sufficient attention in natural language processing.

Method used

By combining a pre-trained language model and an energy model, an energy-based language model training framework is constructed. Training is performed using methods such as noise contrastive estimation and maximum likelihood estimation. The pre-trained language model is used as the skeleton and noise model to improve the model structure and training method.

Benefits of technology

The constructed energy-based language model outperforms ordinary pre-trained language models, avoids label bias, and improves model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940747A_ABST
    Figure CN118940747A_ABST
Patent Text Reader

Abstract

The invention provides a pre-training and energy model combined language model construction method, which comprises the following steps of: constructing an energy-based language model by taking a pre-training language model as a skeleton; and selecting a common language model as a noise model or a proposal model, and carrying out energy-based language model training to obtain a final language model. A pre-training language model is combined with an energy-based language model, the structure and the training method of the model are constructed, three energy model structures with the pre-training language model as a skeleton are provided, and the three structures can utilize original parameters of the pre-training language model and can also utilize the original parameters of the energy-based language model. Common problems, such as label bias, of a common language model can be avoided through a training mode of an energy model; the performance of the energy-based language model trained by the training framework constructed by the invention exceeds that of a common pre-training language model after fine tuning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to a method for constructing a language model combining pre-training and an energy model. Background Art

[0002] Language Model (LM) plays an important role in natural language processing. Its task is to predict the probability of a sentence appearing in all natural languages. In recent years, the emergence of pre-trained language models has greatly increased the upper limit of language models. A pre-trained language model refers to a language model that has been trained with large-scale data in advance. The parameters of this language model can effectively improve the performance of the model on downstream tasks. Currently, commonly used pre-trained language models can be divided into two categories: autoregressive models and non-autoregressive models. An autoregressive model refers to a model in which the task faced during training and testing is to predict the probability of a word from left to right; a non-autoregressive model refers to a model in which the task faced is to predict the probability of a missing word through context. These two types of pre-trained language models have been extremely widely used and have performed very well in a variety of natural language processing tasks.

[0003] Energy based model (EBM) is a generative model introduced from statistical physics. For each sample, the energy model can output the probability of the sample in the entire sample space. For high-dimensional data, the normalization factor required to calculate the probability is often difficult to calculate. In this case, the energy model can only output an unnormalized probability. At present, energy models have applications in fields such as images and texts, but have not received sufficient attention.

[0004] The existing mainstream pre-trained language models are all locally normalized language models, that is, the language model keeps the predicted probability of each word normalized, and the probability of a sentence is equal to the product of the probabilities of each word in the sentence. This locally normalized sequence model will bring about problems such as exposure bias and label bias. Summary of the invention

[0005] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a language model construction method that combines pre-training and energy models. By basing the pre-trained language model on the energy model, the model architecture and training method of previous work are improved, thereby constructing a complete energy-based language model training framework, so that the performance of the energy model exceeds that of the mainstream pre-trained language model.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A method for constructing a language model combining pre-training and an energy model, comprising:

[0008] Using the pre-trained language model as the framework, an energy-based language model is constructed;

[0009] Select a common language model as a noise model or a proposal model, train the energy-based language model, and obtain the final language model.

[0010] Furthermore, the method for constructing a language model combining pre-training and an energy model also includes:

[0011] Construct training data. The training data is in text format. Each piece of data is a sentence of natural language and does not require annotation.

[0012] Furthermore, the pre-trained language model is used as a skeleton, and a corresponding linear layer is added to the pre-trained language model so that the model output is mapped to a scalar space as energy to construct an energy structure. Establishing an energy-based language model includes:

[0013] The energy function uses the autoregressive pre-trained language model as the skeleton, and sums the logits corresponding to the target token at each position to obtain the energy, where token refers to the symbol unit after the natural language is split in the computer, and logit refers to the unnormalized vector output by the classification head of the neural network; it can be expressed as:

[0014]

[0015] Among them, E θ (x) represents the energy function, θ is its parameter, x i represents the i-th token of sentence x, T represents the length of sentence x, and f θ represents an autoregressive model.

[0016] Furthermore, the pre-trained language model is used as a skeleton, and a corresponding linear layer is added to the pre-trained language model so that the model output is mapped to a scalar space as energy to construct an energy structure. Establishing an energy-based language model includes:

[0017] The energy function uses a non-autoregressive model as the skeleton. The latent variable after the model encodes the sentence is sent to a linear layer network to obtain a scalar as energy, which is expressed as follows:

[0018]

[0019] Among them, E θ (x) represents the energy function, θ is its parameter, enc θ (x) represents a non-autoregressive encoder, whose output is a latent variable.

[0020] Furthermore, the pre-trained language model is used as a skeleton, and a corresponding linear layer is added to the pre-trained language model so that the model output is mapped to a scalar space as energy to construct an energy structure. Establishing an energy-based language model includes:

[0021] The energy function uses a non-autoregressive pre-trained language model as the skeleton, uses the language head of the pre-trained masked language model, and uses the logit output of the language head at each token position to define the energy, which can be expressed as:

[0022]

[0023] Among them, E θ (x) represents the energy function, θ is its parameter, g θ (x) represents the masked language model, whose output is the unnormalized logit. Token refers to the symbolic unit after the natural language is split in the computer, and logit refers to the unnormalized vector output by the classification head of the neural network.

[0024] Furthermore, the selecting a common language model as a noise model or a proposal model and performing energy-based language model training includes:

[0025] The training energy model is converted into a binary classification problem using noise contrast estimation. Noise contrast estimation uses a noise distribution to provide suitable noise samples for training. φ (x) indicates that its ultimate optimization goal is:

[0026]

[0027] Where v represents the ratio of the number of noise samples to the number of true samples, represents the unnormalized probability, E θ (x) represents the energy function and 0 is its parameter.

[0028] Furthermore, the selecting a common language model as a noise model or a proposal model and performing energy-based language model training includes:

[0029] The maximum likelihood estimation is used to directly optimize the log-likelihood of the energy model. By derivation and simplification of the energy function, the optimization problem is transformed into a problem of sampling from the energy model. The formula is as follows:

[0030]

[0031] in, is an item that needs to be sampled from the energy model, E θ (x) represents the energy function and θ is its parameter.

[0032] Furthermore, the sampling methods used for sampling from the energy model include independent sampling and importance sampling, and a pre-trained language model is used as the proposal distribution to provide an initial value for sampling.

[0033] Compared with the prior art, the language model construction method combining pre-training and energy model described in the present invention has the following advantages:

[0034] The present invention combines the pre-trained language model with the energy-based language model to construct the model structure and training method. The present invention proposes three energy model structures with the pre-trained language model as the skeleton. These three structures can not only utilize the original parameters of the pre-trained language model, but also avoid the common problems of ordinary language models such as label bias through the training method of the energy model.

[0035] The performance of the energy-based language model trained by the training framework constructed by the present invention exceeds the performance of the ordinary pre-trained language model after fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0037] Figure 1 It is the SumTargetLogit energy structure diagram of the present invention;

[0038] Figure 2 It is the Hidden2Scalar energy structure diagram of the present invention;

[0039] Figure 3 It is the SumTargetLogit energy structure diagram of the present invention;

[0040] Figure 4 This is the WenetSpeech experiment result of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0042] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0043] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0044] Hereinafter, the present invention will be described in detail based on embodiments with reference to the accompanying drawings.

[0045] The present invention provides a method for constructing a language model combining pre-training and an energy model, comprising:

[0046] Using the pre-trained language model as the framework, an energy-based language model is constructed;

[0047] Select a common language model as a noise model or a proposal model, train the energy-based language model, and obtain the final language model.

[0048] Specifically, considering a natural sentence x, the energy-based language model is defined as follows:

[0049]

[0050] Where E θ (x) represents the energy function, θ is its parameter, Z(θ)-∫exp(-E θ (x))dx represents the normalization constant, p θ (x) represents the probability of sentence x. Generally speaking, the normalization constant is difficult to calculate, and the model can only output an unnormalized probability

[0051] Specifically, the present invention constructs three energy structures, including:

[0052] 1)SumTargetLogit

[0053] This energy function uses an autoregressive language model such as GPT-2 as the framework, and sums the logits corresponding to the target token at each position (i.e. the token at the next position) to obtain the energy. Figure 1 As shown, it can be expressed as:

[0054]

[0055] where x i represents the i-th token of sentence x, T represents the length of sentence x, and f θ It represents the autoregressive model, token refers to the symbolic unit after the natural language is split in the computer, and logit refers to the unnormalized vector output by the classification head of the neural network.

[0056] 2) Hidden2Scalar

[0057] This energy function uses a non-autoregressive model such as BERT as the skeleton, and sends the latent variable after the model encodes the sentence into a linear layer network to obtain a scalar as energy, such as Figure 2 As shown, it can be expressed as:

[0058]

[0059] where enc θ (x) represents a non-autoregressive encoder like BERT, whose output is a latent variable, denoted by h in the figure.

[0060] 3) SumTokenLogit

[0061] This energy function uses a non-autoregressive pre-trained language model as its skeleton. Unlike Hidden2Scalar, this energy function uses the language head of a pre-trained masked language model (MLM) and uses the logit output of the language head at each token position to define the energy. This is somewhat similar to SumTargetLogit. Its structure is as follows: Figure 3 As shown, it can be expressed as:

[0062]

[0063] where g θ (x) represents the masked language model, whose output is the unnormalized logit.

[0064] Specifically, since the probability of the energy model is not normalized, its training method is different from the cross entropy training of the traditional language model. The present invention successfully applies two major types of training methods to the energy-based language model.

[0065] 1) Noise Contrastive Estimate (NCE)

[0066] The core idea of ​​noise contrast estimation is to transform the training energy model into a binary classification problem, that is, to let the energy model learn to distinguish between real data and noise data. Therefore, noise contrast estimation requires a noise distribution to provide suitable noise samples for training, generally q φ (x) indicates that its ultimate optimization goal is:

[0067]

[0068] Where v represents the ratio of the number of noise samples to the number of true samples.

[0069] Generally speaking, noise distribution is crucial for NCE training. The noise distribution cannot be too different from the real data, otherwise the noise samples and real samples can be easily distinguished, resulting in premature convergence of the model and poor training results.

[0070] In the actual implementation process, the present invention uses the pre-trained language model GPT-2 as the noise model to generate noise samples, so as to achieve better training results. In addition, in general NCE training, the noise model is fixed and only the parameters of the energy model are updated during the training process without updating the parameters of the noise model; in dynamic noise contrast estimation (Dynamic NCE, DNCE), the parameters of the noise model will also be continuously updated, and the update method is the same as fine-tuning GPT-2. Experiments have found that DNCE often achieves better results than NCE.

[0071] 2) Maximum Likelihood Estimate (MLE)

[0072] The core idea of ​​maximum likelihood estimation is to directly optimize the energy model p θ The log-likelihood of (x) is transformed into a sampling problem from the energy model by taking the derivative of the energy function and simplifying it. It is as follows:

[0073]

[0074] The second term on the right is the one that needs to be sampled from the energy model. As mentioned earlier, the energy model cannot be sampled directly because its probability is not normalized. Therefore, we need to use a special sampling method.

[0075] Two different sampling methods are tried in the present invention: Metropolis Independence Sampling (MIS) and Importance Sampling (IS). These two sampling methods are commonly used methods for sampling from unnormalized probability models.

[0076] It should be noted that both of the above sampling methods require a proposed distribution. The role of the proposed distribution is to provide an initial value for sampling. The basic requirement for the proposed distribution is that it cannot be inconsistent with the target distribution p θ (x) The gap is too far. Similar to NCE, the pre-trained language model GPT-2 is used as the proposal distribution in this invention.

[0077] The present invention combines the pre-trained language model with the energy-based language model, improves the structure and training method of the model, constructs a complete energy-based language model training framework, and ultimately achieves results that are better than those of ordinary pre-trained language model fine-tuning.

[0078] The method of the present invention comprises:

[0079] Prepare training data. The training data is in text format. Each piece of data is a natural language sentence and does not need to be labeled. Figure 4 The text data in the adopted WenetSpeech data set can be used as training data.

[0080] Obtain a pre-trained language model as the skeleton of the energy model. The present invention uses BERT and GPT-2 models, which can also be replaced with other larger pre-trained language models as appropriate;

[0081] Select an energy structure, and based on the pre-trained language model, add corresponding linear layers to map the model output to the scalar space as energy to establish an energy-based language model;

[0082] Choose a common language model as a noise model or proposal model for training noise contrast estimation or maximum likelihood estimation, and train an energy-based language model;

[0083] Save and test the model.

[0084] The effectiveness of the present invention has been demonstrated on two Chinese datasets: AISHELL-1 and WenetSpeech. Figure 4This is the result of rescoring on the WenetSpeech speech dataset. It can be seen that the energy model obtained by using the SumTokenLogit structure and the DNCE training method has a significantly better rescoring result than the fine-tuned BERT and GPT-2 of the same scale.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a language model combining pre-training and an energy model, characterized in that: include: Using the pre-trained language model as the framework, an energy-based language model is constructed; Select a common language model as a noise model or a proposal model, train the energy-based language model, and obtain the final language model.

2. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The language model construction method combining pre-training and energy model also includes: Construct training data. The training data is in text format. Each piece of data is a sentence of natural language and does not require annotation.

3. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The energy-based language model constructed using the pre-trained language model as a framework includes: The energy function uses the autoregressive pre-trained language model as the skeleton, and sums the logits corresponding to the target token at each position to obtain the energy, where token refers to the symbol unit after the natural language is split in the computer, and logit refers to the unnormalized vector output by the classification head of the neural network; it can be expressed as: Among them, E θ (x) represents the energy function, θ is its parameter, x i represents the i-th token of sentence x, T represents the length of sentence x, and f θ represents an autoregressive model.

4. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The energy-based language model constructed using the pre-trained language model as a framework includes: The energy function uses a non-autoregressive model as the skeleton. The latent variable after the model encodes the sentence is sent to a linear layer network to obtain a scalar as energy, which is expressed as follows: Among them, E θ (x) represents the energy function, θ is its parameter, enc θ (x) represents a non-autoregressive encoder, whose output is a latent variable.

5. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The energy-based language model constructed using the pre-trained language model as a framework includes: The energy function uses a non-autoregressive pre-trained language model as the skeleton, uses the language head of the pre-trained masked language model, and uses the logit output of the language head at each token position to define the energy, which can be expressed as: Among them, E θ (x) represents the energy function, θ is its parameter, g θ (x) represents the masked language model, whose output is the unnormalized logit. Token refers to the symbolic unit after the natural language is split in the computer, and logit refers to the unnormalized vector output by the classification head of the neural network.

6. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The selecting a common language model as a noise model or a proposal model and training the energy-based language model includes: The training energy model is converted into a binary classification problem using noise contrast estimation. Noise contrast estimation uses a noise distribution to provide suitable noise samples for training. φ (x) indicates that its ultimate optimization goal is: Where υ represents the ratio of the number of noise samples to the number of true samples, represents the unnormalized probability, E θ (x) represents the energy function and θ is its parameter.

7. The method for constructing a language model combining pre-training and energy model according to claim 1, characterized in that: The selecting a common language model as a noise model or a proposal model and training the energy-based language model includes: The maximum likelihood estimation is used to directly optimize the log-likelihood of the energy model. By derivation and simplification of the energy function, the optimization problem is transformed into a problem of sampling from the energy model. The formula is as follows: in, is an item that needs to be sampled from the energy model, E θ (x) represents the energy function and θ is its parameter.

8. The method for constructing a language model combining pre-training and energy model according to claim 7, characterized in that: The sampling methods used for sampling from the model include independent sampling and importance sampling, and a pre-trained language model is used as the proposal distribution to provide initial values ​​for sampling.