Pre-training Method for Language Representation Model Based on Generator-Discriminator Architecture
Pre-training is performed through the span mask-alternative detection method of the generator-discriminator architecture, the sample data inconsistency problem in pre-training of the language representation model is solved, the applicability and training efficiency of the model in downstream tasks is improved, and the deep language representation model is generated.
Patent Information
- Application Number
- CN202310060651.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-16
- Publication Date
- 2025-07-29
- Estimated Expiration
- 2043-01-16
AI Technical Summary
The existing pre-training methods of language characterization model have inconsistency between sample data between masking strategies and downstream tasks, resulting in the inability to fully utilize sentence knowledge and the slow convergence speed of pre-training.
The generator-discriminator architecture is adopted, and pre-trained through the span mask-alternative detection method. The generator generates mask text, the discriminator judges the alternative position, and optimizes the model using the joint loss function to generate a depth bidirectional language representation model.
The inconsistency problem of sample data is solved, the applicability of the model in downstream tasks and the training convergence speed is improved, the generator module learns phrase-level knowledge, the discriminator module realizes rapid convergence, and generates a deep language representation model.
Smart Images
Figure CN116049405B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for pre-training a language representation model, and in particular to a method for pre-training a language representation model based on a generator-discriminator architecture, belonging to the technical field of natural language processing. Background Art
[0002] The essence of the work of generating a language representation model is to train the parameters in the model through different pre-training methods. The main methods for pre-training the model are: mask language model (MLM) based on the token level, sentence boundary objective prediction (SBO), term frequency-inverse document frequency prediction (TF-IDF), token corruption prediction (TCP), etc. Next sentence prediction (NSP), sentence distance prediction (SDP), sentence order prediction (SO), etc. based on the sentence level. Through previous experiments, it has been verified that the above methods can achieve wide applicability for different downstream tasks by training model parameters. Among them, the Google team published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" in NAACL, which adopted the Encoder structure of the bidirectional Transformer to complete the pre-training methods of the mask language model (MLM) and the next sentence prediction (NSP), achieving a breakthrough in the model for downstream tasks. However, there are problems of random mask strategies and inconsistent sample data between the pre-training method and the downstream task: that is, there is [MASK] in the input samples in the pre-training stage, but there is no such in the input samples of the downstream task, resulting in poor applicability to the downstream task. The Facebook team published "SpanBERT: Improving Pre-training by Representing and Predicting Spans" in ACL, which improved on the basis of BERT and completed the pre-training methods of span mask and span boundary objective, achieving performance improvement in downstream tasks such as extractive question answering. However, there is still a problem that only 15% of the tokens are predicted, resulting in a slow pre-training convergence speed. The Google team published "ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS" in ICLR, which adopted the idea of a generative adversarial network to complete the pre-training method of token replacement prediction (RTD). On the basis of reducing the number of model parameters, the model effect can still be improved. However, there is a problem that the knowledge of the whole sentence cannot be fully utilized in the pre-training stage. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a pre-training method for a language representation model based on a generator-discriminator architecture.
[0004] To solve the above technical problem, the technical solution adopted by the present invention is as follows:
[0005] A pre-training method for a language representation model based on a generator-discriminator architecture, comprising the following steps:
[0006] Step 1: Preprocessing of corpus data:
[0007] The corpus data appears in the form of text sequences: it is trimmed to a preset maximum length, the corpus data is segmented into words, each segmented word is used as a Token, and the corresponding word embedding representation of each Token is found one by one in the vocabulary to obtain a word embedding tensor. According to the different text segments to which the Tokens belong, corresponding sentence chunk tensors are obtained. According to the index position of the Tokens in the text sequence, a position encoding tensor representation is obtained; the word embedding tensors, sentence chunk tensors, and position encoding tensors of each Token are respectively summed to obtain the embedding representation e(x) of each Token on a low-dimensional dense vector. The embedding representations e(x) of each Token on the low-dimensional dense vector are divided into a training data set and a validation data set and stored centrally.
[0008] Step 2: Build a generator-discriminator architecture, including the following specific modules:
[0009] The generator-discriminator architecture includes an input layer, a generator, a discriminator, and an output layer connected in series in sequence;
[0010] The generator and the discriminator have the same structure, both are multi-layer pre-trained text encoder units, which are composed of 6 layers of encoders and 1 layer of hidden units connected in series. Each layer of encoder block includes 1 layer of self-attention layer and 1 layer of feed-forward neural network, and the number of nodes is 256; each layer of encoder is responsible for performing multi-head attention calculation-residual connection, first normalization processing-feed-forward neural network training-residual connection, and second normalization processing on the input Token embedding representation e(x). The hidden layer is responsible for outputting the results; the input layer is responsible for performing span masking processing on the embedding representation of the training example or the validation example, and the output layer is the exit for the final result output.
[0011] Step 3: Pretraining using the span masking - substitution detection method: The input layer masks the training examples and outputs the masked example text to the generator for prediction; the 6 - layer encoder of the generator performs multi - head attention calculation - residual connection, first normalization processing - feed - forward neural network training - residual connection, and second normalization processing on the example text in sequence, fully predicting the position of the masked text according to the context, generating a complete text sequence and outputting it to the discriminator for token substitution detection; the 6 - layer encoder block of the discriminator performs multi - head attention calculation - residual connection, first normalization processing - feed - forward neural network training - residual connection, and second normalization processing on the complete text sequence in sequence, aiming to optimize the loss, judging whether each token is inconsistent with the original input of the model according to the context, performing substitution discrimination on the complete text sequence, and outputting a marked or unmarked token for each token position; the discrimination result is output through the output layer.
[0012] Step 4: Training of the language representation model:
[0013] Training is carried out with the goal of minimizing the combined loss function of the generator and the discriminator; where the generator loss function is L Gen , and the discriminator loss function is L Dis , the sum of the losses of the two modules is used as the overall loss of the model, with min∑(L Gen +L Dis ) as the objective function, and the model is trained by inputting training examples:
[0014] The objective function of the generator is:
[0015]
[0016] The objective function of the discriminator is:
[0017]
[0018] In the formula, the training example x = [x1, …, x n ; n represents the number of tokens after tokenization of the training example; k is the number of randomly covered positions selected, k = [0.15n], x masked represents the text sequence after the token at the covered position is replaced by the span mask [Mask], and x corrupt represents the text sequence output after the generator generates the token with the span mask [Mask].
[0019] Furthermore, it also includes the step of validating the model: Repeatedly execute Steps 3 and 4 using validation examples to fine - tune the model parameters.
[0020] Furthermore, the input layer performs span masking operations according to the geometric distribution, and the training example x = [x1, …, xn is mapped to the vector representation of the context h(x) = [h1, …, h n ; n represents the number of Tokens after tokenizing the training example.
[0021] Furthermore, the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is a decimal between 0 and 1; randomly selects the starting position of each mask according to the uniform distribution, and masks according to the span length l,
[0022] x masked = REPLACE[x, k, [Mask]] (4)
[0023] In the generator module, for the masked position i, the generator outputs the probability distribution of Token x i through the normalization layer:
[0024]
[0025] where h G (x) is the context representation of Token x mapped on the generator;
[0026] In the discriminator module, a corrupted example x is created by replacing the masked Token in the complete text sequence generated by the generator corrupt :
[0027]
[0028]
[0029] where p Gen is the probability distribution output by the generator through the normalization layer (the Token generated by the generator).
[0030] For a given position i, the discriminator predicts the probability that Token x i has been replaced using the sigmoid function:
[0031] D(x, i) = sigmoid(ω T h D (x) i ) (8)
[0032] where ω is the unit vector in the same dimension as h D (x) i , ω T is the transpose of ω, and the result of the inner product operation with h D (x) i is used as the input of the sigmoid function, and h D (x)i It is the context representation of the mapping of Token x on the generator.
[0033] Furthermore, the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.2.
[0034] Furthermore, the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.3.
[0035] Furthermore, the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.4.
[0036] Adopting the above technical solutions, the present invention achieves the following technical effects:
[0037] The present invention solves the problem of inconsistent sample data between the pre-training method and the downstream tasks. While increasing the difficulty of the pre-training method, it reduces the fine-tuning complexity of the downstream tasks. The generator module using span masking allows the model to fully learn phrase-level knowledge, and the discriminator module's binary classification task can achieve rapid convergence of the training model, so as to generate a deep bidirectional language representation model, which is better applicable to downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a schematic structural diagram of the 12-layer Transformer encoder block in Embodiment 1 of the present invention.
[0039] Figure 2 It is an example diagram of the generator-discriminator framework in Embodiment 1 of the present invention.
[0040] Figure 3 It is a structural diagram of the generator-discriminator framework in Embodiment 1 of the present invention.
[0041] Figure 4 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0042] The following embodiments are used to illustrate the present invention.
[0043] Embodiment 1
[0044] A pre-training method for a language representation model based on a generator-discriminator architecture, comprising the following steps:
[0045] Step 1: Corpus data preprocessing:
[0046] Corpus data format conversion: This embodiment is designed based on the TensorFlow framework, and the corpus data is converted into a TFrecord format data file before training.
[0047] The corpus data appears in the form of text sequences: it is trimmed to a preset maximum length of 128, the corpus data is segmented into words, each segmented word is used as a Token, and the corresponding word embedding representation of each Token is found one by one in the vocabulary to obtain a word embedding tensor. According to the different text segments to which the Tokens belong, corresponding sentence-piece tensors are obtained. According to the index position of the Tokens in the text sequence, a position encoding tensor representation is obtained; the word embedding tensors, sentence-piece tensors, and position encoding tensors of each Token are respectively summed to obtain the embedding representation e(x) of each Token on the low-dimensional dense vector. The embedding representations e(x) of each Token on the low-dimensional dense vector are divided into a training data set and a validation data set and stored centrally;
[0048] Step 2: Build a generator-discriminator architecture, including the following specific modules:
[0049] The generator-discriminator architecture includes an input layer, a generator, a discriminator, and an output layer connected in series in sequence;
[0050] The generator and the discriminator have the same structure. Based on the Transformer encoder used in the BERT model, they are both multi-layer pre-trained text encoder units, consisting of 6 layers of encoders and 1 layer of hidden units connected in series. Each layer of encoder block includes 1 layer of self-attention layer and 1 layer of feed-forward neural network, and the number of nodes is 256; each layer of encoder is responsible for performing multi-head attention calculation-residual connection, normalization processing-feed-forward neural network training-residual connection, normalization processing on the embedding representation e(x) of the input Tokens on the low-dimensional dense vector, and the hidden layer is responsible for outputting the results; the input layer is responsible for performing span masking processing on the embedding representations of training examples or validation examples, and the output layer is the outlet for the final result output;
[0051] Step 3: Span masking processing: The input layer inputs the training examples after being damaged by the mask into the generator;
[0052] For the training examples, a span masking operation selected according to the geometric distribution maps the training example x = [x1,..., x n to the vector representation h(x) = [h1,..., h n , where e(x) represents the embedding representation of Token x on the low-dimensional dense vector.
[0053] When performing the mask, the positions to be covered are selected from the n positions in the training example X = [x1,..., x n ;
[0054] Select the span length \(l\) to be masked according to the geometric distribution \(l\sim Geo(p)\), where \(p = 0.2\) is selected. Randomly select the initial position of this segment according to the uniform distribution, and finally perform masking according to the span length \(l\). A total of \(k\) random positions are selected for covering, \(k=[0.15n]\), where \(n\) represents the number of Tokens after segmenting the training example. Subsequently, the selected covering position \(m\) will be replaced by Tokens:
[0055] x masked = REPLACE[x,k,[Mask]] (1)
[0056] Mask is the span mask;
[0057] In the discriminator module, replace the covered Tokens in the generated samples of the generator to create a corrupted example \(x\) corrupt to train the discriminator to predict which positions have been replaced. This process is equivalent to appropriately corrupting the input text through the generator module:
[0058]
[0059]
[0060] In the formula, \(p\) Gen is the probability distribution output by the generator through the normalization layer (Tokens generated by the generator), \(x\) masked represents the text sequence after replacing \(k\) Tokens with [Mask], and a total of \(k\) Tokens are replaced. \(x\) corrupt represents the text sequence output after the generator generates the Tokens masked by [Mask], and is also the input text sequence of the discriminator.
[0061] For a given position \(i\), the discriminator maps the neuron input of the encoding layer to the output end in the hidden layer based on the sigmoid function as the activation function, and predicts whether the Token \(x\) i has been replaced:
[0062] \(D(x,i)=\text{sigmoid}(\omega\) T \(h\) D (x) i ) (4)
[0063] In the formula, for a given position \(i\), the discriminator predicts the probability value of whether the Token \(x\) i has been replaced. sigmoid represents using the sigmoid function as the activation function to map the neuron input of the encoding layer to the output end in the hidden layer. \(\omega\) is the unit vector of the same dimension as \(h\) D (x) i in the same dimension, \(\omega\) Tis the transpose of ω and is inner product with h D (x) i Performing the inner product operation to obtain a result which is convenient as the input of the sigmoid function, h D (x) i is the context representation of the Token x mapped on the generator.
[0064] Training is carried out with the goal of minimizing the combined loss function of the generator and the discriminator. Among them, the generator loss function is L Gen , and the discriminator loss function is L Dis , and the sum of the losses of the two modules is used as the overall loss of the model. Taking min∑(L Gen +L Dis ) as the objective function, inputting training examples to train the model;
[0065]
[0066] D(x corrupt ,i)))(6)
[0067] min∑(L Gen +L Dis )(7)
[0068] The alternative Token recognition of the discriminator is relatively easy compared to the span masking method of the generator. Therefore, a weight μ is added and the weight is optimized through machine learning methods.
[0069] min∑(L Gen +μL Dis )(8)
[0070] The framework model is trained for 50,000 rounds using the Amazon Reviews–Full dataset;
[0071] Repeat steps 3 and 4 with the validation examples, perform 1,000 rounds of validation, and fine-tune the model parameters. Finally, obtain the verification result of the method effectiveness:
[0072] disc_accuracy = 0.90800613
[0073] gen_accuracy = 0.4068668
[0074] loss = 20.326025
[0075] In the formula, gen_accuracy represents the accuracy of the prediction of the covered Token by the generator module, disc_accuracy represents the accuracy of the discriminator module in judging whether the Token is replaced, and loss is the overall loss value of the model, which is the weighted sum of the generator and discriminator losses.
[0076] Example 2
[0077] The difference from Example 1 is that the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.3.
[0078] The framework model is trained for 50,000 rounds and validated for 1,000 rounds using the Billion Word Imputation dataset. Finally, the verification results of the method effectiveness are obtained:
[0079] disc_accuracy = 0.89823365
[0080] gen_accuracy = 0.3267005
[0081] loss = 23.41575
[0082] In the formula, gen_accuracy represents the accuracy of the generator module's prediction of the masked Token, disc_accuracy represents the accuracy of the discriminator module's judgment on whether the Token is replaced, and loss is the overall loss value of the model, which is the weighted sum of the generator and discriminator losses.
[0083] Example 3
[0084] The differences from Examples 1 and 2 are that the input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.4.
[0085] The framework model is trained for 700,000 rounds and validated for 1,000 rounds using the OpenWebText dataset. Finally, the verification results of the method effectiveness are obtained:
[0086] disc_accuracy = 0.97518146
[0087] gen_accuracy = 0.42668855
[0088] loss = 8.831493
[0089] In the formula, gen_accuracy represents the accuracy of the generator module's prediction of the masked Token, disc_accuracy represents the accuracy of the discriminator module's judgment on whether the Token is replaced, and loss is the overall loss value of the model, which is the weighted sum of the generator and discriminator losses.
[0090] The model is fine-tuned using the Recognizing Textual Entailment (RTE) dataset in the General Language Understanding Evaluation (GLUE) benchmark for multi-task natural language understanding created by institutions such as New York University and the University of Washington, and the Microsoft Research Paraphrase Corpus (MRPC) to obtain verification results, demonstrating the effectiveness of the pre-training method.
[0091] On the discriminator, different downstream task datasets are selected for fine-tuning training with a small amount of data on the basis of the original parameters that have completed pre-training, adjusting a small number of parameters to improve the transferability and practicality of the language representation model. RTE contains 2,491 training sets and 3,000 test sets. The task objective is to determine whether a sentence pair is entailed, a binary classification task; MRPC is a text file containing 5,800 sentence pairs, and the task objective is to determine whether the sentence pairs are semantically equivalent. The verification on both datasets uses accuracy as the metric, and the results are as follows:
[0092] RTE (ACC): 61.01
[0093] MRPC (ACC): 81.4
Claims
1. A pre-training method for a language representation model based on a generator-discriminator architecture, characterized in that Including the following steps: Step 1: Preprocessing of corpus data: The corpus data appears in the form of text sequences: trim it to a preset maximum length, segment the corpus data into words, and each segmented word is used as a Token. Look up the corresponding word embedding representation of each Token in the vocabulary one by one to obtain the word embedding tensor. Obtain the corresponding sentence and chunk tensor according to the different text segments to which the Token belongs. According to the index position of the Token in the text sequence, obtain the position encoding tensor representation; sum up the word embedding tensor, sentence and chunk tensor, and position encoding tensor of each Token respectively to obtain the embedding representation e(x) of each Token on the low-dimensional dense vector. Divide the embedding representation e(x) of each Token on the low-dimensional dense vector into a training data set and a validation data set for centralized storage; Step 2: Build a generator-discriminator architecture, including the following specific modules: The generator-discriminator architecture includes an input layer, a generator, a discriminator, and an output layer connected in series in sequence; The generator and the discriminator have the same structure, both are multi-layer pre-trained text encoder units, consisting of 6 layers of encoders and 1 layer of hidden units connected in series. Each layer of encoder block includes 1 layer of self-attention layer and 1 layer of feed-forward neural network, and the number of nodes is 256; each layer of encoder is responsible for performing multi-head attention calculation-residual connection, first normalization processing-feed-forward neural network training-residual connection, and second normalization processing on the input Token embedding representation e(x). The hidden layer is responsible for outputting the results; the input layer is responsible for performing span masking processing on the embedding representation of the training example or validation example, and the output layer is the outlet for the final result output; Step 3: Use the span masking-replacement detection method for pre-training: The input layer masks the training example and outputs the masked example text to the generator for prediction; The 6 layers of encoders of the generator perform multi-head attention calculation-residual connection, first normalization processing-feed-forward neural network training-residual connection, and second normalization processing on the example text in sequence, and fully predict the position of the masked text according to the context to generate a complete text sequence and output it to the discriminator for Token replacement detection; the 6 layers of encoder blocks of the discriminator perform multi-head attention calculation-residual connection, first normalization processing-feed-forward neural network training-residual connection, and second normalization processing on the complete text sequence in sequence. With the goal of optimizing the loss, judge whether each Token is inconsistent with the original input of the model according to the context, perform replacement discrimination on the complete text sequence, and output a marked or unmarked token for each Token position; The discrimination result is output through the output layer; Step 4: Training of the language representation model: Training is carried out with the goal of minimizing the combined loss function of the generator and discriminator; where the generator loss function is L Gen , and the discriminator loss function is L Dis . The sum of the losses of the two modules is used as the overall loss of the model, and min∑(L Gen + L Dis ) is used as the objective function to train the model with input training examples: The objective function of the generator is: The objective function of the discriminator is: where the training example \(x = [x_1, \ldots, x\) n ; \(n\) represents the number of Tokens after tokenizing the training example; \(k\) is the number of randomly selected masking positions, \(k = [0.15n]\), \(x\) masked represents the text sequence after the Token at the masking position is replaced by the span mask [Mask], and \(x\) corrupt represents the text sequence output after the generator generates the Token with the span mask [Mask]; For a given position i, the discriminator predicts the probability that Token x i has been replaced using the sigmoid function: D(x,i) = sigmoid(ω T h D (x) i ) (3) where ω is h D (x) i a unit vector in the same dimension, ω T is the transpose of ω, and the result of the inner product operation with h D (x) i is used as the input of the sigmoid function, and h D (x) i is the context representation of the mapping of Token x on the generator.
2. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 1, wherein: It also includes a model verification step: repeat Steps 3 and 4 using the validation example to fine-tune the model parameters.
3. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 1, characterized in that: The input layer performs a span masking operation according to a geometric distribution, mapping the training example x = [x1, …, x n to the vector representation h(x) = [h1, …, h n ; n represents the number of Tokens after tokenizing the training example.
4. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 3, characterized in that: The input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is a decimal between 0 and 1; randomly select the starting position of each mask according to the uniform distribution and perform masking according to the span length l, x masked = REPLACE[x,k,[Mask]] (4) In the generator module, for the masked position i, the generator outputs the probability distribution of Token x i through the normalization layer: where h G (x) is the context representation of the mapping of Token x on the generator; In the discriminator module, a corrupted example x is created by replacing the masked tokens with the complete text sequence generated by the generator corrupt : where p Gen is the probability distribution of the tokens output by the generator through the normalization layer .
5. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 4, characterized in that: The input layer selects the span length 1 to be masked according to the geometric distribution Geo(p), where p is 0.
3.
6. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 4, characterized in that: The input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.
2.
7. The pre-training method of the language representation model based on the generator-discriminator architecture according to claim 4, characterized in that: The input layer selects the span length l to be masked according to the geometric distribution Geo(p), where p is 0.4.