BERT model training method and system, computer equipment, storage medium and program product

Abstract

The invention provides a BERT model training method and system, computer equipment, a storage medium and a program product, and belongs to the technical field of natural language processing. The method comprises the following steps: respectively generating a plurality of sequences through a plurality of decoding paths by utilizing a pre-constructed BERT model; dividing the plurality of sequences into positive samples and negative samples; sampling the positive sample and the negative sample by using the reference model to construct a preference learning loss function; constructing a regular penalty term loss function; constructing a mask prediction loss function; and determining the total loss of the BERT model according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function to obtain a trained BERT model. By adopting the scheme, the BERT model is guided to learn the influence of different decoding paths on output, and the ability of the BERT model to select paths in a complex decoding space and generate sequences with higher quality is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular, relates to a training method, system, computer device, storage medium and program product of a BERT model. Background Art

[0002] With the development of large language models, such as Llama, Gemini, GPT-4, etc., fields such as humanities and social sciences have developed rapidly. The excellent performance of BERT models in language understanding tasks has long been known, but the development of these models has been relatively slow in recent years, which may be due to the lack of generation capabilities of their original models. Generation tasks require models to be able to perform high-quality inference during sequence generation, and the performance of BERT models in this regard is limited, mainly due to the mismatch between the decoding path between its training and inference processes.

[0003] The main reason why existing BERT models perform poorly in generation tasks can be attributed to the significant differences in the sequence decomposition formats during training and inference. Specifically, autoregressive models can naturally decompose into the task of predicting the next token from left to right in language modeling, and generate sequences in the same order during inference. In contrast, the BERT model adopts a random decomposition method during training, that is, randomly selecting a part of the tokens and replacing them with masked tokens (such as [MASK] tokens) to train the model to predict the contents of these masked tokens. However, during inference, these models often follow certain given criteria to determine the decoding path, gradually generating the target sequence from the fully masked sequence. This inconsistency in the training-inference process leads to a significant performance gap. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a training method, system, computer device, storage medium and program product for a BERT model.

[0005] In a first aspect, the present invention provides a training method for a BERT model, comprising:

[0006] Use the pre-built BERT model to generate multiple sequences with multiple decoding paths;

[0007] Divide multiple sequences into positive samples and negative samples;

[0008] Use the reference model to sample positive and negative samples to construct a preference learning loss function;

[0009] Construct a regularized penalty loss function;

[0010] Construct mask prediction loss function;

[0011] The total loss of the BERT model is determined according to the preference learning loss function, the regularization penalty loss function, and the mask prediction loss function to obtain the trained BERT model.

[0012] Optionally, dividing the multiple sequences into positive samples and negative samples includes:

[0013] Score any two sequences, including the first target sequence and the second target sequence. When the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is used as a positive sample and the second target sequence is used as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is used as a negative sample and the second target sequence is used as a positive sample.

[0014] Optionally, the sampling of positive samples and negative samples using the reference model to construct a preference learning loss function includes:

[0015] Construct the preference learning loss function expression:

[0016]

[0017] Among them, L D is the preference learning loss value; σ represents the sigmoid function; β represents the hyperparameter that controls the preference learning loss; π θ (·) represents the strategy model; π ref (·) indicates the reference model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unshielded part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0018] Optionally, constructing a regularized penalty term loss function includes:

[0019] Construct the regular penalty loss function expression:

[0020]

[0021] Among them, L P is the regular penalty loss value; π ref (·) represents the reference model; π θ (·) represents the strategy model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2represents the unmasked part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0022] Optionally, constructing a mask prediction loss function includes:

[0023] Construct the expression of the mask prediction loss function:

[0024]

[0025] Among them, L M is the mask prediction loss value; Y mask represents the set of tokens that are masked in the input sequence; y i Y mask A single element in Y represents the i-th masked mark; obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input to the model; θ represents the weights and biases to be learned during training.

[0026] Optionally, determining the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function to obtain the trained BERT model includes:

[0027] The total loss L of the BERT model is calculated according to the following formula:

[0028] L=λ1L D +λ2L P +L M ;

[0029] Among them, L D is the preference learning loss value; L P is the loss value of the regular penalty term; L M is the mask prediction loss value; λ1 is the hyperparameter of the preference learning loss; λ2 is the hyperparameter of the regular penalty loss.

[0030] In a second aspect, the present invention provides a training system for a BERT model, comprising:

[0031] A generation module for generating multiple sequences using a pre-built BERT model with multiple decoding paths;

[0032] A partitioning module is used to divide multiple sequences into positive samples and negative samples;

[0033] The first building module is used to sample positive samples and negative samples using a reference model to construct a preference learning loss function;

[0034] The second building module is used to construct a regular penalty term loss function;

[0035] The third building module is used to construct the mask prediction loss function;

[0036] The determination module is used to determine the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function to obtain the trained BERT model.

[0037] In a third aspect, the present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the BERT model training method described in the first aspect.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium, characterized in that it is used to store a computer program; when the computer program is executed by a processor, the steps of the training method of the BERT model described in the first aspect are implemented.

[0039] In a fifth aspect, the present invention provides a computer program product, comprising computer executable instructions or a computer program, which, when executed by a processor, implements the steps of the BERT model training method described in the first aspect.

[0040] The present invention provides a training method, system, computer device, storage medium and program product of a BERT model, wherein the method comprises using a pre-built BERT model to generate multiple sequences with multiple decoding paths respectively; dividing the multiple sequences into positive samples and negative samples; sampling the positive samples and negative samples using a reference model to construct a preference learning loss function; constructing a regularization penalty loss function; constructing a mask prediction loss function; determining the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function, and obtaining a trained BERT model. The above scheme is used to guide the BERT model to learn the influence of different decoding paths on the output, and enhance the ability of the BERT model to select paths in complex decoding space and generate higher quality sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flowchart of a BERT model training method provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a path selection and path optimization method provided by an embodiment of the present invention;

[0044] Figure 3 Results of different decoding path methods provided by embodiments of the present invention in zero-sample common sense reasoning and reading comprehension tasks;

[0045] Figure 4 Result diagrams of different beam search algorithms provided by embodiments of the present invention;

[0046] Figure 5 A result diagram of different λ1 and λ2 in the loss function provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of a BERT model training system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a training method for a BERT model, including:

[0051] Step 101, using a pre-built BERT model to generate multiple sequences using multiple decoding paths.

[0052] In this embodiment, multiple decoding paths are first sampled from the candidate space, and the best path with the highest total prediction probability is selected. Specifically, this embodiment follows most of the settings in the Mask-Predict algorithm, and only modifies the selection scheme of re-masking marks in each decoding step. Figure 2 As shown in , instead of selecting only the token with the lowest prediction probability to enter the only next decoding state (i.e., the first candidate), the first k re-shielded tokens with the lowest prediction probability are allowed to be selected as candidates on each decoding path, where k is the preset number of candidates. However, when the target sequence length N is large, selecting the k tokens with the lowest total prediction probability will lead to a significant increase in the number of searches. Given the total decoding steps T, the target sequence length N, and the number of position candidates k, the total number of searches is Therefore, in order to reduce the search overhead, this embodiment further introduces a simplified version: that is, only one position in the masked part is replaced with one position in the unmasked part to obtain the candidate decoding state. Figure 2As shown, the first candidate sequence obtained by the Mask-Predict algorithm is We often <m> <m> <m> <m> <m> <m> <m> <m>After that, <m>Represents the masking mark, and a token with the largest prediction probability in its masked part (i.e., go) can be selected to replace a token in its unmasked part. In each decoding step, the minimum prediction probability of the second candidate sequence (i.e., often) is obtained to obtain the second candidate sequence state, i.e., We <m> <m>go <m> <m> <m> <m> <m> <m>.

[0053] Step 102: divide the multiple sequences into positive samples and negative samples.

[0054] In this step, any two sequences are scored, and the two sequences include a first target sequence and a second target sequence. When the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is used as a positive sample and the second target sequence is used as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is used as a negative sample and the second target sequence is used as a positive sample.

[0055] This embodiment uses the positive and negative sample pairs obtained by the path selection method to train the BERT model to learn the decoding path preference. Specifically, Figure 2 As shown, given a specific instance, the tokens in multiple target sequences are replaced by mask tokens, denoted as Y mask Randomly sampling two different decoding paths, generating these mask tags in multiple steps, and finally obtaining two different output sequences, Y mask The specific output mark is recorded as and Subsequently, a scoring function (e.g., an exact match accuracy score with the true label) is used to identify specific positive and negative examples. but As a positive sample, As negative samples, and vice versa. Finally, according to the common practice of online direct preference learning algorithms, the positive and negative samples are first sampled with the reference model, and then the DPO loss function is used to update the policy model.

[0056] Step 103, sampling positive samples and negative samples using the reference model to construct a preference learning loss function.

[0057] For example, construct the preference learning loss function expression:

[0058]

[0059] Among them, L D is the preference learning loss value; σ represents the sigmoid function; β represents the hyperparameter controlling the preference learning loss; π θ (·) represents the strategy model; π ref (·) indicates the reference model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unshielded part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0060] Step 104, construct a regularized penalty term loss function.

[0061] Exemplarily, the regular penalty term loss function expression is constructed:

[0062]

[0063] Among them, L P is the regular penalty loss value; π ref (·) represents the reference model; π θ (·) represents the strategy model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unmasked part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0064] Step 105: construct a mask prediction loss function.

[0065] For example, the expression of the mask prediction loss function is constructed as:

[0066]

[0067] Among them, L M Y is the mask prediction loss value; mask represents the set of tokens that are masked in the input sequence; y i Y mask A single element in Y represents the i-th masked mark; obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input to the model; θ represents the weights and biases to be learned during training.

[0068] Step 106, determining the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function, and obtaining a trained BERT model.

[0069] Exemplarily, the total loss L of the BERT model is calculated according to the following formula:

[0070] L=λ1L D +λ2L P +L M .

[0071] Among them, L D is the preference learning loss value; L P is the regular penalty loss value; L M is the mask prediction loss value; λ1 is the hyperparameter of the preference learning loss; λ2 is the hyperparameter of the regular penalty loss.

[0072] This example shows the results of different methods for selecting decoding paths for zero-shot reasoning and reading comprehension tasks. Figure 3 This embodiment uses GeBERT-124M to compare different schemes for determining the decoding path. The rules are as follows: In the result table of zero-shot common sense reasoning and reading comprehension tasks, in order to be consistent with the AR model, this embodiment converts the newly generated tokens (with n new The number of new = 1, that is, only one token is generated each time decoding, the order is from left to right or according to the highest k predicted probabilities, and the total decoding steps are equal to the target sequence length. Then, (1) set n new =2,3,4, the corresponding decoding steps are Where N represents the total target token, indicating that the method is a multi-token-based method; (2) Set the total number of decoding steps to T = 1, 4, 10, and the corresponding This embodiment represents this method as a multi-step-based method. In addition, when n new =1, there are still different rules to achieve specific generation tokens. (3) In addition to the order from left to right, the order from right to left or random can also be added to achieve the final target sequence. This embodiment does not select the token with the highest prediction probability (that is, the order from easy to difficult), but adopts the order from difficult to easy, that is, generates the token with the lowest prediction probability first. These methods can be called multi-order-based methods. (4) In the path selection method proposed in this embodiment, different candidate numbers k can be set. Here, k=2, 3, and 4 are compared, which is expressed as a multi-candidate-based method.

[0073] It can be found that: (1) the performance increases with n new This indicates that setting n new = 1 (consistent with the AR model, which predicts one token in each decoding step) is important for achieving competitive performance. (2) In multi-step based methods, more decoding steps lead to better performance, which also verifies the above observation that in several tasks such as Sciq and SIQA, the length of the target is smaller than the decoding step and the model also predicts one token in each decoding step. In contrast, in tasks with relatively long targets such as PIQA and ARC, the performance of this method still lags behind the left-to-right baseline. (3) In the path selection method, the easy-to-hard order performs best, while other orders (such as right-to-left) lead to significant performance degradation. (4) When using different beams for path selection, the performance of different tasks varies, but the average scores are comparable and all outperform the left-to-right baseline, which shows the effectiveness of the path selection method.

[0074] In addition, this embodiment is also compared with the token-aware beam search, and the results are as follows: Figure 4 As shown in the figure: The path selection method of beam sampling multiple positions to generate multiple candidate outputs is similar to the marker-based beam search algorithm, which has been widely used in AR models. The marker-based beam search algorithm selects more candidate markers during the inference process instead of always selecting the marker with the highest predicted probability, which can significantly improve the performance. It is also extended to the BERT model to allow more markers to be selected in each decoding step. Specifically, a marker is randomly selected in the unmasked part of the target sequence and replaced with a marker whose predicted probability is lower than the first marker in the overall probability distribution. In contrast, the path selection method proposed in this embodiment obtains candidates by selecting markers at different positions, while the marker-based beam search algorithm selects candidates of different predicted markers according to specific positions. This embodiment uses GeBERT-124M to analyze the XSUM dataset, and the results are shown in the figure. Figure 4 As shown, it can be found that the path selection method can achieve continuous performance improvement, while the token-aware beam search algorithm does not work in this case. This failure is attributed to the difference in modeling paradigm between the BERT model and the AR model.

[0075] In addition, this embodiment provides the results of an ablation experiment on path search to explore the impact of different λ1 and λ2 in its final loss function. Figure 5 shown.

[0076] Without position beam, the performance of λ1 in {0.0, 0.1, 0.5, 1} and the performance of λ2 in {0, 1, 5, 10}. Compared with the baseline model (i.e., λ1 = 0.0, λ2 = 0), it can be found that: (1) Both the preference learning loss and the penalty term are necessary conditions for improving performance. For example, in the case of λ1 = 0.5 and λ2 = 0, the performance will decrease. (2) In other cases, the performance is close to each other, and the gap between all indicators is only about 0.1, which shows that there is no need to spend a lot of effort to adjust λ1 and λ2 in this embodiment. The preference learning loss training objective of this embodiment is relatively easy to achieve.

[0077] In summary, the training method of the BERT model provided in this embodiment guides the BERT model to learn the impact of different decoding paths on the output, and enhances the ability of the BERT model to select paths in a complex decoding space and generate higher quality sequences.

[0078] Example 2

[0079] Based on the same inventive concept as Example 1, this example provides a BERT model training system. Since the principle of solving the problem by this system is similar to the BERT model training method provided in the aforementioned Example 1, the implementation of this system can refer to the implementation of the BERT model training method provided in Example 1.

[0080] like Figure 6 As shown in the figure, the training system of the BERT model includes:

[0081] The generation module 10 is used to generate multiple sequences using a pre-built BERT model with multiple decoding paths.

[0082] The division module 20 is used to divide the multiple sequences into positive samples and negative samples.

[0083] The first construction module 30 is used to sample positive samples and negative samples using a reference model to construct a preference learning loss function.

[0084] The second construction module 40 is used to construct a regularized penalty term loss function.

[0085] The third construction module 50 is used to construct a mask prediction loss function.

[0086] The determination module 60 is used to determine the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function to obtain a trained BERT model.

[0087] Exemplarily, the division module includes:

[0088] The scoring unit is used to score any two sequences, where the two sequences include a first target sequence and a second target sequence. When the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is used as a positive sample and the second target sequence is used as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is used as a negative sample and the second target sequence is used as a positive sample.

[0089] Exemplarily, the first building block includes:

[0090] The first building block is used to construct the preference learning loss function expression:

[0091]

[0092] Among them, L D is the preference learning loss value; σ represents the sigmoid function; β represents the hyperparameter that controls the preference learning loss; π θ (·) represents the strategy model; π ref (·) indicates the reference model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unshielded part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0093] Exemplarily, the second building block includes:

[0094] The second building block is used to construct the regular penalty loss function expression:

[0095]

[0096] Among them, L P is the regular penalty loss value; π ref (·) represents the reference model; π θ (·) represents the strategy model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unmasked part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0097] Exemplarily, the third building block includes:

[0098] The third building block is used to construct the expression of the mask prediction loss function:

[0099]

[0100] Among them, L M Y is the mask prediction loss value; mask represents the set of tokens that are masked in the input sequence; y i Y mask A single element in Y represents the i-th masked mark; obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input to the model; θ represents the weights and biases to be learned during training.

[0101] Exemplarily, the determining module includes:

[0102] The calculation unit is used to calculate the total loss L of the BERT model according to the following formula:

[0103] L=λ1L D +λ2L P +L M .

[0104] Among them, L D is the preference learning loss value; L P is the regular penalty loss value; L M is the mask prediction loss value; λ1 is the hyperparameter of the preference learning loss; λ2 is the hyperparameter of the regular penalty loss.

[0105] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0106] Example 3

[0107] This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the BERT model training method described in Example 1 are implemented.

[0108] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0109] Example 4

[0110] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the BERT model training method described in Example 1 are implemented.

[0111] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0112] Example 5

[0113] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the BERT model training method described in Example 1 are implemented.

[0114] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0115] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0116] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0117] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0118] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0119] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0120] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.< / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m>

Claims

1. A training method for a BERT model, characterized in that: include: Use the pre-built BERT model to generate multiple sequences with multiple decoding paths; Divide multiple sequences into positive samples and negative samples; Use the reference model to sample positive and negative samples to construct a preference learning loss function; Construct a regularized penalty loss function; Construct mask prediction loss function; The total loss of the BERT model is determined according to the preference learning loss function, the regularization penalty loss function, and the mask prediction loss function to obtain the trained BERT model.

2. The training method of the BERT model according to claim 1, characterized in that: The dividing the multiple sequences into positive samples and negative samples comprises: Score any two sequences, including the first target sequence and the second target sequence. When the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is used as a positive sample and the second target sequence is used as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is used as a negative sample and the second target sequence is used as a positive sample.

3. The training method of the BERT model according to claim 1, characterized in that: The method of sampling positive samples and negative samples using the reference model to construct a preference learning loss function includes: Construct the preference learning loss function expression: Among them, L D is the preference learning loss value; σ represents the sigmoid function; β represents the hyperparameter that controls the preference learning loss; π θ (·) represents the strategy model; π ref (·) indicates the reference model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unshielded part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

4. The training method of the BERT model according to claim 1, characterized in that: The construction of the regularized penalty term loss function includes: Construct the regular penalty loss function expression: Among them, L P is the regular penalty loss value; π ref (·) represents the reference model; π θ (·) represents the strategy model; Y p represents a positive sample; Y n represents negative samples; Y obs1 represents the unshielded part of the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unmasked part of the negative sample; X2 represents the source sequence corresponding to the negative sample.

5. The training method of the BERT model according to claim 1, characterized in that: The constructing of the mask prediction loss function includes: Construct the expression of the mask prediction loss function: Among them, L M Y is the mask prediction loss value; mask represents the set of tokens that are masked in the input sequence; y i Y mask A single element in Y represents the i-th masked mark; obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input to the model; θ represents the weights and biases to be learned during training.

6. The training method of the BERT model according to claim 1, characterized in that: The total loss of the BERT model is determined according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function to obtain the trained BERT model, including: The total loss L of the BERT model is calculated according to the following formula: L=λ1L D +λ2L P +L M ; Among them, L D is the preference learning loss value; L P is the regular penalty loss value; L M is the mask prediction loss value; λ1 is the hyperparameter of the preference learning loss; λ2 is the hyperparameter of the regular penalty loss.

7. A training system for a BERT model, characterized in that: include: A generation module for generating multiple sequences using a pre-built BERT model with multiple decoding paths; A partitioning module is used to divide multiple sequences into positive samples and negative samples; The first building module is used to sample positive samples and negative samples using a reference model to construct a preference learning loss function; The second building module is used to construct a regular penalty term loss function; The third building module is used to construct the mask prediction loss function; The determination module is used to determine the total loss of the BERT model according to the preference learning loss function, the regularization penalty loss function and the mask prediction loss function to obtain the trained BERT model.

8. A computer device, characterized in that: It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the BERT model training method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program; when the computer program is executed by a processor, the steps of the BERT model training method described in any one of claims 1-6 are implemented.

10. A computer program product, characterized in that It includes computer executable instructions or a computer program, which, when executed by a processor, implements the steps of the BERT model training method described in claims 1-6.

Images