Training method and system of BERT model, computer device, storage medium and program product

Abstract

The application provides a BERT model training method, system, computer device, storage medium and program product, and belongs to the technical field of natural language processing. The method comprises the following steps: using a pre-constructed BERT model to generate a plurality of sequences respectively through a plurality of decoding paths; dividing the plurality of sequences into positive samples and negative samples; sampling the positive samples and the negative samples by using a reference model to construct a preference learning loss function; constructing a regularization penalty term loss function; constructing a mask prediction loss function; determining the total loss of the BERT model according to the preference learning loss function, the regularization penalty term loss function and the mask prediction loss function, and obtaining a trained BERT model. The above scheme is used to guide the BERT model to learn the influence of different decoding paths on the output, and to enhance the ability of the BERT model to select a path in a complex decoding space and generate a higher-quality sequence.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a BERT model training method and system, a computer device, a storage medium and a program product. BACKGROUND

[0002] With the development of large language models such as Llama, Gemini, GPT-4, etc., the fields of humanities and social sciences have developed rapidly. The outstanding performance of BERT models in language understanding tasks has long been known, but in recent years, the development of these models has been relatively slow, which may be due to the lack of generation capability in their original models. Generation tasks require models to be able to make high-quality inferences during sequence generation, while the performance of BERT models in this regard is limited, mainly due to the decoding path mismatch problem between the training and inference processes.

[0003] The main reason for the poor performance of existing BERT models in generation tasks can be attributed to the significant difference in sequence decomposition format between the training and inference processes of the model. Specifically, autoregressive models can naturally decompose into the task of predicting the next token from left to right during language modeling, and generate the sequence in the same order during inference. In contrast, BERT models use a random decomposition method during training, i.e., randomly selecting a portion of tokens and replacing them with mask tokens (such as [MASK] tokens) to train the model to predict the contents of these masked tokens. However, during inference, these models often follow certain given criteria to determine the decoding path, gradually generating the target sequence from the completely masked sequence. This inconsistency between the training and inference processes leads to a significant performance gap. SUMMARY

[0004] The present application provides a BERT model training method, system, computer device, storage medium and program product to address the deficiencies in the prior art.

[0005] In a first aspect, the present application provides a BERT model training method, comprising:

[0006] using a pre-constructed BERT model to generate a plurality of sequences with a plurality of decoding paths;

[0007] dividing the plurality of sequences into positive samples and negative samples;

[0008] sampling the positive samples and negative samples using a reference model to construct a preference learning loss function;

[0009] constructing a regularization penalty term loss function;

[0010] constructing a mask prediction loss function;

[0011] The total loss of the BERT model is determined according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function, and a trained BERT model is obtained.

[0012] Optionally, the dividing the plurality of sequences into positive samples and negative samples comprises:

[0013] Scoring any two sequences, the two sequences comprising a first target sequence and a second target sequence, when the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is taken as a positive sample and the second target sequence is taken as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is taken as a negative sample and the second target sequence is taken as a positive sample.

[0014] Optionally, the sampling the positive samples and the negative samples by using the reference model to construct the preference learning loss function comprises:

[0015] The expression of the preference learning loss function is constructed as:

[0016]

[0017] wherein, L D is a preference learning loss value; sigma represents a sigmoid function; beta represents a hyperparameter for controlling the preference learning loss; pi θ (·) represents a strategy model; pi ref (·) represents a reference model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample.

[0018] Optionally, the constructing the regular penalty term loss function comprises:

[0019] The expression of the regular penalty term loss function is constructed as:

[0020]

[0021] wherein, L P is a regular penalty term loss value; pi ref (·) represents a reference model; pi θ (·) represents a strategy model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2X2 represents the source sequence corresponding to the negative sample.

[0022] Optionally, the mask prediction loss function is constructed, including:

[0023] The expression of the mask prediction loss function is constructed as:

[0024]

[0025] wherein L M is a mask prediction loss value; Y mask represents a set of masked tokens in the input sequence; y i is a single element in Y mask , represents the i-th masked token; Y obs represents a set of tokens in the input sequence that are not masked; X represents the source sequence input into the model; and θ represents the weights and biases that need to be learned in the training process.

[0026] Optionally, the total loss of the BERT model is determined according to the preference learning loss function, the regularization penalty term loss function, and the mask prediction loss function, to obtain the trained BERT model, including:

[0027] The total loss L of the BERT model is calculated according to the following formula:

[0028] L = λ1L D + λ2L P + L M ;

[0029] wherein L D is a preference learning loss value; L P is a regularization penalty term loss value; and L M is a mask prediction loss value; λ1 is a hyperparameter of the preference learning loss; and λ2 is a hyperparameter of the regularization penalty term loss.

[0030] In a second aspect, the present application provides a BERT model training system, including:

[0031] The generation module is configured to generate a plurality of sequences respectively by using a plurality of decoding paths based on a pre-constructed BERT model.

[0032] The division module is configured to divide the plurality of sequences into positive samples and negative samples.

[0033] The first construction module is configured to sample the positive samples and the negative samples by using a reference model, to construct a preference learning loss function.

[0034] The second construction module is configured to construct a regularization penalty term loss function.

[0035] The third construction module is configured to construct a mask prediction loss function.

[0036] The determination module is configured to determine a total loss of the BERT model according to the preference learning loss function, the regularization penalty term loss function, and the mask prediction loss function, to obtain the trained BERT model.

[0037] In a third aspect, the present application provides a computer device, comprising a processor and a memory; wherein the processor implements the steps of the training method of the BERT model according to the first aspect when executing the computer program stored in the memory.

[0038] In a fourth aspect, the present application provides a computer readable storage medium, configured to store a computer program; the computer program is executed by the processor to implement the steps of the training method of the BERT model according to the first aspect.

[0039] In a fifth aspect, the present application provides a computer program product, comprising computer executable instructions or a computer program; the computer executable instructions or the computer program are executed by the processor to implement the steps of the training method of the BERT model according to the first aspect.

[0040] The present application provides a training method, system, computer device, storage medium and program product of a BERT model, wherein the method comprises generating a plurality of sequences by using a pre-constructed BERT model with a plurality of decoding paths; dividing the plurality of sequences into positive samples and negative samples; sampling the positive samples and the negative samples by using a reference model to construct a preference learning loss function; constructing a regularization penalty term loss function; constructing a mask prediction loss function; determining a total loss of the BERT model according to the preference learning loss function, the regularization penalty term loss function and the mask prediction loss function, to obtain a trained BERT model. The above scheme is used to guide the BERT model to learn the influence of different decoding paths on the output, and to enhance the ability of the BERT model to select paths and generate higher quality sequences in a complex decoding space. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0042] Figure 1 A flowchart of a training method of a BERT model provided by an embodiment of the present application;

[0043] Figure 2 A path selection and path optimization method diagram provided by an embodiment of the present application;

[0044] Figure 3 The following diagram shows the results of different decoding path methods provided in the embodiments of the present invention on zero-shot commonsense reasoning and reading comprehension tasks;

[0045] Figure 4 The results of different beam search algorithms provided in the embodiments of the present invention are shown in the figure.

[0046] Figure 5 The results of different λ1 and λ2 in the loss function provided in the embodiments of the present invention are shown in the figure.

[0047] Figure 6 This is a schematic diagram of the structure of a training system for a BERT model provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a training method for a BERT model, including:

[0051] Step 101: Use the pre-built BERT model to generate multiple sequences through multiple decoding paths.

[0052] In this embodiment, multiple decoding paths are first sampled from the candidate space, and the best path with the highest total prediction probability is selected. Specifically, this embodiment follows most of the settings in the Mask-Predict algorithm, only modifying the selection scheme of the re-masking flag in each decoding step. Figure 2 As shown, instead of simply selecting the marker with the lowest predicted probability to enter the unique next decoding state (i.e., the first candidate), it allows selecting the top k re-masking markers with the lowest predicted probabilities as candidates on each decoding path, where k is the preset number of candidates. However, when the target sequence length N is large, selecting the k markers with the lowest total predicted probabilities leads to a significant increase in the number of searches. Given the total decoding steps T, the target sequence length N, and the number of positional candidates k, the total number of searches is... Therefore, to reduce search overhead, this embodiment further introduces a simplified version: that is, simply replacing one position in the masked portion with one position in the unmasked portion is sufficient to obtain the candidate decoding state. For example, such as Figure 2As shown, the first candidate sequence obtained by the Mask-Predict algorithm, i.e., We often <m> <m> <m> <m> <m> <m> <m> <m>rear, wherein <m>On behalf of the cover mark, one token (i.e., go) with the maximum predicted probability in the shielded part can be selected to replace one token in the unshielded part to obtain the minimum predicted probability (i.e., often) of the second candidate sequence at each decoding step to get the second candidate sequence state, i.e., We <m> <m>go <m> <m> <m> <m> <m> <m>.

[0053] Step 102: Divide the multiple sequences into positive samples and negative samples.

[0054] In this step, any two sequences are scored. The two sequences include the first target sequence and the second target sequence. When the score of the first target sequence is greater than the score of the second target sequence, the first target sequence is regarded as a positive sample and the second target sequence is regarded as a negative sample. When the score of the first target sequence is less than the score of the second target sequence, the first target sequence is regarded as a negative sample and the second target sequence is regarded as a positive sample.

[0055] This embodiment uses positive and negative sample pairs obtained through a path selection method for training, allowing the BERT model to learn decoding path preferences. Specifically, as follows... Figure 2 As shown, given a specific instance where the markers in multiple target sequences are replaced with mask markers, denoted as Y. mask Two different decoding paths are randomly sampled, and these mask markers are generated in multiple steps, ultimately resulting in two different output sequences, Y. mask The specific output marker is denoted as and Subsequently, a scoring function (e.g., a score for the exact match accuracy with the true label) is used to identify specific positive and negative samples. If but As a positive sample As negative samples, and vice versa. Finally, following the common practice of online direct preference learning algorithms, positive and negative samples are first sampled using a reference model, and then the policy model is updated using the DPO loss function.

[0056] Step 103: Sample positive and negative samples using the reference model to construct a preference learning loss function.

[0057] For example, construct the expression for the preference learning loss function:

[0058]

[0059] Among them, L D σ represents the preference learning loss value; σ represents the sigmoid function; β represents the hyperparameter controlling the preference learning loss; π θ (·) represents the strategy model; π ref (·) represents the reference model; Y p Y represents a positive sample; n Y represents a negative sample; obs1 X1 represents the unmasked portion of the positive sample; Y represents the source sequence corresponding to the positive sample; obs2 X1 represents the unmasked portion of the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0060] Step 104, constructing a regular penalty term loss function.

[0061] Exemplarily, the expression of the regular penalty term loss function is constructed as follows:

[0062]

[0063] wherein, L P is the regular penalty term loss value; π ref (·) represents the reference model; π θ (·) represents the policy model; Y p represents the positive sample; Y n represents the negative sample; Y obs1 represents the unmasked part in the positive sample; X1 represents the source sequence corresponding to the positive sample; Y obs2 represents the unmasked part in the negative sample; X2 represents the source sequence corresponding to the negative sample.

[0064] Step 105, constructing a mask prediction loss function.

[0065] Exemplarily, the expression of the mask prediction loss function is constructed as follows:

[0066]

[0067] wherein, L M is the mask prediction loss value; Y mask represents the set of masked tokens in the input sequence; y i is a single element in Y mask , representing the i-th masked token; Y obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input into the model; θ represents the weights and biases to be learned in the training process.

[0068] Step 106, determining the total loss of the BERT model according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function, to obtain the trained BERT model.

[0069] Exemplarily, the total loss L of the BERT model is calculated according to the following formula:

[0070] L = λ1L D + λ2L P + L M .

[0071] wherein, L D is the preference learning loss value; L P is the regular penalty term loss value; L M is the mask prediction loss value; λ1 is a hyperparameter of the preference learning loss; and λ2 is a hyperparameter of the regular penalty term loss.

[0072] This embodiment shows the results of different methods for selecting decoding paths for zero-shot reasoning and reading comprehension tasks, as shown in the following table. Figure 3 This embodiment uses GeBERT-124M to compare different schemes for determining decoding paths, with the following rules: in the result table of the zero-shot common sense reasoning and reading comprehension tasks, in order to be consistent with the AR model, this embodiment adjusts the number of newly generated tokens (denoted by n new ) in each decoding step to n new =1, that is, only one token is generated each time, and the order is from left to right or generating tokens according to the highest k prediction probability, and the total decoding step is equal to the target sequence length. Then, (1) set n new =2, 3, 4, and the corresponding decoding steps are where N represents the total target token, and indicates that this method is a multi-token-based method; (2) set the total decoding step to T=1, 4, 10, and the corresponding This embodiment represents this method as a multi-step-based method. In addition, when n new =1, there are still different rules to achieve specific generated tokens. (3) In addition to the left-to-right order, right-to-left or random order can also be added to achieve the final target sequence. This embodiment does not select the token with the highest prediction probability (i.e., from easy to difficult order), but uses the order from difficult to easy, that is, the token with the lowest prediction probability is generated first. These methods can be referred to as multi-oder-based methods. (4) In the path selection method proposed in this embodiment, different candidate numbers k can be set. Here, k=2, 3, 4 are compared, which are referred to as multi-candidate-based methods.

[0073] It can be found that: (1) the performance decreases with the increase of n new , which indicates that setting n new = 1 (consistent with AR model, which predicts one token at each decoding step) is very important to achieve competitive performance. (2) In the multi-step based method, more decoding steps lead to better performance, which also verifies the above observation that in several tasks such as Sciq and SIQA, the length of the target is less than the decoding step, and the model also predicts one token at each decoding step. In contrast, in tasks such as PIQA and ARC that contain relatively long targets, the performance of this method still lags behind the left-to-right baseline. (3) In the path selection method, the order from easy to difficult performs best, while other orders (such as right-to-left) lead to significant performance degradation. (4) When using different beams for path selection, different tasks perform differently, but the average scores are comparable and all outperform the left-to-right baseline, which indicates the effectiveness of the path selection method.

[0074] In addition, the embodiment also conducts a comparison with token-aware beam search, and the results are as shown in Figure 4 The path selection method samples beams at multiple positions to generate multiple candidate outputs, which is similar to the token-based beam search algorithm, which has been widely used in AR models. The token-based beam search algorithm selects more candidate tokens during inference, rather than always selecting the token with the highest prediction probability, which can significantly improve performance. Similarly, it is extended to the BERT model to allow more tokens to be selected at each decoding step. Specifically, a token is randomly selected in the unmasked part of the target sequence and replaced with a token whose prediction probability is lower than the first-ranked token in the overall probability distribution. In contrast, the path selection method proposed in this embodiment selects candidates by selecting tokens at different positions, while the token-based beam search algorithm selects candidates according to specific positions that select different predicted tokens. This embodiment uses GeBERT-124M to conduct analysis experiments on the XSUM dataset, and the results are as shown in Figure 4 It can be found that the path selection method can achieve continuous performance improvement, while the token-aware beam search algorithm does not work in this case. This failure is attributed to the difference in modeling paradigm between BERT and AR models.

[0075] In addition, the embodiment provides the results of the ablation experiment on path search to explore the influence of different λ1 and λ2 in the final loss function. The results are as shown in Figure 5

[0076] ​Without using the position beam, λ1 is in {0.0, 0.1, 0.5, 1} and λ2 is in {0, 1, 5, 10}. Compared with the baseline model (i.e. λ1 = 0.0, λ2 = 0), it can be found that: (1) the preference learning loss and the penalty term are both necessary conditions for improving performance. For example, in the case of λ1 = 0.5 and λ2 = 0, the performance will decrease instead. (2) In other cases, the performance is close to each other, and the gap of all indicators is only about 0.1, which shows that there is no need to spend a lot of effort to adjust λ1 and λ2 in the embodiment. The preference learning loss training target of the embodiment is relatively easy to achieve.

[0077] In summary, the training method of the BERT model provided in the embodiment guides the BERT model to learn the influence of different decoding paths on the output, and enhances the ability of the BERT model to select paths and generate higher quality sequences in a complex decoding space.

[0078] Embodiment 2

[0079] Based on the same inventive concept as embodiment 1, the embodiment provides a training system of a BERT model. Since the principle of the system for solving the problem is similar to the training method of the BERT model provided in the foregoing embodiment 1, the implementation of the system can refer to the implementation of the training method of the BERT model provided in embodiment 1.

[0080] As shown in Figure 6 the training system of the BERT model comprises:

[0081] The generation module 10 is configured to generate a plurality of sequences respectively by using a plurality of decoding paths based on a pre-constructed BERT model.

[0082] The division module 20 is configured to divide the plurality of sequences into positive samples and negative samples.

[0083] The first construction module 30 is configured to sample the positive samples and the negative samples by using a reference model to construct a preference learning loss function.

[0084] The second construction module 40 is configured to construct a regular penalty term loss function.

[0085] The third construction module 50 is configured to construct a mask prediction loss function.

[0086] The determination module 60 is configured to determine a total loss of the BERT model according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function, and obtain a trained BERT model.

[0087] Exemplarily, the division module comprises:

[0088] The scoring unit is configured to score any two sequences, the two sequences including a first target sequence and a second target sequence, and when a score of the first target sequence is greater than a score of the second target sequence, the first target sequence is taken as a positive sample and the second target sequence is taken as a negative sample; when the score of the first target sequence is less than the score of the second target sequence, the first target sequence is taken as a negative sample and the second target sequence is taken as a positive sample.

[0089] Exemplarily, the first construction module comprises:

[0090] The first construction unit is configured to construct a preference learning loss function expression:

[0091]

[0092] wherein, L D is a preference learning loss value; sigma represents a sigmoid function; beta represents a hyperparameter for controlling the preference learning loss; pi θ (·) represents a policy model; pi ref (·) represents a reference model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample.

[0093] Exemplarily, the second construction module comprises:

[0094] The second construction unit is configured to construct a regular penalty term loss function expression:

[0095]

[0096] wherein, L P is a regular penalty term loss value; pi ref (·) represents a reference model; pi θ (·) represents a policy model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample.

[0097] Exemplarily, the third construction module comprises:

[0098] The third construction unit is configured to construct an expression of a mask prediction loss function:

[0099]

[0100] wherein, L M is the mask prediction loss value; Y mask represents a set of masked tokens in the input sequence; y i is a single element in Y mask represents the i-th masked token; Y obs represents a set of unmasked tokens in the input sequence; X represents the source sequence input into the model; and 0 represents the weights and biases to be learned in the training process.

[0101] Exemplarily, the determining module comprises:

[0102] a calculating unit configured to calculate the total loss L of the BERT model according to the following formula:

[0103] L = λ1L D + λ2L P + L M .

[0104] wherein, L D is the preference learning loss value; L P is the regular penalty term loss value; and L M is the mask prediction loss value; λ1 is a hyperparameter of the preference learning loss; and λ2 is a hyperparameter of the regular penalty term loss.

[0105] The more specific working processes of the above-mentioned various modules can refer to the corresponding contents disclosed in Embodiment 1, and will not be repeated here.

[0106] Embodiment 3

[0107] The embodiment provides a computer device comprising a processor and a memory; wherein the processor implements the steps of the training method of the BERT model according to Embodiment 1 when executing the computer program stored in the memory.

[0108] The more specific processes of the above-mentioned method can refer to the corresponding contents disclosed in Embodiment 1, and will not be repeated here.

[0109] Embodiment 4

[0110] The embodiment provides a computer readable storage medium for storing a computer program; the computer program is executed by a processor to implement the steps of the training method of the BERT model according to Embodiment 1.

[0111] The more specific processes of the above-mentioned method can refer to the corresponding contents disclosed in Embodiment 1, and will not be repeated here.

[0112] Embodiment 5

[0113] The embodiment provides a computer program product comprising computer executable instructions or a computer program, which, when executed by a processor, implement the steps of the training method of the BERT model in the embodiment 1.

[0114] More specific processes of the above method can refer to the corresponding content disclosed in the embodiment 1, and details are not described herein.

[0115] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system, device, storage medium and computer program product disclosed in the embodiments, since they correspond to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0116] Those skilled in the art can clearly understand that the technologies in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software products, which can be stored in storage media, such as ROM / RAM, magnetic disks, optical disks, etc., and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments.

[0117] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in computing environments.

[0118] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0119] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0120] The present application has been described in detail by reference to particular embodiments and illustrative examples, but these are not to be construed as limiting the application. Those skilled in the art understand that modifications, equivalent substitutions, improvements and other changes can be made to the technical solutions and embodiments of the present application without departing from the spirit and scope of the present application, and these all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims.< / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m>

Claims

1. A method for training a BERT model, characterized in that, The method comprises the following steps: a plurality of text sequences are generated by using a pre-constructed BERT model through a plurality of decoding paths respectively; the plurality of text sequences are divided into positive samples and negative samples; a reference model is used to sample the positive samples and the negative samples to construct a preference learning loss function, and the expression of the preference learning loss function is: ， wherein, L D is the preference learning loss value; σ represents a sigmoid function; β represents a hyperparameter for controlling the preference learning loss; π θ (·) represents a policy model; π ref (·) represents a reference model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample; a regular penalty term loss function is constructed, and the expression of the regular penalty term loss function is: ， wherein, L P is a regular penalty term loss value; π ref (·) represents a reference model; π θ (·) represents a policy model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample; a mask prediction loss function is constructed, and the expression of the mask prediction loss function is: ， Among them, L M For masking, predict the loss value; Y mask y represents the set of masked markers in the input sequence; i For Y mask A single element in Y represents the i-th masked tag; obs X represents the set of unmasked markers in the input sequence; θ represents the source sequence input into the model; and θ represents the weights and biases that need to be learned during training. a total loss of the BERT model is determined according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function, and a trained BERT model is obtained.

2. The method of claim 1, wherein the BERT model is trained by, The plurality of text sequences are divided into positive samples and negative samples, which comprises: any two text sequences are scored, and the two text sequences comprise a first target text sequence and a second target text sequence; when the score of the first target text sequence is greater than the score of the second target text sequence, the first target text sequence is taken as a positive sample and the second target text sequence is taken as a negative sample; when the score of the first target text sequence is less than the score of the second target text sequence, the first target text sequence is taken as a negative sample and the second target text sequence is taken as a positive sample.

3. The method of claim 1, wherein the BERT model is trained by, The total loss L of the BERT model is calculated according to the following formula: The method comprises the following steps: L = λ1L D + λ2L P + L M ; wherein, L D is a preference learning loss value; L P is a regular penalty term loss value; L M is a mask prediction loss value; λ1 is a hyperparameter of the preference learning loss; and λ2 is a hyperparameter of the regular penalty term loss. 4.A training system of a BERT model, characterized in that, a generation module is configured to generate a plurality of text sequences by using a pre-constructed BERT model through a plurality of decoding paths respectively; a division module is configured to divide the plurality of text sequences into positive samples and negative samples; a first construction module is configured to sample the positive samples and the negative samples by using a reference model to construct a preference learning loss function, and the expression of the preference learning loss function is: a second construction module is configured to construct a regular penalty term loss function, and the expression of the regular penalty term loss function is: ， wherein, L D is the preference learning loss value; σ represents a sigmoid function; β represents a hyperparameter for controlling the preference learning loss; π θ (·) represents a policy model; π ref (·) represents a reference model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1 represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2 represents a source sequence corresponding to the negative sample; a third construction module is configured to construct a mask prediction loss function, and the expression of the mask prediction loss function is: ， wherein, L P is a regular penalty term loss value; p ref (·) represents a reference model; p θ (·) represents a policy model; Y p represents a positive sample; Y n represents a negative sample; Y obs1 represents an unmasked part in the positive sample; X1represents a source sequence corresponding to the positive sample; Y obs2 represents an unmasked part in the negative sample; X2represents a source sequence corresponding to the negative sample; a determination module is configured to determine a total loss of the BERT model according to the preference learning loss function, the regular penalty term loss function and the mask prediction loss function, and obtain a trained BERT model. ， where L M is the masked prediction loss value; Y mask represents the set of masked tokens in the input sequence; y i is a single element in Y mask represents the i-th masked token; Y obs represents the set of unmasked tokens in the input sequence; X represents the source sequence input to the model; and θ represents the weights and biases to be learned during training. The method comprises a processor and a memory; wherein the processor executes a computer program saved in the memory to realize the steps of the training method of the BERT model according to any one of claims 1-3.

5. A computer device, comprising: The computer program is stored in the memory; and the computer program is executed by the processor to realize the steps of the training method of the BERT model according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer executable instructions or the computer program are executed by the processor to realize the steps of the training method of the BERT model according to any one of claims 1-3.

7. A computer program product, characterised in that, ​

Images