Method and device for training language model

By adjusting the training loss in language model training, avoiding excessive penalties on non-preferred output text, the problem of model crash when preferential data is approached in the prior art is solved, and the robustness of the training process and the reliability of the output are improved.

CN119004108APending Publication Date: 2024-11-22ALIPAY (HANGZHOU) INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411123942.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, preference alignment methods may cause language models to crash or output meaningless characters when preference data is approached, resulting in insufficient robustness during training.

Method used

By entering the target input text into the target language model and the pre-trained reference language model, the probability values ​​of the preferred output text and the non-preferred output text are obtained respectively, and the training loss is adjusted to reduce model parameters to avoid excessive penalties on non-preferred output text. Improve model robustness.

Benefits of technology

It effectively avoids unexpected situations when the model is close to the preferred output text and the non-preferred output text, improves the robustness of the training process, and ensures the reliability and effectiveness of the model output.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004108A_ABST
    Figure CN119004108A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a method and device for training a language model, and the method comprises the steps: firstly, inputting a target input text into a target language model and a pre-trained reference language model, and obtaining the probability values of the two models about a first output text and the probability values of the two models about a second output text, the first output text is marked as a preference output text about the target input text relative to the second output text; then, determining a training loss which is negatively correlated with the first loss term and positively correlated with the second loss term; wherein the first loss item comprises a difference between probability values of the two models about the first output text; the second loss item comprises the larger value between the difference between the probability values of the two models with respect to the second output text and 0; and then, by taking reduction of the training loss as a target, adjusting parameters in the target language model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning, and in particular, to methods and devices for training language models. Background Art

[0002] In recent years, large language models (LLMs) trained on large-scale datasets have demonstrated strong capabilities in understanding human intent and reasoning. Large language models are usually pre-trained on large-scale datasets and then fine-tuned according to specific downstream tasks, including preference alignment, to ensure that the model's behavior is consistent with human preferences.

[0003] However, when the preference data pairs used for preference alignment training are close to each other, the preference alignment method in the related art may cause the model to crash, generate repeated output text, or output meaningless characters. Therefore, a better method is needed to improve the robustness of the preference alignment training process. Summary of the invention

[0004] One or more embodiments of this specification describe a method and apparatus for training a language model, ensuring that the model can be correctly trained even when preference data pairs are close to each other, thereby improving the robustness of the preference alignment training process.

[0005] In a first aspect, a method for training a language model is provided, comprising:

[0006] Inputting the target input text into the target language model and the pre-trained reference language model respectively, obtaining probability values ​​of the two models regarding the first output text and the second output text, respectively, wherein the first output text is marked as a preferred output text regarding the target input text relative to the second output text;

[0007] Determining a training loss that is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes a difference between probability values ​​of the two models with respect to the first output text; and the second loss term includes a larger value between the difference between probability values ​​of the two models with respect to the second output text and 0;

[0008] With the goal of reducing the training loss, the parameters in the target language model are adjusted.

[0009] In some possible implementations, the parameters in the target language model are initialized based on the parameters in the pre-trained reference language model.

[0010] In some possible implementations, the first output text and the second output text are obtained in the following manner:

[0011] Input the target input text into the reference language model so that it outputs two output texts;

[0012] A preference evaluation is performed on the two output texts with respect to the target input text, and according to the preference evaluation result, they are respectively determined as the first output text and the second output text.

[0013] In some possible implementations, performing a preference evaluation on the two output texts with respect to the target input text includes:

[0014] Input the two output texts and the target input text into a preference evaluation model for preference evaluation; or,

[0015] The two output texts and the target input text are sent to a manual marking platform, and a preference evaluation result is received.

[0016] In some possible implementations, the target language model outputs a first output text with a first probability value when receiving the target input text; the reference language model outputs a first output text with a second probability value when receiving the target input text; and the first loss term includes a logarithmic value of the ratio of the first probability value to the second probability value.

[0017] In some possible implementations, the target language model outputs a second output text with a third probability value when receiving the target input text; the reference language model outputs a second output text with a fourth probability value when receiving the target input text; and the second loss term includes a larger value between the logarithm of the ratio of the third probability value to the fourth probability value and 0.

[0018] In some possible implementations, the training loss further includes a preset first coefficient; the training loss includes the product of a difference between the first loss term and the second loss term and the first coefficient.

[0019] In a second aspect, a device for training a language model is provided, comprising:

[0020] A probability determination unit is configured to input the target input text into the target language model and the pre-trained reference language model respectively, and obtain the probability values ​​of the two models respectively with respect to the first output text, and the probability values ​​of the two models respectively with respect to the second output text, wherein the first output text is marked as a preferred output text with respect to the target input text relative to the second output text;

[0021] A loss determination unit is configured to determine a training loss, which is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes a difference between probability values ​​of the two models with respect to the first output text; and the second loss term includes a larger value between the difference between probability values ​​of the two models with respect to the second output text and 0;

[0022] The model training unit is configured to adjust the parameters in the target language model with the goal of reducing the training loss.

[0023] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0024] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0025] The method and device for training a language model proposed in the embodiments of this specification, the training samples used by the method include input text, preferred output text, and non-preferred output text. The method first inputs the input text into the target language model and the reference language model to obtain the probability values ​​of the two models regarding the preferred output text, as well as the probability values ​​of the non-preferred output text. When determining the training loss, the difference in the probability values ​​of the two models regarding the preferred output text is directly included in the training loss, and for the non-preferred output text, it is only included in the training loss when the difference in the probability values ​​of the two models regarding the non-preferred output text is greater than or equal to 0. In this way, for similar preference data pairs of preferred output text-non-preferred output text, the non-preferred output text will not be excessively punished when training the model, so as to improve the robustness of the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram showing an implementation scenario of a method for training a language model according to an embodiment;

[0028] Figure 2 A flowchart showing a method for training a language model according to one embodiment;

[0029] Figure 3A schematic diagram showing changes in the first loss term, the second loss term, and the difference between the first loss term and the second loss term during the gradient descent process under different first coefficients;

[0030] Figure 4 A schematic block diagram of an apparatus for training a language model according to an embodiment is shown. DETAILED DESCRIPTION

[0031] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0032] Preference alignment refers to the process of aligning the behavior, output, and decision-making of large language models with human values, moral standards, and expectations, which is mainly reflected in three aspects: practicality, that is, the ability to follow instructions; honesty, avoiding fabrication; and security, avoiding the generation of illegal or harmful information.

[0033] In the preference dataset used for preference alignment, a single training sample often consists of an input text and two output texts, where the output text is the response corresponding to the input text. The two output texts have been pre-labeled, and one of the output texts is more in line with human preferences than the other output text with respect to the input text. This output text is called the preferred output text (hereinafter referred to as the preferred answer), and the other output text is called the non-preferred output text (hereinafter referred to as the non-preferred answer). The training sample can be formally recorded as (input text, preferred answer, non-preferred answer).

[0034] As mentioned above, in the process of preference alignment, when the preference data pairs used are close to each other, that is, when the preference answers and non-preference answers are close to each other, the preference alignment method in the related art may cause the model to crash, produce repeated output text, or output abnormal situations of meaningless characters.

[0035] The main reason for this situation is that the preference alignment method in the related technology often gives a reward to the preferred answer and a penalty of the same size to the non-preferred answer when training the large prediction model. However, when the two are close, the preference labeling will only show that the preferred answer is better than the non-preferred answer, but this does not mean that the non-preferred answer itself is bad and unacceptable. When the related technology is trained, it will fixedly give the non-preferred answer a penalty of the same size as the reward for the preferred answer. When the reward value of the preferred answer is very high, even if the non-preferred answer itself is close to excellent, it will still receive a large and unreasonable penalty. This will reduce the training effect of the model and even cause the model to crash.

[0036] For example, there are two output texts with the same 200 characters, only 3 consecutive characters are different, and the remaining 197 characters are the same, and these two output texts are objectively correct answers to the input text. At this time, if one of them is marked as a non-preferred answer, the relevant technology will give a larger penalty to the correct answer marked as non-preferred, making the model think that the output text that is actually the correct answer is unacceptable, thereby reducing the training effect of preference alignment.

[0037] Based on the above analysis, the embodiments of this specification propose a method for training a language model to overcome the above problems. Figure 1 A schematic diagram of an implementation scenario of a method for training a language model according to an embodiment is shown. Figure 1 In the example, we include the reference language model π as a benchmark ref , and the target language model π to be trained θ . Reference language model π ref is a pre-trained large language model used to constrain the target language model π during the preference alignment training process θ , so that it does not deviate too far from the output of the reference language model. The target language model π θ Also a pre-trained large language model, in some embodiments, the target language model π θ The parameters in the reference language model π ref The target language model π is initialized. θ and the reference language model π ref When receiving input text, it will output the corresponding probability for each output text. When the input text is x and the output text is y, the probability of the two models outputting y after receiving input text x can be recorded as π θ (y|x) and π ref (y|x).

[0038] like Figure 1 As shown, in the process of preference alignment training, a training sample can be (x, y 1 ,y 2 ), where x is the target input text, y 1 and 2 are the first output text and the second output text, y 1 Relative to y 2 is pre-labeled as the preferred output text with respect to the target input text x, and correspondingly, y 2 Output text for non-preference. Input the target input text x into the target language model π θ and the reference language model π ref In the above example, we get the output of the two models about y 1and 2 The probability of . Among them, the target language model π θ The probability of output can be recorded as π θ (y 1 |x) and π θ (y 2 |x), reference language model π ref The probability of output can be recorded as π ref (y 1 |x) and π ref (y 2 |x).

[0039] Then, according to the two models, the preferred output text y 1 The probability π θ (y 1 |x) and π ref (y 1 |x), determine the first loss term L 1 At the same time, according to the two models, the non-preferred output text y 2 The probability π θ (y 2 |x) and π ref (y 2 The difference between |x) and 0, the larger value, determines the second loss term L 2 Based on the first loss term L 1 and the second loss term L 2 The difference between determines the total training loss L, and adjusts the target language model π based on the training loss L θ To complete a round of training. It should be noted that during the training process, the reference language model π ref The parameters in always remain unchanged (i.e., freeze the reference language model π ref ), but only adjust the target language model π θ Parameters in .

[0040] Determine the second loss term L from the above 2 It can be seen from the process that when the target language model π θ The probability of output is π θ (y 2 |x) is smaller than the reference language model π ref The probability of output is π rer (y 2 |x), the target language model π θ Relative to the reference language model π ref Less inclined to output non-preferred output text y 2 The method in the embodiment of this specification does not directly use the difference between the two as the second loss term L 2To penalize the target language model π θ , but instead, the second loss term L 2 The value is 0 to reduce the target language model π θ Output text in non-preference y 2 In this way, it can effectively avoid the situation where y 1 and 2 When approaching, training models may produce unexpected situations.

[0041] The specific implementation steps of the above-mentioned method for training a language model are described below in conjunction with specific embodiments.

[0042] Figure 2 A flowchart of a method for training a language model according to an embodiment is shown. The execution subject of the method can be any platform or server or device cluster with computing and processing capabilities. Figure 2 As shown, the method at least includes: step 202, inputting the target input text into the target language model and the pre-trained reference language model respectively, obtaining the probability values ​​of the two models regarding the first output text, and the probability values ​​of the two models regarding the second output text, wherein the first output text is marked as the preferred output text regarding the target input text relative to the second output text; step 204, determining the training loss, which is negatively correlated with the first loss term and positively correlated with the second loss term; wherein the first loss term includes the difference between the probability values ​​of the two models regarding the first output text; the second loss term includes the larger value between the difference between the probability values ​​of the two models regarding the second output text and 0; step 206, adjusting the parameters in the target language model with the goal of reducing the training loss.

[0043] The specific execution process of each of the above steps is described below.

[0044] First, in step 202, the target input text is input into the target language model and the pre-trained reference language model respectively, and the probability values ​​of the two models regarding the first output text and the probability values ​​of the two models regarding the second output text are obtained, wherein the first output text is marked as the preferred output text with respect to the target input text relative to the second output text.

[0045] Reference language model π ref It can be any pre-trained large language model, which can output the probability value corresponding to each output text after receiving the input text.

[0046] Target language model π θ It is also a pre-trained large language model, which can output the probability value corresponding to each output text after receiving the input text.

[0047] In one embodiment, the target language model π θ The parameters in are based on the pre-trained reference language model π ref The parameters in are initialized. That is, in the initial stage, the target language model π θ and the reference language model π ref have the same parameter values.

[0048] In other embodiments, the same dataset can also be used to pre-train the target language model π θ and the reference language model π ref , so that they have initial parameter values ​​close to each other.

[0049] Target input text x, first output text y 1 , the second output text y 2 The training samples (x, y 1 ,y 2 ) can be any sample in the training set. Among them, the first output text y 1 Relative to the second output text y 2 The preferred output texts are marked as those about the target input text x. The above-mentioned marking work can be done manually, or a preference evaluation model can be first trained using a training set containing human preference samples, and then the preference evaluation model is used to mark the above-mentioned two output texts.

[0050] The first output text y 1 and the second output text y 2 It can be the answer text in the question-answer pair that is obtained in advance and constitutes the target input text x (correspondingly, the target input text x is the question text in the question-answer pair), or it can be generated based on the target input text x using a large language model.

[0051] In one embodiment, the first output text y 1 and the second output text y 2 Obtained through:

[0052] First, the target input text x is input into the reference language model so that it outputs two output texts; then, the two output texts are evaluated for preference with respect to the target input text x, and according to the preference evaluation results, they are respectively determined as the first output text y 1 and the second output text y 2 That is, the output text marked as the preferred output text in the two output texts is determined as the first output text y 1 , and the output text marked as the non-preferred output text is determined as the second output text y 2 .

[0053] In a more specific embodiment, a preference evaluation is performed on the two output texts with respect to the target input text, including: inputting the two output texts and the target input text into a preference evaluation model for preference evaluation; or, sending the two output texts and the target input text to a manual labeling platform, and receiving the preference evaluation result.

[0054] It is understandable that in other embodiments, other large language models may be used to generate two output texts based on the target input text x, which is not limited here.

[0055] In another embodiment, when the two output texts are answer texts in a pre-obtained question-answer pair, preference evaluation can be directly performed on the two output texts with respect to the target input text x, and then, according to the preference evaluation results, they are respectively determined as the first output text y. 1 and the second output text y 2 .

[0056] Target language model π θ When receiving the target input text x, with the first probability value π θ (y 1 |x) output the first output text y 1 , with the third probability value π θ (y 2 |x) output the second output text y 2 .

[0057] Reference language model π ref When receiving the target input text x, the second probability value π ref (y 1 |x) output the first output text y 1 , with the fourth probability value π ref (y 2 |x) output the second output text y 2 .

[0058] Then, in step 204, a training loss is determined, which is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes the difference between the probability values ​​of the two models regarding the first output text; and the second loss term includes a larger value between the difference between the probability values ​​of the two models regarding the second output text and 0.

[0059] Among them, the first loss term L 1 Include the two models about the first output text y 1 The difference between the probability values ​​of θ (y 1 |x) and π ref (y1 |x) is used to measure the target language model π θ Relative to the reference language model π ref In the build preference output text y 1 tendency changes.

[0060] When π θ (y 1 |x) and π ref (y 1 |x) is greater than or equal to 0, indicating that the target language model π θ Relative to the reference language model π ref More inclined to generate preferred output text y 1 , the target language model π should be given θ Corresponding rewards; when π θ (y 1 |x) and π ref (y 1 |x) is less than 0, indicating that the target language model π θ Relative to the reference language model π ref Less inclined to generate preferred output text y 1 , the target language model π should be given θ Corresponding punishment.

[0061] There are many ways to measure π. θ (y 1 |x) and π ref (y 1 |x), for example, calculating the difference, ratio, etc. between them. Further, based on the difference or ratio, the exponent or logarithm can be further calculated to constrain the first loss term L 1 The value range of .

[0062] In one embodiment, the first loss term L 1 Including, the first probability value π θ (y 1 |x) and the second probability value π ref (y 1 |x). As shown in formula (1):

[0063]

[0064] Here, log() is the natural logarithm with the constant e as the base.

[0065] In other embodiments, π θ (y 1 |x) and π ref (y1 The difference between |x) is taken as the first loss term L 1 , that is, L 1 =π θ (y 1 |x)-π ref (y 1 |x); or further exponentialize the difference as the first loss term L 1 ,Right now

[0066] The second loss term L 2 Include the two models about the second output text y 2 The larger value between the difference between the probability values ​​and 0 is π θ (y 2 |x) and π ref (y 2 |x) and 0. Used to measure the target language model π θ Relative to the reference language model π ref In generating non-preferred output text y 2 tendency changes.

[0067] When π θ (y 2 |x) and π ref (y 2 |x) is greater than or equal to 0, indicating that the target language model π θ Relative to the reference language model π ref More inclined to generate non-preferred output text y 2 , the target language model π should be given θ The corresponding penalty; when π θ (y 2 |x) and π ref (y 2 |x) is less than 0, indicating that the target language model π θ Relative to the reference language model π ref Less inclined to generate non-preferred output text y 2 Based on the analysis above, in order to avoid 1 and 2 When approaching, the training model may produce unexpected situations. Here, the scale of the penalty is constrained to 0 instead of a negative value.

[0068] There are many ways to measure π. θ (y 2 |x) and π ref (y 2|x), for example, calculating the difference between them, the logarithm of the ratio, etc.

[0069] In one embodiment, the second loss term L 2 Including, the third probability value π θ (y 2 |x) and the fourth probability value π ref (y 2 The larger value between the logarithm of the ratio of |x) and 0. As shown in formula (2):

[0070]

[0071] Among them, log() is the natural logarithm with the constant e as the base, and max() takes the maximum value in the brackets.

[0072] The training loss L includes the first loss term L 1 With the second loss term L 2 The difference, including L 1 -L 2 Since the gradient descent method is often used to adjust the model parameters, the final value of the training loss L is the same as the first loss term L 1 With the second loss term L 2 The difference L 1 -L 2 Negative correlation, for example, can be L = -(L 1 -L 2 ).

[0073] In some embodiments, the first loss term L 1 With the second loss term L 2 The difference L 1 -L 2 We can continue to add sigmoid functions and / or logarithms to handle very large or very small calculation results and improve the numerical stability of the training loss L. For example, L = -logσ(L 1 -L 2 ), where σ() represents the sigmoid function.

[0074] In some possible implementations, the training loss L further includes a preset first coefficient β; the training loss L includes a first loss term L 1 With the second loss term L 2 The difference L 1 -L 2 The product of L and the first coefficient β. For example, L = -β(L 1 -L 2 ).

[0075] Preferably, in one embodiment, the first coefficient β may be combined with a sigmoid function and a logarithm, and the obtained training loss L is as shown in formula (3):

[0076]

[0077] Among them, log() is the natural logarithm with the constant e as the base, and max() takes the maximum value in the brackets.

[0078] Next, in step 206, the parameters in the target language model are adjusted with the goal of reducing the training loss L.

[0079] Step 206 may, for example, use a gradient descent-based method to adjust the target language model π. θ The parameters in are conventional prior art and will not be described in detail here.

[0080] It should be noted that the above only describes the process of adjusting the parameters in the target language model based on one sample. By selecting samples in the training set multiple times and executing the methods in steps 202 to 206, the target language model can be trained multiple times to improve the effect of preference alignment.

[0081] In other embodiments, a small batch of samples may be selected from the training set, and the training loss of each sample may be obtained according to the method described in the aforementioned steps 202 to 206, and then the training loss of each sample may be averaged, and the average training loss may be used as the training loss of the small batch to adjust the parameters in the target language model. Then, small batches of samples may be selected for multiple rounds and the target language model may be trained to achieve the effect of accelerating the model training speed.

[0082] When samples in the training set are selected multiple times and the process in step 202 to step 206 is executed to train the target language model for multiple rounds, the training loss L generated in the process can be formally expressed as formula (4):

[0083]

[0084] Among them, E() represents the calculation of mathematical expectation, D represents the training set, (x, y 1 ,y 2 )~D represents the process of sampling samples from the training set D.

[0085] The following describes the beneficial effects of the training loss in the embodiments of this specification in combination with specific experimental data. Figure 3 A schematic diagram showing how the first loss term, the second loss term, and the difference between the first loss term and the second loss term change during the gradient descent process under different first coefficients. Figure 3(a), (b), (c), and (d) show the first loss term L when the first coefficient β is 0.02, 0.04, 0.1, and 0.2, respectively. 1 , the second loss term L 2 , and the first loss term L 1 With the second loss term L 2 The difference (L 1 -L 2 ) changes with the increase of training steps.

[0086] exist Figure 3 In each sub-graph (a), (b), (c), (d), the blue curve normalized_rewards / chosen represents the first loss term L 1 The orange curve normalized_rewards / rejected represents the second loss term L 2 The green curve normalized_rewards / margins represents the first loss term L 1 With the second loss term L 2 The difference (L 1 -L 2 ) changes. Figure 3 In each subgraph, the learning rate is uniformly set to 1×10 -6 .

[0087] Depend on Figure 3 From the sub-graphs (a), (b), (c), and (d), we can see that as the number of training rounds increases, the first loss term L used to measure the reward 1 The value of (corresponding to the blue curve) is increasing and is always greater than 0; the second loss term L used to measure the penalty 2 The value of (corresponding to the orange curve) is decreasing and is always less than 0; the value (L 1 -L 2 The curve of ) (corresponding to the green curve) is gradually increasing and is always greater than 0. The experimental results also further confirm the effectiveness of the method in the embodiments of this specification.

[0088] By using the training loss in the embodiment of this specification to perform preference alignment training, for the non-preferred output text in the training sample, the embodiment of this specification effectively prevents the target language model π trained in the related art from being close to each other when the preferred output text and the non-preferred output text are close to each other by truncating the value of the penalty term. θ The probability of output is π θ (y i |x) is smaller than the reference language model π ref The probability of output is πref (y i |x) occurs, i=1,2. When π θ (y i |x)<π ref (y i |x), these decreased probabilities are likely to be assigned to long-tail characters in the character (token) distribution, or to rare characters. The increase in the probability of these characters will lead to the target language model π θ Increasing the probability of these characters appearing during reasoning, or generating character outputs of these symbols, ultimately leads to the target language model π θ Abnormal phenomenon of repeated output or output of unwanted special characters.

[0089] In the samples of preference alignment training, generally speaking, humans prefer the preferred output text to the non-preferred output text, which does not mean that the non-preferred output text is completely wrong. θ Too much penalty is given to non-preferred output text, which in turn affects the target language model π θ The probability distribution over the preferred output text. The impact of this over-penalization leads to the output error of the final model.

[0090] The embodiment of this specification reduces the penalty on non-preferred output text by adding additional restrictions on the penalty, and changes the target language model π θ The probability distribution of θ When the probability distribution is as expected, the embodiments of the present specification reduce the probability skewness on long-tail characters or rare characters in the character distribution, thereby ensuring that the training results are more robust.

[0091] According to an embodiment of another aspect, a device for training a language model is also provided. Figure 4 A schematic block diagram of an apparatus for training a language model according to an embodiment is shown, and the apparatus can be deployed in any device, platform or device cluster with computing and processing capabilities. Figure 4 As shown, the device 400 includes:

[0092] The probability determination unit 402 is configured to input the target input text into the target language model and the pre-trained reference language model respectively, and obtain the probability values ​​of the two models respectively with respect to the first output text, and the probability values ​​of the two models respectively with respect to the second output text, wherein the first output text is marked as the preferred output text with respect to the target input text relative to the second output text;

[0093] The loss determination unit 404 is configured to determine a training loss, which is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes a difference between probability values ​​of the two models with respect to the first output text; and the second loss term includes a larger value between the difference between probability values ​​of the two models with respect to the second output text and 0;

[0094] The model training unit 406 is configured to adjust the parameters in the target language model with the goal of reducing the training loss.

[0095] According to another aspect of the embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any of the above embodiments.

[0096] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the above embodiments is implemented.

[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0098] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0100] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0101] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a language model, comprising: Inputting the target input text into the target language model and the pre-trained reference language model respectively, obtaining probability values ​​of the two models regarding the first output text and the second output text, respectively, wherein the first output text is marked as a preferred output text regarding the target input text relative to the second output text; Determining a training loss that is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes a difference between probability values ​​of the two models with respect to the first output text; and the second loss term includes a larger value between the difference between probability values ​​of the two models with respect to the second output text and 0; With the goal of reducing the training loss, the parameters in the target language model are adjusted.

2. The method according to claim 1, wherein: The parameters in the target language model are initialized based on the parameters in the pre-trained reference language model.

3. The method according to claim 1, wherein: The first output text and the second output text are obtained in the following manner: Input the target input text into the reference language model so that it outputs two output texts; A preference evaluation is performed on the two output texts with respect to the target input text, and according to the preference evaluation result, they are respectively determined as the first output text and the second output text.

4. The method according to claim 3, performing preference evaluation on the two output texts with respect to the target input text, comprising: Inputting the two output texts and the target input text into a preference evaluation model for preference evaluation; or, The two output texts and the target input text are sent to a manual marking platform, and a preference evaluation result is received.

5. The method according to claim 1, wherein: The target language model outputs a first output text with a first probability value when receiving the target input text; When receiving the target input text, the reference language model outputs the first output text with the second probability value; the first loss term includes the logarithmic value of the ratio of the first probability value to the second probability value.

6. The method according to claim 1, wherein: The target language model outputs a second output text with a third probability value when receiving the target input text; When receiving the target input text, the reference language model outputs a second output text with a fourth probability value; the second loss term includes a larger value between a logarithm of a ratio of the third probability value to the fourth probability value and 0.

7. The method according to claim 1, wherein: The training loss also includes a preset first coefficient; the training loss includes the product of the difference between the first loss term and the second loss term and the first coefficient.

8. A device for training a language model, comprising: A probability determination unit is configured to input the target input text into the target language model and the pre-trained reference language model respectively, and obtain the probability values ​​of the two models respectively with respect to the first output text, and the probability values ​​of the two models respectively with respect to the second output text, wherein the first output text is marked as a preferred output text with respect to the target input text relative to the second output text; A loss determination unit is configured to determine a training loss, which is negatively correlated with a first loss term and positively correlated with a second loss term; wherein the first loss term includes a difference between probability values ​​of the two models with respect to the first output text; and the second loss term includes a larger value between the difference between probability values ​​of the two models with respect to the second output text and 0; The model training unit is configured to adjust the parameters in the target language model with the goal of reducing the training loss.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Construction method and device for large language model, question and answer method and electronic equipment

    CN118485148A

Cited By

  • Large language model adaptive multi-preference alignment method and system based on hybrid expert architecture

    CN119783748A