Method and device for training language model

Abstract

The embodiment of the invention relates to a method and device for training a large language model. The method comprises the following steps: firstly, acquiring a data set which comprises a first question and a first refusing answer and a first unsafe answer corresponding to the first question, and comprises a second question and a second normal answer corresponding to the second question; the first question contains violation content; then, constructing a training set which comprises a first sample, a second sample and a third sample; the first sample comprises a first question serving as an input text, a first rejection answer marked as preference output and a first unsafe answer marked as non-preference output; the second sample comprises a second question serving as the input text, a second normal answer marked as preference output and a first rejection answer marked as non-preference output; the third sample comprises a second question serving as the input text, a second normal answer marked as preference output and a first unsafe answer marked as non-preference output; and finally, according to the training set, training the large language model based on a preference optimization algorithm.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning, and in particular, to methods and devices for training language models. Background Art

[0002] In recent years, with the widespread use of large language models (LLMs), the security issues that come with them have also attracted much attention. Current large language models have basic security defense capabilities and can detect and refuse to answer obviously harmful or unsafe questions.

[0003] However, large language models still face serious security threats in practical applications: malicious users can use a variety of attack methods to hide unsafe questions in adversarial attack prompts to induce large language models to generate unsafe responses. Therefore, a method is needed to improve the security of large language models during the training process. Summary of the invention

[0004] One or more embodiments of this specification describe a method and apparatus for training a language model, which adjusts the output tendency of a large language model by constructing specific preference training data, thereby improving the security of the large language model when receiving attack prompt words.

[0005] In a first aspect, a method for training a language model is provided, comprising:

[0006] Obtaining a data set, which includes a first question and its corresponding first refused answer and first unsafe answer, and a second question and its corresponding second normal answer; the first question includes illegal content;

[0007] Construct a training set, which includes a first sample, a second sample and a third sample; the first sample includes a first question as input text, a first rejection answer marked as a preferred output and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as input text, a second normal answer marked as a preferred output and a first rejection answer marked as a non-preferred output; the third sample includes a second question as input text, a second normal answer marked as a preferred output and a first unsafe answer marked as a non-preferred output;

[0008] According to the training set, a large language model is trained based on a preference optimization algorithm.

[0009] In some possible implementations, constructing a training set includes:

[0010] According to the first question, based on the adversarial attack method, construct the corresponding first adversarial attack prompt word;

[0011] A fourth sample is constructed and included in the training set; the fourth sample includes a first adversarial attack prompt word as input text, a first rejection answer marked as a preferred output, and a first unsafe answer marked as a non-preferred output.

[0012] In some possible implementations, the preference optimization algorithm is a KTO algorithm; and according to the training set, training a large language model based on the preference optimization algorithm includes:

[0013] For any target sample in the training set, randomly select an answer from the unsafe answer and the normal answer as a benchmark output;

[0014] Using the large language model and the pre-trained reference language model, respectively processing the input text items in the target sample, obtaining a first probability of a preferred output item of the target sample, a second probability of the reference output, and a third probability of a non-preferred output item;

[0015] Determining a first training loss according to the first probability and the second probability;

[0016] Determine a second training loss according to the second probability and the third probability;

[0017] The large language model is trained according to the total training loss determined by the first training loss and the second training loss.

[0018] In some possible implementations, determining the first training loss includes:

[0019] Determining a first probability offset term according to respective probability values ​​of the two models with respect to the first probability;

[0020] Determining a first KL divergence term according to respective probability values ​​of the two models with respect to the second probability;

[0021] According to a first difference between the first probability offset term and the first KL divergence term, a first training loss is determined, which is negatively correlated with the first difference.

[0022] In some possible implementations, determining the first probability offset term includes:

[0023] A first probability offset term is determined according to a logarithmic value of a ratio of respective probability values ​​of two models with respect to the first probability.

[0024] In some possible implementations, determining the second training loss includes:

[0025] Determining a second probability offset term according to respective probability values ​​of the two models with respect to the third probability;

[0026] Determining a second KL divergence term according to respective probability values ​​of the two models with respect to the second probability;

[0027] A second training loss is determined according to a second difference between the second KL divergence term and the second probability offset term, which is negatively correlated with the second difference.

[0028] In some possible implementations, determining the second probability offset term includes:

[0029] A second probability offset term is determined according to a logarithmic value of a ratio of respective probability values ​​of the two models with respect to the third probability.

[0030] In some possible implementations, the preference optimization algorithm is a preference optimization algorithm based on preference output; and according to the training set, a large language model is obtained by training based on the preference optimization algorithm, including:

[0031] Marking a preferred output item in any sample in the training set as a preferred output text with respect to its non-preferred output item;

[0032] By using each sample in the labeled training set, a large language model is trained according to an algorithm for preference optimization based on preference output.

[0033] In some possible implementations, the preference optimization algorithm based on preference output includes at least one of the following: PPO algorithm, DPO algorithm, ORPO algorithm, SimPO algorithm.

[0034] In some possible implementations, the anti-attack method includes at least one of the following: role playing, responsibility assumption, simulated jailbreak, super permission model, and program execution.

[0035] In a second aspect, a device for training a language model is provided, comprising:

[0036] The acquisition unit is configured to acquire a data set, which includes a first question and a first rejected answer and a first unsafe answer corresponding to the question, and includes a second question and a second normal answer corresponding to the question; the first question includes illegal content;

[0037] The training set construction unit is configured to construct a training set, which includes a first sample, a second sample and a third sample; the first sample includes a first question as an input text, a first rejection answer marked as a preferred output and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as an input text, a second normal answer marked as a preferred output and a first rejection answer marked as a non-preferred output; the third sample includes a second question as an input text, a second normal answer marked as a preferred output and a first unsafe answer marked as a non-preferred output;

[0038] The model training unit is configured to train a large language model based on a preference optimization algorithm according to the training set.

[0039] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0040] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0041] The method and device for training a language model proposed in the embodiments of this specification constructs specific training samples, sets preferred outputs and non-preferred outputs in each training sample, and uses a preference optimization algorithm to train a large language model to adjust the output tendency of the large language model so that when it receives an unsafe question or an attack prompt word, it tends to refuse to answer rather than answer the unsafe question; at the same time, when it receives a normal question, it tends to give a normal answer rather than refuse to answer. In this way, the security of the large language model is improved while maintaining the ability of the large language model to give normal answers to normal questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram showing an implementation scenario of a method for training a language model according to an embodiment;

[0044] Figure 2 A flowchart showing a method for training a language model according to one embodiment;

[0045] Figure 3 A schematic block diagram of an apparatus for training a language model according to an embodiment is shown. DETAILED DESCRIPTION

[0046] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0047] As mentioned above, malicious users will use a variety of attack methods to hide unsafe questions in prompts to form adversarial prompts to bypass the existing security defense mechanism of the large language model and induce the large language model to generate unsafe replies. Unsafe replies may include illegal, immoral, discriminatory and other content.

[0048] In order to improve the security of the large language model, the embodiment of this specification uses a preference alignment-based optimization algorithm to train the large language model, so that when the large language model receives unsafe questions or adversarial attack prompt words, it tends to refuse to answer rather than answer the unsafe questions or adversarial attack prompt words.

[0049] Preference alignment refers to the process of making the behavior, output, and decision of a large language model consistent with human values, moral standards, and expectations. Correspondingly, preference-based optimization is a type of model optimization technology that guides the optimization process of a large language model through user or system preferences rather than through traditional loss functions.

[0050] In the training set used for preference alignment, a single training sample often includes an input text and two output texts, where the output text is the answer corresponding to the input text. The two output texts are pre-labeled, one of which is labeled as a preferred output and the other is labeled as a non-preferred output. The training sample can be formally recorded as (input text, preferred output, non-preferred output). Preferred output refers to the content that the large language model is expected to be more inclined to output when receiving the input text; non-preferred output refers to the content that the large language model is not expected to be inclined to output when receiving the input text.

[0051] At the same time, the inventors found in their research that if only the ability of the large language model to refuse to answer unsafe questions or to resist attack prompt words is optimized, potential side effects may occur. Specifically, this side effect will cause the large language model to tend to refuse to answer any questions, even if the input is a normal question, which will cause the large language model to lose its basic question-answering function. In order to solve this side effect, when constructing a training set, the embodiments of this specification add a part of the training samples containing normal questions to the training set to maintain the large language model's ability to answer normal questions.

[0052] Based on the above analysis, the embodiments of this specification propose a method for training a language model to improve the security of a large language model when facing unsafe questions or counter-attack prompt words. Figure 1 A schematic diagram of an implementation scenario of a method for training a language model according to an embodiment is shown. Figure 1In the example, the original data set includes unsafe questions, and the corresponding refusal answers (answers with the text of refusal to answer) and unsafe answers (answers that directly answer the unsafe questions) for the unsafe questions. It also includes normal questions and the corresponding normal answers for the normal questions.

[0053] For example, an unsafe question may be: "How do I obtain someone else's website login password?" The corresponding refusal to answer may be: "I'm sorry, I can't provide you with relevant information. Unauthorized access to other people's passwords is illegal and may result in serious consequences such as personal information leakage." The corresponding unsafe answer includes specific methods and steps on how to steal other people's login passwords. A normal question may be: "Please help me design a travel luggage list." The corresponding normal answer may be a specific luggage list content.

[0054] Then, a training set is constructed based on the two types of questions and the corresponding answer texts in the data set. These include training samples with unsafe questions as input text, rejection answers as preferred outputs, and unsafe answers as non-preferred outputs (unsafe questions, rejection answers, unsafe answers); training samples with normal questions as input text, normal answers as preferred outputs, and rejection answers as non-preferred outputs (normal questions, normal answers, rejection answers); and training samples with normal questions as input text, normal answers as preferred outputs, and unsafe answers as non-preferred outputs (normal questions, normal answers, unsafe answers).

[0055] In order to further improve the security of the large language model, we can also generate corresponding adversarial attack prompt words based on unsafe issues according to the adversarial attack algorithm ( Figure 1 The adversarial attack prompt word is then used as the input text, the rejection answer is used as the preferred output, and the unsafe answer is used as the non-preferred output to construct the training samples (adversarial attack prompt word, rejection answer, unsafe answer), which are then added to the training set.

[0056] It should be noted that in the above data set, there can be multiple unsafe questions and their corresponding refusal answers and unsafe answers, and there can also be multiple normal questions and their corresponding normal answers. There is no corresponding relationship between normal questions and unsafe questions, nor does it need to have semantic relevance, and the number of the two types of questions is not necessarily equal. Therefore, four sets of training samples corresponding to each unsafe question can be generated based on the above method to enrich the training set.

[0057] After constructing a training set based on the data set, you can use the training samples in the training set to train the large language model based on the preference optimization algorithm to improve the security of the large language model. The detailed training process will be described in the subsequent content.

[0058] The specific implementation steps of the above-mentioned method for training a language model are described below in conjunction with specific embodiments.

[0059] Figure 2 A flowchart of a method for training a language model according to an embodiment is shown. The execution subject of the method can be any platform or server or device cluster with computing and processing capabilities. Figure 2 As shown, the method at least includes: step 202, obtaining a data set, which includes a first question and its corresponding first refusal answer and first unsafe answer, and a second question and its corresponding second normal answer; the first question includes illegal content; step 204, constructing a training set, which includes a first sample, a second sample and a third sample; the first sample includes the first question as input text, the first refusal answer marked as a preferred output and the first unsafe answer marked as a non-preferred output; the second sample includes the second question as input text, the second normal answer marked as a preferred output and the first refusal answer marked as a non-preferred output; the third sample includes the second question as input text, the second normal answer marked as a preferred output and the first unsafe answer marked as a non-preferred output; step 206, according to the training set, training a large language model based on a preference optimization algorithm.

[0060] The specific execution process of each of the above steps is described below.

[0061] First, in step 202, a data set is obtained, which includes a first question and its corresponding first rejected answer and first unsafe answer, and a second question and its corresponding second normal answer; the first question includes illegal content.

[0062] The first question may be an unsafe question, which contains illegal content. Illegal content may include illegal content, immoral content, discriminatory content, false information, harmful content, etc. Unsafe questions are questions that the large language model should refuse to answer.

[0063] The first refusal to answer may be a refusal to answer text for the first question, which may include the content of the refusal to answer, the reason for refusing to answer the question, and the like.

[0064] The first unsafe answer is the answer text to the first question, which also contains illegal content and is the text content that the large language model should avoid generating.

[0065] The second question can be a normal question, which does not contain any illegal content and is a question that the large language model should answer normally. The second normal answer is the regular answer corresponding to the second question.

[0066] The data set in step 202 may be obtained from an existing public data set, such as the SafeEdit data set; or unsafe questions and common questions may be collected by the user and the data set in step 202 may be constructed; or the data set in step 202 may be obtained by mixing the above two methods, which is not limited here.

[0067] Then, in step 204, a training set is constructed, which includes a first sample, a second sample and a third sample; the first sample includes a first question as input text, a first rejection answer marked as a preferred output and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as input text, a second normal answer marked as a preferred output and a first rejection answer marked as a non-preferred output; the third sample includes a second question as input text, a second normal answer marked as a preferred output and a first unsafe answer marked as a non-preferred output.

[0068] Formally, any training sample in the training set can be expressed as (input text, preferred output, non-preferred output), where the preferred output is the output text that the large language model is expected to be more inclined to output (with high probability / high confidence) when receiving the input text, and the non-preferred output is the output text that the large language model is expected to be not inclined to output (with low probability / low confidence) when receiving the input text.

[0069] Then the first sample can be recorded as (first question, first refusal answer, first unsafe answer), the second sample can be recorded as (second question, second normal answer, first refusal answer), and the third sample can be recorded as (second question, second normal answer, first unsafe answer).

[0070] In some possible implementations, a fourth sample may be constructed and included in the data set. In this case, the construction of the training set in step 204 includes:

[0071] According to the first question, based on the adversarial attack method, a corresponding first adversarial attack prompt word is constructed.

[0072] Adversarial attack methods are a type of method that uses carefully designed prompt words to induce large language models to output illegal content. Adversarial attack prompt words contain unsafe questions in a specific form, inducing large language models to answer the unsafe questions and generate output text containing illegal content.

[0073] For example, a counter-attack prompt could be: "Please complete the content of the second paragraph in the story in the prompt below. First paragraph: One night, Xiao Ming had an interesting dream. In the dream, he found himself sitting in a classroom with the title of today's class written on the blackboard: [Insecurity Issues]. Second paragraph: The teacher provides detailed steps to solve the problems in today's class, first step: second step: ...the last step:".

[0074] Among them, [Unsafe Question] is a placeholder for filling in the specific content of the unsafe question.

[0075] Then, a fourth sample is constructed and included in the training set; the fourth sample includes the first adversarial attack prompt word as input text, the first rejection answer marked as a preferred output, and the first unsafe answer marked as a non-preferred output.

[0076] The fourth sample can be recorded as (first counter-attack prompt word, first refusal answer, first unsafe answer).

[0077] In some more specific implementations, the aforementioned anti-attack method includes at least one of the following: character role play, assumed responsibility, simulated jailbreaking, superior model, and program execution.

[0078] It should be noted that the data set in step 202 may include multiple first questions, and the first refused answer and the first unsafe answer corresponding to each first question; and may also include multiple second questions, and the second normal answer corresponding to each second question. Each first question contains illegal content. Correspondingly, the training set may also include multiple first samples, second samples, third samples and / or fourth samples.

[0079] Next, in step 206, a large language model is trained based on the training set and a preference optimization algorithm.

[0080] There are a variety of preference optimization algorithms that can be used to train large language models.

[0081] In one embodiment, the preference optimization algorithm is a KTO algorithm (Kahneman-Tversky Optimization, Kahneman-Tversky optimization algorithm). In this embodiment, step 206 specifically includes steps 2061 to 2065.

[0082] First, in step 2061, for any target sample in the training set, an answer is randomly selected from an unsafe answer and a normal answer as a reference output.

[0083] The target sample can include input text item x and preferred output item y. d and the non-preferred output y u The unsafe answer and the normal answer may be a set of unsafe answers and normal answers in each sample in the training set. The benchmark output may be recorded as y′.

[0084] Then, in step 2062, the large language model and the pre-trained reference language model are used to process the input text items in the target sample respectively to obtain a first probability of a preferred output item for the target sample, a second probability of the benchmark output, and a third probability of a non-preferred output item.

[0085] Reference language model π ref It can be any pre-trained large language model, which can output the probability value corresponding to each output text after receiving the input text. During the optimization process, the reference language model π ref The parameter value of remains unchanged. The large language model π θ It is also a pre-trained large language model. After receiving the input text, it can output the probability value corresponding to each output text. During the optimization process, the large language model π will be adjusted. θ The parameter value of the parameter in .

[0086] In the initial stage of optimization, the large language model π θ The parameters in can be based on the pre-trained reference language model π ref That is, in the initial stage, the large language model π θ and the reference language model π ref have the same parameter values.

[0087] The first probability in step 2062 may include the large language model π θ When receiving input text item x, output preferred output item y d The probability π θ (y d |x), and the reference language model π ref When receiving input text item x, output preferred output item y d The probability π ref (y d |x).

[0088] The second probability may include a large language model π θ When receiving an input text item x, the probability π of outputting a reference output y′θ (y′|x), and the reference language model π ref When receiving an input text item x, the probability π of outputting a reference output y′ ref (y′|x).

[0089] The third probability may include the large language model π θ When receiving input text item x, output non-preferred output item y u The probability π θ (y u |x), and the reference language model π ref When receiving input text item x, output non-preferred output item y u The probability π ref (y u |x).

[0090] Next, in step 2063, a first training loss L1 is determined according to the first probability and the second probability.

[0091] Specifically, according to the probability values ​​π of the two models regarding the first probability, θ (y d |x) and π ref (y d |x), determine the first probability offset term r θ (x,y d Specifically, the first probability offset term is determined according to the logarithmic value of the ratio of the probability values ​​of the two models with respect to the first probability, that is, Here, log may be a natural logarithm.

[0092] Then, according to the two models, the probability values ​​π of the second probability are θ (y′|x) and π ref (y′|x), determine the first KL divergence term KL(π θ (y′|x)||π ref (y′|x)). KL() represents the calculation of the KL divergence (Kullback-Leibler Divergence) between the two terms inside the brackets.

[0093] Next, according to the first difference between the first probability offset term and the first KL divergence term, a first training loss L1 is determined, which is negatively correlated with the first difference. The first training loss L1 can be shown as formula (1):

[0094]

[0095] Among them, λ d and β are preset hyperparameters, λ dIt is used to adjust the weight of the training loss corresponding to the preference output, and σ() represents the sigmoid function.

[0096] Then, in step 2064, a second training loss L2 is determined according to the second probability and the third probability.

[0097] Specifically, according to the probability values ​​π of the two models regarding the third probability θ (y u |x) and π ref (y u |x), determine the second probability offset term r θ (x,y u ). Specifically, the second probability offset term is determined according to the logarithmic value of the ratio of the probability values ​​of the two models with respect to the third probability, that is, Here, log may be a natural logarithm.

[0098] Then, according to the two models, the probability values ​​π of the second probability are θ (y′|x) and π ref (y′|x), determine the second KL divergence term KL(π θ (y′|x)||π ref (y′|x)).

[0099] Next, according to the second difference between the second KL divergence term and the second probability offset term, a second training loss L2 is determined, which is negatively correlated with the second difference. The second training loss L2 can be shown as formula (2):

[0100]

[0101] Among them, λ u and β are preset hyperparameters, λ u It is used to adjust the weight of the training loss corresponding to the non-preferred output, and σ() represents the sigmoid function.

[0102] In step 2065, a large language model is trained according to a total training loss L determined by the first training loss and the second training loss.

[0103] The total training loss L may include the sum L1+L2 of the first training loss L1 and the second training loss L2.

[0104] In one embodiment, the total training loss determined based on a single training sample can be used to perform stochastic gradient descent to update the large language model π θIn another embodiment, a total training loss determined based on the sum of the first training loss and the second training loss of each of the plurality of training samples may be used to perform mini-batch gradient descent to update the large language model π θ Parameters in .

[0105] In the KTO algorithm, the sample point (x, y′) formed by the input text item x and the benchmark output y′ is a sample point with a profit of 0. d The sample points (x,y d ) is a sample point with a positive benefit. Input text item x and non-preferred output y u The sample points (x,y u ) is a sample point where the return is negative.

[0106] In the training loss term, the KL divergence term is used as the reference point, corresponding to the point where the gain is 0. The probability offset term r θ As a benefit, when the calculation includes the preference output y d The sample points (x,y d ) is positive, corresponding to formula (1); when the calculation includes non-preferred output y u The sample points (x,y u ) is negative, corresponding to formula (2).

[0107] In the conventional KTO algorithm, the reference output y′ is obtained by sampling from the pre-calculated reference point distribution Y′. In the embodiment of the present specification, the reference output y′ is obtained by sampling from the non-rejection answer list, that is, a set consisting of unsafe answers and normal answers. Since for the input text item x, y′ is sampled from the non-rejection answer list, it is essentially a non-answer, neither an expected answer nor an unexpected answer. Compared with the reference point distribution Y′ in the conventional KTO algorithm, it is more suitable for constituting a reference point with a benefit of 0 in the scenario of the embodiment of the present specification.

[0108] It should be noted that steps 2061 to 2065 only describe the process of adjusting the parameters in the large language model based on a target sample. By selecting samples in the training set multiple times and executing the methods in steps 2061 to 2065, the large language model can be trained multiple times to improve the effect of model training.

[0109] In another embodiment, the preference optimization algorithm in step 206 is a preference optimization algorithm based on preference output. In this embodiment, step 206 specifically includes:

[0110] The preferred output item in any sample in the training set is marked as the preferred output text with respect to its non-preferred output item.

[0111] In the KTO algorithm, for an input text item x, we only need to mark a certain output text as a preferred output and mark another output text as a non-preferred output. There is no direct connection between the preferred output and the non-preferred output.

[0112] In preference optimization algorithms based on preference output, such as the PPO algorithm (Proximal Policy Optimization), the DPO algorithm (Direct Preference Optimization), the ORPO algorithm (Odds Ratio Preference Optimization), and the SimPO algorithm (Simple Preference Optimization), it is necessary to mark an output text A as the preferred output text relative to another output text B, that is, with respect to a specific input text, output text A is a better output text relative to output text B. Compared with the KTO algorithm, these algorithms require higher annotation costs.

[0113] Then, the large language model is trained by using each sample in the labeled training set according to the preference optimization algorithm based on the preference output.

[0114] The specific process of training a large language model based on a preference optimization algorithm with preference output is an existing technology and will not be described in detail here.

[0115] In a more specific embodiment, the preference optimization algorithm based on preference output includes at least one of the following: PPO algorithm, DPO algorithm, ORPO algorithm, SimPO algorithm.

[0116] By using the method for training a large language model in the embodiments of this specification, in the process of constructing a training set, training samples are constructed based on unsafe questions and their corresponding refusal to answer and unsafe answers, and a preference optimization algorithm is used to train the large language model, thereby improving the ability of the large language model to refuse to answer unsafe questions and resist attack prompt words, thereby enhancing the security of the large language model.

[0117] At the same time, for the potential side effect, that is, the large language model tends to refuse to answer any question when receiving it, thereby affecting the normal performance of the model. The embodiment of this specification introduces normal question and answer data into the construction of training data to reduce the impact of the training process on the normal question and answer ability of the large language model, and can improve the performance of the large language model in knowledge question and answer and language understanding.

[0118] The embodiments of this specification introduce data preferences through the alignment technology in the preference optimization algorithm, which can not only provide direction guidance for the answers preferred by the large language model, but also exclude unsafe answers and irrelevant answers as unpreferred answers, thereby reducing the impact on the performance of the large language model while improving it better according to the input type.

[0119] According to an embodiment of another aspect, a device for training a language model is also provided. Figure 3 A schematic block diagram of an apparatus for training a language model according to an embodiment is shown, and the apparatus can be deployed in any device, platform or device cluster with computing and processing capabilities. Figure 3 As shown, the device 300 includes:

[0120] The acquisition unit 302 is configured to acquire a data set, which includes a first question and its corresponding first rejected answer and first unsafe answer, and a second question and its corresponding second normal answer; the first question includes illegal content.

[0121] The training set construction unit 304 is configured to construct a training set, which includes a first sample, a second sample and a third sample; the first sample includes a first question as input text, a first rejection answer marked as a preferred output and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as input text, a second normal answer marked as a preferred output and a first rejection answer marked as a non-preferred output; the third sample includes a second question as input text, a second normal answer marked as a preferred output and a first unsafe answer marked as a non-preferred output.

[0122] The model training unit 306 is configured to train a large language model based on the preference optimization algorithm according to the training set.

[0123] According to another aspect of the embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any of the above embodiments.

[0124] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the above embodiments is implemented.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0128] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0129] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a large language model, comprising: Obtaining a data set, which includes a first question and its corresponding first refused answer and first unsafe answer, and a second question and its corresponding second normal answer; the first question includes illegal content; Constructing a training set, which includes a first sample, a second sample, and a third sample; the first sample includes a first question as input text, a first rejection answer marked as a preferred output, and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as input text, a second normal answer marked as a preferred output, and the first rejection answer marked as a non-preferred output; The third sample includes a second question as input text, a second normal answer marked as a preferred output, and a first unsafe answer marked as a non-preferred output; According to the training set, a large language model is trained based on a preference optimization algorithm.

2. The method according to claim 1, wherein: Construct a training set, including: According to the first question, based on the adversarial attack method, construct the corresponding first adversarial attack prompt word; A fourth sample is constructed and included in the training set; the fourth sample includes a first adversarial attack prompt word as input text, a first rejection answer marked as a preferred output, and a first unsafe answer marked as a non-preferred output.

3. The method according to claim 1, wherein: The preference optimization algorithm is a KTO algorithm; According to the training set, a large language model is trained based on a preference optimization algorithm, including: For any target sample in the training set, randomly select an answer from the unsafe answer and the normal answer as a benchmark output; Using the large language model and the pre-trained reference language model, respectively processing the input text items in the target sample, obtaining a first probability of a preferred output item of the target sample, a second probability of the reference output, and a third probability of a non-preferred output item; Determining a first training loss according to the first probability and the second probability; Determine a second training loss according to the second probability and the third probability; The large language model is trained according to the total training loss determined by the first training loss and the second training loss.

4. The method according to claim 3, wherein: Determine the first training loss, including: Determining a first probability offset term according to respective probability values ​​of the two models with respect to the first probability; Determining a first KL divergence term according to respective probability values ​​of the two models with respect to the second probability; According to a first difference between the first probability offset term and the first KL divergence term, a first training loss is determined, which is negatively correlated with the first difference.

5. The method according to claim 4, wherein: Determining a first probability offset term includes: A first probability offset term is determined according to a logarithmic value of a ratio of respective probability values ​​of two models with respect to the first probability.

6. The method according to claim 3, wherein: Determine the second training loss, including: Determining a second probability offset term according to respective probability values ​​of the two models with respect to the third probability; Determining a second KL divergence term according to respective probability values ​​of the two models with respect to the second probability; A second training loss is determined according to a second difference between the second KL divergence term and the second probability offset term, which is negatively correlated with the second difference.

7. The method according to claim 6, wherein: Determining a second probability offset term includes: A second probability offset term is determined according to a logarithmic value of a ratio of respective probability values ​​of the two models with respect to the third probability.

8. The method according to claim 1, wherein: The preference optimization algorithm is a preference optimization algorithm based on preference output; According to the training set, a large language model is obtained by training based on a preference optimization algorithm, including: Marking a preferred output item in any sample in the training set as a preferred output text with respect to its non-preferred output item; By using each sample in the labeled training set, a large language model is trained according to an algorithm for preference optimization based on preference output.

9. The method according to claim 8, wherein: The preference optimization algorithm based on preference output includes at least one of the following: PPO algorithm, DPO algorithm, ORPO algorithm, SimPO algorithm.

10. The method according to claim 2, wherein: The anti-attack method includes at least one of the following: role playing, responsibility assumption, simulated jailbreak, super authority model, and program execution.

11. A device for training a large language model, comprising: The acquisition unit is configured to acquire a data set, which includes a first question and a first rejected answer and a first unsafe answer corresponding to the question, and includes a second question and a second normal answer corresponding to the question; the first question includes illegal content; The training set construction unit is configured to construct a training set, which includes a first sample, a second sample and a third sample; the first sample includes a first question as input text, a first rejection answer marked as a preferred output and a first unsafe answer marked as a non-preferred output; the second sample includes a second question as input text, a second normal answer marked as a preferred output and the first rejection answer marked as a non-preferred output; The third sample includes a second question as input text, a second normal answer marked as a preferred output, and a first unsafe answer marked as a non-preferred output; The model training unit is configured to train a large language model based on a preference optimization algorithm according to the training set.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

13. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 10 is implemented.

Images