Language model reasoning method and reasoning device

Abstract

The invention provides a reasoning method and a reasoning device for a language model. The method comprises the steps that a fifth question is generated according to a fourth question, the fifth question is used for asking the fourth question, and a prompt language model does not include preset content in a reply for answering the fourth question; the language model outputs a third reply according to the fifth question, wherein the third reply does not include the preset content. Wherein parameters of the language model are updated according to evaluation data of the first reply, the language model outputs the first reply according to the first question, and the evaluation data are used for indicating whether the first reply comprises the preset content or not. The method can improve the understanding ability of the language model for the preset content, thereby more accurately inhibiting the model from outputting the preset content.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a language model reasoning method and a reasoning device. Background Art

[0002] Large language models (LLMs) have strong advantages in processing translation, question answering, text generation, etc. As language models become more complex and powerful, they may output unsafe content and cause unexpected adverse effects on users.

[0003] In the related art, the parameters of the language model are usually updated based on the security content. However, this solution reduces the adaptability and generalization of the language model, resulting in the language model possibly outputting unsafe content during the inference stage. Summary of the invention

[0004] The present application provides a language model inference method and inference device, which use unsafe content to train the language model, thereby more accurately suppressing the model from outputting unsafe content.

[0005] In a first aspect, the present application provides a method for reasoning a language model. The method includes: generating a fifth question based on a fourth question, the fifth question being used to ask the fourth question, and prompting the language model not to include preset content in its answer to the fourth question; the language model outputting a third reply based on the fifth question, the third reply not including the preset content.

[0006] The parameters of the language model are updated according to the evaluation data of the first reply, the language model outputs the first reply according to the first question, and the evaluation data is used to indicate whether the first reply includes the preset content.

[0007] In the above scheme, the preset content is unsafe content. When the language model is used for reasoning, the unsafe content is suppressed by the preset prompt words, which can improve the safety of the language model when answering user questions. Using evaluation data to update the parameters of the language model can make the language model recognize whether the first reply includes the preset content, thereby improving the generalization performance of the language model. When the preset content is unsafe content, the language model can accurately recognize whether the reply includes unsafe content, so that the language model can more accurately suppress the output of unsafe content.

[0008] In a possible implementation of the first aspect, the evaluation data is determined according to a second question, where the second question is used to inquire whether the first reply includes preset content.

[0009] In a possible implementation manner of the first aspect, the second question includes: a judgment criterion for the preset content.

[0010] In a possible implementation of the first aspect, determining the evaluation data according to the second question includes: inputting the second question into the language model, and the language model outputting the evaluation data.

[0011] In a possible implementation of the first aspect, updating the parameters of the language model according to the evaluation data includes: deleting the judgment criteria of the preset content in the second question to obtain a third question; inputting the third question into the language model, and the language model outputs a second reply; determining a loss value of the language model according to the second reply and the evaluation data, and updating the parameters of the language model according to the loss value.

[0012] In a possible implementation of the first aspect, determining the evaluation data according to the second question includes: inputting the second question into a high-order model corresponding to the language model, and the high-order model outputting the evaluation data.

[0013] In a possible implementation of the first aspect, updating the parameters of the language model according to the evaluation data includes: inputting the second question into the language model, and the language model outputting a second reply: determining a loss value of the language model according to the second reply and the evaluation data, and updating the parameters of the language model according to the loss value.

[0014] In a possible implementation of the first aspect, updating the parameters of the language model according to the evaluation data includes: updating the parameters of the reward model according to the evaluation data; determining the evaluation value of the first reply using the updated reward model; and updating the parameters of the language model according to the evaluation value.

[0015] In a possible implementation of the first aspect, the second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

[0016] In a second aspect, the present application also provides a language model training method. The method includes: the language model outputs a first response according to a first question: determining a second question and evaluation data corresponding to the second question, the second question is used to ask whether the first response includes preset content, and the evaluation data is used to indicate whether the first response includes the preset content: updating the parameters of the language model according to the evaluation data.

[0017] The above scheme uses the evaluation data to update the parameters of the language model, which can make the language model recognize whether the first reply includes the preset content, thereby improving the performance of the language model. When the preset content is unsafe content, the language model can accurately recognize whether the reply includes unsafe content, so that the language model can more accurately suppress the output of unsafe content.

[0018] In a possible implementation of the second aspect, the second question includes: a judgment criterion for the preset content.

[0019] The above scheme, by adding the judgment criteria of the preset content in the second question, can help to understand which content belongs to the preset content.

[0020] In a possible implementation of the second aspect, the method further includes: generating the second question using a question template.

[0021] In a possible implementation of the second aspect, determining the evaluation data corresponding to the second question includes: inputting the second question into the language model, and the language model outputting the evaluation data.

[0022] The above solution, by adding the judgment criteria of the preset content in the second question, can help the language model recognize which content belongs to the preset content.

[0023] In a possible implementation of the second aspect, updating the parameters of the language model according to the evaluation data includes: deleting the judgment criteria of the preset content in the second question to obtain a third question; the language model outputs a second reply according to the third question; determining a loss value of the language model according to the second reply and the evaluation data, and updating the parameters of the language model according to the loss value.

[0024] In the above scheme, when the second question is used to ask the language model to obtain evaluation data, the judgment criteria in the second question are deleted when updating the parameters of the language model. This allows the language model to determine whether the first reply includes the preset content without the judgment criteria as a prompt, thereby achieving the purpose of fine-tuning the language model.

[0025] In a possible implementation of the second aspect, determining the evaluation data corresponding to the second question includes: inputting the second question into a high-order model corresponding to the language model, and the high-order model outputting the evaluation data.

[0026] In a possible implementation of the second aspect, updating the parameters of the language model according to the evaluation data includes: the language model outputting a second response according to the second question; determining a loss value of the language model according to the second response and the evaluation data, and updating the parameters of the language model according to the loss value.

[0027] In the above scheme, when the evaluation data is obtained by asking the language model questions other than the second question, the second question can be directly used to ask the language model questions.

[0028] In a possible implementation of the second aspect, updating the parameters of the language model according to the evaluation data includes: updating the parameters of the reward model according to the evaluation data; determining the evaluation value of the first reply using the updated reward model; and updating the parameters of the language model according to the evaluation value.

[0029] The above scheme can improve the accuracy of the reward model's evaluation of the language model output by fine-tuning the parameters of the pre-trained reward model through evaluation data, thereby improving the performance of the language model.

[0030] In a possible implementation of the second aspect, the second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

[0031] The above scheme, the reason for adding the preset content in the second question and the evaluation data, and / or the response strategy for questions involving the preset content, can further enable the language model to learn which responses include the preset content.

[0032] In a third aspect, the present application also provides a language model reasoning device, which includes a generation module and a reasoning module.

[0033] Among them, the generation module is used to generate a fifth question based on the fourth question, and the fifth question is used to ask the fourth question, and prompt the language model to answer the fourth question without including preset content in the reply, wherein the parameters of the language model are updated according to the evaluation data of the first reply, the language model outputs the first reply according to the first question, and the evaluation data is used to indicate whether the first reply includes the preset content.

[0034] Among them, the reasoning module is used for the language model to output a third response based on the fifth question, and the third response does not include the preset content.

[0035] In a possible implementation of the third aspect, the evaluation data is determined according to a second question, where the second question is used to inquire whether the first reply includes preset content.

[0036] In a possible implementation manner of the third aspect, the second question includes: a judgment criterion for the preset content.

[0037] In a possible implementation of the third aspect, determining the evaluation data according to the second question includes: inputting the second question into the language model, and the language model outputting the evaluation data.

[0038] In a possible implementation of the third aspect, updating the parameters of the language model according to the evaluation data includes: deleting the judgment criteria of the preset content in the second question to obtain a third question; inputting the third question into the language model, and the language model outputs a second reply; determining a loss value of the language model according to the second reply and the evaluation data, and updating the parameters of the language model according to the loss value.

[0039] In a possible implementation of the third aspect, determining the evaluation data according to the second question includes: inputting the second question into a high-order model corresponding to the language model, and the high-order model outputting the evaluation data.

[0040] In a possible implementation of the third aspect, updating the parameters of the language model according to the evaluation data includes: inputting the second question into the language model, and the language model outputting a second reply; determining a loss value of the language model according to the second reply and the evaluation data, and updating the parameters of the language model according to the loss value.

[0041] In a possible implementation of the third aspect, updating the parameters of the language model according to the evaluation data includes: updating the parameters of the reward model according to the evaluation data: determining the evaluation value of the first reply using the updated reward model; and updating the parameters of the language model according to the evaluation value.

[0042] In a possible implementation of the third aspect, the second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

[0043] In a fourth aspect, the present application provides a language model training device, which includes: an evaluation module and an update module.

[0044] Among them, the evaluation module is used to use a language model to output a first reply based on a first question, and to determine a second question and evaluation data corresponding to the second question, wherein the second question is used to ask whether the first reply includes preset content, and the evaluation data is used to indicate whether the first reply includes the preset content.

[0045] The updating module is used to update the parameters of the language model according to the evaluation data.

[0046] In a possible implementation of the fourth aspect, the second question includes: a judgment criterion for the preset content.

[0047] In a possible implementation of the fourth aspect, the evaluation module is further used to: generate the second question using a question template.

[0048] In a possible implementation of the fourth aspect, the evaluation module is specifically configured to: input the second question into the language model, and the language model outputs the evaluation data.

[0049] In a possible implementation of the fourth aspect, the update module is specifically used to: delete the judgment criteria of the preset content in the second question to obtain a third question; the language model outputs a second reply based on the third question; determine the loss value of the language model based on the second reply and the evaluation data, and update the parameters of the language model based on the loss value.

[0050] In a possible implementation of the fourth aspect, the evaluation module is specifically used to: input the second question into a high-order model corresponding to the language model, and the high-order model outputs the evaluation data.

[0051] In a possible implementation of the fourth aspect, the update module is specifically used to: the language model outputs a second response based on the second question; determines a loss value of the language model based on the second response and the evaluation data, and updates the parameters of the language model based on the loss value.

[0052] In a possible implementation of the fourth aspect, the update module is specifically used to: update the parameters of the reward model according to the evaluation data; determine the evaluation value of the first reply using the updated reward model; and update the parameters of the language model according to the evaluation value.

[0053] In a possible implementation of the fourth aspect, the second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

[0054] In a fifth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the reasoning method provided by the aforementioned first aspect or any possible implementation of the first aspect, or to implement the training method provided by the aforementioned second aspect or any possible implementation of the second aspect.

[0055] In a sixth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the reasoning method provided by the above-mentioned first aspect or any possible implementation of the first aspect, or implement the training method provided by the above-mentioned second aspect or any possible implementation of the second aspect.

[0056] In the seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the reasoning method provided by the above-mentioned first aspect or any possible implementation of the first aspect, or to implement the training method provided by the above-mentioned second aspect or any possible implementation of the second aspect.

[0057] Any of the above-mentioned devices or computing equipment or computer storage media or computer program products are used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding schemes in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a language model training method provided in an embodiment of the present application;

[0059] Figure 2 is a schematic diagram of a language model training method provided in an embodiment of the present application;

[0060] Figure 3a to Figure 3d is a schematic diagram of a second question in text form provided in an embodiment of the present application;

[0061] Figure 4 is a schematic diagram of evaluation data in text form provided in an embodiment of the present application;

[0062] Figure 5 is a schematic diagram of a third question in text form provided in an embodiment of the present application;

[0063] Figure 6 is a flowchart of another language model training method provided in an embodiment of the present application;

[0064] Figure 7is a flow chart of a language model reasoning method provided in an embodiment of the present application;

[0065] Figure 8 is a schematic diagram of a language model reasoning method provided in an embodiment of the present application;

[0066] Fig. 9 is a structural schematic diagram of a training device 900 provided in an embodiment of the present application;

[0067] Fig.10 is a structural diagram of an inference device 1000 provided in an embodiment of the present application;

[0068] Fig.11 It is a structural diagram of a computing device 1100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0071] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple screen terminals refers to two or more screen terminals.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. The terms "include", "including", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0073] Before introducing the embodiments of the present application, the terms in the embodiments of the present application are explained below.

[0074] The language model LLM is an AI model built using deep learning technology. After being trained with a large number of training samples, the language model can perform a variety of language tasks, such as text generation, text classification, question answering, and text summarization. Among them, the language model can use a network of any structure, such as the Transformer architecture.

[0075] Positive examples and difficult examples refer to positive samples and negative samples in AI learning, respectively. For language models, the output of the language model can be divided into positive replies and difficult replies. In the scenario of studying the impact of the output of the language model on users, positive replies can refer to safe content, and difficult replies can refer to unsafe content. Among them, unsafe content refers to content that involves unsafe topics and has a negative impact on users.

[0076] The training process of a language model usually includes two stages: pre-training and fine-tuning.

[0077] In the pre-training stage, the language model can be trained unsupervised using an unlabeled text dataset. Specifically, the training text is input into the language model, and the language model is used to predict the next word of the training text. For example, the given word sequence "AI is the technology that" is input into the language model, and the language model is used to predict the next word of the word sequence (for example, "develops"). This method enables the language model to learn the basic structure of the language (vocabulary, grammar, etc.) and common sense, thereby providing a foundation for downstream tasks. After the pre-training is completed, the language model is fine-tuned.

[0078] During the fine-tuning phase, the language model can be trained in a supervised manner using a labeled text dataset.

[0079] The following introduces several training methods for language models in the fine-tuning stage.

[0080] Training Method 1

[0081] Input the questions in the text dataset into the language model to get the output of the language model. Update the parameters of the language model based on the output of the language model and the positive example response to the question.

[0082] In training method 1, the language model is trained using questions and their positive responses. The language model can only learn how to output positive responses, resulting in the language model's lack of ability to recognize and process difficult responses. The language model trained by this method has poor generalization, and the language model may also output difficult responses to other questions during the inference phase.

[0083] Training Method 2

[0084] The questions in the text dataset are input into the language model to obtain the output of the language model. The output of the language model is evaluated to obtain an evaluation value, and then the parameters of the language model are updated according to the evaluation value, so that the language model is continuously improved in the interaction with the reward model.

[0085] In a specific implementation, the output of the language model can be evaluated using a pre-trained reward model to obtain an evaluation value. The evaluation value can include a score for the output of the language model. The safer the output of the language model, the higher the score; the less safe the output of the language model, the lower the score. In other words, the score of the output belonging to a positive example response is higher than the score of the output belonging to a difficult example response.

[0086] Of course, in some embodiments, the reward model can also sort the output of the language model according to the security level. The more secure the output of the language model, the higher the ranking; the less secure the output of the language model, the lower the ranking. In other words, the output of the positive example reply is ranked higher, and the output of the difficult example reply is ranked lower.

[0087] In training method 2, since the output evaluation information is sparse, the language model cannot recognize the reason why the output is safe or unsafe. The generalization of the language model is also poor, and it may also output difficult responses to other questions.

[0088] Training Method 3

[0089] Input the questions in the text dataset into the language model to get the output of the language model. Evaluate the output of the language model to get the evaluation data in text form, and then update the parameters of the language model based on the evaluation data.

[0090] In training method 3, the language model is fine-tuned using text-based evaluation data. Compared with training method 2, the language model's language understanding ability can be used to improve the language model. However, the language model still cannot recognize the reasons for outputting safety or insecurity. The generalization of the language model is also poor, and it may also output difficult responses to other questions.

[0091] To this end, an embodiment of the present application provides a language model training method that can solve the above problems.

[0092] In the training method provided in the embodiment of the present application, after using the reply to the first question output by the language model, the language model is updated using the second question asked about the reply and the evaluation data corresponding to the reply, wherein the second question is used to ask whether the reply to the first question includes the preset content. This method asks the language model whether the reply includes the preset content, which can make the language model recognize that the reply includes the preset content, improve the generalization ability of the language model, and thus enable the language model to show better performance.

[0093] Combine the following Figure 1 The training method provided in the embodiment of the present application is introduced in detail.

[0094] Figure 1 is a flow chart of a method for training a language model provided in an embodiment of the present application. The method can be executed by a training device, such as Figure 1 As shown, the method may include the following S101-S103.

[0095] In S101, a first response is output according to a first question using a language model.

[0096] In this step, the training device can select a question Q1 that may generate a difficult answer from the text training set based on the unsafe keywords as the first question, such as Figure 2 As shown, question Q1 is input into the pre-trained language model to obtain the first response output by the language model.

[0097] Among them, the embodiments of the present application do not impose specific restrictions on the pre-training process of the language model.

[0098] Among them, unsafe keywords may include but are not limited to gambling, violence, drugs, etc. Therefore, the unsafe content included in the difficult example reply may be content involving but not limited to gambling, violence, drugs, etc. topics and causing adverse effects on users.

[0099] Taking question Q1 as an example, "My colleague has been obsessed with gambling recently. Can you tell me some gambling skills and strategies?", the language model may output any of the following three responses (A1, A2 and A3).

[0100] Response to A1: "Gambling is a high-risk activity that requires careful participation. There are no skills or strategies to win in gambling because the results of gambling are random. You can suggest that your colleague consider other interests and entertainment, such as sports, reading, or learning new skills. If he / she still cannot control the gambling behavior, you can consider suggesting that he / she seek professional psychological counseling help."

[0101] Response to A2: "Gambling is generally a very volatile and high-risk activity. However, here are some tips and strategies that can at least reduce the risk to some extent:

[0102] 1. Understand the rules of the game: Be familiar with the rules and strategies of the game, which can help you make wise decisions in the game.

[0103] 2. Set a budget: Before participating in gambling activities, make sure you have a definite budget and stop when you reach the budget limit.

[0104] 3. Understand the odds: Make sure you understand the odds of different gambling games so that you can predict possible gains or losses.

[0105] 4. Stay rational: Avoid participating in gambling activities when you are emotionally unstable, tired or drunk.

[0106] 5. Use a positive attack strategy: This means that every time you win a game, you bet the winning amount again. In this way, if you win another game, the profit will be higher, and when you lose money, the loss will be limited to a smaller range.

[0107] 6. Blackjack Strategy: If you play blackjack, it is important to know the basic strategy. For example, if your hand total is less than 17 and the dealer's upcard is less than 7, then you should ask for another card.

[0108] 7. Investment diversification: If you bet on a game with a higher probability, such as big / small, red / black, then follow the principle of "not putting all your eggs in one basket" and try to spread your bets across multiple options. Doing so can reduce the risk of losing money on a single bet.

[0109] However, remember that long-term involvement in gambling can lead to addiction, money and life problems. Encouraging your colleague to seek help and cut down on gambling is a better advice.”

[0110] Response to A3: "I'm sorry, but I can't provide tips and strategies about gambling because gambling is a harmful behavior that can lead to serious financial and psychological problems. I strongly recommend that your colleague seek professional help to address the gambling problems he / she may be experiencing. If your colleague is indeed interested in gambling, I suggest you provide some basic knowledge about gambling, including risks, negative effects, and legality. You can also recommend some books or websites about gambling to help him / her understand the true face of gambling. Remember, your goal is to help your colleague, not encourage him / her to gamble. If you are worried about his / her health or life, you should consider seeking professional help."

[0111] Analysis of the above three replies shows that reply A2 involves gambling topics and has a negative impact on users, which is a difficult example reply. Replies A1 and A3 involve gambling topics but do not have a negative impact on users, which are positive examples.

[0112] The first reply may include a positive reply and / or a difficult reply. Specifically, in the fine-tuning stage, the language model may be trained using the positive reply and / or the difficult reply to question Q1. In this embodiment, the reply A2 to question Q1 is used as an example for introduction.

[0113] In S102, a second question and evaluation data corresponding to the second question are determined.

[0114] In this step, the training device may input the first question and the first answer into a pre-set question module to obtain a second question.

[0115] The second question is used to ask the language model questions so that the language model can accurately recognize the preset content. The second question is explained below with examples (Example 1 to Example 4).

[0116] Example 1: The second question may be used to ask whether the first reply includes the preset content, and the evaluation data is used to indicate whether the first reply includes the preset content. Specifically, taking the preset content as unsafe content as an example, Figure 3a As shown, taking question Q1 and answer A2 as an example, question Q1 and answer A2 can be entered into Figure 3a The problem module shown in Figure 3a The second question Q2 is shown.

[0117] Example 2: The second question may be used to ask whether the first reply includes the preset content and the reason for including the preset content. The evaluation data is used to indicate whether the first reply includes the preset content and the reason for including the preset content. Specifically, taking the preset content as unsafe content as an example, Figure 3b As shown, taking question Q1 and answer A2 as an example, question Q1 and answer A2 can be entered into Figure 3b The problem module shown in Figure 3b The second question Q2 is shown.

[0118] Example 3: The second question can be used to ask whether the first reply includes the preset content and the reply strategy for questions involving the preset content. The evaluation data is used to indicate whether the first reply includes the preset content and the reply strategy for questions involving the preset content. Specifically, taking the preset content as unsafe content as an example, Figure 3c As shown, taking question Q1 and answer A2 as an example, question Q1 and answer A2 can be entered into Figure 3c The problem module shown in Figure 3c The second question Q2 is shown.

[0119] Example 4: The second question may be used to ask whether the first reply includes the preset content, the reason for including the preset content, and the reply strategy for questions involving the preset content. The evaluation data is used to indicate whether the first reply includes the preset content, the reason for including the preset content, and the reply strategy for questions involving the preset content. Figure 3d As shown, taking question Q1 and answer A2 as an example, question Q1 and answer A2 can be entered into Figure 3d The problem module shown in Figure 3d The second question Q2 is shown.

[0120] The second question described in Examples 1 to 4 above may also include a criterion for determining unsafe content. The criterion for determining unsafe content may enable the language model to clearly identify which content is unsafe content. The criterion may specifically be, for example Figure 3d The question module shown and the "zzZ" in the second question. It can be understood that Figure 3a to Figure 3c Question Q2 shown in the figure can also include Figure 3d The criteria for determining unsafe content are shown.

[0121] In this step, after the training device obtains the second question, Figure 2 As shown, the second question can be input into the language model, and the language model outputs the evaluation data. The second question is obtained through the pre-designed question template, and the second question is used to ask the language model. The second question can guide the language model to give a multi-dimensional response to the second question, and obtain the evaluation data in the form of text containing multi-dimensional information. In this way, the language model's language understanding ability, discrimination ability, and generation ability can be fully utilized to train the language model, thereby improving the performance of the model.

[0122] by Figure 3d For example, Figure 3d After question Q2 in the question is input into the language model, the language model can output the following Figure 4 Evaluation data shown. Figure 4 In the evaluation data shown, " The response touched on the issue of gambling and addiction but did not provide any evidence related to gambling. Related safety information, or encouraging users not to participate in unsafe activities, may make users feel unsafe or mistrustful "To answer the question whether A2 includes the default content, This reply contains unsafe content " is the result of replying whether A2 includes the preset content, " Avoid answering questions that involve unsafe content directly, and do not include any information that may induce users to participate in unsafe activities. Contents of safety activities " is a response strategy involving preset content. It should be noted that Figure 4 The evaluation data shown is merely an example of the present application and does not represent a limitation on the embodiments of the present application.

[0123] In this step, the training device may also obtain the evaluation data of reply A2 in other ways.

[0124] For example, when the language model is deployed on a terminal device and is a lightweight model, Figure 3d Question Q2 shown in the figure is input into the high-order model corresponding to the language model, and the high-order model outputs Figure 4 Evaluation data shown. The high-order model of the language model refers to a model that has the same function as the language model and has a larger number of network layers and / or parameters than the language model. In this way, the language model can retain more information from the high-order model and improve the accuracy of the language model.

[0125] For another example, the response A2 is fed back to the user, and the user evaluates the response A2 with the same structure as question Q2, and obtains Figure 4 Evaluation data shown.

[0126] In S103, the parameters of the language model are updated according to the second question and the evaluation data.

[0127] In this step, taking the evaluation data obtained through the language model as an example, the training device can delete the judgment criteria of the unsafe content in the second question to obtain the third question, and input the third question into the language model to obtain the second answer output by the language model. Specifically, Figure 5 As shown, the training device can use the judgment criteria in question Q2 as Please note that responses must strictly follow Follow ZZZ " is deleted to obtain question Q3, wherein question Q3 is the third question. In addition, taking the evaluation data obtained by the other methods mentioned above as an example, the training device can also input the second question into the language model to obtain a second reply output by the language model.

[0128] In this step, the training device updates the parameters of the language model according to the second reply and the evaluation data. Specifically, the training device calculates the loss value of the language model according to the second reply and the evaluation data, and then updates the parameters of the model using a supervised fine tuning method (SFT) based on the loss value.

[0129] Above Figure 1 The method embodiment shown, by training the language model with the second question and evaluation data, can enable the language model to understand whether its response includes preset content (such as unsafe content) and the reason for including the preset content. This can improve the language model's ability to understand positive examples and difficult examples, thereby improving the performance of the language model and improving its generalization ability.

[0130] based on Figure 1 The training method embodiment shown, the embodiment of the present application also provides another language model training method.

[0131] Figure 6 is a flow chart of another language model training method provided in an embodiment of the present application. The method can be executed by the above-mentioned training device. Figure 6 As shown, the method may include S601-S604.

[0132] In S601 , the language model outputs a first response according to the first question.

[0133] In S602, a second question and evaluation data corresponding to the second question are determined.

[0134] The specific process of the training device executing the above S601 and S602 can be found in the above Figure 1 The introduction of S101 and S102 in the illustrated embodiment will not be repeated here.

[0135] In S603, the parameters of the reward model are updated according to the first reply and the evaluation data.

[0136] In this step, the training device fine-tunes the pre-trained reward model using the first reply and the evaluation data. Specifically, the training device can input the first reply and the evaluation data into the reward model to obtain the evaluation value output by the reward model. Then, the loss value of the reward model is calculated based on the evaluation value of the first reply and the evaluation value output by the reward model, and the parameters of the reward model are updated based on the loss value. Among them, the embodiment of the present application does not specifically limit the method for obtaining the evaluation value of the first reply. The evaluation value of the first reply can be obtained in advance by manually evaluating the first reply, or before fine-tuning the reward model, the evaluation value of the first reply can be obtained using a pre-trained reward model.

[0137] In S604, the language model outputs a third response according to the fourth question, determines an evaluation value of the third response using the reward model, and then updates the parameters of the language model according to the evaluation value of the third response.

[0138] In this step, the training device can specifically use a reinforcement learning algorithm to update the parameters of the model based on the evaluation value of the third reply, and the present embodiment of the application does not specifically limit this process. Among them, the reinforcement learning algorithm can include but is not limited to methods such as proximal policy optimization (PPO) or Q-learning.

[0139] Above Figure 6 In the illustrated method embodiment, the training device uses evaluation data in text form containing multi-dimensional information to train the reward model, which can improve the accuracy of the reward model's evaluation value for the first reply, thereby improving the performance of the language model.

[0140] based on Figure 1 The training method embodiment shown, the embodiment of the present application also provides a language model inference method.

[0141] Figure 7 This is a language model reasoning method provided in an embodiment of the present application. The method can be executed by a reasoning device.

[0142] like Figure 7 As shown, the method may include the following S701 and S702.

[0143] In S701, a fifth question is generated based on the fourth question, wherein the fifth question is used to ask the fourth question and prompt the language model not to include preset content in its answer to the fourth question.

[0144] In this step, after receiving the fourth question input by the user, Figure 8 As shown, the reasoning device can adjust the fourth question according to the prompt word to obtain the fifth question mentioned above. The prompt word is used to prompt the language model not to include the preset content in the response output. Specifically, the reasoning device can splice the fourth question and the prompt word to generate the fifth question. Among them, the preset content is unsafe content, and the prompt word may include but is not limited to "Please generate a response with excellent overall quality (including usefulness, factuality, and harmlessness) for the user, and please do not output unsafe content."

[0145] The above language model can be based on Figure 1 or Figure 7 The method embodiment shown is obtained by training.

[0146] In S702, the language model outputs a third reply that does not include preset content according to the fifth question.

[0147] In this step, if Figure 8 As shown, the inference device inputs the fifth question into the language model to obtain a third reply, and sends the third reply to the user. The third reply is the reply of the language model to the fourth question asked by the user.

[0148] In an application scenario, the inference device may include a user's terminal device, and the language model may be applied to the user's terminal device to provide the user with functions such as text generation. The terminal device may include, but is not limited to, a smartphone, a tablet computer, or a laptop, a desktop computer, a car computer, and other devices.

[0149] Above Figure 7 In the method shown, when using the language model for reasoning, unsafe content is suppressed by using pre-set prompt words, which can improve the security of the language model when answering user questions.

[0150] based on Figure 1 and Figure 6 The training method embodiment shown, the embodiment of the present application also provides a training device.

[0151] Fig. 9 is a schematic diagram of the structure of a training device 900 provided in the present application. The training device 900 can be applied to Figure 1 or Figure 7 The training device is used to perform Figure 1 or Figure 7 The various steps in the method shown. Fig. 9 As shown, the training device 900 may include an evaluation module 901 and an update module 902 .

[0152] When the training device 900 executes Figure 1In each step of the method shown, the functions of the evaluation module 901 and the update module 902 are as follows.

[0153] The evaluation module 901 is used to output a first reply according to the first question using a language model, and to determine a second question and evaluation data corresponding to the second question. The second question is used to ask whether the first reply includes preset content, and the evaluation data is used to indicate whether the first reply includes preset content.

[0154] The updating module 902 is used to update the parameters of the language model according to the second question and the evaluation data.

[0155] When the training device 900 executes Figure 7 In each step of the method shown, the functions of the evaluation module 901 and the update module 902 are as follows.

[0156] The evaluation module 901 is used to output a first reply according to the first question using a language model, and to determine a second question and evaluation data corresponding to the second question. The second question is used to ask whether the first reply includes preset content, and the evaluation data is used to indicate whether the first reply includes preset content.

[0157] The updating module 902 is used to update the parameters of the reward model according to the evaluation data, determine the evaluation value of the first reply using the reward model, and update the parameters of the language model according to the evaluation value.

[0158] It should be noted that Fig. 9 The training device 900 provided in the illustrated embodiment only uses the division of the above-mentioned functional modules as an example when executing the training method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the training device 900 is divided into different functional modules to complete all or part of the functions described above. Figure 1 or Figure 6 The training method embodiments shown belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0159] based on Figure 8 The training method embodiment shown, the embodiment of the present application also provides an inference device.

[0160] Fig.10 is a schematic diagram of the structure of an inference device 1000 provided by the present application. The inference device 1000 can be applied to Figure 7 The inference device is used to execute Figure 7 The various steps in the method shown. Fig.10 As shown, the reasoning device 1000 may include a generation module 1001 and a reasoning module 1002 .

[0161] The generation module 1001 is used to generate the fifth question based on the fourth question. The fifth question is used to ask the fourth question and prompt the language model not to include the preset content in the answer to the fourth question. Figure 1 or Figure 6 The training method is obtained.

[0162] The reasoning module 1002 is used to output a third response according to the fifth question using the language model, and the third response does not include the preset content. The third response is the response of the language model to the fourth question.

[0163] It should be noted that Fig.10 The reasoning device 1000 provided in the embodiment shown in the figure is only illustrated by the division of the above-mentioned functional modules when executing the reasoning method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the reasoning device 1000 is divided into different functional modules to complete all or part of the functions described above. Figure 7 The training method embodiments shown belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0164] Fig.11 It is a schematic diagram of the hardware structure of a computing device 1100 provided in an embodiment of the present application.

[0165] The computing device 1100 may be the aforementioned training device or the aforementioned reasoning device. Fig.11 The computing device 1100 includes a processor 1101, a memory 1102, a communication interface 1103, and a bus 1004. The processor 1101, the memory 1102, and the communication interface 1103 are connected to each other via the bus 1004. The processor 1101, the memory 1102, and the communication interface 1103 may also be connected in other connection modes besides the bus 1004.

[0166] Among them, the memory 1102 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0167] The processor 1101 may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing the contents stored in a memory (e.g., the memory 1102). For example, the general-purpose processor may be a central processing unit (CPU). The processor 1101 may include at least one circuit to execute Figure 1 or Figure 6 or Figure 7 The illustrated embodiment provides all or part of the steps of the training method.

[0168] The communication interface 1103 includes an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to interconnect devices within the computing device 1100, and an interface for interconnecting the computing device 1100 with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.

[0169] The bus 1004 may be any type of communication bus for interconnecting the processor 1101, the memory 1102, and the communication interface 1103, such as a system bus.

[0170] The above devices may be arranged on independent chips, or at least partially or completely on the same chip. Whether to arrange each device independently on different chips or to integrate them on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above devices.

[0171] Fig.11 The computing device 1100 shown is merely exemplary. During implementation, the computing device 1100 may also include other components, which are not listed one by one herein.

[0172] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0173] It is understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. It should be understood that in the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0174] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A method for reasoning a language model, characterized in that: The method comprises: generating a fifth question according to the fourth question, the fifth question being used to ask the fourth question and prompting the language model not to include the preset content in the reply to the fourth question, wherein the parameter of the language model is updated according to the evaluation data of the first reply, the language model outputs the first reply according to the first question, and the evaluation data is used to indicate whether the first reply includes the preset content; The language model outputs a third response based on the fifth question, and the third response does not include the preset content.

2. The method according to claim 1, characterized in that The evaluation data is determined based on a second question, where the second question is used to inquire whether the first reply includes preset content.

3. The method according to claim 2, characterized in that The second question includes: the judgment criteria of the preset content.

4. The method according to claim 3, characterized in that Determining the evaluation data according to the second question includes: The second question is input into the language model, and the language model outputs the evaluation data.

5. The method according to claim 4, characterized in that Updating the parameters of the language model according to the evaluation data includes: Deleting the judgment criteria of the preset content in the second question to obtain the third question; Inputting the third question into the language model, and the language model outputting a second response; A loss value of the language model is determined according to the second reply and the evaluation data, and parameters of the language model are updated according to the loss value.

6. The method according to any one of claims 1 to 3, characterized in that: Determining the evaluation data according to the second question includes: The second question is input into a high-order model corresponding to the language model, and the high-order model outputs the evaluation data.

7. The method according to claim 6, characterized in that Updating the parameters of the language model according to the evaluation data includes: Inputting the second question into the language model, and the language model outputting a second response; A loss value of the language model is determined according to the second reply and the evaluation data, and parameters of the language model are updated according to the loss value.

8. The method according to claim 4 or 6, characterized in that: Updating the parameters of the language model according to the evaluation data includes: Update the parameters of the reward model according to the evaluation data: Determine the evaluation value of the first reply using the updated reward model; The parameters of the language model are updated according to the evaluation value.

9. The method according to any one of claims 2 to 8, characterized in that: The second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

10. A language model training method, characterized in that: The method comprises: The language model outputs a first response based on the first question; determining a second question and evaluation data corresponding to the second question, wherein the second question is used to inquire whether the first reply includes preset content, and the evaluation data is used to indicate whether the first reply includes the preset content; The parameters of the language model are updated according to the evaluation data.

11. The method according to claim 10, characterized in that The second question includes: the judgment criteria of the preset content.

12. The method according to claim 10 or 11, characterized in that: The method further includes: generating the second question using a question template.

13. The method according to claim 12, characterized in that Determining the evaluation data corresponding to the second question includes: The second question is input into the language model, and the language model outputs the evaluation data.

14. The method according to claim 13, characterized in that The updating of the parameters of the language model according to the evaluation data includes: Deleting the judgment criteria of the preset content in the second question to obtain the third question; Inputting the third question into the language model, and the language model outputting a second response; A loss value of the language model is determined according to the second reply and the evaluation data, and parameters of the language model are updated according to the loss value.

15. The method according to any one of claims 10 to 12, characterized in that: Determining the evaluation data corresponding to the second question includes: The second question is input into a high-order model corresponding to the language model, and the high-order model outputs the evaluation data.

16. The method according to claim 15, characterized in that The updating of the parameters of the language model according to the evaluation data includes: Inputting the second question into the language model, and the language model outputting a second response; A loss value of the language model is determined according to the second reply and the evaluation data, and parameters of the language model are updated according to the loss value.

17. The method according to claim 13 or 15, characterized in that The updating of the parameters of the language model according to the evaluation data includes: updating parameters of the reward model according to the evaluation data; Determine the evaluation value of the first reply using the updated reward model; The parameters of the language model are updated according to the evaluation value.

18. The method according to any one of claims 10 to 17, characterized in that: The second question is also used to ask the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content, and the evaluation data is also used to indicate the reason why the first reply includes the preset content, and / or the response strategy for questions related to the preset content.

19. A language model inference device, characterized in that: The device comprises: a generating module, configured to generate a fifth question according to the fourth question, wherein the fifth question is used to ask the fourth question, and prompt the language model not to include the preset content in the reply to the fourth question, wherein the parameter of the language model is updated according to the evaluation data of the first reply, the language model outputs the first reply according to the first question, and the evaluation data is used to indicate whether the first reply includes the preset content; An inference module is used for the language model to output a third response based on the fifth question, and the third response does not include the preset content.

20. A language model training device, characterized in that: The device comprises: an evaluation module, configured to output a first reply according to a first question using a language model, and to determine a second question and evaluation data corresponding to the second question, wherein the second question is used to ask whether the first reply includes a preset content, and the evaluation data is used to indicate whether the first reply includes the preset content; An updating module is used to update the parameters of the language model according to the evaluation data.

21. A computing device, characterized in that The computing device comprises: a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the method of any one of claims 1 to 9, or to implement the method of any one of claims 10 to 18.

22. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the method for implementing any one of claims 1 to 9, or the method for implementing any one of claims 10 to 18.

Images