Large language model value alignment method and device

Abstract

The embodiment of the invention discloses a large language model value alignment method. The method comprises the steps of obtaining a plurality of question query texts and corresponding candidate answer pools; determining a reward score of each candidate answer in the candidate answer pool, and generating a training sample data set; determining the answer probability of each candidate answer, calculating a first reward expectation of each question query text according to the answer probability of the plurality of candidate answers and the reward score, and calculating a second reward expectation of all question query texts according to all the first reward expectations; performing value alignment adjustment on the large language model according to the second reward expectation and the training sample data set so as to adjust model parameters, and judging whether a preset convergence condition is met or not; if not, generating a plurality of new candidate answers of each question query text according to the adjusted large language model, updating the candidate answer pool according to the plurality of new candidate answers, and skipping to execute reward score determination; if yes, the target large language model is generated, and the model value alignment capability is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method and device for aligning the value of a large language model. Background Art

[0002] The evolution of large model system capabilities refers to the process in which the model gradually migrates and adapts from simple pre-training tasks to more complex and open application scenarios, and finally possesses multiple system functions. However, the evolution of current multimodal large model systems is often uncontrollable. The main reason is that the model relies on endogenous logic for non-deterministic probabilistic generation. This generation method leads to uncontrollable phenomena such as hallucinations and cannot ensure the security and reliability of the generated content. In recent years, with the rapid development of large language models (LLMs), especially in tasks such as text generation, dialogue systems, and machine translation, emergent capabilities have become a significant phenomenon. Although this emergent capability has brought about an improvement in model performance, it also brings potential risks. When generating complex content, the model may generate text that is biased, discriminatory, or contrary to mainstream values. This unpredictable emergence phenomenon exacerbates the uncontrollability of model output, which in turn raises serious ethical and social issues. Therefore, establishing a trustworthy theoretical framework for the adaptive evolution of multimodal large model system capabilities, especially a formal knowledge expression and language alignment method for content security and high reliability, has become an important scientific issue that needs to be solved urgently. Summary of the invention

[0003] In view of the above problems, the present application is proposed to provide a large language model value alignment method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of an embodiment of the present application, a large language model value alignment method is provided, including:

[0005] S1, obtaining multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers;

[0006] S2, determining a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores constitute a training sample data set;

[0007] S3, for each candidate answer of each question query text, determine the answer probability corresponding to the candidate answer, calculate the first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate the second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text;

[0008] S4, performing value alignment adjustment on the large language model according to the second reward expectation and the training sample data set to adjust the model parameters and determine whether the predetermined convergence condition is met; if not, executing S5; if yes, executing S6;

[0009] S5, generating multiple new candidate answers corresponding to each question query text according to the adjusted large language model, updating the candidate answer pool according to the multiple new candidate answers, and jumping to execute S2;

[0010] S6, generate a target large language model.

[0011] Furthermore, calculating the first reward expectation corresponding to the question query text according to the answer probabilities and the corresponding reward scores corresponding to the multiple candidate answers, and calculating the second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text further includes:

[0012] The first reward expectation is calculated based on the following formula:

[0013]

[0014] Among them, E i is the first reward expectation corresponding to the i-th question query text, is the answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, Ai is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0015] The second reward expectation is calculated using the following formula:

[0016]

[0017] Among them, J(θ) is the second reward expectation, and N is the number of question query texts.

[0018] Furthermore, for each candidate answer of each question query text, determining the answer probability corresponding to the candidate answer further includes:

[0019] For each candidate answer of each question query text, perform word segmentation processing on the candidate answer to obtain multiple word segmentation results;

[0020] Determine the probability of occurrence of each word segmentation result;

[0021] According to the word segmentation occurrence probabilities corresponding to multiple word segmentation results, the answer probabilities corresponding to the candidate answers are calculated.

[0022] Furthermore, after determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes:

[0023] Filter the candidate answers in the candidate answer pool according to the reward scores;

[0024] The training sample data set consisting of multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores further includes:

[0025] A plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and a reward score constitute a training sample data set.

[0026] Furthermore, calculating the first reward expectation corresponding to the question query text according to the answer probabilities and the corresponding reward scores corresponding to the multiple candidate answers further includes:

[0027] Normalize the answer probability of each candidate answer for each question query text;

[0028] The first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities of multiple candidate answers and the corresponding reward scores.

[0029] Furthermore, the predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iteration executions is greater than or equal to a preset number of iterations.

[0030] According to another aspect of an embodiment of the present application, a large language model value alignment device is provided, including:

[0031] An acquisition module is adapted to acquire multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers;

[0032] A determination module, adapted to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein a plurality of question query texts, candidate answer pools corresponding to the plurality of question query texts, and reward scores constitute a training sample data set;

[0033] A calculation module, adapted to determine, for each candidate answer of each question query text, an answer probability corresponding to the candidate answer, calculate a first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate a second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text;

[0034] An alignment adjustment module, adapted to perform value alignment adjustment on the large language model according to the second reward expectation and the training sample data set, so as to adjust the model parameters;

[0035] A judgment module, adapted to judge whether a predetermined convergence condition is met; if not, triggering the execution of the update module; if so, triggering the execution of the generation module;

[0036] An updating module, adapted to generate multiple new candidate answers corresponding to each question query text according to the adjusted large language model, update the candidate answer pool according to the multiple new candidate answers, and trigger the execution of the determining module;

[0037] The generation module is suitable for generating a target large language model.

[0038] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0039] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned large language model value alignment method.

[0040] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned large language model value alignment method.

[0041] According to another aspect of an embodiment of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned large language model value alignment method.

[0042] According to the large language model value alignment method and device provided in the embodiment of the present application, the self-iteration method is used to continuously update and generate higher-quality data that conforms to mainstream value preferences for further optimization of the model. With the support of higher-quality data that conforms to mainstream value preferences, the value alignment capability of the large language model can be further improved. This makes the values ​​of the large language model consistent with human values, avoids the risk of the large language model outputting harmful information, and improves the security and controllability of the large language model.

[0043] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 A schematic diagram of a process of a large language model value alignment method according to an embodiment of the present application is shown;

[0046] Figure 2 Schematic diagram of the process of aligning the value of a large language model;

[0047] Figure 3 A structural block diagram of a large language model value alignment device according to an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] Figure 1 A schematic diagram of a large language model value alignment method according to an embodiment of the present application is shown. Figure 2 Schematic diagram of the process of aligning the value of a large language model, combined with Figure 1 and Figure 2 , the method comprises the following steps:

[0051] Step S101, obtaining multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers.

[0052] Specifically, there are many methods and approaches to obtain multiple question query texts and candidate answer pools corresponding to each question query text, for example, by crawling web page content, collecting comments and posts on social media, and requiring human annotators to provide specific answers according to specific requirements. These datasets can be applied to various tasks, such as dialogue generation, text summarization tasks, and sentiment analysis.

[0053] In addition, multiple question query texts can be collected, and then the open source pre-trained large language model can be used to generate corresponding candidate answers for the question query texts. Specifically, given a question query text, it is input into the pre-trained large language model, so that the pre-trained large language model can generate multiple different answers according to different sampling strategies, thereby obtaining multiple question query texts and multiple candidate answers corresponding to each question query text, which can effectively reduce the cost of manual collection, reduce the workload of manual data collection, and also improve data quality. Each question query text corresponds to a candidate answer pool, which contains multiple candidate answers.

[0054] It should be noted that the pre-trained large language model is an open source, trained model, and there is no need to pre-train the large language model here.

[0055] Step S102, determining a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein a plurality of question query texts, candidate answer pools corresponding to the plurality of question query texts, and reward scores constitute a training sample data set.

[0056] Since each candidate answer in the candidate answer pool may be the best answer to the question query text, after obtaining multiple question query texts and the candidate answer pool corresponding to each question query text, it is necessary to determine the reward score corresponding to each candidate answer of each question query text, where the reward score is used to characterize the quality of the candidate answer. In other words, the reward score reflects the mainstream value preference of the candidate answer. The higher the reward score, the more the candidate answer conforms to the mainstream values. The lower the reward score, the more the candidate answer deviates from the mainstream values, that is, the less it conforms to the mainstream values.

[0057] In an optional implementation, a pre-trained reward model can be used to determine the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text. For example, each candidate answer corresponding to the question query text is input into the reward model, and the reward model parses each candidate answer to output the reward score corresponding to each candidate answer.

[0058] Among them, the reward model can be trained by the following method: for each candidate answer of each question query text, determine its feedback annotation (such as the ranking or score set for each candidate answer), then, supervised learning is performed based on the question query text, its corresponding candidate answers and these feedback annotations, and a neural network model is trained so that it can predict the reward score under the given text. The training goal is to minimize the difference between the predicted reward score and the feedback annotation. For example, the question query text and its corresponding candidate answers can be input into the neural network model for training to obtain the predicted reward score corresponding to each candidate answer, and the model parameters of the neural network model are updated according to the loss between the predicted reward score and the feedback annotation. The above steps are iteratively performed until the difference between the predicted reward score and the feedback annotation is minimized, and the model training is completed, thereby obtaining a trained reward model. After the training is completed, the reward model can be used in the subsequent reinforcement learning stage to give scores to the candidate answers output by the large language model, thereby guiding model optimization.

[0059] In addition to using the reward model to determine the reward score of each selected answer, each candidate answer can also be scored based on preset rules to determine the reward score corresponding to the candidate answer. I will not go into details here.

[0060] After determining the reward score corresponding to each candidate answer, multiple question query texts, candidate answer pools corresponding to multiple question query texts, and reward scores constitute a training sample data set, where the training sample data set can be expressed as D(Q,A,R), where Q represents the question query text, A represents the candidate answer pool corresponding to the question query text, the candidate answer pool contains multiple candidate answers, and R represents the reward score of each candidate answer. This training sample data set fully records the mainstream value preference information and provides a powerful feedback mechanism for model training, enabling the model to continuously improve its generated text to better meet mainstream values. This process plays a vital role in training and fine-tuning to ensure that the quality of the generated candidate answers is improved.

[0061] Step S103, for each candidate answer of each question query text, determine the answer probability corresponding to the candidate answer, calculate the first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to multiple candidate answers, and calculate the second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text.

[0062] Specifically, for each candidate answer of each question query text, we first need to determine the probability of this candidate answer being asked, that is, the answer probability. For example, for each candidate answer of each question query text, we can perform word segmentation processing on the candidate answer to obtain multiple word segmentation results. Through word segmentation processing, the candidate answer is divided into multiple words and / or words. Then, using the large language model, we can calculate the text of the current word segmentation and its corresponding occurrence probability under the condition of the previous word segmentation sequence. The word segmentation occurrence probability can be expressed as π θ (y i,j,k |x i ), x i is the query text of the i-th question, y i,j,k is the kth word segmentation result of the jth candidate answer of the i-th question query text, π θ (y i,j,k |x i ) is the word segmentation probability of the kth word segmentation result of the jth candidate answer of the ith question query text, and θ is the model parameter; after determining the word segmentation probability corresponding to each word segmentation result, the answer probability corresponding to the candidate answer can be calculated based on the word segmentation probabilities corresponding to multiple word segmentation results. For example, the product of all word segmentation probabilities is calculated, and the product result is determined as the answer probability corresponding to the candidate answer.

[0063] In an optional implementation, the answer probability of each candidate answer to each question query text may be normalized first. For example, the answer probability may be normalized using the following formula:

[0064]

[0065] Where T is the temperature parameter used to control the smoothness of the probability distribution, π θ is a large language model with θ as a parameter, which is used to predict the probability of word segmentation and sentence correspondence. exp is an exponential function, log is a logarithmic function, and y i,j The jth candidate answer corresponding to the i-th question query text. Intuitively, the candidate answer with a higher reward score should have a larger answer probability.

[0066] Then, the first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities and corresponding reward scores of the multiple candidate answers. Different question query texts may have different reward score ranges. In order to better reflect the relative quality of each candidate answer, the reward scores may also be normalized.

[0067] Of course, other methods can also be used to determine the answer probability of each candidate answer, which will not be elaborated here.

[0068] Then, the first reward expectation corresponding to the question query text is calculated based on the answer probabilities and corresponding reward scores of the multiple candidate answers in the question query text. For example, the first reward expectation can be calculated based on the following formula:

[0069]

[0070] Among them, E i is the first reward expectation corresponding to the i-th question query text, is the answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, A i is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0071] For example, suppose a question query text x 1 , its candidate answer pool A 1 There are three candidate answers, namely y 1,1 ,y 1,2 ,y 1,3 , the answer probabilities corresponding to each candidate answer are determined by the above method to be 0.4, 0.3 and 0.3 respectively, and the reward scores corresponding to the three candidate answers are 0.5, 0.6 and 0.7 respectively, so the question query text x can be calculated using the above formula 1 The corresponding first reward expectation is (0.4*0.5+0.3*0.6+0.3*0.7) / =0.17.

[0072] For all question query texts in the training sample data set, the corresponding first reward expectations can be calculated. The second reward expectations corresponding to all question query texts in the training sample data set are calculated based on the first reward expectations corresponding to each question query text. The second reward expectation is the average of the sum of all first reward expectations. For example, the second reward expectation is calculated using the following formula:

[0073]

[0074] Where J(θ) is the second reward expectation, and N is the number of question query texts. In this embodiment, J(θ) is the objective function of the large language model. The definition of directly includes the parameters θ that need to be trained, so optimizing this objective function is equivalent to letting the model learn a better parameter θ.

[0075] By maximizing this second reward expectation, the model learns in the direction with the fastest gradient increase, thus achieving the purpose of parameter learning.

[0076] In order to further explore how the objective function achieves policy optimization, the following gradient derivation is given below:

[0077]

[0078] where π θ (y i,j |x i ,A i ) is in the training strategy π θ Next candidate answer y i,j The probability of Represents the gradient value of the model prediction probability, E i is at π θ The expected value of the reward for all candidate answers to the query text of the i-th question, It means that the gradient is in the whole x i and i,j The expected value in the sample space.

[0079] In the process of finding the derivative corresponding to the objective function, we first calculate a decentralized reward value (R(x,y)-E i ), and then calculate a normalized gradient value Finally, the expected value of the product of the two is calculated in the entire sample space, which is the gradient value corresponding to the entire objective function. By increasing the contribution of the response with high reward value in the entire gradient descent process, the model learns along the direction of the maximum gradient, so that the model learns the representation that is most conducive to maximizing the reward.

[0080] Step S104, performing value alignment adjustment on the large language model according to the second reward expectation and the training sample data set to adjust the model parameters and determine whether the predetermined convergence condition is met; if not, executing step S105; if yes, executing step S106;

[0081] Specifically, value alignment refers to the process of aligning the values ​​of a model with its environment. That is, the model should be able to correctly understand and express the basic values ​​and ethical standards of human society, and avoid risky behaviors that violate common sense and morality.

[0082] For example, the large language model can be trained for a certain number of steps based on the second reward expectation and the training sample data set, and the model training process can adjust the model parameters. After completing the value alignment adjustment, it is necessary to determine whether the predetermined convergence conditions are met, wherein the predetermined convergence conditions further include: the second reward expectation is greater than or equal to the preset reward expectation threshold, and / or the number of iterations is greater than or equal to the preset number of iterations. If the predetermined convergence conditions are met, the model training is considered to be completed, and the execution can be skipped to step S106. If the predetermined convergence conditions are not met, the execution jumps to step S105.

[0083] Step S105, generating multiple new candidate answers corresponding to each question query text according to the adjusted large language model, updating the candidate answer pool according to the multiple new candidate answers, and jumping to step S102;

[0084] When the predetermined convergence conditions are not met, it is necessary to continue to adjust the value alignment of the model. In order to improve the performance of the model, the adjusted large language model can be used online to generate multiple new candidate answers corresponding to each question query text, and the old candidate answers in the candidate answer pool are replaced with multiple new candidate answers, which helps to form a higher quality candidate answer pool. Then jump to step S102, calculate the reward scores corresponding to the multiple new candidate answers again, and realize a new round of data sampling.

[0085] Step S106: Generate a target large language model.

[0086] Specifically, when the second reward expectation is greater than or equal to the preset reward expectation threshold, and / or the number of iterations is greater than or equal to the preset number of iterations, the model training can be stopped and the current model can be regarded as the final version output to generate the target large language model.

[0087] In an optional implementation of the present application, after determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes:

[0088] The candidate answers in the candidate answer pool are filtered according to the reward score. By filtering, some high-quality candidate answers can be retained, while some low-quality candidate answers can be filtered out. For example, a reward score threshold can be set, and the reward score corresponding to the candidate answer can be compared with the preset reward score threshold. The candidate answers with reward scores greater than or equal to the preset reward score threshold are retained, and the candidate answers with reward scores less than the preset reward score threshold are filtered out. This can better ensure that the model is updated on a higher quality data set, while reducing the risk of model divergence. Therefore, multiple question query texts, the filtered candidate answer pools corresponding to the multiple question query texts, and the reward scores constitute a training sample data set.

[0089] This application determines the training sample data set (Q, A, R), and then the target strategy is initialized to a pre-trained language model and trained for a certain number of steps according to the above objective function. Once the policy update in the current iteration loop is completed, the strategy is reinitialized to the model adjusted for value alignment, and then the newly sampled candidate answers are used to replace the old candidate answers in the candidate answer pool. This helps to form a higher quality candidate answer pool, and then these candidate answers are scored using the reward model to complete a new round of data sampling. The alternating iterative process of data sampling and policy training can continuously improve policy performance, because higher quality samples help improve the strategy, thereby achieving more accurate task execution and answer generation. The following is a brief explanation of the detailed algorithm:

[0090] Algorithm: A self-reinforcing strategy for maximizing rewards during sampling and training. An iteration step is defined as the entire process of generating data once using the sampling strategy and training the Iterate step using the objective function J(θ).

[0091]

[0092] This application constructs a column-level optimization framework that allows offline rewards to be directly integrated into the optimization objective. By maximizing the overall reward of a diverse pool of candidate answers in an iterative manner, the preferred mode is learned. This method has significant advantages, does not require complex parameter adjustments, and exhibits excellent performance, especially in dialogue and summary tasks. It has important potential for improving the performance of the model and expanding its application areas.

[0093] According to the method provided in the embodiment of the present application, the self-iteration method is used to continuously update and generate higher-quality data that conforms to mainstream value preferences for further optimization of the model. With the support of higher-quality data that conforms to mainstream value preferences, the value alignment ability of the large language model can be further improved. In this way, the values ​​of the large language model are consistent with human values, avoiding the risk of the large language model outputting harmful information and improving the security and controllability of the large language model.

[0094] Figure 3 FIG. 4 shows a structural block diagram of a large language model value alignment device according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0095] The acquisition module 301 is adapted to acquire multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers;

[0096] A determination module 302 is adapted to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein a plurality of question query texts, candidate answer pools corresponding to the plurality of question query texts, and reward scores constitute a training sample data set;

[0097] A calculation module 303 is adapted to determine, for each candidate answer of each question query text, an answer probability corresponding to the candidate answer, calculate a first reward expectation corresponding to the question query text according to the answer probabilities corresponding to the multiple candidate answers and the corresponding reward scores, and calculate a second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text;

[0098] An alignment adjustment module 304 is adapted to perform value alignment adjustment on the large language model according to the second reward expectation and the training sample data set, so as to adjust the model parameters;

[0099] The judgment module 305 is adapted to judge whether the predetermined convergence condition is met; if not, the update module is triggered to execute; if so, the generation module is triggered to execute;

[0100] An updating module 306, adapted to generate a plurality of new candidate answers corresponding to each question query text according to the adjusted large language model, update the candidate answer pool according to the plurality of new candidate answers, and trigger the execution of the determining module;

[0101] The generation module 307 is adapted to generate a target large language model.

[0102] Optionally, the calculation module is further adapted to: calculate the first reward expectation based on the following formula:

[0103]

[0104] Among them, E i is the first reward expectation corresponding to the i-th question query text, is the answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, Ai is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0105] The second reward expectation is calculated using the following formula:

[0106]

[0107] Among them, J(θ) is the second reward expectation, and N is the number of question query texts.

[0108] Optionally, the calculation module is further adapted to: for each candidate answer of each question query text, perform word segmentation processing on the candidate answer to obtain multiple word segmentation results;

[0109] Determine the probability of occurrence of each word segmentation result;

[0110] According to the word segmentation occurrence probabilities corresponding to multiple word segmentation results, the answer probabilities corresponding to the candidate answers are calculated.

[0111] Optionally, the device further comprises: a filtering module adapted to filter the candidate answers in the candidate answer pool according to the reward score;

[0112] The determination module is further adapted to: a plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and reward scores constitute a training sample data set.

[0113] Optionally, the calculation module is further adapted to: normalize the answer probability of each candidate answer to each question query text;

[0114] The first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities of multiple candidate answers and the corresponding reward scores.

[0115] Optionally, the predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iterations is greater than or equal to a preset number of iterations.

[0116] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0117] According to the device provided in the embodiment of the present application, a self-iterative method is used to continuously update and generate higher-quality data that conforms to mainstream value preferences for further optimization of the model. With the support of higher-quality data that conforms to mainstream value preferences, the value alignment capability of the large language model can be further improved. This makes the values ​​of the large language model consistent with human values, avoids the risk of the large language model outputting harmful information, and improves the security and controllability of the large language model.

[0118] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the large language model value alignment method in any of the above method embodiments.

[0119] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the large language model value alignment method in any of the above method embodiments.

[0120] Figure 4 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0121] like Figure 4 As shown, the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0122] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned large language model value alignment method embodiment for computing devices.

[0123] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0124] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0125] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0126] Program 410 can be specifically used to enable processor 402 to execute the large language model value alignment method in any of the above-mentioned method embodiments. The specific implementation of each step in program 410 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned large language model value alignment embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0127] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0128] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0129] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0130] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0131] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0132] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0133] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A large language model value alignment method, comprising: S1, obtaining multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers; S2, determining a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores constitute a training sample data set; S3, for each candidate answer of each question query text, determine the answer probability corresponding to the candidate answer, calculate the first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate the second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text; S4, performing value alignment adjustment on the large language model according to the second reward expectation and the training sample data set to adjust the model parameters and determine whether a predetermined convergence condition is met; if not, executing S5; if yes, executing S6; S5, generating multiple new candidate answers corresponding to each question query text according to the adjusted large language model, updating the candidate answer pool according to the multiple new candidate answers, and jumping to execute S2; S6, generate a target large language model.

2. The method according to claim 1, wherein: The step of calculating the first reward expectation corresponding to the question query text according to the answer probabilities and the corresponding reward scores corresponding to the plurality of candidate answers, and calculating the second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text further includes: The first reward expectation is calculated based on the following formula: Among them, E i is the first reward expectation corresponding to the i-th question query text, is the answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, Ai is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model; The second reward expectation is calculated using the following formula: Among them, J(θ) is the second reward expectation, and N is the number of question query texts.

3. The method according to claim 1 or 2, wherein: For each candidate answer of each question query text, determining the answer probability corresponding to the candidate answer further includes: For each candidate answer of each question query text, perform word segmentation processing on the candidate answer to obtain multiple word segmentation results; Determine the probability of occurrence of each word segmentation result; The answer probability corresponding to the candidate answer is calculated according to the word segmentation occurrence probabilities corresponding to the multiple word segmentation results.

4. The method according to any one of claims 1 to 3, wherein: After determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes: Filter the candidate answers in the candidate answer pool according to the reward scores; The training sample data set consisting of multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores further includes: A plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and a reward score constitute a training sample data set.

5. The method according to any one of claims 1 to 4, wherein: The calculating of the first reward expectation corresponding to the question query text according to the answer probabilities and the reward scores corresponding to the plurality of candidate answers further comprises: Normalize the answer probability of each candidate answer for each question query text; The first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities of multiple candidate answers and the corresponding reward scores.

6. The method according to any one of claims 1 to 5, wherein: The predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iterations is greater than or equal to a preset number of iterations.

7. A large language model value alignment device, comprising: An acquisition module is adapted to acquire a plurality of question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: a plurality of candidate answers; A determination module, adapted to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein a plurality of question query texts, candidate answer pools corresponding to the plurality of question query texts, and reward scores constitute a training sample data set; A calculation module, adapted to determine, for each candidate answer of each question query text, an answer probability corresponding to the candidate answer, calculate a first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate a second reward expectation corresponding to all question query texts in the training sample data set according to the first reward expectation corresponding to each question query text; An alignment adjustment module, adapted to perform value alignment adjustment on the large language model according to the second reward expectation and the training sample data set, so as to adjust the model parameters; A judgment module, adapted to judge whether a predetermined convergence condition is met; if not, triggering the execution of the update module; if so, triggering the execution of the generation module; An updating module, adapted to generate a plurality of new candidate answers corresponding to each question query text according to the adjusted large language model, update the candidate answer pool according to the plurality of new candidate answers, and trigger the determination module to execute; The generation module is suitable for generating a target large language model.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the large language model value alignment method as described in any one of claims 1-6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the large language model value alignment method as described in any one of claims 1-6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the large language model value alignment method as described in any one of claims 1-6.

Images