Large language model value alignment method and device

Abstract

An embodiment of the present application discloses a large language model value alignment method, including: obtaining multiple question query texts and corresponding candidate answer pools; determining a reward score for each candidate answer in the candidate answer pool and generating a training sample data set; determining the answer probability of each candidate answer, calculating a first reward expectation for each question query text based on the answer probabilities and reward scores of the multiple candidate answers, and calculating a second reward expectation for all question query texts based on all first reward expectations; performing value alignment adjustment on the large language model based on the second reward expectation and the training sample data set to adjust model parameters, and determining whether a predetermined convergence condition is met; if not, generating multiple new candidate answers for each question query text based on the adjusted large language model, updating the candidate answer pool based on the multiple new candidate answers, and jumping to determine the reward score; if so, generating a target large language model, thereby improving the model value alignment capability.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method and device for aligning the value of a large language model. Background Art

[0002] The evolution of large model system capabilities refers to the process by which models evolve from simple pre-training tasks to gradually migrating and adapting to more complex and open application scenarios, ultimately acquiring diverse system functions. However, the evolution of current multimodal large model systems is often uncontrollable. This is primarily due to the models' reliance on inherent logic for non-deterministic probabilistic generation. This generational approach leads to uncontrollable phenomena such as hallucinations, and cannot guarantee the security and reliability of generated content. In recent years, with the rapid development of large language models (LLMs), emergent capabilities have become a prominent phenomenon, particularly in tasks such as text generation, dialogue systems, and machine translation. While this emergent capability has improved model performance, it also carries potential risks. When generating complex content, models may generate text that is biased, discriminatory, or violates mainstream values. This unpredictable emergent phenomenon exacerbates the uncontrollability of model outputs, raising serious ethical and social issues. Therefore, establishing a trustworthy, adaptive theoretical framework for the evolution of multimodal large model system capabilities, particularly formal knowledge representation and language alignment methods for content security and high reliability, has become an urgent scientific challenge. Summary of the Invention

[0003] In view of the above problems, the present application is proposed to provide a large language model value alignment method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of an embodiment of the present application, a large language model value alignment method is provided, including:

[0005] S1, obtaining multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers;

[0006] S2, determining a reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, wherein multiple question query texts, the candidate answer pools corresponding to the multiple question query texts, and the reward scores constitute a training sample dataset;

[0007] S3, for each candidate answer to each question query text, determining the answer probability corresponding to the candidate answer, calculating the first reward expectation corresponding to the question query text based on the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculating the second reward expectation corresponding to all question query texts in the training sample dataset based on the first reward expectation corresponding to each question query text;

[0008] S4, performing value alignment adjustment on the large language model based on the second reward expectation and the training sample dataset to adjust the model parameters and determine whether a predetermined convergence condition is met; if not, executing S5; if so, executing S6;

[0009] S5: Generate multiple new candidate answers corresponding to each question query text based on the adjusted large language model, update the candidate answer pool based on the multiple new candidate answers, and jump to S2;

[0010] S6, generate the target large language model.

[0011] Furthermore, calculating a first reward expectation corresponding to the question query text according to the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculating a second reward expectation corresponding to all question query texts in the training sample dataset according to the first reward expectation corresponding to each question query text further includes:

[0012] The first reward expectation is calculated based on the following formula:

[0013]

[0014] Among them, E i is the first reward expectation corresponding to the i-th question query text, The answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, Ai is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0015] The second reward expectation is calculated using the following formula:

[0016]

[0017] Where J(θ) is the second reward expectation and N is the number of question query texts.

[0018] Furthermore, for each candidate answer of each question query text, determining the answer probability corresponding to the candidate answer further includes:

[0019] For each candidate answer of each question query text, perform word segmentation processing on the candidate answer to obtain multiple word segmentation results;

[0020] Determine the probability of occurrence of each word segmentation result;

[0021] According to the word segmentation probability corresponding to multiple word segmentation results, the answer probability corresponding to the candidate answer is calculated.

[0022] Furthermore, after determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes:

[0023] Filter the candidate answers in the candidate answer pool according to the reward scores;

[0024] The training sample dataset comprising multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores further includes:

[0025] A plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and a reward score constitute a training sample data set.

[0026] Furthermore, calculating the first reward expectation corresponding to the question query text based on the answer probabilities and the corresponding reward scores corresponding to the multiple candidate answers further includes:

[0027] Normalize the answer probability of each candidate answer for each question query text;

[0028] The first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities and corresponding reward scores of multiple candidate answers.

[0029] Furthermore, the predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iterative executions is greater than or equal to a preset number of iterations.

[0030] According to another aspect of an embodiment of the present application, a large language model value alignment device is provided, comprising:

[0031] An acquisition module is adapted to acquire a plurality of question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: a plurality of candidate answers;

[0032] a determination module adapted to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores constitute a training sample dataset;

[0033] a calculation module adapted to determine, for each candidate answer to each question query text, an answer probability corresponding to the candidate answer, calculate a first reward expectation corresponding to the question query text based on the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate a second reward expectation corresponding to all question query texts in the training sample dataset based on the first reward expectation corresponding to each question query text;

[0034] An alignment adjustment module, adapted to perform value alignment adjustment on the large language model according to the second reward expectation and the training sample dataset, so as to adjust the model parameters;

[0035] A judgment module, adapted to judge whether a predetermined convergence condition is satisfied; if not, triggering the execution of the update module; if so, triggering the execution of the generation module;

[0036] An update module is adapted to generate multiple new candidate answers corresponding to each question query text based on the adjusted large language model, update the candidate answer pool based on the multiple new candidate answers, and trigger the execution of the determination module;

[0037] The generation module is suitable for generating a target large language model.

[0038] According to another aspect of an embodiment of the present application, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0039] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned large language model value alignment method.

[0040] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored. The executable instruction enables a processor to perform operations corresponding to the above-mentioned large language model value alignment method.

[0041] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned large language model value alignment method.

[0042] The large language model value alignment method and apparatus provided in the embodiments of this application continuously updates through self-iteration, generating higher-quality data that aligns with mainstream value preferences for further model optimization. This higher-quality data, in line with mainstream value preferences, further enhances the large language model's value alignment capabilities. This ensures that the values of the large language model align with human values, avoids the risk of the large language model outputting harmful information, and improves its security and controllability.

[0043] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the embodiments of the present application. The same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 A schematic diagram of a process for aligning the value of a large language model according to an embodiment of the present application is shown;

[0046] Figure 2 A flowchart for value alignment of large language models;

[0047] Figure 3 A structural block diagram of a large language model value alignment device according to an embodiment of the present application is shown;

[0048] Figure 4 A schematic structural diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] Figure 1 A flow chart of a large language model value alignment method according to an embodiment of the present application is shown. Figure 2 A flowchart for value alignment of large language models, combined with Figure 1 and Figure 2 , the method comprises the following steps:

[0051] Step S101: Acquire multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers.

[0052] Specifically, there are various methods and approaches for obtaining multiple question query texts and the candidate answer pool corresponding to each question query text. For example, this can be achieved by crawling web content, collecting comments and posts on social media, and requiring human annotators to provide specific answers according to specific requirements. These datasets can be applied to various tasks, such as dialogue generation, text summarization, and sentiment analysis.

[0053] In addition, multiple question query texts can be collected and then used to generate corresponding candidate answers for the question query texts using an open-source pre-trained large language model. Specifically, given a question query text, it is input into the pre-trained large language model, which allows the pre-trained large language model to generate multiple different answers based on different sampling strategies. This allows multiple question query texts and multiple candidate answers corresponding to each question query text to be obtained, effectively reducing the cost of manual collection, alleviating the workload of manual data collection, and also improving data quality. Each question query text corresponds to a candidate answer pool, which contains multiple candidate answers.

[0054] It should be noted that the pre-trained large language model is an open source, already trained model, and there is no need to pre-train the large language model here.

[0055] Step S102: determine the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text. Multiple question query texts, the candidate answer pools corresponding to the multiple question query texts, and the reward scores constitute a training sample data set.

[0056] Since each candidate answer in the candidate answer pool may be the best answer to the question query text, after obtaining multiple question query texts and the candidate answer pool corresponding to each question query text, it is necessary to determine the reward score corresponding to each candidate answer for each question query text, where the reward score is used to characterize the quality of the candidate answer. In other words, the reward score reflects the mainstream value preference of the candidate answer. The higher the reward score, the more the candidate answer conforms to the mainstream values. The lower the reward score, the more the candidate answer deviates from the mainstream values, that is, the less it conforms to the mainstream values.

[0057] In an optional embodiment, a pre-trained reward model can be used to determine the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text. For example, each candidate answer corresponding to the question query text is input into the reward model, and the reward model parses each candidate answer to output the reward score corresponding to each candidate answer.

[0058] Among them, the reward model can be trained by the following method: for each candidate answer of each question query text, its feedback annotation is determined (such as the ranking or score set for each candidate answer), and then, supervised learning is performed based on the question query text, its corresponding candidate answers and these feedback annotations to train a neural network model so that it can predict the reward score under the given text. The training goal is to minimize the difference between the predicted reward score and the feedback annotation. For example, the question query text and its corresponding candidate answers can be input into the neural network model for training to obtain the predicted reward score corresponding to each candidate answer. The model parameters of the neural network model are updated according to the loss between the predicted reward score and the feedback annotation. The above steps are iteratively performed until the difference between the predicted reward score and the feedback annotation is minimized, and the model training is completed to obtain a trained reward model. After the training is completed, the reward model can be used in the subsequent reinforcement learning stage to give scores to the candidate answers output by the large language model, thereby guiding model optimization.

[0059] In addition to using the reward model to determine the reward score for each selected answer, you can also score each candidate answer based on preset rules to determine the reward score corresponding to the candidate answer. I will not go into too much detail here.

[0060] After determining the reward score for each candidate answer, multiple question query texts, the candidate answer pool corresponding to the multiple question query texts, and the reward scores constitute a training sample dataset. The training sample dataset can be represented as D(Q,A,R), where Q represents the question query text, A represents the candidate answer pool corresponding to the question query text, which contains multiple candidate answers, and R represents the reward score for each candidate answer. This training sample dataset fully records mainstream value preference information and provides a powerful feedback mechanism for model training, enabling the model to continuously improve its generated text to better meet mainstream values. This process plays a crucial role in training and fine-tuning to ensure the quality of the generated candidate answers is improved.

[0061] Step S103: For each candidate answer of each question query text, determine the answer probability corresponding to the candidate answer, calculate the first reward expectation corresponding to the question query text based on the answer probabilities and corresponding reward scores corresponding to multiple candidate answers, and calculate the second reward expectation corresponding to all question query texts in the training sample data set based on the first reward expectation corresponding to each question query text.

[0062] Specifically, for each candidate answer to each question query text, we first need to determine the probability of this candidate answer being asked, that is, the answer probability. For example, for each candidate answer to each question query text, we can perform word segmentation processing on the candidate answer to obtain multiple word segmentation results. Through word segmentation processing, the candidate answer is divided into multiple words and / or words. Then, using the large language model, we can calculate the text of the current word segmentation and its corresponding occurrence probability under the given previous word segmentation sequence. The word segmentation probability can be expressed as π θ (y i,j,k |x i ), x i is the query text of the i-th question, y i,j,k is the kth word segmentation result of the jth candidate answer of the i-th question query text, π θ (y i,j,k |x i ) is the word segmentation probability of the kth word segmentation result of the jth candidate answer to the i-th question query text, and θ is the model parameter; after determining the word segmentation probability corresponding to each word segmentation result, the answer probability corresponding to the candidate answer can be calculated based on the word segmentation probability corresponding to multiple word segmentation results. For example, the product of all word segmentation probabilities is calculated, and the product result is determined as the answer probability corresponding to the candidate answer.

[0063] In an optional implementation, the answer probability of each candidate answer to each question query text may be normalized first. For example, the answer probability may be normalized using the following formula:

[0064]

[0065] Where T is the temperature parameter used to control the smoothness of the probability distribution, π θ is a large language model with θ as a parameter, which is used to predict the probability of word segmentation and sentence correspondence. exp is an exponential function, log is a logarithmic function, and y i,j For the jth candidate answer corresponding to the i-th question query text, intuitively, the candidate answer with a higher reward score should have a larger answer probability.

[0066] Next, the first expected reward for the query text is calculated based on the normalized answer probabilities and corresponding reward scores of the multiple candidate answers. Different query texts may have different reward score ranges. To better reflect the relative quality of each candidate answer, the reward scores can also be normalized.

[0067] Of course, other methods can also be used to determine the answer probability of each candidate answer, which will not be elaborated here.

[0068] Then, the first reward expectation corresponding to the question query text is calculated based on the answer probabilities and corresponding reward scores of the multiple candidate answers in the question query text. For example, the first reward expectation can be calculated based on the following formula:

[0069]

[0070] Among them, E i is the first reward expectation corresponding to the i-th question query text, The answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, A i is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0071] For example, suppose a question query text x1, its candidate answer pool A1 contains three candidate answers, namely y 1,1 、y 1,2 、y 1,3 , using the above method, it is determined that the answer probabilities corresponding to each candidate answer are 0.4, 0.3 and 0.3 respectively. At the same time, the reward scores corresponding to these three candidate answers are 0.5, 0.6 and 0.7 respectively. Therefore, the above formula can be used to calculate that the first reward expectation corresponding to the question query text x1 is (0.4*0.5+0.3*0.6+0.3*0.7) / =0.17.

[0072] For all question query texts in the training sample dataset, the corresponding first reward expectation can be calculated. The second reward expectation corresponding to all question query texts in the training sample dataset is calculated based on the first reward expectation corresponding to each question query text. The second reward expectation is the average of the sum of all first reward expectations. For example, the second reward expectation is calculated using the following formula:

[0073]

[0074] Where J(θ) is the second reward expectation, and N is the number of question query texts. In this embodiment, J(θ) is the objective function of the large language model. The definition of directly includes the parameters θ that need to be trained, so optimizing this objective function is equivalent to letting the model learn a better parameter θ.

[0075] By maximizing this second reward expectation, the model learns in the direction of the fastest gradient increase, achieving the purpose of parameter learning.

[0076] To further explore how the objective function achieves policy optimization, the following gradient derivation is given below:

[0077]

[0078] where π θ (y i,j |x i ,A i ) is in the training strategy π θ Next candidate answer y i,j The probability of Represents the gradient value of the model prediction probability, E i is in π θ The expected value of the reward for all candidate answers to the query text of the i-th question, Indicates that the gradient is over the entire x i and y i,j The expected value in the sample space.

[0079] In the process of finding the derivative of the objective function, we first calculate a decentralized reward value (R(x,y)-E i ), and then calculate a normalized gradient value Finally, the expected value of the product of the two over the entire sample space is calculated, which is the gradient of the entire objective function. By increasing the contribution of responses with high reward values during the gradient descent process, the model learns along the direction of the maximum gradient, thereby learning the representation that is most conducive to maximizing reward.

[0080] Step S104: perform value alignment adjustment on the large language model based on the second reward expectation and the training sample dataset to adjust the model parameters and determine whether the predetermined convergence condition is met; if not, execute step S105; if so, execute step S106;

[0081] Specifically, value alignment refers to the process of aligning the model's values with its environment. That is, the model should be able to correctly understand and express the basic values and ethical standards of human society, and avoid risky behaviors that violate common sense and morality.

[0082] For example, the large language model can be trained for a certain number of steps based on the second reward expectation and the training sample data set, and the model training process can achieve the adjustment of the model parameters. After completing the value alignment adjustment, it is necessary to determine whether the predetermined convergence conditions are met, where the predetermined convergence conditions further include: the second reward expectation is greater than or equal to the preset reward expectation threshold, and / or the number of iterations is greater than or equal to the preset number of iterations. If the predetermined convergence conditions are met, the model training is considered to be completed and the process can jump to step S106. If the predetermined convergence conditions are not met, the process jumps to step S105.

[0083] Step S105: Generate multiple new candidate answers corresponding to each question query text based on the adjusted large language model, update the candidate answer pool based on the multiple new candidate answers, and jump to step S102;

[0084] If the predetermined convergence conditions are not met, the model value alignment adjustment needs to be continued. In order to improve the performance of the model, the adjusted large language model can be used online to generate multiple new candidate answers corresponding to each question query text, and the old candidate answers in the candidate answer pool are replaced with multiple new candidate answers, which helps to form a higher quality candidate answer pool. Then jump to step S102, calculate the reward scores corresponding to the multiple new candidate answers again, and realize a new round of data sampling.

[0085] Step S106: Generate a target large language model.

[0086] Specifically, when the second reward expectation is greater than or equal to the preset reward expectation threshold, and / or the number of iterations is greater than or equal to the preset number of iterations, the model training can be stopped and the current model can be regarded as the final version output to generate the target large language model.

[0087] In an optional embodiment of the present application, after determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes:

[0088] The candidate answers in the candidate answer pool are filtered according to the reward score. By filtering, some high-quality candidate answers can be retained, while some low-quality candidate answers can be filtered out. For example, a reward score threshold can be set, and the reward score corresponding to the candidate answer is compared with the preset reward score threshold. The candidate answers with reward scores greater than or equal to the preset reward score threshold are retained, and the candidate answers with reward scores less than the preset reward score threshold are filtered out. This can better ensure that the model is updated on a higher quality data set, while reducing the risk of model divergence. Thus, multiple question query texts, the filtered candidate answer pools corresponding to the multiple question query texts, and the reward scores constitute the training sample data set.

[0089] This application determines the training sample data set (Q, A, R), and then the target strategy is initialized to a pre-trained language model and trained for a certain number of steps according to the above objective function. Once the policy update in the current iterative loop is completed, the strategy is reinitialized to the value-aligned model, and then the newly sampled candidate answers are used to replace the old candidate answers in the candidate answer pool. This helps to form a higher-quality candidate answer pool, and then the reward model is used to score these candidate answers to complete a new round of data sampling. The alternating iterative process of data sampling and policy training can continuously improve policy performance, because higher-quality samples help improve the strategy, thereby achieving more accurate task execution and answer generation. The following is a brief explanation of its detailed algorithm:

[0090] Algorithm: A self-reinforcing strategy for maximizing rewards during sampling and training. An iteration step is defined as the entire process of generating data once using the sampling strategy and training the Iterate step using the objective function J(θ).

[0091]

[0092] This application constructs a column-level optimization framework that allows for the direct integration of offline rewards into the optimization objective. By iteratively maximizing the overall reward across a diverse pool of candidate answers, the preferred pattern is learned. This approach offers significant advantages, requiring no complex parameter tuning and demonstrating excellent performance, particularly in conversational and summarization tasks. It holds significant potential for improving model performance and expanding its application areas.

[0093] The method provided in the embodiments of this application continuously updates through self-iteration, generating higher-quality data that aligns with mainstream value preferences for further model optimization. This higher-quality data, coupled with mainstream value preferences, further enhances the value alignment capabilities of the large language model. This ensures that the values of the large language model align with human values, avoids the risk of the large language model outputting harmful information, and improves its security and controllability.

[0094] Figure 3 FIG. 1 shows a structural block diagram of a large language model value alignment device according to an embodiment of the present application. Figure 3 As shown, the device includes:

[0095] The acquisition module 301 is adapted to acquire multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers;

[0096] A determination module 302 is adapted to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein the plurality of question query texts, the candidate answer pools corresponding to the plurality of question query texts, and the reward scores constitute a training sample dataset;

[0097] A calculation module 303 is adapted to determine, for each candidate answer to each question query text, an answer probability corresponding to the candidate answer, calculate a first expected reward corresponding to the question query text based on the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate a second expected reward corresponding to all question query texts in the training sample dataset based on the first expected reward corresponding to each question query text;

[0098] An alignment adjustment module 304 is adapted to perform value alignment adjustment on the large language model according to the second reward expectation and the training sample dataset, so as to adjust the model parameters;

[0099] The judgment module 305 is adapted to judge whether a predetermined convergence condition is satisfied; if not, the update module is triggered to execute; if so, the generation module is triggered to execute;

[0100] An updating module 306 is adapted to generate multiple new candidate answers corresponding to each question query text based on the adjusted large language model, update the candidate answer pool based on the multiple new candidate answers, and trigger the execution of the determining module;

[0101] The generation module 307 is adapted to generate a target large language model.

[0102] Optionally, the calculation module is further adapted to calculate the first reward expectation based on the following formula:

[0103]

[0104] Among them, E i is the first reward expectation corresponding to the i-th question query text, The answer probability corresponding to the jth candidate answer of the i-th question query text, R(x i ,y i,j ) is the reward score corresponding to the jth candidate answer of the i-th question query text, Ai is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model;

[0105] The second reward expectation is calculated using the following formula:

[0106]

[0107] Where J(θ) is the second reward expectation and N is the number of question query texts.

[0108] Optionally, the calculation module is further adapted to: perform word segmentation processing on each candidate answer of each question query text to obtain multiple word segmentation results;

[0109] Determine the probability of occurrence of each word segmentation result;

[0110] According to the word segmentation probability corresponding to multiple word segmentation results, the answer probability corresponding to the candidate answer is calculated.

[0111] Optionally, the device further comprises: a filtering module adapted to filter candidate answers in the candidate answer pool according to the reward score;

[0112] The determination module is further adapted to: a plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and reward scores constitute a training sample data set.

[0113] Optionally, the calculation module is further adapted to: normalize the answer probability of each candidate answer to each question query text;

[0114] The first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities and corresponding reward scores of multiple candidate answers.

[0115] Optionally, the predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iterative executions is greater than or equal to a preset number of iterations.

[0116] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0117] The device provided in the embodiments of this application continuously updates through self-iteration, generating higher-quality data that aligns with mainstream value preferences for further model optimization. This higher-quality data, coupled with mainstream value preferences, further enhances the value alignment capabilities of the large language model. This ensures that the values of the large language model align with human values, avoids the risk of the large language model outputting harmful information, and improves its security and controllability.

[0118] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the large language model value alignment method in any of the above method embodiments.

[0119] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the large language model value alignment method in any of the above method embodiments.

[0120] Figure 4 A schematic structural diagram of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0121] like Figure 4 As shown, the computing device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0122] Processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other devices, such as clients or other server network elements. Processor 402 is used to execute program 410, which may specifically perform the steps described in the embodiment of the large language model value alignment method for a computing device.

[0123] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0124] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0125] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0126] Program 410 can be specifically used to cause processor 402 to execute the large language model value alignment method in any of the above-mentioned method embodiments. The specific implementation of each step in program 410 can refer to the corresponding descriptions of the corresponding steps and units in the above-mentioned large language model value alignment embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-mentioned devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiments, and will not be repeated here.

[0127] The algorithm and display provided herein are not inherently relevant to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the embodiments of the present application are not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the embodiments of the present application described herein, and the description of the specific languages above is for the purpose of disclosing the best mode of implementation of the embodiments of the present application.

[0128] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0129] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the present application require more features than the features explicitly recited in each claim. More precisely, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the embodiments of the present application.

[0130] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0131] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features that are included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0132] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for performing a part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0133] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The embodiments of the present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A large language model value alignment method, comprising: S1, obtaining multiple question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: multiple candidate answers; S2, determining a reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, wherein multiple question query texts, the candidate answer pools corresponding to the multiple question query texts, and the reward scores constitute a training sample dataset; S3, for each candidate answer of each question query text, determining the answer probability corresponding to the candidate answer, calculating the first reward expectation corresponding to the question query text based on the answer probabilities and corresponding reward scores of the multiple candidate answers, and calculating the second reward expectation corresponding to all question query texts in the training sample dataset based on the first reward expectation corresponding to each question query text; S4, performing value alignment adjustment on the large language model based on the second reward expectation and the training sample dataset to adjust the model parameters and determine whether a predetermined convergence condition is met; if not, executing S5; if so, executing S6; S5, generating multiple new candidate answers corresponding to each question query text based on the adjusted large language model, updating the candidate answer pool based on the multiple new candidate answers, and jumping to S2; S6, generate the target large language model; The step of calculating the first reward expectation corresponding to the question query text based on the answer probabilities and the corresponding reward scores corresponding to the plurality of candidate answers, and calculating the second reward expectation corresponding to all question query texts in the training sample dataset based on the first reward expectation corresponding to each question query text further includes: The first reward expectation is calculated based on the following formula: Among them, E i is the first reward expectation corresponding to the i-th question query text, The answer probability corresponding to the jth candidate answer of the i-th question query text, The reward score corresponding to the jth candidate answer of the i-th question query text, A i is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model; The second reward expectation is calculated using the following formula: in, is the second reward expectation, and N is the number of question query texts.

2. The method according to claim 1, wherein For each candidate answer to each question query text, determining the answer probability corresponding to the candidate answer further includes: For each candidate answer of each question query text, perform word segmentation processing on the candidate answer to obtain multiple word segmentation results; Determine the probability of occurrence of each word segmentation result; The answer probability corresponding to the candidate answer is calculated based on the word segmentation occurrence probabilities corresponding to the multiple word segmentation results.

3. The method according to claim 1 or 2, wherein: After determining the reward score corresponding to each candidate answer in the candidate answer pool corresponding to each question query text, the method further includes: Filter the candidate answers in the candidate answer pool according to the reward scores; The training sample dataset comprising multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores further includes: A plurality of question query texts, a filtered candidate answer pool corresponding to the plurality of question query texts, and a reward score constitute a training sample data set.

4. The method according to claim 1 or 2, wherein: Calculating the first reward expectation corresponding to the question query text based on the answer probabilities and reward scores corresponding to the plurality of candidate answers further includes: Normalize the answer probability of each candidate answer for each question query text; A first reward expectation corresponding to the question query text is calculated based on the normalized answer probabilities and corresponding reward scores of the multiple candidate answers.

5. The method according to claim 1 or 2, wherein: The predetermined convergence condition further includes: the second reward expectation is greater than or equal to a preset reward expectation threshold, and / or the number of iterations is greater than or equal to a preset number of iterations.

6. A large language model value alignment device, comprising: An acquisition module is configured to acquire a plurality of question query texts and a candidate answer pool corresponding to each question query text, wherein the candidate answer pool includes: a plurality of candidate answers; A determination module is used to determine a reward score corresponding to each candidate answer in a candidate answer pool corresponding to each question query text, wherein multiple question query texts, candidate answer pools corresponding to the multiple question query texts, and reward scores constitute a training sample data set; A calculation module is used to determine, for each candidate answer to each question query text, an answer probability corresponding to the candidate answer, calculate a first reward expectation corresponding to the question query text based on the answer probabilities and corresponding reward scores corresponding to the multiple candidate answers, and calculate a second reward expectation corresponding to all question query texts in the training sample data set based on the first reward expectation corresponding to each question query text; an alignment adjustment module, configured to perform value alignment adjustment on the large language model based on the second reward expectation and the training sample dataset, so as to adjust the model parameters; The judgment module is used to judge whether the predetermined convergence conditions are met; if not, the update module is triggered to execute; if so, the generation module is triggered to execute; An updating module is configured to generate multiple new candidate answers corresponding to each question query text based on the adjusted large language model, update the candidate answer pool based on the multiple new candidate answers, and trigger the execution of the determining module; Generation module, used to generate the target large language model; The calculation module is further configured to calculate the first reward expectation based on the following formula: Among them, E i is the first reward expectation corresponding to the i-th question query text, The answer probability corresponding to the jth candidate answer of the i-th question query text, The reward score corresponding to the jth candidate answer of the i-th question query text, A i is the candidate answer pool corresponding to the i-th question query text, M is the number of candidate answers in the candidate answer pool of the i-th question query text, and θ is the model parameter of the large language model; The second reward expectation is calculated using the following formula: in, is the second reward expectation, and N is the number of question query texts.

7. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the large language model value alignment method according to any one of claims 1 to 5.

8. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the large language model value alignment method as described in any one of claims 1-5.

9. A computer program product comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the large language model value alignment method according to any one of claims 1 to 5.

Images