Large language model optimization method and optimization device

Abstract

The application relates to the technical field of artificial intelligence, and provides a large language model optimization method and an optimization device. The method obtains a candidate answer by inputting a question in a pre-training large language model, calculates a first reward value, a second reward value and a third reward value of the candidate answer, wherein the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the degree of fit of the candidate answer and the question, and the third reward value is used to represent the alignment degree of the candidate answer and user preferences, then the first reward value, the second reward value and the third reward value are multi-head fused to obtain a target reward value, and finally the pre-training large language model is trained based on the target reward value to obtain an optimized large language model, the importance coefficients of the reward mechanisms of different levels of the large language model are adaptively learned in a multi-head layered manner, the model training efficiency is improved, and the generation quality of the trained model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for optimizing large language models. Background Technology

[0002] In digital interactive systems based on LLM (Large Language Model), existing RL (Reinforcement Learning) fine-tuning techniques face the following key challenges:

[0003] 1. Insufficient granularity of reward signals: Existing RL frameworks typically use sequence-level reward signals, which cannot capture the fine-grained optimization direction of each token during language generation, resulting in slow model convergence and easy getting trapped in local optima.

[0004] 2. High cost of manual annotation: RLHF (Reinforcement Learning with Human Feedback) relies on manually annotated preference data to build reward models. When dealing with dynamically changing user behavior patterns (such as the migration of hot topics on social media), the annotation cost increases exponentially.

[0005] 3. Low efficiency in strategy exploration: Traditional strategy gradient methods use a random strategy with a fixed temperature coefficient during token sampling, which makes it difficult to balance the diversity and security of generated content (such as compliance control of promotional language in e-commerce scenarios).

[0006] 4. Real-time feedback delay issue: Existing offline reward modeling methods cannot effectively handle dynamic user interaction data streams, causing model updates to lag behind changes in user behavior patterns.

[0007] Therefore, it is necessary to improve the reinforcement learning optimization method for LLM models in order to reduce their training costs, improve training efficiency, and enhance the quality of the trained models. Summary of the Invention

[0008] In view of this, embodiments of this application provide a method and apparatus for optimizing large language models, in order to solve the technical problems of low training efficiency and insufficient accuracy of the trained model caused by unreasonable reward rule design in the optimization of large language models in the prior art.

[0009] A first aspect of this application provides a method for optimizing a large language model, comprising:

[0010] Input a question into a pre-trained large language model and obtain candidate answers;

[0011] Calculate the reward value of the candidate answer; the reward value includes at least a first reward value, a second reward value, and a third reward value. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preferences.

[0012] The first, second, and third reward values ​​are merged from multiple sources to obtain the target reward value;

[0013] The pre-trained large language model is trained based on the target reward value, and the optimized large language model is obtained.

[0014] A second aspect of this application provides a large language model optimization apparatus, comprising:

[0015] The input module is configured to take a question as input into a pre-trained large language model and obtain candidate answers.

[0016] The calculation module is configured to calculate the reward value of the candidate answer; the reward value includes at least a first reward value, a second reward value and a third reward value, the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preferences;

[0017] The fusion module is configured to perform multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain the target reward value;

[0018] The calculation module is also configured to dynamically calculate the temperature coefficient of the current step based on the preset divergence value of the pre-trained large language model and the generated position;

[0019] The update module is configured to update the temperature coefficient based on a preset temperature control strategy;

[0020] The training module is configured to train a pre-trained large language model based on the target reward value and the updated temperature coefficient, resulting in an optimized large language model.

[0021] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment obtains candidate answers by inputting a question into a pre-trained large language model, calculates a first reward value, a second reward value, and a third reward value for the candidate answers, where the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preference. Then, the first reward value, the second reward value, and the third reward value are fused by multiple heads to obtain a target reward value. Finally, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model. This realizes the adaptive learning of the importance coefficients of the reward mechanism at different levels of the large language model in a multi-head hierarchical manner, thereby improving the model training efficiency and enhancing the generation quality of the trained model. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a large language model optimization method provided in an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating the method for determining the first reward value provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart illustrating the method for determining a second reward value provided in an embodiment of this application.

[0026] Figure 4 This is a flowchart illustrating the method for determining a third reward value provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating the method for obtaining a target reward value by multi-head fusion of a first reward value, a second reward value, and a third reward value, as provided in an embodiment of this application.

[0028] Figure 6 This is a flowchart illustrating the method for determining the basic reward coefficient of the first reward value provided in an embodiment of this application.

[0029] Figure 7 This is a flowchart illustrating the method for determining the fit reward coefficient of the second reward value according to an embodiment of this application.

[0030] Figure 8This is a flowchart illustrating the method for determining the preference alignment reward coefficient of the third reward value provided in an embodiment of this application.

[0031] Figure 9 This is a flowchart illustrating another large language model optimization method provided in an embodiment of this application.

[0032] Figure 10 This is a schematic diagram of a large language model optimization device provided in an embodiment of this application.

[0033] Figure 11 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for optimizing a large language model according to an embodiment of this application.

[0036] As mentioned above, in LLM-based digital interactive systems, existing RL fine-tuning techniques face challenges such as insufficient granularity of reward signals, high cost of manual annotation, low efficiency of policy exploration, difficulty in balancing the diversity and security of generated content with random policies using fixed temperature coefficients, and real-time feedback latency.

[0037] Among related technologies, reinforcement learning methods for optimizing large language models mainly include the following technical approaches:

[0038] 1. Traditional Reinforcement Learning Algorithms: These algorithms employ general-purpose algorithms such as PPO (Proximal Policy Optimization) or DDPG (Deep Deterministic Policy Gradient) to drive model optimization through holistic reward functions, such as BLEU (Bilingual Evaluation Understudy) scores and response relevance scores. While widely used in dialogue system optimization, these methods suffer from reward sparsity issues.

[0039] 2. RLHF: This approach trains a reward model using manually labeled preference data to guide a policy model in generating content that aligns with human values. OpenAI's InstructGPT uses this method, but it faces bottlenecks such as high labeling costs and poor dynamic adaptability.

[0040] 3. Imitation Learning Framework: This approach utilizes Behavior Cloning or GAIL (Generative Adversarial Imitation Learning) to mimic expert demonstration data, such as DeepMind's Sparrow dialogue system. These methods rely on high-quality expert data and struggle with open-domain generation tasks.

[0041] 4. Rule-based reward design: This approach constructs a deterministic reward function using predefined grammatical rules (such as entity coverage and sentence structure regularity). It is commonly used in vertical domain dialogue systems, such as IBM Watson Assistant. However, rule maintenance is costly and its generalization ability is limited.

[0042] 5. Automatic reward modeling techniques: For example, Meta's PEER (Plan, Execute, Express, Review) framework attempts to generate synthetic feedback through a self-training mechanism, but in complex scenarios, it is prone to reward hacking, that is, the model obtains high scores by deceiving the reward function but reduces the actual effect.

[0043] The above solutions have significant limitations in the following aspects:

[0044] 1. Traditional reinforcement learning algorithms such as PPO have a single dimension in their reward signal design, relying only on sequence-level (e.g., BLEU scores) or round-level reward signals, failing to capture local optimization directions during token generation. The sparsity of rewards leads to excessively large variance in policy gradient estimation, requiring a large number of interaction samples for convergence. Random sampling strategies with a fixed temperature coefficient are difficult to balance in marketing copywriting and creative generation scenarios.

[0045] 2. The cost of manual annotation in RLHF is increasing exponentially, the timeliness of the annotated data is poor, and the subjective bias of the annotators leads to overfitting of the reward model.

[0046] 3. Rule-based reward design: Rule maintenance costs increase superlinearly with scenario complexity; every 10 new grammatical rules require additional computational overhead. The deterministic reward function limits the exploration space of the policy model, leading to a decrease in diversity metrics in personalized recommendation scenarios. It cannot handle implicit semantic constraints (such as cultural sensitivity), resulting in insufficient rule coverage.

[0047] 5. Automatic reward modeling techniques (such as PEER) suffer from semantic drift due to their self-training mechanism. After multiple iterations, the KL divergence between the reward signal and the true target increases. Reward hacking is severe, and the model may construct high-frequency reward keywords to boost false rewards.

[0048] In view of this, this application provides a method for optimizing a large language model. The method involves inputting a question into a pre-trained large language model to obtain candidate answers, calculating a first reward value, a second reward value, and a third reward value for each candidate answer. The first reward value represents the basic reward of the candidate answer, the second reward value represents the fit between the candidate answer and the question, and the third reward value represents the alignment between the candidate answer and user preferences. Then, the first, second, and third reward values ​​are fused using a multi-head fusion process to obtain a target reward value. Finally, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model. This method achieves adaptive learning of the importance coefficients of reward mechanisms at different levels of the large language model using a multi-head hierarchical approach, thereby improving model training efficiency and enhancing the quality of the generated model.

[0049] Figure 1 This is a flowchart illustrating a large language model optimization method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0050] In step S101, the question is input into the pre-trained large language model to obtain candidate answers.

[0051] In step S102, the reward value for the candidate answer is calculated.

[0052] The reward value includes at least a first reward value, a second reward value, and a third reward value. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preferences.

[0053] In step S103, the first reward value, the second reward value, and the third reward value are fused together to obtain the target reward value.

[0054] In step S104, a pre-trained large language model is trained based on the target reward value to obtain an optimized large language model.

[0055] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain computing capabilities.

[0056] In some embodiments of this application, a pre-trained large language model can be obtained, and a question can be input into the pre-trained large language model to obtain candidate answers to the question. In one example, the number of candidate answers may include one or more.

[0057] In some embodiments of this application, a reward value for a candidate answer can be calculated, which includes at least a first reward value, a second reward value, and a third reward value. The first reward value characterizes the basic reward of the candidate answer, the second reward value characterizes the relevance of the candidate answer to the question, and the third reward value characterizes the alignment of the candidate answer with user preferences.

[0058] In other words, the first reward value can be used to represent the basic reward of the candidate answer, which may include at least one of the following: language quality reward, grammatical correctness reward, and logical coherence reward. The second reward value is used to represent the fit between the candidate answer and the question, especially when the question includes specific prompts or context, and the specific context and / or topic can be derived from these prompts; this indicates whether the candidate answer is topic-related and context-appropriate. The third reward value is used to represent the alignment between the candidate answer and user preferences, where user preferences can be determined based on a pre-trained user preference reward model. These first, second, and third reward values ​​represent different dimensions of textual creative information, and thus represent different levels of reward values.

[0059] In some embodiments of this application, the first reward value, the second reward value, and the third reward value can be fused using a multi-head method to obtain a target reward value. Then, the pre-trained large language model can be trained based on the target reward value to obtain the optimized large language model.

[0060] According to the technical solution provided in the embodiments of this application, candidate answers are obtained by inputting a question into a pre-trained large language model. A first reward value, a second reward value, and a third reward value of the candidate answer are calculated. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preference. Then, the first reward value, the second reward value, and the third reward value are fused by multiple heads to obtain a target reward value. Finally, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model. This realizes the adaptive learning of the importance coefficients of the reward mechanism at different levels of the large language model in a multi-head hierarchical manner, thereby improving the model training efficiency and enhancing the generation quality of the trained model.

[0061] Figure 2 This is a flowchart illustrating the method for determining a first reward value provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0062] In step S201, the diversity reference word sequence in the candidate answer is determined, and the proportion of the diversity reference word sequence in all words is determined.

[0063] In step S202, the diversity of candidate answers is determined based on this proportion.

[0064] In step S203, the perplexity of the candidate answer is determined.

[0065] In step S204, a first reward value is determined based on diversity and confusion.

[0066] In some embodiments of this application, when determining the first reward value, a diversity reference word sequence in the candidate answer can be determined first. This diversity reference word sequence can be a sequence of n consecutive words in the candidate answer, where n is a positive integer and its value is set according to actual needs. For example, if n is 1, the diversity reference word sequence can be all words in the candidate answer; if n is 2, the diversity reference word sequence can be a sequence of all words in the candidate answer consisting of two consecutive words, such as the word sequence {a, b}, where a and b represent different words. And so on.

[0067] The number of diversity reference lexical sequences can be denoted as the number of n-grams. This number represents the number of different n-grams included in the candidate answer, i.e., the number of diversity reference lexical sequences in the candidate answer. For example, if a candidate answer includes A n-grams, then the number of diversity reference lexical sequences is A.

[0068] Next, we can determine the proportion of the diversity reference lexical sequence in the total number of lexical units. In one example, if the total number of lexical units in the candidate answer is A, that is, the candidate answer includes A n-gram lexical units, such as A1 1-gram lexical units, A2 2-gram lexical units, ... and An m-gram lexical units, where m is a positive integer greater than or equal to n; and at the same time, the diversity reference lexical sequence appears B times, that is, the n-gram lexical units appear B times in the candidate answer, then the proportion is B / A, where A and B are both positive integers, and A is greater than B.

[0069] The diversity of candidate answers can be determined based on this proportion. The larger the proportion, the richer the diversity of candidate answers. Therefore, provided that diversity is positively correlated with this proportion, the diversity of candidate answers can be determined based on this proportion through threshold comparison, table lookup, or function calculation. Other methods can also be used to determine the diversity of candidate answers, which are not limited here.

[0070] In some embodiments of this application, the perplexity of candidate answers can also be determined. The perplexity of a large language model is a parameter that measures the model's predictive ability on test data, and it can be determined by calculating the exponent of the cross-entropy on the test set. In some embodiments of this application, the test set can be the word sequences corresponding to the candidate answers.

[0071] In some embodiments of this application, the first reward value can be determined based on the diversity and perplexity. In one example, the first reward value can be determined based on diversity and perplexity by looking up a table. In another example, the first reward value can also be determined based on diversity and perplexity by using a trained prediction model. Alternatively, other methods can be used to determine the first reward value based on diversity and perplexity, which are not limited here.

[0072] In some implementations, the first reward value is denoted as R. base The definite diversity is denoted as dintinsinct-n, and the perplexity is denoted as PPL. -1 Then R base =dintinsinct-n+PPL -1 .

[0073] Figure 3 This is a flowchart illustrating the method for determining a second reward value provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0074] In step S301, the embedding vectors of each word in the candidate answer are obtained.

[0075] In step S302, the target prompt words in the question are obtained.

[0076] Among them, the target cue words are cue words that are related to at least one of the context and the theme.

[0077] In step S303, the similarity between the embedded vector and the target prompt word is determined as the second reward value.

[0078] In some embodiments of this application, when determining the second reward value, the embedding vectors of each word in the candidate answer can be obtained first, and then the target prompt word in the question can be obtained. The target prompt word is a prompt word that is related to at least one of the context and topic. Finally, a pre-trained similarity calculation model can be used to calculate the similarity between the embedding vector and the target prompt word, and the calculated similarity can be used as the second reward value.

[0079] In one example, if the target prompt word is denoted as "prompt" and the embedding vector of each word in the candidate answer is denoted as "sample", then the second reward value R is... topic For R topic=simility(prompt, sample) where simility() is the similarity calculation model.

[0080] Figure 4 This is a flowchart illustrating the method for determining a third reward value provided in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0081] In step S401, the target user is obtained.

[0082] The target users are those users of the optimized large language model.

[0083] In step S402, the target user's preference reward model is obtained.

[0084] The target user's preference reward model is trained based on target sample data, which is data labeled based on the target user's preferences.

[0085] In step S403, the candidate answers are scored using the target user's preference reward model to obtain a third reward value.

[0086] In some embodiments of this application, when determining the third reward value, the target user can be obtained first. The target user is the user group that the optimized large language model is intended for, such as a brand.

[0087] Then, the target user's preference reward model can be obtained. This model is trained on target sample data, which is data labeled based on the target user's preferences. In other words, a preference reward model can be maintained for the target user, used to label data according to their preferences, and then the labeled data is used to train and update the model in real time. This improves the relevance and accuracy of data labeling, and by labeling data solely based on the target user's preferences, it also reduces the amount of data to be labeled, thus increasing labeling efficiency.

[0088] Finally, the candidate answers can be scored using the target user's preference reward model to obtain a third reward value.

[0089] In one example, let the third reward value be denoted as R. hf Then R hf =RewardModel(sample), where RewardModel() is the target user's preference reward model.

[0090] By adopting the technical solution provided in the embodiments of this application, reward values ​​can be calculated from basic dimensions such as the text quality of candidate answers, the relevance of candidate answers to questions, and the alignment of candidate answers with user preferences, making the reward model of the constructed large language model more accurate, thereby improving model training efficiency and the generation quality of the trained model.

[0091] Figure 5 This is a flowchart illustrating the method for obtaining a target reward value by multi-head fusion of a first reward value, a second reward value, and a third reward value, as provided in an embodiment of this application. Figure 5 As shown, the method includes the following steps:

[0092] In step S501, the base reward coefficient of the first reward value, the fit reward coefficient of the second reward value, and the preference alignment reward coefficient of the third reward value are determined respectively.

[0093] In step S502, the product of the first reward value and the basic reward coefficient is determined as the first reward component, the product of the second reward value and the fit reward coefficient is determined as the second reward component, and the third reward value and the preference alignment reward coefficient are determined as the third reward component.

[0094] In step S503, the sum of the first reward component, the second reward component, and the third reward component is determined as the target reward value.

[0095] In some embodiments of this application, after determining the first reward value, the second reward value, and the third reward value, the base reward coefficient of the first reward value, the fit reward coefficient of the second reward value, and the preference alignment reward coefficient of the third reward value can also be determined respectively.

[0096] Then, the product of the first reward value and the base reward coefficient is determined as the first reward component; the product of the second reward value and the fit reward coefficient is determined as the second reward component; and the product of the third reward value and the preference alignment reward coefficient is determined as the third reward component. Finally, the sum of the first, second, and third reward components is determined as the target reward value.

[0097] In one example, if we denote the base reward coefficient as α, the fit reward coefficient as β, and the preference alignment reward coefficient as γ, then R total =α*R base +β*R topic +γ*R hf , where R total The target reward value.

[0098] Figure 6 This is a flowchart illustrating the method for determining the basic reward coefficient of the first reward value provided in an embodiment of this application. Figure 6 As shown, the method includes the following steps:

[0099] In step S601, a linear projection is applied to the hidden state of each word in the candidate answer to obtain the first instant reward coefficient.

[0100] The first instant reward coefficient is greater than or equal to 0 and less than or equal to 1.

[0101] In step S602, the first instant reward coefficient is determined as the base reward coefficient.

[0102] In some embodiments of this application, the basic reward coefficient for determining the first reward value may be obtained by applying a linear projection to the hidden state h_t of each token of the candidate answer to obtain a first instant reward coefficient. Then, this first instant reward coefficient is determined as the basic reward coefficient. .

[0103] Figure 7 This is a flowchart illustrating the method for determining the fit reward coefficient of the second reward value according to an embodiment of this application. Figure 7 As shown, the method includes the following steps:

[0104] In step S701, the candidate answers are split into sentences, and a learnable embedding vector is added to the end of each split sentence.

[0105] In step S702, the hidden state of each learnable embedding vector position is obtained.

[0106] In step S703, the hidden states of the learnable embedding vector positions are mapped to the same dimensional space as sentence lexical units through a linear layer.

[0107] In step S704, the importance weights of each sentence word are generated by a normalized exponential function, and the weighted sentence vector is obtained based on the importance weights of each sentence word.

[0108] In step S705, the weighted sentence vectors are subjected to gating fusion to obtain the fit reward coefficient.

[0109] In some embodiments of this application, determining the fit reward coefficient for the second reward value can be achieved by first splitting the candidate answers into sentences and adding a learnable embedding vector to the end of each sentence. Then, the hidden states at the positions of each learnable embedding vector are obtained. Next, a linear layer maps the hidden states at the positions of the learnable embedding vectors to the same dimensional space as the sentence words, and a normalized exponential Softmax function is used to generate importance weights for each sentence word. Based on these importance weights, a weighted sentence vector is obtained. Finally, gating and fusing the weighted sentence vector yields the fit reward coefficient.

[0110] In other words, candidate answers can be split into sentences, and a [CLS_S] token, which is a learnable embedding vector, can be added to the end of each sentence. The hidden state h_s at each [CLS_S] token position is extracted, and then the importance and authority of each token within the sentence are calculated using a linear layer and a softmax function. Finally, the weighted sentence vectors are gated and fused to obtain the fit reward coefficient. .

[0111] Figure 8 This is a flowchart illustrating the method for determining the preference alignment reward coefficient for a third reward value, as provided in an embodiment of this application. Figure 8 As shown, the method includes the following steps:

[0112] In step S801, a sequence end marker word is added to the end of the candidate answer.

[0113] The sequence end marker lexical unit aggregates the overall linguistic information of the candidate answers.

[0114] In step S802, the hidden state of the sequence end marker lexicon is mapped to the same dimensional space as the candidate answer vector through a linear layer.

[0115] In step S803, importance weights of candidate answer vectors are generated using a normalized exponential function.

[0116] In step S804, the importance weight of the candidate answer vector is determined as the preference alignment reward coefficient.

[0117] In some embodiments of this application, the preference alignment reward coefficient for determining the third reward value can be achieved by first adding a sequence end marker to the end of the candidate answer, which aggregates the overall linguistic information of the candidate answer. Then, a linear layer maps the hidden state of the sequence end marker to the same dimensional space as the candidate answer vector, and a normalized exponential Softmax function is used to generate the importance weights of the candidate answer vector. Finally, the importance weights of the candidate answer vector are determined as the preference alignment reward coefficient.

[0118] In other words, the preference alignment reward coefficient can be obtained by performing regression calculation on the last token vector [EOS] of the candidate answer through a linear layer and a softmax function. .

[0119] The technical solution provided in this application can adaptively learn the reward coefficients of different reward mechanisms through a multi-head approach to distinguish the importance of different reward mechanisms under different circumstances, thereby further improving the accuracy of the large model reward model, increasing model training efficiency, and improving the generation accuracy of the trained model.

[0120] Figure 9 This is a flowchart illustrating another large language model optimization method provided in an embodiment of this application. Specifically, steps S901 to S903 in the embodiment shown in Figure 9 are... Figure 1 Steps S101 to S103 in the illustrated embodiment are basically the same and will not be repeated here. Figure 9 As shown, the method also includes the following steps:

[0121] In step S904, the temperature coefficient of the current step is dynamically calculated based on the preset divergence value of the pre-trained large language model and the generated position.

[0122] In step S905, the temperature coefficient is updated based on a preset temperature control strategy.

[0123] In step S906, a pre-trained large language model is trained based on the target reward value and the updated temperature coefficient to obtain the optimized large language model.

[0124] In some embodiments of this application, a dynamic temperature coefficient control strategy can also be used to adjust the temperature coefficient during model training in real time. In one example, the temperature coefficient for the current step can be dynamically calculated first based on the preset divergence and generation position of the pre-trained large language model, and then the temperature coefficient can be updated based on the preset temperature control strategy.

[0125] To construct an exploration-utilization tradeoff model, a high temperature value can be maintained in the early stages of model generation to promote diversity, while the temperature value can be gradually reduced in the middle and later stages to ensure safety. The early stages of model generation can be the period when the model training epochs *t* are less than a preset threshold, while the middle and later stages can be the period when the model training epochs *t* are greater than or equal to the preset threshold. In one example, when *t* < 5, the temperature coefficient can be set to approximately 1.2; while when *t* >= 5, the temperature coefficient can be gradually reduced to 0.5.

[0126] Therefore, updating the temperature coefficient based on the preset temperature control strategy can be done as follows: if the model runtime is less than the preset time threshold, determine whether the difference between the calculated temperature coefficient of the current step and the set high temperature value is less than the first preset difference value. If so, there is no need to update the temperature coefficient; otherwise, adjust the preset divergence value or other parameters, and recalculate the temperature coefficient of the current step until the difference between the calculated temperature coefficient of the current step and the set high temperature value is less than the first preset difference value.

[0127] On the other hand, if the model runtime is greater than or equal to the preset time threshold, it can be determined whether the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step. If so, there is no need to update the temperature coefficient. Otherwise, the preset divergence value or other parameters are adjusted, and the temperature coefficient of the current step is recalculated until the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step.

[0128] In some embodiments of this application, a decreasing step size for the temperature coefficient can also be set. That is, after calculating the temperature coefficient of the current step and determining that the temperature coefficient of the current step is less than the temperature coefficient of the previous step, it can also be determined whether the difference between the temperature coefficient of the current step and the temperature coefficient of the previous step is less than a second preset difference value. If not, the temperature coefficient of the current step is recalculated until the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step, and the difference between the temperature coefficient of the current step and the temperature coefficient of the previous step is less than the second preset difference value.

[0129] By adopting dynamic temperature control, the stability of model training can be further improved, the compliance error rate during model training can be significantly reduced, and the diversity of statements generated during model generation can be increased.

[0130] By adopting the technical solution provided in the embodiments of this application, a hierarchical reinforcement learning framework including a three-layer optimization structure is established. Each layer focuses on different dimensions of textual creative information, and the importance coefficients of different reward mechanisms at different levels are adaptively learned in a multi-head manner. This achieves the optimization of the large language model, improves the training efficiency during the optimization of the large language model, reduces the need for manual annotation, and improves the generation accuracy of the model after training through dynamic temperature control, while maintaining the diversity of generated sentences.

[0131] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0132] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0133] Figure 10 This is a schematic diagram of a large language model optimization device provided in an embodiment of this application. Figure 10 As shown, the device includes:

[0134] Input module 1001 is configured to input a question into a pre-trained large language model and obtain candidate answers.

[0135] The calculation module 1002 is configured to calculate the reward value of the candidate answer; the reward value includes at least a first reward value, a second reward value and a third reward value, the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preference.

[0136] The fusion module 1003 is configured to perform multi-head fusion of the first reward value, the second reward value and the third reward value to obtain the target reward value.

[0137] The calculation module 1002 is also configured to dynamically calculate the temperature coefficient of the current step based on the preset divergence value of the pre-trained large language model and the generated position.

[0138] The update module 1004 is configured to update the temperature coefficient based on a preset temperature control strategy.

[0139] Training module 1005 is configured to train a pre-trained large language model based on the target reward value and the updated temperature coefficient, resulting in an optimized large language model.

[0140] According to the technical solution provided in the embodiments of this application, candidate answers are obtained by inputting a question into a pre-trained large language model. A first reward value, a second reward value, and a third reward value of the candidate answer are calculated. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the alignment between the candidate answer and the user's preference. Then, the first reward value, the second reward value, and the third reward value are fused by multiple heads to obtain the target reward value. At the same time, a dynamic temperature coefficient strategy is used. Finally, the pre-trained large language model is trained based on the target reward value and the real-time temperature coefficient to obtain an optimized large language model. This realizes the adaptive learning of the importance coefficients of the reward mechanism at different levels of the large language model in a multi-head hierarchical manner, thereby improving the model training efficiency and enhancing the generation quality of the trained model.

[0141] In some implementations, the first reward value is determined as follows: determining the diversity reference word sequence in the candidate answer, determining the proportion of the diversity reference word sequence in all words; determining the diversity of the candidate answer based on the proportion; determining the perplexity of the candidate answer; and determining the first reward value based on the diversity and perplexity.

[0142] In some implementations, the second reward value is determined as follows: the embedding vectors of each word in the candidate answer are obtained; the target prompt word in the question is obtained, wherein the target prompt word is a prompt word that is related to at least one of the context and the topic; and the similarity between the embedding vector and the target prompt word is determined as the second reward value.

[0143] In some implementations, the third reward value is determined as follows: the target user is obtained, which is the user that the optimized large language model is intended for; the target user's preference reward model is obtained, which is trained based on the target sample data, which is data labeled based on the target user's preferences; the candidate answer is scored using the target user's preference reward model to obtain the third reward value.

[0144] In some implementations, the target reward value is obtained by multi-head fusion of the first reward value, the second reward value, and the third reward value, including: determining the base reward coefficient of the first reward value, the fit reward coefficient of the second reward value, and the preference alignment reward coefficient of the third reward value respectively; determining the product of the first reward value and the base reward coefficient as the first reward component, the product of the second reward value and the fit reward coefficient as the second reward component, and the product of the third reward value and the preference alignment reward coefficient as the third reward component; and determining the sum of the first reward component, the second reward component, and the third reward component as the target reward value.

[0145] In some implementations, the base reward coefficient of the first reward value is determined as follows: a linear projection is applied to the hidden state of each word in the candidate answer to obtain the first instant reward coefficient; the first instant reward coefficient is greater than or equal to 0 and less than or equal to 1; the first instant reward coefficient is determined as the base reward coefficient.

[0146] In some implementations, the fit reward coefficient of the second reward value is determined as follows: candidate answers are split into sentences, and a learnable embedding vector is added to the end of each sentence; the hidden state at each learnable embedding vector position is obtained; the hidden state at each learnable embedding vector position is mapped to the same dimension space as sentence lexical units through a linear layer; the importance weight of each sentence lexical unit is generated through a normalized exponent Softmax function, and a weighted sentence vector is obtained based on the importance weight of each sentence lexical unit; the weighted sentence vector is gated and fused to obtain the fit reward coefficient.

[0147] In some implementations, the preference alignment reward coefficient of the third reward value is determined as follows: a sequence end marker word is added to the end of the candidate answer, which aggregates the overall linguistic information of the candidate answer; the hidden state of the sequence end marker word is mapped to the same dimensional space as the candidate answer vector through a linear layer; the importance weight of the candidate answer vector is generated through the normalized exponential Softmax function; and the importance weight of the candidate answer vector is determined as the preference alignment reward coefficient.

[0148] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0149] Figure 11 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 11 of this embodiment includes: a processor 1101, a memory 1102, and a computer program 1103 stored in the memory 1102 and executable on the processor 1101. When the processor 1101 executes the computer program 1103, it implements the steps in the various method embodiments described above. Alternatively, when the processor 1101 executes the computer program 1103, it implements the functions of each module / unit in the various device embodiments described above.

[0150] Electronic device 11 may be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 11 may include, but is not limited to, processor 1101 and memory 1102. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 11 and does not constitute a limitation on electronic device 11. It may include more or fewer components than shown, or different components.

[0151] The processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0152] The memory 1102 can be an internal storage unit of the electronic device 11, such as a hard disk or RAM of the electronic device 11. The memory 1102 can also be an external storage device of the electronic device 11, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc., equipped on the electronic device 11. The memory 1102 can also include both internal and external storage units of the electronic device 11. The memory 1102 is used to store computer programs and other programs and data required by the electronic device.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0155] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A large language model optimization method, characterized in that, The method comprises the following steps: inputting a question into a pre-trained large language model to obtain a candidate answer; calculating a reward value of the candidate answer; the reward value comprises at least a first reward value, a second reward value and a third reward value, the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the degree of fit between the candidate answer and the question, and the third reward value is used to represent the alignment degree of the candidate answer and the user preference; multi-head fusion is performed on the first reward value, the second reward value and the third reward value to obtain a target reward value; training the pre-trained large language model based on the target reward value to obtain an optimized large language model; dynamically calculating a temperature coefficient of the current step based on a preset divergence value and a generation position of the pre-trained large language model; updating the temperature coefficient based on a preset temperature control strategy; training the pre-trained large language model based on the target reward value and the updated temperature coefficient to obtain an optimized large language model; the first reward value is determined in the following way: determining a diversity reference token sequence in the candidate answer, and determining the proportion of the number of the diversity reference token sequence in all tokens; determining the diversity of the candidate answer based on the proportion; determining the perplexity of the candidate answer; determining the first reward value based on the diversity and the perplexity; the second reward value is determined in the following way: obtaining an embedding vector of each token in the candidate answer; obtaining a target prompt word in the question, the target prompt word being a prompt word related to at least one of the context and the theme; determining the similarity between the embedding vector and the target prompt word as the second reward value; the third reward value is determined in the following way: obtaining a target user, the target user being a user facing the optimized large language model; obtaining a preference reward model of the target user, the preference reward model of the target user being trained based on target sample data, the target sample data being data annotated based on the target user's preference; scoring the candidate answer using the preference reward model of the target user to obtain the third reward value.

2. The method of claim 1, wherein, Multi-head fusion of the first reward value, the second reward value and the third reward value to obtain a target reward value comprises: determining a basic reward coefficient of the first reward value, a fit degree reward coefficient of the second reward value and a preference alignment reward coefficient of the third reward value, respectively; determining the product of the first reward value and the basic reward coefficient as a first reward component, the product of the second reward value and the fit degree reward coefficient as a second reward component, and the product of the third reward value and the preference alignment reward coefficient as a third reward component; determining the sum of the first reward component, the second reward component and the third reward component as the target reward value.

3. The method of claim 2, wherein, The basic reward coefficient of the first reward value is determined in the following way: applying linear projection to the hidden state of each token of the candidate answer to obtain a first instant reward coefficient; the first instant reward coefficient is greater than or equal to 0 and less than or equal to 1; determining the first instant reward coefficient as the basic reward coefficient.

4. The method of claim 2, wherein, The degree-of-fit reward coefficient of the second reward value is determined in the following manner: The candidate answer is split according to sentences, and a learnable embedding vector is added at the end of each split sentence; Hidden states of each learnable embedding vector position are obtained; The hidden states of the learnable embedding vector positions are mapped to the same dimensional space as the sentence word elements through a linear layer; An importance weight of each sentence word element is generated through a normalized exponential Softmax function, and a weighted sentence vector is obtained based on the importance weight of each sentence word element; The weighted sentence vector is gate fused to obtain the degree-of-fit reward coefficient.

5. The method of claim 2, wherein, The preference alignment reward coefficient of the third reward value is determined in the following manner: A sequence end marker word element is added at the end of the candidate answer, and the sequence end marker word element aggregates the overall language information of the candidate answer; The hidden state of the sequence end marker word element is mapped to the same dimensional space as the candidate answer vector through a linear layer; An importance weight of the candidate answer vector is generated through a normalized exponential Softmax function; The importance weight of the candidate answer vector is determined as the preference alignment reward coefficient. 6.A large language model optimization apparatus, characterized by comprising: It comprises: An input module configured to input a question in a pre-trained large language model to obtain a candidate answer; A calculation module configured to calculate a reward value of the candidate answer; The reward value at least includes a first reward value, a second reward value and a third reward value, the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the degree of fit of the candidate answer with the question, and the third reward value is used to represent the alignment degree of the candidate answer with the user preference; A fusion module configured to multi-head fuse the first reward value, the second reward value and the third reward value to obtain a target reward value; The calculation module is further configured to dynamically calculate a temperature coefficient of the current step based on a preset divergence value of the pre-trained large language model; An update module configured to update the temperature coefficient based on a preset temperature control strategy; A training module configured to train the pre-trained large language model based on the target reward value and the updated temperature coefficient to obtain an optimized large language model; The calculation module is specifically configured to determine the first reward value in the following manner: determining a diversity reference word element sequence in the candidate answer, and determining the proportion of the diversity reference word element sequence number in all word elements; Based on the proportion, the diversity of the candidate answer is determined; the perplexity of the candidate answer is determined; and the first reward value is determined based on the diversity and perplexity; The calculation module is specifically configured to determine the second reward value in the following manner: obtaining an embedding vector of each word element in the candidate answer; obtaining a target prompt word in the question, the target prompt word being a prompt word related to at least one of the context and the theme; and determining the similarity between the embedding vector and the target prompt word as the second reward value; The computing module is specifically configured to determine the third reward value in the following manner: obtaining a target user, the target user being a user to which the optimized large language model faces; obtaining a preference reward model of the target user, the preference reward model of the target user being trained based on target sample data, the target sample data being data annotated based on a preference of the target user; and scoring the candidate answer using the preference reward model of the target user to obtain the third reward value. ​

Images