Large language model optimization method and optimization device

Abstract

The invention relates to the technical field of artificial intelligence, and provides a large language model optimization method and device. According to the method, a candidate answer is obtained by inputting a question in a pre-trained large language model, a first reward value, a second reward value and a third reward value of the candidate answer are calculated, the first reward value is used for representing a basic reward of the candidate answer, the second reward value is used for representing the integrating degree of the candidate answer and the question, and the third reward value is used for representing the integrating degree of the candidate answer and the question. The third reward value is used for representing the alignment degree of the candidate answers and the user preferences, then performing multi-head fusion on the first reward value, the second reward value and the third reward value to obtain a target reward value, and finally training a pre-trained large language model based on the target reward value to obtain an optimized large language model. The importance coefficients of different levels of reward mechanisms of the large language model are adaptively learned in a multi-head layering manner, so that the model training efficiency is improved, and the generation quality of the trained model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large language model optimization method and optimization device. Background Art

[0002] In digital interaction systems based on Large Language Models (LLMs), existing RL (Reinforcement Learning) fine-tuning techniques face the following key challenges: 1. Insufficient granularity of reward signals: Existing RL frameworks typically use sequence-level reward signals, which cannot capture the fine-grained optimization direction of each token in the language generation process. This results in slow model convergence and a tendency to fall into local optimal solutions.

[0003] 2. High cost of manual labeling: RLHF (Reinforcement Learning with Human Feedback) relies on manually labeled preference data to build reward models. When responding to dynamically changing user behavior patterns (such as the migration of hot topics on social media), there is a problem of exponentially growing labeling costs.

[0004] 3. Inefficient strategy exploration: Traditional policy gradient methods use a random strategy with a fixed temperature coefficient during token sampling, which makes it difficult to balance the diversity and security of generated content (such as compliance control of promotional language in e-commerce scenarios).

[0005] 4. Real-time feedback delay problem: Existing offline reward modeling methods cannot effectively handle dynamic user interaction data streams, causing model updates to lag behind changes in user behavior patterns.

[0006] In view of this, it is necessary to improve the reinforcement learning optimization method of the LLM model to reduce its training cost, improve training efficiency and the quality of the trained model. Summary of the Invention

[0007] In view of this, the embodiments of the present application provide a large language model optimization method and optimization device to solve the technical problems in the prior art of large language model optimization, such as low training efficiency and insufficient model accuracy after training due to unreasonable reward rule design.

[0008] A first aspect of the embodiments of the present application provides a large language model optimization method, comprising: Input questions into the pre-trained large language model and obtain candidate answers; Calculate the reward value of the candidate answer; the reward value includes at least a first reward value, a second reward value, and a third reward value. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the degree of alignment between the candidate answer and the user's preference. Perform multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain the target reward value; The pre-trained large language model is trained based on the target reward value to obtain an optimized large language model.

[0009] A second aspect of the embodiments of the present application provides a large language model optimization device, including: The input module is configured to input questions into the pre-trained large language model and obtain candidate answers; a calculation module configured to calculate a reward value for a candidate answer; the reward value includes at least a first reward value, a second reward value, and a third reward value, wherein the first reward value is used to represent a base reward for the candidate answer, the second reward value is used to represent a degree of fit between the candidate answer and the question, and the third reward value is used to represent a degree of alignment between the candidate answer and the user's preference; a fusion module configured to perform multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain a target reward value; The calculation module is further configured to dynamically calculate a temperature coefficient of a current step based on a preset divergence value and a generation position of a pre-trained large language model; an updating module configured to update the temperature coefficient based on a preset temperature control strategy; The training module is configured to train the pre-trained large language model based on the target reward value and the updated temperature coefficient to obtain an optimized large language model.

[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain candidate answers by inputting questions into a pre-trained large language model, and calculate the first reward value, second reward value and third reward value of the candidate answers, wherein the first reward value is used to characterize the basic reward of the candidate answer, the second reward value is used to characterize the fit between the candidate answer and the question, and the third reward value is used to characterize the degree of alignment between the candidate answer and the user preference, and then the first reward value, the second reward value and the third reward value are multi-headedly fused to obtain the target reward value, and finally the pre-trained large language model is trained based on the target reward value to obtain the optimized large language model, thereby realizing the adaptive learning of the importance coefficients of the reward mechanisms at different levels of the large language model in a multi-head layered manner, thereby improving the model training efficiency and improving the generation quality of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a flow chart of a large language model optimization method provided in an embodiment of the present application.

[0013] Figure 2 This is a flowchart of a method for determining a first reward value provided in an embodiment of the present application.

[0014] Figure 3 This is a flowchart of a method for determining a second reward value provided in an embodiment of the present application.

[0015] Figure 4 This is a flowchart of a method for determining a third reward value provided in an embodiment of the present application.

[0016] Figure 5 This is a flow chart of a method for performing multi-head fusion of a first reward value, a second reward value, and a third reward value to obtain a target reward value, as provided in an embodiment of the present application.

[0017] Figure 6 This is a flow chart of a method for determining a basic reward coefficient of a first reward value provided in an embodiment of the present application.

[0018] Figure 7 This is a flow chart of a method for determining a compatibility reward coefficient of a second reward value provided in an embodiment of the present application.

[0019] Figure 8 4 is a flow chart of a method for determining a preference alignment reward coefficient of a third reward value provided in an embodiment of the present application.

[0020] Figure 9 This is a flowchart of another large language model optimization method provided in an embodiment of the present application.

[0021] Figure 10 Schematic diagram of a large language model optimization device provided in an embodiment of the present application.

[0022] Figure 11 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0024] A large language model optimization method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0025] As mentioned above, in LLM-based digital interaction systems, existing RL fine-tuning techniques face challenges such as insufficient granularity of reward signals, high manual labeling costs, inefficient strategy exploration, the difficulty of using random strategies with fixed temperature coefficients to balance the diversity and security of generated content, and real-time feedback delays.

[0026] Among related technologies, reinforcement learning methods for optimizing large language models mainly include the following technical routes: 1. Traditional reinforcement learning algorithms: These employ general-purpose algorithms such as PPO (Proximal Policy Optimization) or DDPG (Deep Deterministic Policy Gradient). These algorithms drive model optimization by designing holistic reward functions such as BLEU (Bilingual Evaluation Understudy) scores and response relevance scores. These methods are widely used in dialogue system optimization, but suffer from reward sparsity.

[0027] 2. RLHF: This approach uses manually labeled preference data to train a reward model, guiding the policy model to generate content that aligns with human values. OpenAI's InstructGPT uses this approach, but it faces bottlenecks such as high labeling costs and poor dynamic adaptability.

[0028] 3. Imitation learning frameworks: These use behavior cloning or GAIL (Generative Adversarial Imitation Learning) to imitate expert demonstration data, such as DeepMind's Sparrow dialogue system. These methods rely on high-quality expert data and are difficult to handle for open-domain generation tasks.

[0029] 4. Rule-based reward design: This approach builds a deterministic reward function based on predefined grammatical rules (such as entity coverage and sentence structure standardization). This is commonly used in vertical dialogue systems, such as IBM Watson Assistant. However, this approach has high rule maintenance costs and limited generalization capabilities.

[0030] 5. Automatic reward modeling technology: Meta's PEER (Plan, Execute, Express, Review) framework attempts to generate synthetic feedback through a self-training mechanism. However, in complex scenarios, it is prone to reward hacking, where the model cheats the reward function to obtain high scores but reduces actual effectiveness.

[0031] The above solutions have significant limitations in the following aspects: 1. Traditional reinforcement learning algorithms, such as PPO, employ a single-dimensional reward signal design, relying solely on sequence-level (such as BLEU scores) or episode-level reward signals. This fails to capture local optimization trends during token generation. Reward sparsity leads to excessive variance in policy gradient estimates, requiring a high number of interaction samples for convergence. Random sampling strategies with fixed temperature coefficients struggle to achieve balanced results in creative marketing copy generation scenarios.

[0032] 2. The cost of manual labeling for RLHF is growing exponentially, the timeliness of labeled data is poor, and the subjective bias of the labelers leads to overfitting of the reward model.

[0033] 3. Rule-based reward design: The cost of maintaining rules increases superlinearly with scenario complexity. Every 10 new grammar rules requires increased computing power. Deterministic reward functions limit the exploration space of policy models, resulting in reduced diversity metrics in personalized recommendation scenarios. Implicit semantic constraints (such as cultural sensitivities) cannot be handled, resulting in insufficient rule coverage.

[0034] 5. The self-training mechanism of automatic reward modeling technologies (such as PEER) can cause semantic drift, increasing the KL divergence between the reward signal and the true target after multiple iterations. Reward hacking is a serious phenomenon, where the model may create false rewards by constructing high-frequency reward keywords.

[0035] In view of this, an embodiment of the present application provides a method for managing a large language model optimization, by inputting a question into a pre-trained large language model to obtain a candidate answer, calculating a first reward value, a second reward value, and a third reward value for the candidate answer, wherein the first reward value is used to characterize the basic reward of the candidate answer, the second reward value is used to characterize the fit between the candidate answer and the question, and the third reward value is used to characterize the degree of alignment between the candidate answer and the user preference, and then multi-head fusion is performed on the first reward value, the second reward value, and the third reward value to obtain a target reward value. Finally, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model, thereby realizing adaptive learning of the importance coefficients of the reward mechanism at different levels of the large language model in a multi-head layered manner, thereby improving the model training efficiency and improving the generation quality of the trained model.

[0036] Figure 1 This is a flow chart of a large language model optimization method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps: In step S101, a question is input into the pre-trained large language model to obtain candidate answers.

[0037] In step S102 , the reward value of the candidate answer is calculated.

[0038] Among them, the reward value includes at least a first reward value, a second reward value and a third reward value. The first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the degree of alignment between the candidate answer and the user's preference.

[0039] In step S103, the first reward value, the second reward value, and the third reward value are multi-fused to obtain a target reward value.

[0040] In step S104, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model.

[0041] In some embodiments of the present application, the method may be executed by a server, or by a terminal device with certain computing capabilities.

[0042] In some embodiments of the present application, a pre-trained large language model may be obtained, and a question may be input into the pre-trained large language model to obtain candidate answers to the question. In one example, the number of candidate answers may include one or more.

[0043] In some embodiments of the present application, a reward value for a candidate answer may be calculated, which includes at least a first reward value, a second reward value, and a third reward value. The first reward value represents the base reward for the candidate answer, the second reward value represents the degree of fit between the candidate answer and the question, and the third reward value represents the degree of alignment between the candidate answer and the user's preferences.

[0044] That is to say, the first reward value can be used to represent the basic reward for the candidate answer, which may include at least one of the language quality reward, grammatical correctness and logical coherence reward for the candidate answer. The second reward value is used to represent the fit between the candidate answer and the question, especially when the question includes specific prompt words or context, and the specific context and / or theme can be derived from the prompt words, to represent whether the candidate answer fits the theme and conforms to the context. The third reward value is used to represent the degree of alignment between the candidate answer and the user preference, where the user preference can be determined based on a pre-trained user preference reward model. The first reward value, the second reward value and the third reward value respectively represent different dimensions of textual creative information, and thus represent different levels of reward values.

[0045] In some embodiments of the present application, the first reward value, the second reward value, and the third reward value may be multi-fused to obtain a target reward value. Then, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model.

[0046] According to the technical solution provided in the embodiments of the present application, by inputting a question into a pre-trained large language model to obtain a candidate answer, a first reward value, a second reward value, and a third reward value of the candidate answer are calculated, wherein the first reward value is used to characterize the basic reward of the candidate answer, the second reward value is used to characterize the fit between the candidate answer and the question, and the third reward value is used to characterize the degree of alignment between the candidate answer and the user preference. Then, the first reward value, the second reward value, and the third reward value are multi-head fused to obtain a target reward value. Finally, the pre-trained large language model is trained based on the target reward value to obtain an optimized large language model, thereby realizing adaptive learning of the importance coefficients of the reward mechanisms at different levels of the large language model in a multi-head layered manner, thereby improving the model training efficiency and improving the generation quality of the trained model.

[0047] Figure 2 This is a flow chart of the method for determining the first reward value provided in the embodiment of the present application. Figure 2 As shown, the method includes the following steps: In step S201 , a diverse reference word-gram sequence in a candidate answer is determined, and a proportion of the diverse reference word-gram sequence in all word-grams is determined.

[0048] In step S202 , the diversity of the candidate answers is determined based on the proportions.

[0049] In step S203, the perplexity of the candidate answer is determined.

[0050] In step S204 , a first reward value is determined based on the diversity and the perplexity.

[0051] In some embodiments of the present application, when determining the first reward value, a diversity reference word sequence in the candidate answer can be first determined. The diversity reference word sequence can be a sequence consisting of n consecutive word-grams in the candidate answer, where n is a positive integer and its value is set according to actual needs. For example, if n is 1, the diversity reference word sequence can be all word-grams in the candidate answer; if n is 2, the diversity reference word sequence can be a word-gram sequence consisting of all two consecutive word-grams in the candidate answer, such as the word-gram sequence {a, b}, where a and b represent different word-grams. And so on.

[0052] The number of diverse reference word sequences can be expressed as the number of n-grams. This n-gram number represents the number of distinct n-grams in the candidate answer, i.e., the number of diverse reference word sequences in the candidate answer. For example, if the candidate answer includes A n-grams, then the number of diverse reference word sequences is A.

[0053] Next, the proportion of the diversity reference word sequence in the total number of words can be determined. In one example, if the total number of words in the candidate answer is A, that is, the candidate answer includes A n-gram words, such as A1 1-gram words, A2 2-gram words, ..., and An m-gram words, where m is a positive integer greater than or equal to n; and at the same time, the diversity reference word sequence appears B times, that is, the n-gram words appear B times in the candidate answer, then the proportion is B / A, where A and B are both positive integers, and A is greater than B.

[0054] The diversity of the candidate answers can be determined based on the percentage. The larger the percentage, the greater the diversity of the candidate answers. Therefore, under the premise of ensuring a positive correlation between diversity and the percentage, the diversity of the candidate answers can be determined based on the percentage by threshold comparison, table lookup, or function calculation. Other methods can also be used to determine the diversity of the candidate answers, which are not limited here.

[0055] In some embodiments of the present application, the perplexity of the candidate answer can also be determined. The perplexity of a large language model is a parameter that measures the model's predictive ability for test data and can be determined by calculating the cross-entropy index on a test set. In some embodiments of the present application, the test set can be the word sequence corresponding to the candidate answer.

[0056] In certain embodiments of the present application, a first reward value may be determined based on the diversity and perplexity. In one example, the first reward value may be determined based on the diversity and perplexity by looking up a table. In another example, the first reward value may be determined based on the diversity and perplexity using a trained prediction model. Alternatively, other methods may be used to determine the first reward value based on the diversity and perplexity, which are not limited herein.

[0057] In some implementations, if the first reward value is recorded as R base , the determined diversity is recorded as dintinsinct-n, and the perplexity is recorded as PPL -1 , then R base =dintinsinct-n+PPL -1 .

[0058] Figure 3 : is a flow chart of the method for determining the second reward value provided in the embodiment of the present application. Figure 3 As shown, the method includes the following steps: In step S301, the embedding vector of each word in the candidate answer is obtained.

[0059] In step S302, the target prompt word in the question is obtained.

[0060] The target prompt word is a prompt word related to at least one of the context and the theme.

[0061] In step S303, the similarity between the embedding vector and the target prompt word is determined as a second reward value.

[0062] In some embodiments of the present application, when determining the second reward value, the embedding vector of each word in the candidate answer can be first obtained, and then the target prompt word in the question can be obtained. The target prompt word is a prompt word related to at least one of the context and the topic. Finally, the similarity between the embedding vector and the target prompt word can be calculated using a pre-trained similarity calculation model, and the calculated similarity is used as the second reward value.

[0063] In an example, if the target prompt word is recorded as prompt and the embedding vector of each word in the candidate answer is recorded as sample, then the second reward value R topic R topic =simility(prompt, sample) where simility() is the similarity calculation model.

[0064] Figure 4 : is a flow chart of the method for determining the third reward value provided in the embodiment of the present application. Figure 4 As shown, the method includes the following steps: In step S401, the target user is obtained.

[0065] Among them, the target users are the users targeted by the optimized large language model.

[0066] In step S402, the target user's preference reward model is obtained.

[0067] The target user's preference reward model is trained based on target sample data, and the target sample data is data labeled based on the target user's preferences.

[0068] In step S403, the candidate answers are scored using the target user's preference reward model to obtain a third reward value.

[0069] In some embodiments of the present application, when determining the third reward value, a target user may be first obtained, wherein the target user is the user targeted by the optimized large language model, such as a brand.

[0070] The target user's preference reward model can then be obtained. This target user's preference reward model is trained based on target sample data, which is data labeled based on the target user's preferences. In other words, a preference reward model can be maintained for the target user. This preference reward model is used to label data based on the target user's preferences, and the labeled data is then used to train and update the model in real time. This improves the relevance and accuracy of data labeling. Furthermore, by labeling data based solely on the target user's preferences, the amount of data labeling can be reduced, improving labeling efficiency.

[0071] Finally, the target user's preference reward model can be used to score the candidate answers to obtain a third reward value.

[0072] In one example, if the third reward value is recorded as R hf , then R hf =RewardModel(sample), where RewardModel() is the target user's preferred reward model.

[0073] By adopting the technical solution provided in the embodiments of the present application, the reward value can be calculated based on basic dimensions such as the text quality of the candidate answers, the dimension of the fit between the candidate answers and the questions, and the dimension of whether the candidate answers are aligned with user preferences, so that the reward model of the constructed large language model is more accurate, thereby improving the model training efficiency and the generation quality of the trained model.

[0074] Figure 5 This is a flow chart of a method for obtaining a target reward value by multi-merging the first reward value, the second reward value, and the third reward value provided in an embodiment of the present application. Figure 5 As shown, the method includes the following steps: In step S501 , a base reward coefficient of the first reward value, a compatibility reward coefficient of the second reward value, and a preference alignment reward coefficient of the third reward value are determined respectively.

[0075] In step S502, the product of the first reward value and the basic reward coefficient is determined as the first reward component, the product of the second reward value and the fit reward coefficient is determined as the second reward component, and the third reward value and the preference alignment reward coefficient is determined as the third reward component.

[0076] In step S503 , the sum of the first reward component, the second reward component, and the third reward component is determined as the target reward value.

[0077] In some embodiments of the present application, after determining the first reward value, the second reward value, and the third reward value, the base reward coefficient of the first reward value, the fit reward coefficient of the second reward value, and the preference alignment reward coefficient of the third reward value may also be determined respectively.

[0078] Next, the product of the first reward value and the base reward coefficient is determined as the first reward component, the product of the second reward value and the fit reward coefficient is determined as the second reward component, and the third reward value and the preference alignment reward coefficient is determined as the third reward component. Finally, the sum of the first, second, and third reward components is determined as the target reward value.

[0079] In an example, if the base reward coefficient is recorded as α, the fit reward coefficient is recorded as β, and the preference alignment reward coefficient is recorded as γ, then R total =α*R base +β*R topic +γ*R hf , where R total is the target reward value.

[0080] Figure 6 : is a flow chart of a method for determining a basic reward coefficient of a first reward value provided by an embodiment of the present application. Figure 6 As shown, the method includes the following steps: In step S601, a linear projection is applied to the hidden state of each word-gram of the candidate answer to obtain a first immediate reward coefficient.

[0081] The first instant reward coefficient is greater than or equal to 0 and less than or equal to 1.

[0082] In step S602, a first instant reward coefficient is determined as a basic reward coefficient.

[0083] In some embodiments of the present application, the basic reward coefficient for determining the first reward value may be to apply a linear projection to the hidden state h_t of each token of the candidate answer to obtain a first immediate reward coefficient. The first immediate reward coefficient is then determined as the basic reward coefficient. .

[0084] Figure 7 This is a flow chart of the method for determining the compatibility reward coefficient of the second reward value provided in the embodiment of the present application. Figure 7 As shown, the method includes the following steps: In step S701, the candidate answers are split into sentences, and a learnable embedding vector is added to the end of each sentence after the split.

[0085] In step S702, the hidden state of each learnable embedding vector position is obtained.

[0086] In step S703, the hidden state of the learnable embedding vector position is mapped to the same dimensional space as the sentence word through a linear layer.

[0087] In step S704 , the importance weight of each sentence word is generated by a normalized exponential function, and a weighted sentence vector is obtained based on the importance weight of each sentence word.

[0088] In step S705, gate fusion is performed on the weighted sentence vector to obtain a fit reward coefficient.

[0089] In some embodiments of the present application, the fit reward coefficient for determining the second reward value may be to first split the candidate answer into sentences and add a learnable embedding vector to the end of each sentence after the split. Then obtain the hidden state of each learnable embedding vector position. Next, the hidden state of the learnable embedding vector position is mapped to the same dimensional space as the sentence word through a linear layer, and the importance weight of each sentence word is generated through a normalized exponential Softmax function, and a weighted sentence vector is obtained based on the importance weight of each sentence word. Finally, the weighted sentence vector is gated and fused to obtain the fit reward coefficient.

[0090] In other words, the candidate answers can be split into sentences and [CLS_S] token is added at the end of each sentence. The [CLS_S] token is a learnable embedding vector. The hidden state h_s of each [CLS_S] token position is extracted, and then the importance of each token in the sentence is calculated through a linear layer and a Softmax function. Finally, the weighted sentence vector is gated and fused to obtain the fit reward coefficient. .

[0091] Figure 8: is a flow chart of a method for determining a preference alignment reward coefficient of a third reward value provided by an embodiment of the present application. Figure 8 As shown, the method includes the following steps: In step S801, a sequence end marker word is added to the end of the candidate answer.

[0092] The sequence end marker word aggregates the overall language information of the candidate answers.

[0093] In step S802, the hidden state of the end-of-sequence marker word is mapped to the same dimensional space as the candidate answer vector through a linear layer.

[0094] In step S803 , the importance weight of the candidate answer vector is generated by a normalized exponential function.

[0095] In step S804 , the importance weight of the candidate answer vector is determined as a preference alignment reward coefficient.

[0096] In some embodiments of the present application, the preference alignment reward coefficient for determining the third reward value may be determined by first adding an end-of-sequence marker token to the end of the candidate answer. This end-of-sequence marker token aggregates the overall linguistic information of the candidate answer. Then, using a linear layer, the hidden state of the end-of-sequence marker token is mapped to the same dimensional space as the candidate answer vector, and a normalized exponential Softmax function is used to generate the importance weight of the candidate answer vector. Finally, the importance weight of the candidate answer vector is determined as the preference alignment reward coefficient.

[0097] That is to say, the last token vector [EOS] of the candidate answer can be regressed through the linear layer and the Softmax function to obtain the preference alignment reward coefficient .

[0098] By adopting the technical solution provided in the embodiment of the present application, the reward coefficients of reward mechanisms at different levels can be adaptively learned in a multi-head manner to distinguish the importance of reward mechanisms at different levels in different situations, thereby further improving the accuracy of the large-model reward model, improving the model training efficiency, and improving the generation accuracy of the trained model.

[0099] Figure 9 9 is a flow chart of another large language model optimization method provided by the embodiment of the present application. Figure 1 Steps S101 to S103 in the embodiment shown are basically the same and will not be described in detail here. Figure 9 As shown, the method further includes the following steps: In step S904 , the temperature coefficient of the current step is dynamically calculated based on the preset divergence value and the generation position of the pre-trained large language model.

[0100] In step S905 , the temperature coefficient is updated based on a preset temperature control strategy.

[0101] In step S906 , the pre-trained large language model is trained based on the target reward value and the updated temperature coefficient to obtain an optimized large language model.

[0102] In some embodiments of the present application, a dynamic temperature coefficient control strategy can also be used to adjust the temperature coefficient during model training in real time. In one example, the temperature coefficient of the current step can be dynamically calculated based on the preset divergence and generation position of the pre-trained large language model, and then updated based on the preset temperature control strategy.

[0103] To model the exploration-exploitation tradeoff, a high temperature value can be set in the early stages of model generation to promote diversity, while the temperature value can be gradually lowered in the middle and late stages of model generation to ensure safety. The early stages of model generation can be when the number of model training rounds, t, is less than a preset threshold, while the middle and late stages of model generation can be when the number of model training rounds, t, is greater than or equal to the preset threshold. In one example, the temperature coefficient can be set to approximately 1.2 when t < 5, and gradually lowered to 0.5 when t >= 5.

[0104] Therefore, updating the temperature coefficient based on the preset temperature control strategy can be that if the model running time is less than the preset time threshold, it is determined whether the difference between the calculated temperature coefficient of the current step and the set high temperature value is less than the first preset difference value. If so, there is no need to update the temperature coefficient. Otherwise, the preset divergence value or other parameters are adjusted, and the temperature coefficient of the current step is recalculated until the difference between the calculated temperature coefficient of the current step and the set high temperature value is less than the first preset difference value.

[0105] On the other hand, if the model running time is greater than or equal to the preset time threshold, it can be determined whether the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step. If so, there is no need to update the temperature coefficient. Otherwise, adjust the preset divergence value or other parameters and recalculate the temperature coefficient of the current step until the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step.

[0106] In some embodiments of the present application, a decreasing step size of the temperature coefficient can also be set. That is, after calculating the temperature coefficient of the current step and determining that the temperature coefficient of the current step is less than the temperature coefficient of the previous step, it can also be determined whether the difference between the temperature coefficient of the current step and the temperature coefficient of the previous step is less than a second preset difference value. If not, the temperature coefficient of the current step is recalculated until the calculated temperature coefficient of the current step is less than the temperature coefficient of the previous step, and the difference between the temperature coefficient of the current step and the temperature coefficient of the previous step is less than the second preset difference value.

[0107] The use of dynamic temperature control can further improve the stability of model training, significantly reduce the compliance error rate during model training, and increase the statement diversity during model generation.

[0108] By adopting the technical solution provided in the embodiments of the present application, a hierarchical reinforcement learning framework is established, including a three-layer optimization structure. Each layer focuses on textual creative information of different dimensions, and the importance coefficients of reward mechanisms at different levels are adaptively learned in a multi-headed manner. This achieves the optimization of large language models, improves the training efficiency during optimization of large language models, reduces the need for manual labeling, and improves the generation accuracy of the trained model through dynamic temperature control, while maintaining the diversity of generated sentences.

[0109] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0110] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0111] Figure 10 Schematic diagram of a large language model optimization device provided in an embodiment of the present application. Figure 10 As shown, the device includes: The input module 1001 is configured to input a question into the pre-trained large language model to obtain candidate answers.

[0112] Calculation module 1002 is configured to calculate the reward value of the candidate answer; the reward value includes at least a first reward value, a second reward value and a third reward value, the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the fit between the candidate answer and the question, and the third reward value is used to represent the degree of alignment between the candidate answer and the user's preference.

[0113] The fusion module 1003 is configured to perform multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain a target reward value.

[0114] The calculation module 1002 is further configured to dynamically calculate the temperature coefficient of the current step based on the preset divergence value and the generation position of the pre-trained large language model.

[0115] The updating module 1004 is configured to update the temperature coefficient based on a preset temperature control strategy.

[0116] The training module 1005 is configured to train the pre-trained large language model based on the target reward value and the updated temperature coefficient to obtain an optimized large language model.

[0117] According to the technical solution provided in the embodiments of the present application, a question is input into a pre-trained large language model to obtain a candidate answer, and a first reward value, a second reward value, and a third reward value of the candidate answer are calculated, wherein the first reward value is used to characterize the basic reward of the candidate answer, the second reward value is used to characterize the fit between the candidate answer and the question, and the third reward value is used to characterize the degree of alignment between the candidate answer and the user preference. The first reward value, the second reward value, and the third reward value are then multi-headedly fused to obtain a target reward value. At the same time, a dynamic temperature coefficient strategy is utilized. Finally, the pre-trained large language model is trained based on the target reward value and the real-time temperature coefficient to obtain an optimized large language model, thereby realizing adaptive learning of the importance coefficients of the reward mechanisms at different levels of the large language model in a multi-head layered manner, thereby improving the model training efficiency and the generation quality of the trained model.

[0118] In some embodiments, the first reward value is determined in the following manner: determining the diversity reference word sequence in the candidate answer, determining the proportion of the number of diversity reference word sequences in all words; determining the diversity of the candidate answer based on the proportion; determining the perplexity of the candidate answer; and determining the first reward value based on the diversity and perplexity.

[0119] In some embodiments, the second reward value is determined as follows: obtaining an embedding vector for each word in the candidate answer; obtaining a target prompt word in the question, where the target prompt word is a prompt word related to at least one of the context and the topic; and determining the similarity between the embedding vector and the target prompt word as the second reward value.

[0120] In some embodiments, the third reward value is determined as follows: obtain the target user, where the target user is the user targeted by the optimized large language model; obtain the target user's preference reward model, where the target user's preference reward model is trained based on target sample data, where the target sample data is data labeled based on the target user's preferences; use the target user's preference reward model to score the candidate answers to obtain the third reward value.

[0121] In some embodiments, the first reward value, the second reward value, and the third reward value are multi-fused to obtain a target reward value, including: respectively determining a basic reward coefficient for the first reward value, a fit reward coefficient for the second reward value, and a preference alignment reward coefficient for the third reward value; determining the product of the first reward value and the basic reward coefficient as the first reward component, the product of the second reward value and the fit reward coefficient as the second reward component, and the third reward value and the preference alignment reward coefficient as the third reward component; and determining the sum of the first reward component, the second reward component, and the third reward component as the target reward value.

[0122] In some embodiments, the base reward coefficient of the first reward value is determined as follows: a linear projection is applied to the hidden state of each word unit of the candidate answer to obtain a first immediate reward coefficient; the first immediate reward coefficient is greater than or equal to 0 and less than or equal to 1; and the first immediate reward coefficient is determined as the base reward coefficient.

[0123] In some embodiments, the fit reward coefficient of the second reward value is determined as follows: the candidate answers are split into sentences, and a learnable embedding vector is added to the end of each sentence after the split; the hidden state of each learnable embedding vector position is obtained; the hidden state of the learnable embedding vector position is mapped to the same dimensional space as the sentence word unit through a linear layer; the importance weight of each sentence word unit is generated through a normalized exponential Softmax function, and a weighted sentence vector is obtained based on the importance weight of each sentence word unit; the weighted sentence vector is gated and fused to obtain the fit reward coefficient.

[0124] In some embodiments, the preference alignment reward coefficient of the third reward value is determined as follows: a sequence end marker word is added to the end of the candidate answer, and the sequence end marker word aggregates the overall language information of the candidate answer; the hidden state of the sequence end marker word is mapped to the same dimensional space as the candidate answer vector through a linear layer; the importance weight of the candidate answer vector is generated through a normalized exponential Softmax function; and the importance weight of the candidate answer vector is determined as the preference alignment reward coefficient.

[0125] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0126] Figure 11 Schematic diagram of an electronic device provided in an embodiment of the present application. Figure 11As shown, the electronic device 11 of this embodiment includes: a processor 1101, a memory 1102, and a computer program 1103 stored in the memory 1102 and executable by the processor 1101. When the processor 1101 executes the computer program 1103, the steps of the above-described method embodiments are implemented. Alternatively, when the processor 1101 executes the computer program 1103, the functions of the modules / units in the above-described device embodiments are implemented.

[0127] The electronic device 11 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 11 may include but is not limited to a processor 1101 and a memory 1102. Those skilled in the art will appreciate that Figure 11 The electronic device 11 is merely an example and does not limit the electronic device 11 , and may include more or fewer components than shown in the figure, or different components.

[0128] The processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0129] Memory 1102 can be an internal storage unit of electronic device 11, such as a hard drive or memory of electronic device 11. Memory 1102 can also be an external storage device of electronic device 11, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Memory 1102 can also include both an internal storage unit of electronic device 11 and an external storage device. Memory 1102 is used to store computer programs and other programs and data required by the electronic device.

[0130] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A large language model optimization method, characterized in that: include: Input questions into the pre-trained large language model and obtain candidate answers; Calculating a reward value for the candidate answer; The reward value includes at least a first reward value, a second reward value, and a third reward value, wherein the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the degree of fit between the candidate answer and the question, and the third reward value is used to represent the degree of alignment between the candidate answer and the user's preference; Performing multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain a target reward value; The pre-trained large language model is trained based on the target reward value to obtain an optimized large language model.

2. The method according to claim 1, characterized in that The first reward value is determined in the following manner: Determine a diverse reference word sequence in the candidate answer, and determine a proportion of the diverse reference word sequence in all word units; determining the diversity of the candidate answers based on the proportions; Determining the perplexity of the candidate answer; The first reward value is determined based on the diversity and the perplexity.

3. The method according to claim 1, characterized in that The second reward value is determined in the following manner: Obtaining an embedding vector for each word in the candidate answer; Obtaining a target prompt word in the question, wherein the target prompt word is a prompt word related to at least one of the context and the topic; Determine the similarity between the embedding vector and the target prompt word as the second reward value.

4. The method according to claim 1, wherein The third reward value is determined in the following manner: Obtaining target users, where the target users are users targeted by the optimized large language model; Obtaining a preference reward model for the target user, where the preference reward model for the target user is trained based on target sample data, where the target sample data is data labeled based on the target user's preferences; The candidate answers are scored using the target user's preference reward model to obtain the third reward value.

5. The method according to claim 1, wherein Performing multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain a target reward value includes: Determine a base reward coefficient for the first reward value, a fit reward coefficient for the second reward value, and a preference alignment reward coefficient for the third reward value respectively; Determine the product of the first reward value and the basic reward coefficient as a first reward component, the product of the second reward value and the compatibility reward coefficient as a second reward component, and the third reward value and the preference alignment reward coefficient as a third reward component; The sum of the first reward component, the second reward component, and the third reward component is determined to be the target reward value.

6. The method according to claim 5, characterized in that The basic bonus coefficient of the first bonus value is determined as follows: Applying a linear projection to the hidden state of each word-gram of the candidate answer to obtain a first immediate reward coefficient; the first immediate reward coefficient is greater than or equal to 0 and less than or equal to 1; The first instant reward coefficient is determined as the basic reward coefficient.

7. The method according to claim 5, characterized in that The compatibility bonus coefficient of the second bonus value is determined as follows: Split the candidate answer into sentences and add a learnable embedding vector to the end of each sentence after the split; Get the hidden state of each learnable embedding vector position; Mapping the hidden state of the learnable embedding vector position to the same dimensional space as the sentence word through a linear layer; The importance weight of each sentence word is generated by the normalized exponential Softmax function, and the weighted sentence vector is obtained based on the importance weight of each sentence word; Gated fusion is performed on the weighted sentence vector to obtain the fit reward coefficient.

8. The method according to claim 5, characterized in that The preference alignment reward coefficient of the third reward value is determined as follows: Adding a sequence end marker word at the end of the candidate answer, wherein the sequence end marker word aggregates the overall language information of the candidate answer; Mapping the hidden state of the end-of-sequence token to the same dimensional space as the candidate answer vector through a linear layer; Generating the importance weight of the candidate answer vector by a normalized exponential Softmax function; The importance weight of the candidate answer vector is determined as the preference alignment reward coefficient.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Dynamically calculating a temperature coefficient of a current step based on a preset divergence value and a generation position of the pre-trained large language model; updating the temperature coefficient based on a preset temperature control strategy; The pre-trained large language model is trained based on the target reward value and the updated temperature coefficient to obtain an optimized large language model.

10. A large language model optimization device, characterized in that: include: The input module is configured to input questions into the pre-trained large language model and obtain candidate answers; A calculation module, configured to calculate the reward value of the candidate answer; The reward value includes at least a first reward value, a second reward value, and a third reward value, wherein the first reward value is used to represent the basic reward of the candidate answer, the second reward value is used to represent the degree of fit between the candidate answer and the question, and the third reward value is used to represent the degree of alignment between the candidate answer and the user's preference; a fusion module configured to perform multi-head fusion on the first reward value, the second reward value, and the third reward value to obtain a target reward value; The calculation module is further configured to dynamically calculate a temperature coefficient of a current step based on a preset divergence value and a generation position of the pre-trained large language model; an updating module configured to update the temperature coefficient based on a preset temperature control strategy; The training module is configured to train the pre-trained large language model based on the target reward value and the updated temperature coefficient to obtain an optimized large language model.

Images