Model training method and device, equipment and storage medium

Abstract

The invention discloses a model training method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps of generating at least one output text based on an input prompt text through a reference strategy model; for each output text, acquiring context entropy information of the output text; based on the context entropy information of the output text, determining update weight information of the output text, the update weight information of the output text comprising an update weight of at least one lexical element in the output text, the update weight of the lexical element being used for indicating an influence degree of the lexical element on parameters of the current strategy model; and adjusting parameters of the current strategy model based on the updated weight information of each output text. According to the method, the lexical-level fine adjustment of the parameters of the current strategy model is realized, so that the learning efficiency of the current strategy model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, apparatus, device, and storage medium. Background Technology

[0002] Currently, with the development of computer technology, large language models are widely used in various fields. Large language models require extensive training to achieve powerful language understanding capabilities.

[0003] In related technologies, a large language model is used to generate at least one output text based on input prompt text. Then, a reward model is used to score the at least one output text to obtain a reward value. Based on the reward value of the at least one output text, the parameters of the large language model are adjusted to train the large language model.

[0004] The aforementioned technologies evaluate the output text as a whole and adjust the output text level of the large language model. This may lead to poor learning performance at key inference steps during training, resulting in slow learning efficiency of the large language model. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, and storage medium. The technical solutions provided by this application are as follows: According to one aspect of the embodiments of this application, a model training method is provided, the method comprising: Based on the input prompt text, at least one output text is generated using a reference policy model, which is used to assist in training the current policy model. For each output text, obtain the context entropy information of the output text, the context entropy information including the context entropy of at least one word in the output text, the context entropy of the word being used to indicate the uncertainty of the reference policy model when generating the word; Based on the context entropy information of the output text, the update weight information of the output text is determined. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model. Based on the updated weight information of each of the output texts, the parameters of the current policy model are adjusted.

[0006] According to one aspect of the embodiments of this application, a model training apparatus is provided, the apparatus comprising: The generation module is used to generate at least one output text based on the input prompt text using a reference policy model, wherein the reference policy model is used to assist in training the current policy model. The acquisition module is configured to acquire context entropy information of each output text, wherein the context entropy information includes the context entropy of at least one word in the output text, and the context entropy of the word is used to indicate the uncertainty of the reference policy model when generating the word; The determination module is used to determine the update weight information of the output text based on the context entropy information of the output text. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model. The adjustment module is used to adjust the parameters of the current strategy model based on the updated weight information of each of the output texts.

[0007] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described model training method.

[0008] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described model training method.

[0009] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being loaded and executed by a processor to implement the above-described model training method.

[0010] The technical solutions provided in this application have at least the following beneficial effects: After generating at least one output text using a reference policy model, the context entropy of each word in these output texts is obtained to assess the uncertainty of the reference policy model in generating each word. Then, based on the context entropy of each word, the update weight of each word is determined, enabling fine-tuning of the parameters of the current policy model at the word level. By considering the uncertainty of each word during generation and assigning update weights to it, the influence of each word on the parameter updates of the current policy model is determined, allowing for targeted training of the large language model and thus improving the learning efficiency of the current policy model. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application; Figure 2 This is a flowchart of a model training method provided in one embodiment of this application; Figure 3 This is a flowchart illustrating the determination and updating of weight information according to an embodiment of this application; Figure 4 This is a block diagram of a model training apparatus provided in one embodiment of this application; Figure 5 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0013] Please refer to Figure 1 The diagram illustrates a computer system provided in one embodiment of this application. The computer system may include a model training device 10 and a model usage device 20.

[0014] The model training device 10 is an electronic device with data computation, processing, and storage functions. In some embodiments, the model training device 10 can be either a terminal device or a server. In this application, the model training device 10 is used to assist in training the current policy model using a reference policy model.

[0015] In some embodiments, the training process of the current policy model is as follows (this is only a brief description; for a detailed training process, please refer to [link to documentation]): Obtaining input prompt text; generating output text based on the input prompt text using a reference policy model; determining the context entropy information of the output text based on the output text, wherein the context entropy information of the output text includes the context entropy of at least one word element included in the output text, and the context entropy of the word element is used to indicate the uncertainty of the reference policy model when generating the word element; determining the update weight of the word element based on the context entropy of the word element; adjusting the parameters of the current policy model based on the update weight information of at least one output text (the update weight information of the output text includes the update weight of at least one word element included in the output text), to obtain the trained current policy model. Optionally, both the current policy model and the reference policy model are large language models. Optionally, when the current policy model meets the condition for stopping updates, the updating of the parameters of the current policy model is stopped, and the trained current policy model is obtained.

[0016] The model-using device 20 is an electronic device with data computing, processing, and storage functions. In some embodiments, the model-using device 20 can be either a terminal device or a server. In this application, the model-using device 20 uses the trained current policy model to output corresponding output text based on the input prompt text.

[0017] The model training device 10 and the model usage device 20 can be two separate devices or the same device.

[0018] In some embodiments, the terminal device may include, but is not limited to, at least one of the following: personal computer, laptop computer, tablet computer, smartphone, vehicle terminal, etc., and may also include other terminal devices, which are not limited in this application embodiment.

[0019] In some embodiments, the server may include, but is not limited to, at least one of the following: physical server, server cluster, distributed system, cloud server, edge server, etc., and may also include other types of servers, which are not limited in this application embodiment.

[0020] In related technologies, PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) are used to optimize large language models during the reinforcement learning stage (such as RLHF (Reinforcement Learning from Human Feedback) and RLAIF (Reinforcement Learning from AI Feedback)).

[0021] PPO introduces a "clipping mechanism" to limit the importance ratio between the new and old policies, preventing training from crashing due to excessively large policy update steps.

[0022] GRPO removes the (penalty) network from PPO and reduces computational resource consumption by calculating the relative reward advantage by generating a group of outputs for the same prompt.

[0023] The aforementioned techniques employ "hard clipping." In long-chain reasoning tasks, when a large language model experiences "insight" at a critical step (i.e., the probability of the new policy is significantly higher than that of the old policy), hard clipping forcibly truncates these high-value gradient signals to a constant, leading to vanishing gradients (signal loss). This makes it difficult for the large language model to effectively learn key reasoning transitions.

[0024] In another related technology, hard truncation is not adopted, and a soft gating factor based on the Sigmoid function is introduced. When the Importance Ratio deviates from 1, the soft gating factor will decay smoothly, thus limiting the update amplitude while preserving the gradient direction. In addition, this related technology introduces an asymmetric temperature parameter, imposing different degrees of constraints on positive and negative advantages. However, the soft gating factor in this related technology is context-independent. It only decays according to the Importance Ratio and cannot distinguish the semantic importance of Tokens.

[0025] For Case 1, for functional Tokens such as "de" (的) and "shi" (是), if there is a large deviation in the Importance Ratio, it is usually noise and should be strongly suppressed. For Case 2, for key nodes in the inference logic (such as intermediate results of math problems, key functions of code), if there is a large deviation in the Importance Ratio, it often represents that the model has learned the key logic and should be given a greater update weight (i.e., less attenuation).

[0026] Since this related technology treats the above two cases equally, it results in hurting the learning efficiency of key inference steps while suppressing noise, limiting the convergence speed and the ultimate upper limit of large language models in complex logic tasks.

[0027] That is to say, the above related technology at least includes the following technical problems: (1) Lack of semantic awareness: The related technology applies the same attenuation rule to all Tokens, ignoring the importance differences of Tokens in the chain of thought (e.g., key inference steps vs. stop words).

[0028] (2) Sensitive to noise: For Tokens with an extremely low base probability (the old strategy is very uncertain), a small probability fluctuation will lead to a huge Importance Ratio, and it may wrongly retain these "pseudo-insight" signals caused by randomness.

[0029] (3) Insufficient utilization of key signals: When there is a real logical breakthrough in the large language model, due to a large deviation of the Importance Ratio, the general attenuation mechanism may overly weaken the gradient of this step, resulting in a decrease in learning efficiency.

[0030] Please refer to Figure 2 which shows a flowchart of the model training method provided by an embodiment of the present application. The execution subject of each step of this method can be a computer device. For example, this computer device can be Figure 1The model training device 10 in the computer system shown. The method may include at least one of the following steps (210-240): Step 210: Generate at least one output text based on the input prompt text using a reference policy model. The reference policy model is used to assist in training the current policy model.

[0031] The reference policy model is a stable model relative to the current policy model, used to constrain the update direction of the parameters of the current policy model. It can be understood that the reference policy model can provide benchmark information for training the current policy model, guiding and stabilizing its learning. The current policy model is the optimization object of the model training method provided in this application.

[0032] Both the reference policy model and the current policy model are language models, meaning they have the ability to process natural language. Therefore, the reference policy model can process the input prompt text to generate at least one output text.

[0033] In some embodiments, both the reference policy model and the current policy model are large language models. Optionally, the reference policy model and the current policy model can be the same large language model or different large language models.

[0034] In some embodiments, the source of the reference policy model may include, but is not limited to, at least one of the following: the historical current policy model with frozen parameters, the pre-trained model, the current policy model obtained from the previous training round, etc., and may also include other sources, which are not limited in this application embodiment.

[0035] The historical current policy model refers to a historical version of the current policy model prior to the current training. The historical current policy model with frozen parameters refers to a historical current policy model with fixed parameters. Optionally, before the current training, the parameters of the reference policy model and the current policy model are the same.

[0036] A pre-trained model refers to a model pre-trained using large-scale training data, whose parameters already possess general language modeling capabilities before being trained on a specific task. Optionally, the pre-trained model can be obtained by pre-training the current policy model. Optionally, the pre-trained model can differ from the current policy model.

[0037] The current policy model obtained from the previous training round refers to using the current policy model obtained after the previous training round as a reference policy model to assist the current policy model in the next round of training.

[0038] The input prompt text is used to indicate the target task to be completed by the reference strategy model. In some embodiments, the input prompt text may include, but is not limited to, at least one of the following: questions, instructions, context information, task descriptions, etc., and may also include other content, which is not limited in this application embodiment.

[0039] The output text is a sequence of texts generated by the reference policy model after completing the target task based on the input prompt text.

[0040] In some embodiments, when the number of input texts is equal to 1, a single sampling method is used through the reference policy model to obtain the output text based solely on the input prompt text. Single sampling means obtaining only one output text based on the input prompt text.

[0041] In some embodiments, when the number of output texts is greater than one, an autoregressive sampling method is used through the reference policy model to obtain multiple (two or more) output texts based on the input prompt text. Autoregressive sampling means that the input prompt text is first input into the reference policy model, the reference policy model outputs the first output text, and then the input prompt text and the previously obtained output text are input into the reference policy model together, finally obtaining multiple different output texts.

[0042] For example, the first input prompt text is input into the reference policy model, resulting in output text A. The second input prompt text and output text A are input together into the reference policy model, resulting in output text B. The third input prompt text, output text A, and output text B are input together into the reference policy model, resulting in output text C. At this point, three different output texts are obtained: output text A, output text B, and output text C.

[0043] Step 220: For each output text, obtain the context entropy information of the output text. The context entropy information includes the context entropy of at least one word in the output text. The context entropy of the word is used to indicate the uncertainty of the reference policy model when generating words.

[0044] The contextual entropy information of the output text describes the uncertainty of the reference policy model at various generation positions during the generation of the output text. Uncertainty in generating tokens refers to the degree of hesitation in selecting candidate tokens at the current generation position based on the input prompt text and the currently generated token sequence. A generation position, also known as a generation time step, refers to the sequence position in the autoregressive generation process where the reference policy model predicts and selects the next token based on the already generated token sequence. Each generation position corresponds to one token. In other words, the contextual entropy of a token describes the uncertainty of the reference policy model in evaluating that token based on the current context at a specific time step in generating the output text. A token is the smallest unit that a language model processes and understands natural language. The specific form of a token depends on the word segmentation algorithm of the language model.

[0045] For any given word in the output text, the higher the context entropy of that word, the more uncertain the reference policy model is in generating it. In this case, the reference policy model may have many candidate words when generating that word, making that generation position highly likely to be crucial for solving the aforementioned objective task. In other words, the word is highly critical; generating a word at that position can lead to different inference paths, thus affecting the accuracy of the final output text.

[0046] The smaller the context entropy of a lexical unit, the more certain the reference policy model is when generating that lexical unit. In this case, the lexical unit might be a syntactic connector, a functional lexical unit, or a fixed lexical unit in a templated expression. Therefore, this generation position is unlikely to be crucial for solving the aforementioned objective task. In other words, the lexical unit is of low importance; the lexical unit generated at this position has little impact on subsequent reasoning and the accuracy of the final output text.

[0047] In some embodiments, reference probability distribution information of the output text is obtained, which is used to indicate the generation probability distribution of the reference policy model when generating the output text; based on the reference probability distribution information of the output text, the context entropy information of the output text is determined.

[0048] The reference probability distribution information of the output text reflects the probability preference of the reference policy model for the generation path during the generation of the output text. This reference probability distribution information is generated simultaneously by the reference policy model when generating the output text.

[0049] The output text generation probability distribution describes the probability that the reference policy model will generate different results when generating this output text. In some embodiments, the reference probability distribution information of the output text is generated by the reference policy model.

[0050] Since the reference probability distribution information of the output text reflects the probability preference of the reference policy model for the generation path during the generation of the output text, the uncertainty of the reference policy model at each generation position of the output text can be determined based on the reference probability distribution information of the output text.

[0051] The above method, by obtaining the reference probability distribution information of the output text and further calculating the context entropy, can quantify the uncertainty of the reference policy model when generating each output position. This helps to analyze the sensitivity of the reference policy model to context dependencies, thereby enhancing the interpretability and analyzability of the text generation process.

[0052] In some embodiments, the reference probability distribution information of the output text includes lexical reference probability information of at least one lexical in the output text, wherein the lexical reference probability information of the lexical is used to indicate the generation probability distribution of the reference strategy model for at least one candidate lexical when generating lexical; for each lexical in the output text, the context entropy of the lexical is determined based on the lexical reference probability information of the lexical; and the context entropy of at least one lexical in the output text is determined as the context entropy information of the output text.

[0053] In other words, the reference probability distribution information of the output text is used to indicate the generation probability distribution of the reference policy model at each generation position of the output text.

[0054] In some embodiments, when the reference strategy model is a large language model, for any word in the output text, when the reference strategy model generates the word, the word reference probability information of the word is generated; when the reference strategy model generates the output text, the word reference probability information of at least one word in the output text is determined as the reference probability distribution information of the output text.

[0055] Because large language models generate each word in the output text progressively through autoregression, and for any given generation position, they predict the word at that position based on previously generated words, this process generates the generation probability distribution of at least one candidate word at that position. Based on this distribution, the word at that position is then determined from among these candidate words. In other words, when determining the word at a given generation position, the reference probability information for that word can be obtained.

[0056] The lexical reference probability information includes the generation probability of at least one candidate lexical. The generation probability of a candidate lexical refers to the probability of generating the candidate lexical at the generation position in the context of that generation position.

[0057] In some embodiments, the candidate word set is the same for different generation positions, and the candidate word set for each generation position includes at least one candidate word for that generation position. In this case, the candidate word set for each generation position is the vocabulary of the reference strategy model. That is, when determining the word at each generation position, it is necessary to predict at least one candidate word from the vocabulary of the reference strategy model.

[0058] In some embodiments, the Shannon entropy of a word is determined based on its word reference probability information. The Shannon entropy of a word is used to indicate the degree of uncertainty in generating the word at its corresponding generation position. The context entropy of a word is then determined based on its Shannon entropy. Optionally, the Shannon entropy of a word is determined as its context entropy.

[0059] For example, lexical Context entropy The calculation formula is as follows:

[0060] in, Indicates the first in the output text One generation location (or generation time step). Indicates the first The generation probability of any candidate lexical unit from at least one candidate lexical unit at each generation position. In this case, the lexical unit... Context entropy, i.e., lexical Shannon entropy.

[0061] In other words, lexicon Context entropy By generating the location Calculate the generation probability of at least one candidate word. With log probability Multiply; and calculate the generation probabilities of these candidate lexical units. With log probability Add the products together to get the cumulative probability value; finally, the negative of this cumulative probability value is determined as the word element. Context entropy .

[0062] In some embodiments, a reference generation probability of a word is determined based on its reference probability information. This reference generation probability indicates the likelihood that the reference strategy model will generate the word during the output text generation process. The logarithm of the reciprocal of the reference generation probability is taken to obtain the word's surprise level. Based on this surprise level, the context entropy of the word is determined. Optionally, the logarithm of the word's reference generation probability is used as the context entropy of the word.

[0063] For example, lexical Context entropy The calculation formula is as follows:

[0064] in, Indicates word elements The reference generation probability.

[0065] In other words, the surprise factor is directly measured using the logarithm of the inverse of the reference generation probability of a word element, and this logarithm is used to determine the context entropy of that word element. Surprise factor represents the degree of unexpectedness when an event occurs, and is also used to represent the uncertainty when the reference policy model generates that word element.

[0066] The above method summarizes the contextual entropy at the lexical level into the contextual entropy information of the output text, which helps to understand the decision-making logic and generation tendency of the reference strategy model in different contexts. It can intuitively reflect the degree of contribution of different lexical units to the generation process of the output text, thereby enhancing the analyzability of the output text.

[0067] Step 230: Based on the context entropy information of the output text, determine the update weight information of the output text. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model.

[0068] Since the update weight information of each output text includes the update weight of each word in the output text, and the update weight of each word indicates the degree of influence on the parameters of the current policy model, it means that each word in the output text will affect the parameter update of the current policy model.

[0069] It is understandable that a larger update weight for a word element indicates a greater impact of that word element on the parameter updates of the current policy model; conversely, a smaller update weight for a word element indicates a smaller impact. In other words, this method allows for fine-tuning of the parameters of the current policy model at the word element level.

[0070] In some embodiments, please refer to Figure 3 The flowchart of determining updated weight information provided in one embodiment of this application is shown. Step 230 further includes the following sub-steps (231-234).

[0071] Sub-step 231: For each word in the output text, determine the correction coefficient of the word based on the context entropy of the word. The correction coefficient of the word is used to indicate the criticality of the word when the reference strategy model generates the output text.

[0072] By leveraging the uncertainty (context entropy) exhibited by the reference policy model when generating each lexical unit, a numerical correction coefficient is assigned to that lexical unit to measure its criticality in the generation process. This correction coefficient subsequently influences the updated weights corresponding to that lexical unit during training.

[0073] When a significant deviation occurs between the new strategy (current strategy model) and the old strategy (reference strategy model), to prevent gradient explosion during training and subsequent instability, the gradient update gradient for the current strategy model needs to be reduced. However, not all large deviations in generation are detrimental to the training of the current strategy model. When the reference strategy model hesitates significantly when generating a certain word, this generation position is a critical turning point in solving the target task. In this case, the generated word is highly critical, and the deviation of the current strategy model is a valid learning signal, which should be protected. When the context entropy of a word is high, it indicates that the reference strategy model hesitated when generating that word, thus the generation position corresponding to that word is a relatively critical position in solving the target task. That is, in this embodiment, the higher the context entropy of a word, the more critical its generation position; conversely, the lower the context entropy of a word, the less critical its generation position.

[0074] Since the context entropy of a lexical unit represents its uncertainty, a correction coefficient is assigned to the lexical unit based on its context entropy. This increases the positive impact of the lexical unit on the parameter updates of the current policy model at lexical units with higher context entropy. In other words, the context entropy of a lexical unit is directly proportional to its correction coefficient. The higher the context entropy of a lexical unit, the higher its correction coefficient; conversely, the lower the context entropy, the lower its correction coefficient. Optionally, the correction coefficient is an integer greater than or equal to 1.

[0075] In some embodiments, a pre-defined maximum and minimum context entropy are obtained; and a correction coefficient for the lexical is determined based on the maximum and minimum context entropy and the context entropy of the lexical.

[0076] The maximum and minimum context entropy are used to normalize the context entropy of each word, that is, to normalize the context entropy of the word to a certain numerical range.

[0077] Optionally, the maximum context entropy is the theoretical upper limit of context entropy, and the minimum context entropy is the theoretical lower limit of context entropy. The theoretical upper limit of context entropy refers to the theoretically largest context entropy of a lexical unit, representing a highly uncertain case. Optionally, the theoretical upper limit of context entropy is determined based on the vocabulary size of the reference strategy model. For example, the logarithm of the vocabulary size of the reference strategy model can be used to determine the maximum context entropy. Optionally, the theoretical upper limit of context entropy is determined based on the maximum context length that the reference strategy model can handle. For example, the logarithm of the maximum context length that the reference strategy model can handle can be used to determine the maximum context entropy. The theoretical lower limit of context entropy refers to the theoretically smallest context entropy of a lexical unit, representing a highly certain case. Optionally, the minimum context entropy is 0.

[0078] Optionally, the maximum and minimum context entropy can be set by relevant technical personnel based on experimental experience.

[0079] In some embodiments, a gain strength parameter is obtained, which is used to adjust the influence of the context entropy of the lexical on the correction coefficient; the correction coefficient of the lexical is determined based on the gain strength parameter, the maximum context entropy, the minimum context entropy, and the context entropy of the lexical.

[0080] The gain strength parameter controls the degree to which high-entropy terms (terms with higher contextual entropy) amplify the impact of low-entropy terms (terms with lower contextual entropy) on the parameter updates of the current policy model. The gain strength coefficient can enhance the learning effect at key generation positions while ensuring training stability. For example, the gain strength parameter can be 0.5. It should be noted that the gain strength parameter can also be set to other values, which will not be elaborated upon here.

[0081] For example, lexical Correction coefficient The calculation formula is as follows:

[0082] in, Indicates the first in the output text One generation location (or generation time step). as a word element Context entropy, To minimize context entropy, For the maximum context entropy, This is the gain intensity parameter. Optionally, .

[0083] Based on the previous example, if the word element Context entropy The larger the value, the more significant the word element. Correction coefficient The larger; if the word element Context entropy The smaller the value, the more significant the word element. Correction coefficient The closer it is to 1.

[0084] The above approach, because the context entropy of a lexical unit can measure the degree of hesitation when generating that lexical unit, processes the context entropy of each lexical unit based on the maximum and minimum context entropy to dynamically adjust the correction coefficient of each lexical unit. This automatically identifies and protects the learning signal brought by lexical units generated at key generation positions, enhancing the learning ability and efficiency of the current policy model at these key generation positions. By protecting the learning signal at key generation positions, the current policy model can extract more effective information from the learning signal at these key generation positions, reducing the number of samplings required for the current policy model to learn a particular piece of knowledge.

[0085] Sub-step 232: Obtain the soft-gating weights of the lexical units. The soft-gating weights of the lexical units are used to control the adjustment range of the parameters of the current strategy model.

[0086] The soft-gating weights of lexical units are used to smoothly adjust the magnitude of the lexical unit's influence on the parameter updates of the current policy model. Optionally, the soft-gating weights of lexical units are continuously adjustable weight factors.

[0087] In some embodiments, a reference generation probability of a word is obtained, which indicates the likelihood of a reference policy model generating a word during the generation of output text; a current generation probability of a word is obtained, which indicates the likelihood of a current policy model generating a word during the generation of output text; based on the reference generation probability and the current generation probability of the word, a word importance ratio is calculated, which indicates the degree of deviation between the reference policy model and the current policy model when generating words; and based on the word importance ratio, the soft gating weight of the word is determined.

[0088] The reference generation probability of a lexical can be obtained from the lexical reference probability information described above. Since the lexical reference probability information includes the generation probability of at least one candidate lexical when generating that lexical, and that lexical is selected from at least one candidate lexical, the reference generation probability of that lexical can be determined from it.

[0089] In some embodiments, the current generation probability of the word is predicted using the current policy model. Optionally, the current generation probability of the word is predicted using the current policy model based on the input text corresponding to the output text and the preceding word of the word. The input text corresponding to the output text refers to the input information that is input to the reference policy model to generate the output text. Optionally, the input text corresponding to the output text includes input prompt text and at least one preceding output text, which is the output text generated before the current output text is generated. Since the output text is not generated by the current policy model, the current policy model needs to re-predict the generation probability (i.e., the current generation probability) of each generation position (word) of the output text.

[0090] The lexical importance ratio is used to measure the degree of deviation of the new strategy (current strategy model) from the old strategy (reference strategy model) in terms of preference for the same lexical.

[0091] In some embodiments, the ratio of the current generation probability of a word element to the reference generation probability of the word element is determined as the importance ratio of that word element. In this case, when the importance ratio of a word element is equal to 1, the predictions of the current policy model and the reference policy model at the generation position corresponding to that word element are consistent; when the importance ratio of a word element is greater than 1, the current policy model's preference for that word element at the generation position is greater than the reference policy model's preference for that word element at the generation position; when the importance ratio of a word element is less than 1, the current policy model's preference for that word element at the generation position is less than the reference policy model's preference for that word element at the generation position.

[0092] For example, lexical Importance ratio The calculation formula is as follows:

[0093] in, Indicates the first in the output text One generation location (or generation time step). Indicates word elements The current generation probability, Indicates word elements Reference generation probability. Lexicon Current generation probability , For the current strategy model, This indicates the currently generated output text. Indicates the input prompt text The next generated Each word element. Current generation probability , For reference strategy model, This indicates the currently generated output text. Indicates the input prompt text The next generated Each word element.

[0094] In some embodiments, a temperature hyperparameter is obtained, which is used to control the steepness of the decay of word update weights; based on the temperature hyperparameter and the word importance ratio, the soft-gated weights of the words are determined. Optionally, the temperature hyperparameter is adjustable.

[0095] In some embodiments, an sigmoid decay function is employed to determine the soft-gated weights of terms based on the temperature hyperparameter and the term importance ratio. The sigmoid decay function is a continuous, smooth mapping function that maps the input deviation to weight values, with the output value changing smoothly within a specified range, its curve exhibiting an "S" shape. The steepness of the decay can be controlled by the temperature hyperparameter, ensuring that when the importance ratio is close to 1 (i.e., the difference between the new and old strategies is small), the soft-gated weights are close to 1, thus preserving most of the gradient signal; when the importance ratio deviates significantly from 1, the soft-gated weights gradually decrease, but do not directly return to zero like hard truncation. This effectively solves the gradient vanishing problem caused by hard truncation in related techniques, ensuring the training signal is continuous and smooth, and preserving sufficient update amplitude at key generation positions.

[0096] For example, lexical soft-gating weights The calculation formula is as follows:

[0097] in, Indicates the first in the output text One generation location (or generation time step). Indicates word elements Importance ratio This represents the tolerance interval hyperparameter. This indicates temperature hyperparameters. It is an sigmoid function. For example, It can be set to 0.2 to indicate that it is allowed. exist It can change freely within its range.

[0098] when hour, (Preserve gradient); when hour, It varies freely within a smooth range. It gradually decreases rather than being directly truncated to 0.

[0099] The above method, by calculating the importance ratio of lexical units, can accurately quantify the difference between the current policy model and the reference policy model in generating each lexical unit. Then, based on the importance ratio of the lexical unit, the soft-gating weight of that lexical unit is determined. This allows for smooth adjustment of the parameter updates of the current policy model based on the importance ratio, thus maintaining the stability of the current policy model's training. It should be noted that the execution order of sub-steps 231 and 232 is not limited; sub-step 231 can be executed first, followed by sub-step 232, or vice versa, or they can be executed in parallel.

[0100] Sub-step 233: Determine the update weight of the word based on the correction coefficient and soft gating weight of the word.

[0101] Lexical correction coefficients are used to enhance the training signals at key generation positions in the output text, while lexical soft-gating weights are used to maintain the stability of the current policy model's training. By combining lexical correction coefficients and lexical soft-gating weights, the update weights of lexical units are determined, thus maintaining a balance between the stability of the current policy model's learning and training at key generation positions.

[0102] In some embodiments, the product of the word's correction coefficient and the word's soft-gating weight is determined as the word's update weight.

[0103] For example, lexical Update weights The calculation formula is as follows:

[0104] in, Indicates the first in the output text One generation location (or generation time step). Indicates word elements Soft gating weights, Indicates word elements The correction factor.

[0105] When a word has a high contextual entropy and a high importance ratio, its soft-gating weight is relatively small (meaning it will significantly reduce the gradient update of the current policy model). In this case, combining the word's correction coefficient can partially offset the reduction caused by the high importance ratio, thus protecting the learning signal at key generation positions.

[0106] In some embodiments, substeps 232 and 233 can be replaced by substep 235 as follows (in Figure 3(Not shown in the text): Based on the context entropy of the word, determine the soft-gating weight of the word; based on the soft-gating weight of the word and / or the correction coefficient of the word, determine the update weight of the word.

[0107] In other words, in this approach, the context entropy of a lexical unit is used to adjust its soft-gating weights. Optionally, the temperature hyperparameter is adjusted based on the lexical unit's context entropy to obtain the adjusted temperature hyperparameter; the soft-gating weights of the lexical unit are then determined based on the temperature hyperparameter. In other words, the temperature hyperparameter is dynamically adjusted based on the lexical unit's context entropy.

[0108] It should be noted that after adjusting the temperature hyperparameter, the method for calculating the soft-gating weights of the tokens is the same as described above, so it will not be repeated here.

[0109] In some embodiments, a preset adjustment function is used to adjust the temperature hyperparameter based on the context entropy of the lexical, resulting in the adjusted temperature hyperparameter. The preset adjustment function is used to control the temperature hyperparameter to be proportional to the context entropy of the lexical. That is, the larger the context entropy of the lexical, the larger the temperature hyperparameter, and the flatter the curve of the sigmoid function.

[0110] Optionally, the soft gating weight of a word element can be determined as the update weight of that word element.

[0111] Optionally, the update weight of a word can be determined based on its soft-gating weight and correction coefficient.

[0112] The above method, by mapping context entropy to soft-gated weights, can dynamically increase or decrease the adjustment intensity based on the context entropy of the lexical, thereby controlling the degree of influence of the lexical on the parameter update of the current policy model.

[0113] For simple lexical units that are relatively certain in the current policy model, if the generation probability of the lexical unit in the current policy model differs significantly from that in the reference policy model (a sudden change may occur, possibly due to noise), the update weight assigned to the lexical unit is also small because the context entropy of the lexical unit is small. This strictly controls the decay of the current policy model and prevents the capability of the current policy model from degrading.

[0114] Sub-step 234: The update weight of at least one word in the output text is determined as the update weight information of the output text.

[0115] In other words, the update weights of at least one token in the output text are aggregated into the update weight information of the output text.

[0116] The above method, by combining the correction coefficient of the word and the soft gating weight of the word, determines the update weight of the word, so that the stability of the current policy model in learning and training key generation positions is maintained.

[0117] Step 240: Adjust the parameters of the current policy model based on the updated weight information of each output text.

[0118] In other words, the current policy model is trained based on the updated weight information of each output text to obtain the adjusted current policy model.

[0119] In some embodiments, if the adjusted current policy model meets the training stop condition, the adjusted current policy model is determined as the trained current policy model. The training stop condition refers to the condition for stopping the training of the current policy model. In some embodiments, the training stop condition may include at least one of the following: the training epochs of the current policy model reach a first preset epoch, or the loss function value of the current policy model is less than or equal to a preset threshold. The first preset epoch and the preset threshold are preset by those skilled in the art.

[0120] In some embodiments, when the reference policy model meets the update conditions, the parameters of the reference policy model are updated based on the adjusted parameters of the current policy model to obtain the updated reference policy model. The update conditions refer to the conditions for updating the reference policy model. Optionally, the update conditions include the current policy model reaching a second preset number of training epochs. The second preset number of epochs is pre-set by those skilled in the art and is less than the first preset number of epochs.

[0121] In some embodiments, the loss function value of the current policy model is determined based on the updated weight information of each output text; and the parameters of the current policy model are adjusted based on the loss function value.

[0122] The loss function value is used to guide the update direction and magnitude of the current policy model.

[0123] In some embodiments, the loss value of each output text is obtained, and the loss value of the output text includes the word loss value of at least one word in the output text. The word loss value of the word is used to indicate the degree of difference between the generation probability of the current policy model and the reference policy model for that word. Based on the loss value of each output text and the update weight information of each output text, the loss function value of the current policy model is determined.

[0124] Optionally, based on the updated weight information of each output text, the loss values ​​of each output text are weighted and averaged to obtain the loss function value.

[0125] In some embodiments, the parameters of the current policy model are adjusted using gradient descent based on the loss function value.

[0126] The above method constructs a weighted loss function based on the updated weight information of each output text, and adjusts the parameters of the current policy model based on the value of the loss function, thereby achieving targeted adjustment of the parameters of the current policy model and improving the learning efficiency of the current policy model.

[0127] In some embodiments, a reward score is obtained for each output text, which is used to indicate the degree of matching between the output text and the target task indicated by the input prompt text; the loss function value of the current policy model is determined based on the updated weight information of each output text, the reward score of each output text, and the importance information of each output text; wherein, the importance information of the output text includes the importance ratio of at least one word in the output text, and the importance ratio of the word is used to indicate the degree of deviation between the reference policy model and the current policy model when generating words.

[0128] In some embodiments, the advantage value of each output text is determined based on the reward score of each output text, and the advantage value of the output text is used to evaluate the quality of the output text; the loss function value of the current policy model is determined based on the updated weight information of each output text, the advantage value of each output text, and the importance information of each output text.

[0129] In some embodiments, a reward mean is determined based on the reward score of each output text, where the reward mean is the average of the reward scores of each output text; a reward standard deviation is determined based on the reward score of each output text, where the reward standard deviation is the standard deviation of the reward scores of each output text; and for each output text, an advantage value is determined based on the reward mean, the reward standard deviation, and the reward score of that output text.

[0130] For example, output text Advantages The calculation formula is as follows:

[0131] in, Indicates output text The reward points, This represents the average reward. Indicates the standard deviation of the reward. It is a very small positive number, so we avoid having a denominator of zero.

[0132] In some embodiments, for each word in each output text, the word loss value is determined based on the word advantage value and the word importance ratio of the word; and the loss values ​​of each output text are weighted and averaged based on the updated weight information of each output text to obtain the loss function value.

[0133] In some embodiments, based on the advantage value of the output text, the lexical advantage value of each word in the output text is determined, and the lexical advantage value of a word is used to indicate the degree of superiority or inferiority of that word. Optionally, the lexical advantage value of each word in the output text is the advantage value of the output text. Optionally, the lexical advantage value of each word in the output text is the average of the advantage values ​​of the output text. Dividing the advantage value of the output text by the number of words in the output text yields the average advantage value of the output text, which is the lexical advantage value of each word in the output text.

[0134] In some embodiments, a reward model or a rule-based judge scores each output text to obtain a reward score for each output text. The reward model is an evaluation model built on machine learning or deep learning, used to predict and evaluate the overall quality of the output text given the input prompt text. The rule-based judge is an automated evaluation module built on pre-defined decision rules, constraints, or logical expressions, used to determine or score whether the output text meets the explicit constraints of the target task.

[0135] For example, the loss function value The calculation formula is as follows:

[0136] Where j represents the first... Each word element, Indicates the first The importance ratio of each word element Indicates the first The update weight of each word element, Indicates the first The lexical advantage value of each lexical unit. This indicates the total number of words in at least one output text.

[0137] The above method constructs a weighted loss function based on the updated weight information of each output text, the reward score of each output text, and the importance information of each output text, and adjusts the parameters of the current policy model based on the value of the loss function, thereby achieving targeted adjustment of the parameters of the current policy model and improving the learning efficiency of the current policy model.

[0138] Furthermore, the loss function value is determined by combining the update weights and importance information of the output text. The importance information of the output text reflects the degree of deviation of the current policy model from the reference policy model when generating the output text, while the update weights reflect the update value based on each word in each output text. These two dimensions jointly constrain the parameter updates of the current policy model, rather than applying a one-size-fits-all approach to generation positions with high deviations. This makes the training process of the current policy model more in line with human learning intuition—focusing on learning under uncertainty while maintaining stability on deterministic knowledge. This approach helps maintain the stability of the current policy model's training and improves its learning efficiency at key generation positions. It enables the current policy model to learn to handle complex logical reasoning while maintaining the cognition of deterministic knowledge.

[0139] In summary, the technical solution provided in this application, after generating at least one output text through a reference policy model, obtains the context entropy of each word in these output texts to assess the uncertainty of the reference policy model in generating each word. Then, based on the context entropy of each word, it determines the update weight of each word, thereby achieving word-level fine-tuning of the parameters of the current policy model. By considering the uncertainty of each word during generation and assigning update weights to it, the influence of each word on the parameter update of the current policy model is determined, allowing for targeted training of the large language model and thus improving the learning efficiency of the current policy model.

[0140] The following describes another way to determine the contextual entropy information of the output text.

[0141] Next, we will introduce how to determine the contextual entropy information of the output text based on attention information. That is, the reference policy model must at least have an attention mechanism. In this embodiment, the reference policy model is a large language model. It should be noted that the reference policy model can also be other models with attention mechanisms.

[0142] In some embodiments, attention information of the output text is obtained, which is used to indicate the degree of attention of the reference policy model to at least one word in the output text; based on the attention information of the output text, context entropy information of the output text is determined.

[0143] The attention information in the output text reflects the degree of attention the reference policy model pays to a word at a given generation location within the context of that word. This attention information is generated simultaneously with the reference policy model when it generates the output text.

[0144] The degree of attention between lexical units refers to the degree of dependence on other lexical units in the output text when generating a particular lexical unit.

[0145] The attention information of the output text reflects the degree to which the reference policy model depends on each position in the context when generating the output text. The concentration or dispersion of attention distribution can quantify the uncertainty of the generated output text, and therefore can be used to calculate the context entropy information of the output text and to identify key positions for solving the target task.

[0146] The above method obtains the attention information of the output text and further calculates the context entropy. Since the attention information of the output text can reflect the contextual dependencies between words in the output text, the uncertainty of the reference policy model in generating each output position can be quantified based on the attention information of the output text. This helps to analyze the sensitivity of the reference policy model to contextual dependencies, thereby enhancing the interpretability and analyzability of the text generation process.

[0147] In some embodiments, the attention information of the output text includes attention weight information of at least one word in the output text. The attention weight information of the word includes the attention weight of at least one preceding word, which is a word generated before the generated word. The attention weight of the preceding word is used to indicate the importance of the preceding word to the generated word. For each word in the output text, the context entropy of the word is determined based on the attention weight information of the word. The context entropy of at least one word in the output text is determined as the context entropy information of the output text.

[0148] The attention weight information for each word element reflects the degree of attention paid to each preceding word element. In other words, the attention weight of a preceding word element reflects its contribution to the generation of the current word element. A larger attention weight for a preceding word element indicates that it is more important (i.e., contributes more) to the generation of the current word element; a smaller attention weight indicates that it is less important (i.e., contributes less) to the generation of the current word element.

[0149] In some embodiments, for any word in the output text, when the reference policy model generates the word, attention weight information of the word is generated; when the reference policy model generates the output text, the attention weight information of at least one word in the output text is determined as the attention information of the output text.

[0150] Because large language models generate each word in the output text progressively through autoregression, and for any given generation position, predict the word at that position based on previously generated words, attention weight information is generated for that position. Based on this attention weight information, the word at that position is determined from at least one candidate word. In other words, when determining the word at a given generation position, the attention weight information for that position can be obtained.

[0151] In some embodiments, for each word in the output text, the attention weights of at least one preceding word are summed to determine the sum of attention weights for that word; and the context entropy of that word is determined based on the sum of attention weights. Optionally, the sum of attention weights for that word is determined as the context entropy of that word.

[0152] For example, lexical Context entropy The calculation formula is as follows:

[0153] in, Indicates the first One generation location (or generation time step). Indicates the first The attention weights of each generation position with respect to the i-th preceding word. It is a positive integer.

[0154] In other words, lexicon Context entropy Is this the location of generation? The value is obtained by summing the attention weights of at least one preceding word.

[0155] In the above method, the attention weight information of a word reflects the degree of dependence of that word on its preceding word during generation. Based on these attention weights, the context entropy of each word can be calculated, thereby obtaining the uncertainty in accurately generating that word.

[0156] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0157] Please refer to Figure 4 This diagram illustrates a block diagram of a model training apparatus according to an embodiment of this application. The apparatus has the functions described above, which can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the model training device 10 described above, or it can be disposed within the model training device 10. Figure 4As shown, the device 400 may include a generation module 410, an acquisition module 420, a determination module 430, and an adjustment module 440.

[0158] The generation module 410 is used to generate at least one output text based on the input prompt text using a reference policy model, wherein the reference policy model is used to assist in training the current policy model.

[0159] The acquisition module 420 is configured to acquire context entropy information of each output text, the context entropy information including the context entropy of at least one word in the output text, the context entropy of the word being used to indicate the uncertainty of the reference policy model when generating the word.

[0160] The determination module 430 is used to determine the update weight information of the output text based on the context entropy information of the output text. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model.

[0161] The adjustment module 440 is used to adjust the parameters of the current strategy model based on the updated weight information of each of the output texts.

[0162] In some embodiments, the acquisition module 420 is used to acquire reference probability distribution information of the output text, wherein the reference probability distribution information of the output text is used to indicate the generation probability distribution of the reference policy model when generating the output text; and to determine the context entropy information of the output text based on the reference probability distribution information of the output text.

[0163] In some embodiments, the reference probability distribution information of the output text includes lexical reference probability information of at least one lexical in the output text, wherein the lexical reference probability information of the lexical is used to indicate the generation probability distribution of the reference strategy model for at least one candidate lexical when generating the lexical; the acquisition module 420 is further configured to determine the context entropy of each lexical in the output text based on the lexical reference probability information of the lexical; and to determine the context entropy of at least one lexical in the output text as the context entropy information of the output text.

[0164] In some embodiments, the acquisition module 420 is configured to acquire attention information of the output text, wherein the attention information of the output text is used to indicate the degree of attention paid by the reference policy model to at least one word in the output text; and to determine the context entropy information of the output text based on the attention information of the output text.

[0165] In some embodiments, the attention information of the output text includes attention weight information of at least one word in the output text, the attention weight information of the word includes the attention weight of at least one preceding word of the word, the preceding word being a word generated before the word is generated, and the attention weight of the preceding word is used to indicate the importance of the preceding word to the generation of the word; the acquisition module 420 is further configured to, for each word in the output text, determine the context entropy of the word based on the attention weight information of the word; and determine the context entropy of at least one word in the output text as the context entropy information of the output text.

[0166] In some embodiments, the determining module 430 is configured to: for each word in the output text, determine a correction coefficient for the word based on the context entropy of the word, wherein the correction coefficient is used to indicate the criticality of the word in the reference strategy model when generating the output text; obtain a soft-gating weight for the word, wherein the soft-gating weight is used to control the adjustment range of the parameters of the current strategy model; determine an update weight for the word based on the correction coefficient and the soft-gating weight of the word; and determine the update weight of at least one word in the output text as the update weight information of the output text.

[0167] In some embodiments, the determining module 430 is further configured to obtain a pre-set maximum context entropy and a minimum context entropy; and to determine the correction coefficient of the lexical based on the maximum context entropy, the minimum context entropy, and the context entropy of the lexical.

[0168] In some embodiments, the determining module 430 is further configured to: obtain a reference generation probability of the word element, the reference generation probability of the word element being used to indicate the probability that the reference strategy model will generate the word element during the generation of the output text; obtain a current generation probability of the word element, the current generation probability of the word element being used to indicate the probability that the current strategy model will generate the word element during the generation of the output text; calculate an importance ratio of the word element based on the reference generation probability and the current generation probability of the word element, the importance ratio of the word element being used to indicate the degree of deviation between the reference strategy model and the current strategy model when generating the word element; and determine a soft gating weight of the word element based on the importance ratio of the word element.

[0169] In some embodiments, the determining module 430 is further configured to determine the updated weight of the word as the product of the correction coefficient of the word and the soft gating weight of the word.

[0170] In some embodiments, the determining module 430 is further configured to determine the soft-gating weight of the lexical unit based on the correction coefficient of the lexical unit; and to determine the update weight of the lexical unit based on the soft-gating weight of the lexical unit.

[0171] In some embodiments, the adjustment module 440 is used to determine the loss function value of the current policy model based on the updated weight information of each of the output texts; and to adjust the parameters of the current policy model based on the loss function value.

[0172] In some embodiments, the adjustment module 440 is further configured to obtain the reward score of each of the output texts, the reward score of the output texts being used to indicate the degree of matching between the output texts and the target task indicated by the input prompt texts; and to determine the loss function value of the current policy model based on the updated weight information of each of the output texts, the reward score of each of the output texts, and the importance information of each of the output texts; wherein the importance information of the output texts includes the importance ratio of at least one word in the output texts, the importance ratio of the word being used to indicate the degree of deviation between the reference policy model and the current policy model when generating the word.

[0173] In summary, the technical solution provided in this application, after generating at least one output text through a reference policy model, obtains the context entropy of each word in these output texts to assess the uncertainty of the reference policy model in generating each word. Then, based on the context entropy of each word, it determines the update weight of each word, thereby achieving word-level fine-tuning of the parameters of the current policy model. By considering the uncertainty of each word during generation and assigning update weights to it, the influence of each word on the parameter update of the current policy model is determined, allowing for targeted training of the large language model and thus improving the learning efficiency of the current policy model.

[0174] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0175] Please refer to Figure 5 This diagram illustrates a structural block diagram of a computer device 500 provided in one embodiment of this application. The computer device 500 may be... Figure 1The model training device 10 in the computer system shown is used to implement the model training method provided in the above embodiments. Specifically: Typically, computer device 500 includes a processor 510 and a memory 520.

[0176] Processor 510 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 510 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 510 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 510 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 510 may also include an AI processor for handling computational operations related to machine learning.

[0177] The memory 520 may include one or more computer-readable storage media, which may be non-transitory. The memory 520 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 520 are used to store a computer program configured to be executed by one or more processors to implement the model training method described above.

[0178] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the computer device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0179] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the above-described model training method. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0180] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the model training method described above.

[0181] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0182] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: Based on the input prompt text, at least one output text is generated using a reference policy model, which is used to assist in training the current policy model. For each output text, obtain the context entropy information of the output text, the context entropy information including the context entropy of at least one word in the output text, the context entropy of the word being used to indicate the uncertainty of the reference policy model when generating the word; Based on the context entropy information of the output text, the update weight information of the output text is determined. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model. Based on the updated weight information of each of the output texts, the parameters of the current policy model are adjusted.

2. The method according to claim 1, characterized in that, The step of obtaining the context entropy information of the output text includes: Obtain reference probability distribution information of the output text, which is used to indicate the generation probability distribution of the reference policy model when generating the output text; Based on the reference probability distribution information of the output text, the context entropy information of the output text is determined.

3. The method according to claim 2, characterized in that, The reference probability distribution information of the output text includes the lexical reference probability information of at least one word in the output text. The lexical reference probability information of the word is used to indicate the generation probability distribution of the reference strategy model for at least one candidate word when generating the word. Determining the context entropy information of the output text based on the reference probability distribution information of the output text includes: For each word in the output text, the context entropy of the word is determined based on the word reference probability information of the word. The context entropy of at least one word in the output text is determined as the context entropy information of the output text.

4. The method according to claim 1, characterized in that, The step of obtaining the context entropy information of the output text includes: Obtain attention information of the output text, which is used to indicate the degree of attention the reference policy model pays to at least one word in the output text; Based on the attention information of the output text, the context entropy information of the output text is determined.

5. The method according to claim 4, characterized in that, The attention information of the output text includes attention weight information of at least one word in the output text. The attention weight information of the word includes the attention weight of at least one preceding word of the word. The preceding word is a word generated before the word is generated. The attention weight of the preceding word is used to indicate the importance of the preceding word in the generation of the word. Determining the context entropy information of the output text based on the attention information of the output text includes: For each word in the output text, the context entropy of the word is determined based on the attention weight information of the word. The context entropy of at least one word in the output text is determined as the context entropy information of the output text.

6. The method according to claim 1, characterized in that, Determining the update weight information of the output text based on the context entropy information of the output text includes: For each word in the output text, a correction coefficient is determined based on the context entropy of the word. The correction coefficient of the word is used to indicate the criticality of the word when the reference strategy model generates the output text. Obtain the soft-gating weights of the lexical units, which are used to control the adjustment range of the parameters of the current policy model; The update weight of the word is determined based on the correction coefficient of the word and the soft gating weight of the word. The update weight of at least one word in the output text is determined as the update weight information of the output text.

7. The method according to claim 6, characterized in that, Determining the correction coefficient of the lexical based on its context entropy includes: Get the pre-defined maximum and minimum context entropy; The correction coefficient of the lexical is determined based on the maximum context entropy, the minimum context entropy, and the context entropy of the lexical.

8. The method according to claim 6, characterized in that, The process of obtaining the soft-gated weights of the lexical units includes: Obtain the reference generation probability of the lexical unit, which is used to indicate the probability that the reference strategy model will generate the lexical unit during the process of generating the output text; Obtain the current generation probability of the word element, which is used to indicate the probability that the current policy model will generate the word element in the process of generating the output text; Based on the reference generation probability and the current generation probability of the word, the importance ratio of the word is calculated. The importance ratio of the word is used to indicate the degree of deviation between the reference strategy model and the current strategy model when generating the word. The soft gating weight of the lexical is determined based on the importance ratio of the lexical.

9. The method according to claim 6, characterized in that, The step of determining the update weight of a word based on its correction coefficient and its soft-gating weight includes: The product of the correction coefficient of the lexical element and the soft gating weight of the lexical element is determined as the update weight of the lexical element.

10. The method according to any one of claims 1 to 9, characterized in that, The step of adjusting the parameters of the current policy model based on the updated weight information of each of the output texts includes: Based on the updated weight information of each of the output texts, the loss function value of the current policy model is determined; Based on the loss function value, adjust the parameters of the current policy model.

11. The method according to claim 10, characterized in that, The step of determining the loss function value of the current policy model based on the updated weight information of each of the output texts includes: Obtain a reward score for each of the output texts, wherein the reward score of the output text is used to indicate the degree of matching between the output text and the target task indicated by the input prompt text; Based on the updated weight information of each output text, the reward score of each output text, and the importance information of each output text, the loss function value of the current policy model is determined; wherein, the importance information of the output text includes the importance ratio of at least one word in the output text, and the importance ratio of the word is used to indicate the degree of deviation between the reference policy model and the current policy model when generating the word.

12. A model training device, characterized in that, The device includes: The generation module is used to generate at least one output text based on the input prompt text using a reference policy model, wherein the reference policy model is used to assist in training the current policy model. The acquisition module is configured to acquire context entropy information of each output text, wherein the context entropy information includes the context entropy of at least one word in the output text, and the context entropy of the word is used to indicate the uncertainty of the reference policy model when generating the word; The determination module is used to determine the update weight information of the output text based on the context entropy information of the output text. The update weight information of the output text includes the update weight of at least one word in the output text. The update weight of the word is used to indicate the degree of influence of the word on the parameters of the current policy model. The adjustment module is used to adjust the parameters of the current strategy model based on the updated weight information of each of the output texts.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 11.

Images