Training methods, devices, electronic equipment, media, and products for inference models

Abstract

This application discloses a training method, apparatus, electronic device, medium, and product for a reasoning model, relating to the field of artificial intelligence technology. The method includes obtaining the entropy value of each word, and based on the entropy value of each word, obtaining an entropy average sequence of the reasoning process. The entropy value and entropy average sequence are used to identify multiple reasoning stages in the reasoning process, with different reasoning lengths for different stages. Based on the entropy average sequence, a reasoning length reward value is obtained, fully considering the stage characteristics of the reasoning process and ensuring its accuracy. By ensuring that the reasoning answer is consistent with the standard reasoning answer, and then obtaining the reasoning length reward value based on the entropy average sequence of the reasoning process, the reasoning length can be limited based on the accuracy of the reasoning answer, avoiding situations where the reasoning process is short but the reasoning answer is inaccurate, thus ensuring the accuracy of the reasoning answer. Therefore, the accuracy of the reasoning result can be ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus, electronic devices, media and products for training inference models. Background Technology

[0002] Chain of Thought (CoT) reasoning mechanisms improve the processing capability of complex tasks by breaking down complex problems into a series of intermediate reasoning steps. Building on this, Long Chain-of-Thought (LCoT) models, compared to CoT, can handle even more complex reasoning problems with more steps, typically characterized by the potential use of multi-round self-verification mechanisms. However, multi-round self-verification mechanisms can lead to overthinking, gradually becoming a key bottleneck affecting the model's efficiency and accuracy.

[0003] Currently, related technologies use the final length of the reasoning process as a reward to avoid excessively long inference processes, thus addressing the problem of excessively long inference processes in inference models based on reinforcement learning. However, these technologies do not consider the stage characteristics of the inference process and cannot guarantee the accuracy of the inference results. Summary of the Invention

[0004] This application provides training methods, apparatus, electronic devices, media, and products for inference models, in order to at least solve the problem that the accuracy of inference results cannot be guaranteed in related technologies.

[0005] This application provides a training method for a reasoning model, comprising: collecting multiple historical reasoning results for any reasoning problem; wherein each historical reasoning result includes a reasoning process and a reasoning answer; the reasoning process includes multiple lexical units; obtaining the entropy value of each lexical unit; obtaining the entropy average sequence of the reasoning process based on the entropy value of each lexical unit; wherein the entropy value and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process; determining the accuracy reward value of each historical reasoning answer based on the reasoning answer; if the reasoning answer is consistent with the standard reasoning answer for any reasoning problem, obtaining the reasoning length reward value of the reasoning process based on the entropy average sequence of the reasoning process; training the reasoning model to be trained based on the accuracy reward value and the reasoning length reward value to obtain a trained reasoning model; wherein the trained reasoning model is used to output the reasoning process and the reasoning answer based on the problem to be reasoned, and when any lexical unit is generated during the reasoning process, determining whether to terminate the reasoning process based on the entropy value and the entropy average sequence of the current lexical unit to avoid over-reasoning stages during the reasoning process.

[0006] This application also provides a training device for a reasoning model, comprising: a collection module for collecting multiple historical reasoning results for any reasoning problem; wherein each historical reasoning result includes a reasoning process and a reasoning answer; the reasoning process includes multiple lexical units; a first acquisition module for acquiring the entropy value of each lexical unit; a second acquisition module for obtaining the entropy average sequence of the reasoning process based on the entropy value of each lexical unit; wherein the entropy value and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process; a determination module for determining the accuracy reward value of each historical reasoning answer based on the reasoning answer; a third acquisition module for acquiring the reasoning length reward value of the reasoning process based on the entropy average sequence of the reasoning process if the reasoning answer is consistent with the standard reasoning answer of any reasoning problem; and a training module for training the reasoning model to be trained based on the accuracy reward value and the reasoning length reward value to obtain a trained reasoning model; wherein the trained reasoning model is used to output the reasoning process and the reasoning answer based on the problem to be reasoned, and when any lexical unit is generated during the reasoning process, it determines whether to terminate the reasoning process based on the entropy value and the entropy average sequence of the current lexical unit to avoid over-reasoning stages during the reasoning process.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the training method of any of the above-described inference models.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the training method for any of the above-described inference models.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of training a training method for any of the above-described inference models.

[0010] This application obtains the entropy value of each lexical unit, which is used to quantify the uncertainty in generating each lexical unit. Based on the entropy value of each lexical unit, an entropy average sequence of the reasoning process is obtained. The entropy value and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process, with different reasoning lengths for different stages. Based on the entropy average sequence, a reasoning length reward value is obtained, fully considering the stage characteristics of the reasoning process and ensuring its accuracy. While ensuring the reasoning answer is consistent with the standard reasoning answer, the reasoning length reward value is obtained based on the entropy average sequence of the reasoning process. This limits the reasoning length of the reasoning process while maintaining the accuracy of the reasoning answer, avoiding situations where the reasoning process is short but the reasoning answer is inaccurate, thus ensuring the accuracy of the reasoning answer. Therefore, ensuring the accuracy of the reasoning process while ensuring the accuracy of the reasoning answer ensures the accuracy of the reasoning result. Furthermore, based on ensuring the accuracy of the reasoning answer, the reasoning length reward value is obtained. The reasoning model to be trained is then trained based on the accuracy reward value and the reasoning length reward value, resulting in a well-trained reasoning model that ensures both the accuracy of the reasoning result and avoids overthinking in the reasoning process, improving reasoning efficiency. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram illustrating a scenario for training a reasoning model provided in an embodiment of this application;

[0013] Figure 2 Flowchart of the training method for the inference model provided in the embodiments of this application Figure 1 ;

[0014] Figure 3 Flowchart of the training method for the inference model provided in the embodiments of this application Figure 2 ;

[0015] Figure 4 A schematic diagram of the structure of a training device for the inference model provided in an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To address the technical problems in related technologies, this application proposes the following technical concept: While reinforcement learning addresses the issue of excessively long reasoning processes in reasoning models by considering the final length of the reasoning process as a reward, it fails to consider the stage characteristics of the reasoning process, thus failing to guarantee the accuracy of the reasoning result. The inventors devised a method to identify multiple reasoning stages in the reasoning process. Specifically, they obtained the entropy value of each word, which is used to quantify the uncertainty in generating each word. Based on the entropy value of each word, an entropy average sequence of the reasoning process is obtained. The entropy value and the entropy average sequence are used to identify multiple reasoning stages in the process, with different reasoning lengths for different stages. Based on the entropy average sequence, a reasoning length reward value is obtained, fully considering the stage characteristics of the reasoning process and ensuring its accuracy. Furthermore, if the reasoning answer is accurate, another reasoning length reward value is obtained. Based on the accuracy reward value and the reasoning length reward value, the reasoning model to be trained is trained to obtain a well-trained reasoning model that ensures both the accuracy of the reasoning result and avoids overthinking during the reasoning process, thus improving reasoning efficiency.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the training method of the combined inference model depends is described here.

[0022] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating a training method for the inference model provided in this application embodiment. The electronic device provided in this embodiment includes: a receiving device 101, a processing device 102, and a display device 103.

[0023] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the training method of the inference model. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0024] In its implementation, the receiving device 101 can be an input / output interface or a communication interface, capable of receiving multiple historical reasoning results for any given reasoning problem. Each historical reasoning result includes a reasoning process and a reasoning answer; the reasoning process includes multiple lexical units.

[0025] The processing device 102 can perform a series of processes on multiple lexical units to obtain the accuracy reward value of the reasoning answer and the reasoning length reward value of the reasoning process; based on the accuracy reward value and the reasoning length reward value, the reasoning model to be trained is trained to obtain the trained reasoning model.

[0026] The display device 103 can be used to display the reasoning process and reasoning answer output by the trained reasoning model.

[0027] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0028] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0029] Figure 2 Flowchart of the training method for the inference model provided in the embodiments of this application Figure 1 ,like Figure 2 As shown, embodiments of this application provide a method for training an inference model, which is described in detail below:

[0030] S201: Collect multiple historical reasoning results for any reasoning problem; each historical reasoning result includes the reasoning process and the reasoning answer; the reasoning process includes multiple word units.

[0031] In this embodiment, any reasoning problem refers to the target reasoning problem that the reasoning model to be trained needs to handle. Historical reasoning results refer to the different reasoning paths generated by the reasoning model to be trained when handling the target reasoning problem, such as different problem-solving steps and different analysis logics, including the reasoning process and the reasoning answer.

[0032] In this embodiment, the reasoning process is a sequence of lexical units generated by the reasoning model to be trained, including multiple lexical units; the reasoning answer is the conclusion finally output by the reasoning process.

[0033] S202: Obtain the entropy value of each word element.

[0034] In this embodiment, entropy is used to quantify the uncertainty in generating any given word. A higher entropy value indicates less confidence in the model's ability to generate the current word, while a lower entropy value indicates greater confidence. This provides a basis for multiple inference stages in the subsequent recognition and reasoning process.

[0035] Specifically, the formula for obtaining the entropy value of each word element is:

[0036]

[0037]

[0038] In the formula, Let V represent the entropy value of the t-th lexical unit, and let V represent the number of lexical units in the vocabulary. This represents the probability that the t-th generated word is the j-th word in the vocabulary; This indicates that the t-th generated lexical is the log-odd value of the j-th lexical in the vocabulary, and T represents the temperature parameter used to control the smoothness of the probability distribution. This represents the log-probability value that the t-th generated lexicon is the k-th lexicon in the vocabulary.

[0039] Optionally, the temperature parameter can be set to 0.6. The temperature parameter is used to control the smoothness of the probability distribution. The smaller the temperature parameter, the more concentrated the probability is on a few words; the larger the temperature parameter, the more uniform the probability distribution.

[0040] S203: Based on the entropy value of each word, obtain the entropy average sequence of the reasoning process; where the entropy value and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process.

[0041] In this embodiment, the multiple reasoning stages include a reasoning exploration stage, a reasoning decision stage, and an over-reasoning stage. The reasoning exploration stage is the initial stage of reasoning, aiming to try different problem-solving approaches. At this stage, the understanding of the correct reasoning process is uncertain, requiring multiple explorations to confirm a feasible reasoning process. The reasoning decision stage is the concluding stage of reasoning, aiming to confirm the correct reasoning process and output the final reasoning answer. At this point, the model has found a feasible reasoning process, and no further exploration is needed. The over-reasoning stage is an abnormal stage of reasoning. Essentially, it occurs when the decision stage has been entered, and the correct reasoning process and answer have been found, but the model continues to generate highly uncertain terms, leading to redundancy in the reasoning process. Subsequent embodiments describe the use of entropy values ​​and entropy-average sequences to identify multiple reasoning stages in the reasoning process.

[0042] In this embodiment, the reasoning length is different at different reasoning stages. In related technologies, the final reasoning length only reflects the total reasoning length of the reasoning process and cannot determine whether the total reasoning length comes from the necessary derivation in the reasoning exploration stage or the redundant generation after entering the reasoning decision stage. Therefore, it will lead to inaccurate reasoning results.

[0043] Specifically, step S203 includes Sa~Sd:

[0044] Sa: Determine the average entropy of each word based on its entropy value.

[0045] In this embodiment, when calculating the average entropy of any word, the average of multiple entropy values ​​from the first word to the current word is calculated to obtain the average entropy of the current word.

[0046] For example, if when calculating the 5th word element, multiple entropy values ​​from the first word element to the current 5th word element are obtained first, namely [0.15, 0.18, 0.14, 0.11, 0.09], then the average entropy of the 5th word element is (0.15+0.18+0.14+0.11+0.09) / 5=0.134.

[0047] Specifically, the formula for determining the average entropy of each word element based on its entropy value is as follows:

[0048]

[0049] In the formula, The entropy average is represented by t, where t represents the t-th word. This represents the entropy value of the i-th word. This represents the summation of entropy values ​​from the first lexicon to the t-th lexicon.

[0050] Sb: Based on the average entropy of each word, obtain the initial entropy average sequence of the reasoning process.

[0051] In this embodiment, the entropy average of each word is arranged according to the generation order of the word to form an initial entropy average sequence.

[0052] For example, the initial entropy average sequence is The initial sequence dimension of the initial entropy-averaged sequence is L.

[0053] Sc: Get the initial sequence dimension of the initial entropy average sequence.

[0054] In this embodiment, the initial sequence dimension is the number of entropy average values ​​in the initial entropy average sequence.

[0055] Sd: Transform the initial entropy average sequence into an entropy average sequence of a preset sequence dimension. When calculating the entropy average of each entropy average in the preset sequence dimension entropy average sequence, perform the following steps.

[0056] In this embodiment, the number of lexical units and the number of entropy average values ​​are different in different reasoning processes, that is, the initial sequence dimension of the initial entropy average value sequence is also different. For example, the initial sequence dimension of reasoning process A is 5, and the initial sequence dimension of reasoning process B is 7. The initial sequence dimension needs to be unified to the preset sequence dimension.

[0057] For example, the preset sequence dimension is M, and the sequence with the preset sequence dimension and the entropy average value is... .

[0058] Specifically, step Sd includes Sd1 to Sd3:

[0059] Sd1: Determine the formula for calculating the current entropy average based on the initial sequence dimension, the preset sequence dimension, and the sequence number of the current entropy average in the entropy average sequence of the preset sequence dimension.

[0060] In this embodiment, the initial sequence dimension is L, and the preset sequence dimension is M. When calculating the m-th entropy average value in the sequence of entropy average values ​​of the preset sequence dimension, the sequence number is m.

[0061] Specifically, obtain the sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension, and multiply it with the initial sequence dimension; obtain the quotient of the product with the preset sequence dimension.

[0062] In this embodiment, the sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension is multiplied by the initial sequence dimension and is m×L; the quotient of the product and the preset sequence dimension is m×L / M.

[0063] In this embodiment, if the quotient is an integer, the formula for calculating the current average entropy is:

[0064]

[0065] In the formula, Let m represent the current average entropy, L represent the sequence number of the current average entropy, M represent the initial sequence dimension, and L represent the preset sequence dimension. Represents the first digit in the sequence with the initial entropy average. The average entropy of each sequence number.

[0066] In this embodiment, the quotient is an integer, that is, m×L / M is an integer.

[0067] In this embodiment, if the quotient is not an integer, the formula for calculating the current average entropy is:

[0068]

[0069] In the formula, Let m represent the current average entropy, L represent the initial sequence dimension, M represent the preset sequence dimension, and n represent the integer part of the quotient. Represents the first digit in the sequence with the initial entropy average. The average entropy corresponding to each sequence number Represents the first digit in the sequence with the initial entropy average. The average entropy of each sequence number.

[0070] In this embodiment, the quotient is a non-integer, that is, m×L / M is a non-integer.

[0071] When the quotient is not an integer, the current average entropy is calculated using a linear interpolation method.

[0072] Sd2: Calculate the current average entropy according to the formula for calculating the current average entropy.

[0073] Sd3: After the entropy average values ​​of each sequence in the preset sequence dimension are calculated, the entropy average value sequence of the reasoning process is obtained.

[0074] In this embodiment, after calculating the entropy average value of each entropy average value in the sequence of entropy average values ​​of the preset sequence dimension, the sequence is sorted according to the sequence number of each entropy average value to obtain the entropy average value sequence of the reasoning process.

[0075] S204: Determine the accuracy reward value for each historical reasoning answer based on the reasoning answer.

[0076] Specifically, if the reasoning answer is consistent with the standard reasoning answer of any reasoning question, the accuracy reward value of each historical reasoning answer is the first value; if the reasoning answer is inconsistent with the standard reasoning answer of any reasoning question, the accuracy reward value of each historical reasoning answer is the second value.

[0077] In this embodiment, if the reasoning answer is consistent with the standard reasoning answer of any reasoning question, the accuracy reward value is a first value, which can be 1; if the reasoning answer is inconsistent with the standard reasoning answer of any reasoning question, the accuracy reward value is a second value, which can be 0.

[0078] In this embodiment, "consistent with the standard inference answer" rather than "exact text match" means that the core conclusion of the inference answer is consistent with the standard inference answer. For example, for a coding task, consistency with the standard inference answer means passing the designed unit tests, not that it is exactly the same as the standard inference answer; for a math task, the inference answer only needs to be mathematically equivalent to the standard inference answer. For example, x=3 and 3 are solutions to the equation, so the answers to the math problem are equivalent.

[0079] S205: If the reasoning answer is consistent with the standard reasoning answer of any reasoning problem, then obtain the reasoning length reward value of the reasoning process based on the entropy average sequence of the reasoning process.

[0080] In this embodiment, the accuracy reward value and the inference length reward value are used to guide the inference model to be trained to learn correct inference strategies with fewer inference processes.

[0081] In this embodiment, the determination of whether to obtain the reasoning length reward value is based on the accuracy of the reasoning answer. When the reasoning answer matches the standard reasoning answer for any reasoning problem, it indicates that the reasoning answer is accurate. Obtaining the reasoning length reward value is only meaningful when the reasoning answer is accurate. When the reasoning answer does not match the standard reasoning answer for any reasoning problem, it indicates that the reasoning answer is inaccurate, and therefore, the reasoning length reward value is not obtained. This is because obtaining the reasoning length reward value is meaningless when the reasoning answer is inaccurate; it only limits the length of the reasoning process, but the reasoning answer is inaccurate.

[0082] Specifically, the formula for obtaining the reasoning length reward value of the reasoning process based on the entropy average sequence of the reasoning process is as follows:

[0083]

[0084] In the formula, The reasoning length reward value represents the reasoning process. The entropy average of the last word in the sequence representing the entropy average of the reasoning process.

[0085] In this embodiment, the reasoning length reward value of the reasoning process is calculated based on the entropy average value of the last word in the entropy average value sequence of the reasoning process. An exponential function is used to map it to the [0,2] interval, so that the lower the entropy average value, the lower the exploration degree of the reasoning process and the higher the reasoning length reward value.

[0086] In this embodiment, if the reasoning answer is consistent with the standard reasoning answer of any reasoning question, the reasoning length reward value of the reasoning process is obtained. This ensures the positiveness of the reasoning length reward value and amplifies the impact of the entropy average difference on the reasoning length reward value through the exponential function. This enhances the guiding role of the reward signal on the reasoning model to be trained and guides the reasoning model to learn the correct reasoning strategy with fewer reasoning processes.

[0087] In this embodiment, for simple problems, such as elementary school math problems, the average entropy of the reasoning process decreases rapidly, and the process terminates promptly after entering the reasoning decision-making stage, resulting in a short reasoning length. For complex problems, such as math competition problems, the average entropy of the reasoning process continuously increases during the reasoning exploration stage and then decreases slowly after sufficient exploration, leading to an appropriately longer reasoning length. This approach can adaptively match the complexity of the problem and exhibits strong generalization ability.

[0088] S206: Train the inference model to be trained based on the accuracy reward value and the inference length reward value to obtain a trained inference model; the trained inference model is used to output the inference process and the inference answer based on the question to be inferred. When any word is generated during the inference process, determine whether to terminate the inference process based on the entropy value and the entropy average sequence of the current word to avoid the over-inference stage during the inference process.

[0089] Specifically, step S206 includes Se~Sg:

[0090] Se: Determine the total reward value for each historical reasoning result based on the accuracy reward value and the reasoning length reward value.

[0091] In this embodiment, if the accuracy reward value is 0, that is, the reasoning answer is inaccurate, the reasoning length reward value is not obtained, and the total reward value is 0, so as to avoid the reasoning model to be trained from generating short and incorrect reasoning results in order to obtain the reasoning length reward value.

[0092] In this embodiment, if the accuracy reward value is 1, that is, the reasoning answer is accurate, then the reasoning length reward value needs to be obtained. The total reward value = 1 + reasoning length reward value. Under the premise of ensuring accuracy, the reasoning model to be trained can obtain a higher reward by reducing the reasoning length reward value, thereby avoiding over-reasoning.

[0093] Sf: Standardize the total reward value of each historical reasoning result to obtain the relative reward value of each historical reasoning result.

[0094] In this embodiment, the formula for standardization is:

[0095]

[0096] In the formula, Let represent the relative reward value of each historical reasoning result, r represent the total reward value of each historical reasoning result, and represent the average of the total reward values ​​of multiple historical reasoning results. The standard deviation of the total reward value of multiple historical inference results. It is a constant. To prevent the denominator from being 0.

[0097] In this embodiment, the relative reward values ​​of each historical reasoning result after standardization follow a normal distribution with a mean of 0 and a standard deviation of 1, which facilitates training stability during the training process.

[0098] Sg: Based on the relative reward values ​​of each historical reasoning result, a group relative strategy optimization algorithm is used to train the reasoning model to be trained, and a trained reasoning model is obtained.

[0099] In this embodiment, the relative reward values ​​of multiple historical inference results are used as a sample. The Group Relative Policy Optimization (GRPO) algorithm is employed to optimize the parameters of the inference model to be trained through gradient descent, strengthening policies with positive relative reward values ​​and weakening policies with negative relative reward values. Strengthening policies with positive relative reward values ​​increases the probability of generating accurate and exploratory inference processes, while weakening policies with negative relative reward values ​​reduces the probability of generating inaccurate or overexploratory inference processes.

[0100] In summary, the entropy value of each word is obtained, which quantifies the uncertainty in generating each word. Based on the entropy value of each word, an entropy average sequence of the reasoning process is obtained. The entropy value and entropy average sequence are used to identify multiple reasoning stages in the process, each with a different reasoning length. Based on the entropy average sequence, a reasoning length reward value is obtained, fully considering the stage characteristics of the reasoning process and ensuring its accuracy. While ensuring the reasoning answer is consistent with the standard reasoning answer, the reasoning length reward value is obtained based on the entropy average sequence of the reasoning process. This limits the reasoning length while maintaining the accuracy of the reasoning answer, preventing situations where the reasoning process is short but the answer is inaccurate, thus ensuring the accuracy of the reasoning answer. Therefore, ensuring the accuracy of the reasoning process while ensuring the accuracy of the reasoning answer ensures the accuracy of the reasoning result. Furthermore, based on ensuring the accuracy of the reasoning answer, the reasoning length reward value is obtained. The reasoning model to be trained is then trained using both the accuracy reward value and the reasoning length reward value, resulting in a well-trained reasoning model that ensures both the accuracy of the reasoning result and avoids overthinking during the reasoning process, improving reasoning efficiency.

[0101] Figure 3 Flowchart of the training method for the inference model provided in the embodiments of this application Figure 2 ,like Figure 3 As shown, the inference process of the trained inference model is mainly introduced, and detailed below:

[0102] S301: Input the problem to be reasoned into the trained reasoning model. During the reasoning process, when the trained reasoning model generates any word, it obtains the entropy value of the current word.

[0103] S302: Based on the entropy value of the current word element, obtain the entropy average sequence of the current reasoning process; wherein the entropy average sequence includes the entropy average of multiple words elements.

[0104] In this embodiment, the method for obtaining the entropy value and the entropy average value sequence is the same as that in the above embodiment, and will not be repeated here.

[0105] S303: Based on the entropy average sequence of the current reasoning process, determine whether the reasoning termination condition is met.

[0106] S304: If the reasoning termination condition is met, the reasoning of the trained reasoning model is terminated, and the reasoning process and reasoning answer of the problem to be reasoned are obtained; wherein the reasoning termination condition includes: in the entropy average value sequence of the current reasoning process, there is a first preset number of word units whose entropy average value increases sequentially; or the reasoning answer has been generated, and after the reasoning answer is generated, there is a second preset number of word units whose entropy average value decreases sequentially.

[0107] In this embodiment, if the entropy average value of a first preset number of words in the current inference process sequentially increases, it is determined that there may be over-inference or inference failure, the inference is forcibly terminated, and the current inference result is output. Optionally, the first preset number can be 20.

[0108] In this embodiment, after the reasoning answer has been generated, and the entropy average value of a second preset number of words in the current reasoning process sequence decreases sequentially, it is determined that valid reasoning has been completed, avoiding the over-reasoning stage, the reasoning is terminated, and the reasoning result is output. Optionally, the second preset number can be 3.

[0109] In this embodiment, the steps for identifying multiple inference stages in the inference process based on the entropy value and the entropy average value sequence include Sh~Sj:

[0110] Sh: Starting from the first word in the reasoning process, if there is a third consecutive word whose entropy value exceeds the preset entropy threshold, then the reasoning process corresponding to the third consecutive word is determined as the reasoning exploration stage.

[0111] In this embodiment, the characteristic of the reasoning exploration phase is that the entropy value of the word element is continuously higher than the preset entropy threshold, indicating that the reasoning model is in a state of high uncertainty.

[0112] Optionally, the third preset number can be 5. If the entropy value of 5 consecutive words exceeds the preset entropy threshold, then the reasoning process corresponding to the third preset number of consecutive words is determined as the reasoning exploration stage.

[0113] Si: After the reasoning exploration phase, if there is a fourth consecutive preset number of word units with an entropy value lower than the preset entropy threshold, then the reasoning process corresponding to the fourth consecutive preset number of word units will be determined as the reasoning decision phase.

[0114] In this embodiment, the characteristic of the reasoning decision stage is that the entropy value of the word element is continuously lower than the preset entropy value threshold, indicating that the reasoning model is in a state of low uncertainty and has converged to a definite reasoning process.

[0115] Optionally, the fourth preset number can be 5. If the entropy value of 5 consecutive words is lower than the preset entropy threshold, then the reasoning process corresponding to the fourth preset number of consecutive words is determined as the reasoning decision stage.

[0116] Sj: After the reasoning decision stage, if the entropy value of a fifth consecutive preset number of words exceeds the preset entropy value threshold, the reasoning process corresponding to the fifth consecutive preset number of words is determined as an over-reasoning stage; or after the reasoning decision stage, if the entropy average value sequence of the reasoning process has a sixth consecutive preset number of entropy average value increments that exceed the preset entropy average value increment threshold, the reasoning process corresponding to the sixth consecutive preset number of entropy average values ​​is determined as an over-reasoning stage.

[0117] Optionally, the fifth preset quantity can be 5, and the sixth preset quantity can be 10.

[0118] In this embodiment, the reasoning decision-making stage is characterized by the entropy value of the word being continuously lower than a preset entropy threshold. Therefore, after entering the reasoning decision-making stage, if the entropy value of a fifth consecutive preset number of word elements exceeds the preset entropy threshold, it indicates that word elements with high uncertainty are still being generated, leading to redundancy in the reasoning process.

[0119] In this embodiment, the reasoning decision-making stage is characterized by the entropy value of each word consistently being below a preset entropy threshold. Therefore, after entering the reasoning decision-making stage, the average entropy value of each word should show a continuous decreasing trend or stabilize at a low level. If there is a sixth consecutive preset number of entropy value increments in the entropy value sequence that exceeds the preset entropy value increment threshold, it indicates that words with high uncertainty are still being generated, leading to redundancy in the reasoning process, i.e., over-reasoning.

[0120] In this embodiment, a statistical method is used to determine the preset entropy threshold, calculate the entropy values ​​of the lexical units of multiple correct reasoning result samples, and take the 90th percentile as the preset entropy threshold.

[0121] Optionally, the preset entropy average increment threshold can be set to 0.05.

[0122] S305: During the reasoning process, delete the words in the entropy average sequence whose entropy average exceeds the preset entropy average threshold to obtain the final reasoning process and reasoning answer.

[0123] Optionally, the generated reasoning process can be post-processed to remove redundant content, such as repeated verification steps and irrelevant explanations, while retaining the core reasoning process and the reasoning answer. The post-processing rules are designed based on the entropy average: removing the reasoning process corresponding to words with an entropy average greater than a preset entropy average threshold, and retaining the reasoning process corresponding to words with an entropy average less than the preset entropy average threshold. This can further reduce the length of the reasoning process by 15%-20% without affecting accuracy.

[0124] Optionally, a preset entropy average threshold can be determined using statistical methods, the entropy average of the lexical units of multiple correct reasoning result samples can be calculated, and the 90th percentile can be taken as the preset entropy average threshold.

[0125] In summary, in the entropy average sequence of the current reasoning process, if the entropy average of a first preset number of words increases sequentially, it indicates a possible over-reasoning or reasoning failure. Once a reasoning answer has been generated, and after its generation, if the entropy average of a second preset number of words in the current reasoning process decreases sequentially, it indicates that effective reasoning has been completed, avoiding the over-reasoning stage. By determining whether the reasoning termination condition is met based on the entropy average sequence of the current reasoning process, and deciding whether to terminate the reasoning, the over-reasoning stage can be avoided, effectively improving reasoning efficiency. Furthermore, entropy is used to quantify the uncertainty when generating any word. Entropy and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process. Through the entropy values ​​of each word, the entropy average sequence, preset entropy thresholds, and preset entropy average increment thresholds, the reasoning exploration stage, reasoning decision-making stage, and over-reasoning stage can be accurately identified.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0127] Figure 4 This is a schematic diagram of the structure of a training device for the inference model provided in an embodiment of this application. Figure 4As shown, embodiments of this application also provide a training device for an inference model, including: a data acquisition module 401, a first acquisition module 402, a second acquisition module 403, a determination module 404, a third acquisition module 405, and a training module 406.

[0128] The acquisition module 401 is used to acquire multiple historical reasoning results for any reasoning problem; each historical reasoning result includes a reasoning process and a reasoning answer; the reasoning process includes multiple word units.

[0129] The first acquisition module 402 is used to acquire the entropy value of each word element.

[0130] The second acquisition module 403 is used to obtain the entropy average sequence of the reasoning process based on the entropy value of each word; wherein the entropy value and the entropy average sequence are used to identify multiple reasoning stages in the reasoning process.

[0131] Module 404 is used to determine the accuracy reward value for each historical reasoning answer based on the reasoning answer.

[0132] The third acquisition module 405 is used to obtain the reasoning length reward value of the reasoning process based on the entropy average sequence of the reasoning process if the reasoning answer is consistent with the standard reasoning answer of any reasoning problem.

[0133] Training module 406 is used to train the inference model to be trained based on the accuracy reward value and the inference length reward value to obtain a trained inference model. The trained inference model is used to output the inference process and the inference answer based on the question to be inferred. When any word is generated during the inference process, the inference process is terminated based on the entropy value and the entropy average sequence of the current word to avoid the over-inference stage during the inference process.

[0134] In one possible implementation, the training module 406 is specifically used to: determine the total reward value of each historical inference result based on the accuracy reward value and the inference length reward value; standardize the total reward value of each historical inference result to obtain the relative reward value of each historical inference result; and train the inference model to be trained using a group relative strategy optimization algorithm based on the relative reward value of each historical inference result to obtain the trained inference model.

[0135] In one possible implementation, the second acquisition module 403 is specifically used for: determining the average entropy of each word based on its entropy value; obtaining an initial entropy average sequence for the reasoning process based on the average entropy of each word; acquiring the initial sequence dimension of the initial entropy average sequence; converting the initial entropy average sequence into an entropy average sequence of a preset sequence dimension; and performing the following steps when calculating each entropy average in the entropy average sequence of the preset sequence dimension: the steps include: determining the calculation formula for the current entropy average based on the initial sequence dimension, the preset sequence dimension, and the sequence number of the current entropy average in the entropy average sequence of the preset sequence dimension; calculating the current entropy average according to the calculation formula; and obtaining the entropy average sequence for the reasoning process after the calculation of each entropy average in the entropy average sequence of the preset sequence dimension is completed.

[0136] In one possible implementation, the second acquisition module 403 is further configured to: acquire the sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension, and the product of the sequence number and the initial sequence dimension; acquire the quotient of the product and the preset sequence dimension; if the quotient is an integer, the formula for calculating the current entropy average value is:

[0137]

[0138] In the formula, Let m represent the current average entropy, L represent the sequence number of the current average entropy, M represent the initial sequence dimension, and L represent the preset sequence dimension. Represents the first digit in the sequence with the initial entropy average. The average entropy of each sequence number.

[0139] If the quotient is not an integer, then the formula for calculating the current average entropy is:

[0140]

[0141] In the formula, Let m represent the current average entropy, L represent the initial sequence dimension, M represent the preset sequence dimension, and n represent the integer part of the quotient. Represents the first digit in the sequence with the initial entropy average. The average entropy corresponding to each sequence number Represents the first digit in the sequence with the initial entropy average. The average entropy of each sequence number.

[0142] In one possible implementation, the formula for the first acquisition module 402 is:

[0143]

[0144]

[0145] In the formula, Let V represent the entropy value of the t-th lexical unit, and let V represent the number of lexical units in the vocabulary. This represents the probability that the t-th generated word is the j-th word in the vocabulary; This indicates that the t-th generated lexical is the log-odd value of the j-th lexical in the vocabulary, and T represents the temperature parameter used to control the smoothness of the probability distribution. This represents the log-probability value that the t-th generated lexicon is the k-th lexicon in the vocabulary.

[0146] In one possible implementation, the formula for the third acquisition module 405 is:

[0147]

[0148] In the formula, The reasoning length reward value represents the reasoning process. The entropy average of the last word in the sequence representing the entropy average of the reasoning process.

[0149] In one possible implementation, the determining module 404 is specifically configured to: if the reasoning answer is consistent with the standard reasoning answer of any reasoning question, then the accuracy reward value of each historical reasoning answer is a first value; if the reasoning answer is inconsistent with the standard reasoning answer of any reasoning question, then the accuracy reward value of each historical reasoning answer is a second value.

[0150] In one possible implementation, the training device for the reasoning model further includes a first judgment module. The first judgment module is specifically used for: inputting the problem to be reasoned into the trained reasoning model; during the reasoning process, when the trained reasoning model generates any word element, obtaining the entropy value of the current word element; obtaining the entropy average sequence of the current reasoning process based on the entropy value of the current word element; wherein the entropy average sequence includes the entropy averages of multiple word elements; determining whether the reasoning termination condition is met based on the entropy average sequence of the current reasoning process; if the reasoning termination condition is met, terminating the reasoning of the trained reasoning model, and obtaining the reasoning process and reasoning answer for the problem to be reasoned; wherein the reasoning termination condition includes: in the entropy average sequence of the current reasoning process, there exists a first preset number of word elements whose entropy average values ​​sequentially increase; or a reasoning answer has been generated, and after generating the reasoning answer, in the entropy average sequence of the current reasoning process, there exists a second preset number of word elements whose entropy average values ​​sequentially decrease; during the reasoning process, deleting word elements in the entropy average sequence whose entropy average values ​​exceed a preset entropy average threshold, and obtaining the final reasoning process and reasoning answer.

[0151] In one possible implementation, the training device for the inference model further includes a second judgment module. The second judgment module is specifically configured to: starting from the first word in the inference process, if the entropy value of a third consecutive preset number of words exceeds a preset entropy threshold, then the inference process corresponding to the third consecutive preset number of words is determined as an inference exploration stage; after the inference exploration stage, if the entropy value of a fourth consecutive preset number of words is lower than the preset entropy threshold, then the inference process corresponding to the fourth consecutive preset number of words is determined as an inference decision stage; after the inference decision stage, if the entropy value of a fifth consecutive preset number of words again exceeds the preset entropy threshold, then the inference process corresponding to the fifth consecutive preset number of words is determined as an over-inference stage; or after the inference decision stage, if the entropy average value sequence of the inference process contains a sixth consecutive preset number of entropy average value increments exceeding a preset entropy average value increment threshold, then the inference process corresponding to the sixth consecutive preset number of entropy average values ​​is determined as an over-inference stage.

[0152] For a description of the features in the embodiment corresponding to the training device of the inference model, please refer to the relevant description of the embodiment corresponding to the training method of the inference model, which will not be repeated here.

[0153] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0154] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described training method embodiment for the inference model.

[0155] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0156] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0157] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0158] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0159] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in the training method embodiments of any of the above-described inference models at runtime.

[0160] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0161] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the training method embodiments of any of the above-described inference models.

[0162] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the training method embodiments of any of the above-described inference models.

[0163] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] The foregoing has provided a detailed description of the training method, apparatus, electronic device, medium, and product for a reasoning model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for training a reasoning model, characterized in that, The method comprises: acquiring, by a receiving device of an electronic device, a plurality of historical reasoning results for any reasoning question; wherein each historical reasoning result comprises a reasoning process and a reasoning answer; the reasoning process comprises a plurality of word units; wherein the reasoning question is a reasoning question in a mathematical question and answer scenario; the reasoning process is different problem solving steps and different analysis logics in the mathematical question and answer scenario; and the reasoning answer is a conclusion of the reasoning question in the mathematical question and answer scenario; acquiring, by a processing device of the electronic device, an entropy value of each word unit; the processing device obtains an entropy average value sequence of the reasoning process according to the entropy values of the word units; wherein the entropy values and the entropy average value sequence are used to identify a plurality of reasoning stages in the reasoning process; the processing device determines an accuracy reward value of each historical reasoning answer according to the reasoning answer; if the reasoning answer is consistent with a standard reasoning answer of the reasoning question, the processing device acquires a reasoning length reward value of the reasoning process according to the entropy average value sequence of the reasoning process; the processing device trains a reasoning model to be trained according to the accuracy reward value and the reasoning length reward value, and obtains a trained reasoning model; wherein the trained reasoning model is used to output a reasoning process and a reasoning answer according to a reasoning question to be reasoned, and to determine whether to terminate the reasoning process when generating any word unit in the reasoning process according to the entropy value of the current word unit and the entropy average value sequence, so as to avoid over-reasoning stage in the reasoning process.

2. The method of claim 1, wherein, the processing device trains a reasoning model to be trained according to the accuracy reward value and the reasoning length reward value, and obtains a trained reasoning model, comprising: the processing device determines a total reward value of each historical reasoning result according to the accuracy reward value and the reasoning length reward value; the processing device performs standardization processing on the total reward value of each historical reasoning result to obtain a relative reward value of each historical reasoning result; the processing device trains the reasoning model to be trained according to the relative reward value of each historical reasoning result by using a group relative strategy optimization algorithm, and obtains the trained reasoning model.

3. The method of claim 1, wherein, the processing device obtains the entropy average value sequence of the reasoning process according to the entropy values of the word units, comprising: the processing device determines the entropy average value of each word unit according to the entropy values of the word units; the processing device obtains an initial entropy average value sequence of the reasoning process according to the entropy average values of the word units; the processing device acquires an initial sequence dimension of the initial entropy average value sequence; the processing device converts the initial entropy average value sequence into an entropy average value sequence of a preset sequence dimension, and performs the following steps when calculating each entropy average value in the entropy average value sequence of the preset sequence dimension; the steps comprise: the processing device determines a calculation formula of the current entropy average value according to the initial sequence dimension, the preset sequence dimension, and a sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension; the processing device calculates the current entropy average value according to the calculation formula of the current entropy average value; After each entropy average value in the entropy average value sequence of the preset sequence dimension is calculated, the entropy average value sequence of the inference process is obtained.

4. The method of claim 3, wherein, The processing apparatus determines a calculation formula of the current entropy average value according to the initial sequence dimension, the preset sequence dimension, and a sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension, including: The processing apparatus obtains a product of the sequence number of the current entropy average value in the entropy average value sequence of the preset sequence dimension and the initial sequence dimension; The processing apparatus obtains a quotient of the product and the preset sequence dimension; If the quotient is an integer, the calculation formula of the current entropy average value is: In the formula, represents the current entropy average value, m represents the sequence number of the current entropy average value, L represents the initial sequence dimension, M represents a preset sequence dimension, represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence; and represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence; and If the quotient is not an integer, the calculation formula of the current entropy average value is: In the formula, represents the current entropy average value, m represents the sequence number of the current entropy average value, L represents the initial sequence dimension, M represents a preset sequence dimension, n represents the value of the integer part of the quotient, represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence, represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence, represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence, represents the entropy average value corresponding to the sequence number m in the initial entropy average sequence.

5. The method of claim 1, wherein, The processing apparatus obtains the entropy value of each token, and the formula is: wherein, denotes the entropy value of the t-th token, V denotes the number of tokens in the vocabulary, denotes the probability that the t-th generated token is the j-th token in the vocabulary; denotes the log-likelihood value that the t-th generated token is the j-th token in the vocabulary, T denotes a temperature parameter for controlling the degree of smoothing of the probability distribution, denotes the log-likelihood value that the t-th generated token is the k-th token in the vocabulary.

6. The method of claim 1, wherein, The processing apparatus obtains the inference length reward value of the inference process according to the entropy average value sequence of the inference process, and the formula is: wherein represents the inference length reward value of the inference process, represents the entropy average value of the last word element in the sequence of entropy average values of the inference process.

7. The method of claim 1, wherein, The processing apparatus determines the accuracy reward value of each historical inference answer according to the inference answer, including: If the inference answer is consistent with the standard inference answer of any inference question, the accuracy reward value of each historical inference answer is a first numerical value; If the inference answer is not consistent with the standard inference answer of any inference question, the accuracy reward value of each historical inference answer is a second numerical value.

8. The method of claim 1, wherein, After the trained inference model is obtained, the method further includes: The processing apparatus inputs the problem to be inferred into the trained inference model, and the trained inference model obtains the entropy value of the current token when generating any token in the inference process; The processing apparatus obtains the entropy average value sequence of the current inference process according to the entropy value of the current token, wherein the entropy average value sequence includes the entropy average values of multiple tokens; The processing apparatus determines whether the inference termination condition is met according to the entropy average value sequence of the current inference process; If the processing apparatus determines that the inference termination condition is met, the inference of the trained inference model is terminated, and the inference process and the inference answer of the problem to be inferred are obtained; wherein the inference termination condition includes that in the entropy average value sequence of the current inference process, the entropy average values of a first preset number of tokens are sequentially increased, or the inference answer is generated, and after the inference answer is generated, the entropy average values of a second preset number of tokens in the entropy average value sequence of the current inference process are sequentially decreased; In the inference process, the processing apparatus deletes the tokens with the entropy average values exceeding a preset entropy average value threshold in the entropy average value sequence, and obtains the final inference process and the inference answer.

9. The method of claim 1, wherein, The method further includes: Starting from the first token in the inference process, if the entropy values of a continuous third preset number of tokens exceed a preset entropy value threshold, the processing apparatus determines the inference process corresponding to the continuous third preset number of tokens as an inference exploration stage; After the inference exploration stage, if the entropy values of a continuous fourth preset number of tokens are lower than the preset entropy value threshold, the processing apparatus determines the inference process corresponding to the continuous fourth preset number of tokens as an inference decision stage; After the reasoning decision stage, if the entropy values of the fifth preset number of continuous word elements again exceed the preset entropy value threshold, the processing device determines the reasoning process corresponding to the fifth preset number of continuous word elements as an over-reasoning stage; or after the reasoning decision stage, if the increments of the sixth preset number of continuous entropy average values in the entropy average value sequence of the reasoning process exceed the preset entropy average value increment threshold, the reasoning process corresponding to the sixth preset number of continuous entropy average values is determined as the over-reasoning stage.

10. An electronic device, comprising: The method comprises the steps of: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the training method of the reasoning model according to any one of claims 1-9.

Images