Model inference method, electronic device, and storage medium

Abstract

The application discloses a model reasoning method, an electronic device and a storage medium, and relates to the technical field of artificial intelligence. In the model reasoning process, various kinds of information are comprehensively used to correct the logarithmic probability of a candidate word element to be generated, so that the model can avoid switching ideas in the later reasoning stage, the reasoning process can be shortened due to the frequent switching of reasoning ideas, the accuracy of the model reasoning result is ensured, and the waste of model reasoning resources is avoided.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model reasoning method, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have achieved remarkable results in fields such as natural language processing. Among them, long thought chain large language models have strong complex reasoning capabilities, but they often have excessively long reasoning processes even for simple problems due to frequent switching between different reasoning approaches, thus wasting the model's reasoning resources.

[0003] In related technologies, computational resources are typically limited for models, such as forcibly interrupting model inference when the inference length reaches a preset upper limit. However, forcibly interrupting model inference prevents the model from fully thinking and reasoning, limiting the ability of long-thinking-chain models to continuously explore multiple inference strategies when dealing with complex tasks, thereby reducing the accuracy of the model's inference results. Summary of the Invention

[0004] This application provides a model inference method, electronic device, and storage medium to at least solve the problem in related technologies that it is impossible to ensure the accuracy of model inference results while avoiding the waste of model inference resources.

[0005] This application provides a model reasoning method, including:

[0006] Obtain the target inference model;

[0007] During the reasoning process of the target reasoning model in response to the user-input reasoning question, the initial log odds corresponding to multiple candidate words to be generated are determined based on the generated reasoning process; wherein, the generated reasoning process includes multiple words that have been generated in sequence;

[0008] For any candidate word to be generated, the target log odds of the candidate word to be generated are determined based on the initial log odds corresponding to the candidate word, the preset logic switching word set, the preset log odds correction constraints, and the distance between the current reasoning position and the reasoning start.

[0009] Based on the target log probability of each candidate word to be generated, the target word to be generated is determined, so as to obtain the target reasoning process of the target reasoning model.

[0010] This application also provides a model inference apparatus, including:

[0011] The acquisition module is used to acquire the target inference model;

[0012] The first determining module is used to determine the initial log odds corresponding to multiple candidate words to be generated based on the generated reasoning process during the reasoning process of the target reasoning model in response to the reasoning question input by the user; wherein the generated reasoning process includes multiple words that have been generated in sequence.

[0013] The second determining module is used to determine the target log probability of any candidate word to be generated based on the initial log probability corresponding to the candidate word, the preset logic switching word set, the preset log probability correction constraint, and the distance between the current reasoning position and the reasoning start.

[0014] The reasoning module is used to determine the target word to be generated based on the target log probability of each candidate word to be generated, so as to obtain the target reasoning process of the target reasoning model.

[0015] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model reasoning methods.

[0016] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model reasoning methods.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model reasoning methods.

[0018] By incorporating various information during the model inference process, the log probability of the candidate words to be generated is corrected to avoid switching the model's approach in the later stages of inference. This avoids the inference process becoming too long due to frequent switching of inference approaches, ensuring the accuracy of the model's inference results while avoiding the waste of model inference resources. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of the model inference system on which the embodiments of this application are based;

[0021] Figure 2 A flowchart illustrating the model inference method provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram of the structure of the model inference device provided in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] With the rapid development of artificial intelligence technology, large language models (LLMs) have achieved remarkable results in the field of natural language processing. Among them, long-thinking-chain LLMs, such as Deepseek's r1, have demonstrated powerful complex reasoning capabilities. They can mimic the deep thinking process of humans, continuously exploring multiple reasoning strategies, reflecting on decisions, and iteratively optimizing solutions when dealing with complex tasks by using more computing resources, longer thinking time, and longer thought sequences. These models provide new approaches and methods for solving practical problems in complex reasoning tasks, such as mathematical problem solving and code generation. However, in practical applications, LLMs like r1 have also revealed some problems, affecting their further performance improvement and widespread application.

[0027] The thought process of long thought chain models is uncontrolled, frequently switching between different reasoning approaches. Each attempt fails to fully explore to arrive at a definitively correct conclusion, leading to numerous intermediate reasoning steps, prolonged reasoning time, and wasted computational resources. Furthermore, the uncontrolled output during reasoning makes it difficult to allocate computational budgets reasonably based on task requirements and available resources. This is especially problematic for simple problems, where the generated reasoning may be excessively long, further wasting computational resources and resulting in poor efficiency and performance in real-world applications.

[0028] Therefore, how to obtain the final solution to a problem under limited computing resources, that is, within a certain range of reasoning constraints, is roughly equivalent to the reasoning waiting time that users can accept, and also roughly equivalent to the reasoning cost that users can accept.

[0029] Currently, two approaches are commonly used to address the aforementioned technical issues: instruction-following and early exit from inference. Regarding instruction-following, current methods tend to use cue word engineering to guide the model to generate thought processes that meet resource constraints. For example, when resources are limited, cue words like "Please solve this problem through a single thought process, and be sure not to switch ideas" are used to guide the model to generate fewer output sequences. Conversely, when resources are ample but accuracy is more critical, cue words like "Verify each direction in detail, and don't give up halfway" are used to guide the model to generate longer thought sequences, improving accuracy. However, while cue word engineering provides guidance, this method heavily relies on the superior instruction-following capabilities of large models. Existing long thought chain models exhibit unstable instruction-following performance, often failing. Furthermore, instruction-following capabilities cannot achieve fine-grained control over the length of the model's output sequence, i.e., it cannot finely control the model's inference cost. Regarding early exit from inference, current methods attempt to control costs by forcing the model to exit when approaching the inference resource cost limit (approaching the acceptable output sequence length) by generating a special token indicating the end of inference (e.g., "Therefore, the final answer is:"). This approach constrains the length of the output sequence of the long thought chain model to some extent. However, it is obvious that this rigid design strategy abruptly truncates the reasoning process, causing the model to be unable to think and reason completely. This limits the ability of the resource-conditional long thought chain model to continuously explore the advantages of multiple reasoning strategies when dealing with complex tasks, resulting in reduced accuracy and negatively impacting the user interpretability of the model's output.

[0030] To address the aforementioned technical problems, embodiments of this application provide a model inference method, an electronic device, and a storage medium. In this method, various information is integrated during the model inference process to correct the logarithmic probability of the candidate lexical units to be generated, thereby avoiding the model switching its approach in the later stages of inference. This also avoids the inference process becoming too long due to frequent switching of inference approaches, ensuring the accuracy of the model inference results while avoiding the waste of model inference resources.

[0031] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The specific application environment architecture or specific hardware architecture on which the execution of the model inference method depends is described here.

[0033] First, the structure of the model inference system on which this application is based will be described:

[0034] The model reasoning method, electronic device, and storage medium provided in this application are suitable for optimizing reasoning models with long thought chains, so as to ensure the accuracy of the reasoning answer while ensuring that the reasoning process is not too long. Figure 1 The diagram shows the structure of the model inference system based on the embodiments of this application, mainly including a data acquisition device, an inference model, and a model inference device. The data acquisition device is used to acquire the model training sample set during the model optimization phase and to acquire the user-inputted inference question during the model inference phase. The inference model to be optimized is used to optimize the inference model using the model training sample set. After model optimization is completed, the inference model infers the user-inputted inference question. During its inference process, the model inference device corrects the log probability of the candidate words to be generated to avoid the model switching its approach in the later stages of inference, thus avoiding excessively long inference processes due to frequent switching of inference approaches.

[0035] This application provides a model reasoning method for optimizing reasoning models with long thought chains, ensuring the accuracy of the reasoning answer while minimizing the length of the reasoning process. The execution subject of this application is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used to optimize reasoning models with long thought chains.

[0036] like Figure 2 The diagram shown is a flowchart illustrating the model inference method provided in an embodiment of this application. The method includes:

[0037] Step 201: Obtain the target inference model.

[0038] The target reasoning model is used to reason about code problems and data problems, and finally obtain the reasoning results.

[0039] Step 202: During the process of the target reasoning model reasoning in response to the user's input reasoning question, determine the initial log odds corresponding to multiple candidate words to be generated based on the generated reasoning process.

[0040] The generated reasoning process includes multiple lexical units that have been generated in sequence.

[0041] It should be noted that the log odds is the original log probability output by the model for each candidate word to be generated during the inference process, based on the generated words. Its value reflects the initial tendency of the model to generate that word.

[0042] Specifically, the normal reasoning process of the target inference model is a typical autoregressive process. When the current t-th token is generated, the log odds vector of the model output layer is first obtained. The log odds vector consists of the initial log odds corresponding to multiple candidate tokens to be generated. The length of the log odds vector is consistent with the length of the vocabulary during model training.

[0043] Step 203: For any candidate word to be generated, determine the target log probability of the candidate word to be generated based on the initial log probability corresponding to the candidate word, the preset logic switching word set, the preset log probability correction constraint, and the distance between the current reasoning position and the reasoning start.

[0044] It should be noted that the reason why the sequence length of long-thinking-chain large language models is difficult to control during the reasoning process is that multiple reasoning strategies will be tried, previous strategies will be reflected upon and iterative optimizations will be performed during the reasoning process. These attempts at over-validation and path switching may be redundant, which do not help with accuracy but consume a lot of computing resources.

[0045] The preset set of reasoning switching terms includes terms related to reasoning switching and path reflection, such as "but," "etc.," "however," "or," "possibly," "maybe," "again," "let me again," and "let us wait again." When any term from the preset set of reasoning switching terms appears, it will cause a switch in the model's reasoning approach, which may result in redundancy in the reasoning process.

[0046] Specifically, in order to avoid frequent switching of ideas in the target reasoning model during the reasoning process, the log odds of the candidate words to be generated are adjusted during the model reasoning process to determine the target log odds of the candidate words to be generated.

[0047] Step 204: Determine the target word to be generated based on the target log probability of each candidate word to be generated, so as to obtain the target reasoning process of the target reasoning model.

[0048] Specifically, the target log odds alter the probability of generating a word shift term within a specified inference range, prompting the model to fully explore problem-solving approaches in the early stages and output the final conclusion as early as possible in the later stages. The probability distribution of each candidate word term to be generated can be calculated based on the following formula:

[0049]

[0050] in, This represents the probability that the t-th word is w when generating the t-th word based on the first (t-1) generated words, i.e., the candidate word to be generated is w. Indicates candidate lexical units to be generated The target log-odds ratio is given by V, where V is the vocabulary used during model training. After obtaining this probability distribution, the word to be generated with the highest speech probability distribution is taken as the t-th word (the target word to be generated).

[0051] Based on the above embodiments, as one implementable approach, in one embodiment, obtaining the target inference model includes:

[0052] Step 2011: Obtain the training sample set of the inference model to be optimized and the target model; wherein, each training sample of the target model in the training sample set includes at least a prompt word, an inference length threshold, and a standard answer;

[0053] Step 2012: Based on the target model training sample set, iteratively optimize the inference model to be optimized until the target inference model is obtained.

[0054] The target model training sample set includes at least a prompt word, an inference length threshold, and a standard answer for each target model training sample. The inference model to be optimized is a pre-trained inference language model. Taking a mathematical problem as an example, the standard answer is the reference answer to that mathematical problem. The inference model to be optimized is a pre-trained large language model. Large language models are typically based on the Transformer architecture and utilize self-attention mechanisms to understand and process the complex relationships between words in text, enabling the understanding and generation of natural language text, such as question-and-answer and translation. In the process of generating natural language text, one word is generated at a time, and the preceding context is considered each time a new word is generated.

[0055] Specifically, since the target model's training samples include at least prompts, an inference length threshold, and a standard answer, model optimization is constrained in terms of both inference length and answer accuracy. Therefore, through iterative optimization of the inference model to be optimized, the final target inference model can output the correct answer within the specified inference length. That is, the target inference model can obtain accurate inference answers while ensuring that the inference process is not too long, thus avoiding waste of inference resources.

[0056] Based on the above embodiments, as an implementable approach, in one embodiment, the method further includes:

[0057] Step 301: Obtain the original model training sample set; wherein each original model training sample in the original model training sample set includes at least a prompt word and a standard answer;

[0058] Step 302: For any original model training sample, determine the inference length threshold based on the standard inference length and preset fluctuation value corresponding to the standard answer in the original model training sample;

[0059] Step 303: Add the inference length threshold to the original model training samples to obtain the target model training samples.

[0060] Specifically, first, obtain the original model training sample set D, which contains N original model training samples. Each sample in D contains both a prompt and a standard answer. As a prompt word, Enter the standard answer corresponding to the prompt word.

[0061] Then, the existing fine-tuning dataset was modified. To achieve length control, each input prompt word... The modification process involves adding a target length instruction (inference length threshold) to the end. The specific form is as follows: to indicate passage Each token completes the reasoning, among which... , This represents the preset fluctuation value, used to characterize the minimum allowable deviation in the model's inference length. for The number of tokens, i.e. Indicates the standard inference length. ,Right now For from 0 and Integers sampled uniformly from the middle By setting a preset value, a new dataset is obtained. This means obtaining the training sample set for the target model.

[0062] Based on the above embodiments, as an implementable approach, in one embodiment, the inference model to be optimized is iteratively optimized based on the target model training sample set until the target inference model is obtained, including:

[0063] Step 20121: For any target model training sample in the target model training sample set, obtain the prompt words of the reasoning model to be optimized in response to the target model training sample, and determine the sample reasoning process and sample reasoning answer.

[0064] Step 20122: Determine the current comprehensive reward of the reasoning model to be optimized based on the length of the sample reasoning process, the sample reasoning answer, the reasoning length threshold, and the standard answer.

[0065] Step 20123: Based on the current comprehensive reward of the inference model to be optimized, iteratively optimize the inference model to be optimized until the target inference model is obtained.

[0066] It should be noted that the current comprehensive reward is a reward value calculated based on the correctness of the combined sample reasoning answer and the compliance of the sample reasoning process length, and is used to guide the direction and intensity of model parameter adjustments.

[0067] Specifically, after the inference model to be optimized has been iteratively optimized based on the training sample set of the target model, a test sample set can be obtained when the preset threshold of the iteration optimization rounds is reached, so as to verify whether the iteratively optimized model can be used as the target inference model.

[0068] Specifically, the primary indicator of the long thought chain model's task capability remains ensuring the correctness of the reasoning process. Based on this, completing the reasoning within a user-acceptable fixed output sequence length is considered optimal, with the further away from the fixed limit, the better. Conversely, exceeding this limit is considered a situation requiring suppression and restriction. Therefore, the iteratively optimized model can be scored based on the following formula:

[0069]

[0070] in, This represents the scoring result of the iteratively optimized model. When the scoring result reaches the preset standard, the model is determined to be a suitable target inference model. This indicates the number of test samples contained in the test sample set. Indicates that it is for the test sample The test results Represents the natural base. Indicates that it is for the test sample The actual length of the reasoning process, that is, the length of the output sequence of a long thought chain model. Indicates test sample The inference length value, i.e. The model output sequence length is defined as the length acceptable to the user. That is, if the model output sequence exceeds the user-accepted sequence length, even if the large model outputs the correct answer, its capability index will still be less than 1, and the greater the exceedance, the closer it gets to 0. When the model output sequence is exactly equal to the user-accepted sequence length and the large model outputs the correct answer, the capability index is 1. When the model completes reasoning within the user-accepted sequence length and obtains the correct result, its capability index is greater than 1. On the other hand, if the large model cannot obtain the correct answer, its capability index score is 0. Furthermore, multiple samples are averaged to ensure the positive definiteness of the capability assessment results.

[0071] Specifically, in one embodiment, the current length reward of the reasoning model to be optimized can be determined based on the deviation between the length of the sample reasoning process and the reasoning length threshold; the current correctness reward of the reasoning model to be optimized can be determined based on the deviation between the sample reasoning answer and the standard answer; and the current comprehensive reward of the reasoning model to be optimized can be determined based on the current length reward and the current correctness reward.

[0072] Specifically, a reward function can be defined. That is, the reward function consists of two parts: correctness reward. and length bonus .in, This represents the sample reasoning answer. This represents the standard answer. Indicates the length of the sample reasoning process. This indicates the inference length threshold.

[0073] in, , It is a bivariate function, if If the value is 1, then its value is 1; otherwise, it is 0.

[0074] Specifically, in one embodiment, the unverified length reward of the inference model to be optimized can be determined based on the deviation between the length of the sample inference process and the inference length threshold; whether the unverified length reward belongs to a preset value range can be verified; if the unverified length reward belongs to the preset value range, the unverified length reward can be used as the current length reward.

[0075] The reward for the length to be verified is set to a value of 1. , , A constant used for scaling length bonuses. 0 < <1 is a constant used to ensure that even if an answer has a slight resource constraint excess, it is still favored over an incorrect answer as long as it is correct, in order to focus on ensuring the accuracy of the model's reasoning.

[0076] Accordingly, in one embodiment, if the reward for the length to be verified is lower than a preset value range, the preset lower limit of the preset value range can be used as the current length reward.

[0077] Among them, the preset lower limit value is negatively correlated with the degree of requirement for the correctness of the reasoning answer in the model iterative optimization.

[0078] Accordingly, in one embodiment, if the reward for the length to be verified is higher than a preset value range, the preset upper limit of the preset value range can be used as the current length reward.

[0079] Among them, the preset upper limit value is positively correlated with the length requirement of the inference process for model iteration optimization.

[0080] Specifically, ,in, This represents the lower limit of a preset value range. This indicates the upper limit of the preset value range. and This is a constant used to adjust the trade-off between generating the correct answer and meeting the length threshold. In practical applications, a lower value is preferred. When correctness is paramount, higher values ​​are prioritized for generating the correct answer, while lower values ​​are preferred. The value then enforces length constraints more strictly. For truncation function, when The value is greater than hour, The value of the function is ;when The value is less than hour, The value of the function is .

[0081] Through this reward function, the model learns during training to generate correct answers while minimizing the length of the inference sequence to approach the target length (length threshold). This design also implements a soft constraint on the output length, gradually penalizing outputs exceeding the target length rather than directly truncating them, ensuring proper gradient propagation in reinforcement learning. Therefore, the reward function serves a dual purpose: firstly, it encourages the model to generate correct answers while implicitly favoring concise inference paths when shorter outputs are requested; secondly, it continuously incentivizes the model to match the specified target length even when fewer tokens (terms) can be used to generate the correct answer.

[0082] Based on the above embodiments, as an implementable approach, in one embodiment, for any candidate word to be generated, the target log probability of the candidate word to be generated is determined according to the initial log probability corresponding to the candidate word, the preset logic switching word set, the preset log probability correction constraint, and the distance between the current inference position and the inference start, including:

[0083] Step 2031: For any candidate word to be generated, determine whether the candidate word to be generated belongs to the preset idea switching word set;

[0084] Step 2032: If the candidate word to be generated belongs to the preset reasoning switching word set, determine the target log probability of the candidate word to be generated based on the initial log probability corresponding to the candidate word to be generated, the preset log probability correction constraint, and the distance between the current reasoning position and the reasoning start.

[0085] It should be noted that the fundamental purpose of modifying the log odds of the candidate lexical units to be generated in this application embodiment is to suppress the ability to switch between different reasoning strategies multiple times in the later stages of the reasoning process, thereby enabling more flexible and efficient handling of the computational cost of reasoning.

[0086] Accordingly, in one embodiment, if the candidate word to be generated does not belong to the preset idea switching word set, the initial log odds corresponding to the candidate word to be generated are used as the target log odds.

[0087] Specifically, if the candidate word to be generated does not belong to the preset idea switching word set, the log probability of the candidate word to be generated will not be corrected to avoid wasting log probability correction resources.

[0088] Specifically, in one embodiment, the inference progress correction value corresponding to the current inference position can be determined based on the preset inference progress correction value constraint and the distance between the current inference position and the inference start point; it can be determined whether the absolute value of the initial logarithmic probability of the candidate word to be generated is less than the preset standard threshold; if the absolute value of the initial logarithmic probability is not less than the preset standard threshold, the initial logarithmic probability is determined as the multiplication penalty standard; the first penalty coefficient is determined based on the positive or negative sign of the initial logarithmic probability; and the target logarithmic probability of the candidate word to be generated is determined based on the multiplication penalty standard, the inference progress correction value, the first penalty coefficient, and the initial logarithmic probability.

[0089] Among them, the preset logarithmic probability correction constraint includes the preset inference progress correction value constraint and the preset standard definition value.

[0090] It should be noted that the reasoning progress correction value determines the strength of the logarithmic probability correction, also known as the penalty intensity. The penalty intensity is reduced in the early stages of reasoning to encourage exploration, while the penalty intensity is increased in the later stages to suppress strategy switching. The preset standard threshold is a threshold used to judge the initial logarithmic probability of candidate words to be generated, distinguishing the initial generation tendency of that word in the model, and thus selecting different correction methods. The first penalty coefficient is a coefficient set based on the positive or negative initial logarithmic probability. Positive logarithmic probabilities (prone to generation) are weakened, while negative logarithmic probabilities (prone to non-generation) are strengthened, to further precisely control the generation probability of strategy-switching words.

[0091] Specifically, in one embodiment, the inference progress correction value corresponding to the current inference position is determined based on the preset inference progress correction upper limit value, the preset inference progress correction lower limit value, the inference length upper limit value, and the distance between the current inference position and the inference start.

[0092] The preset inference progress correction value constraints include the preset inference progress correction upper limit, the preset inference progress correction lower limit, and the preset inference length upper limit.

[0093] It should be noted that the preset inference progress correction upper limit represents the upper limit of the correction intensity for logic switching terms, which can suppress logic switching with the strongest intensity in the later stages of inference. The preset inference progress correction lower limit represents the lower limit of the correction intensity for logic switching terms, which can allow logic exploration with the weakest intensity in the early stages of inference. The inference length upper limit is a quantitative indicator of resource constraints, used to limit the total number of terms generated during the inference process (inference length) from exceeding this value, otherwise it will lead to wasted computing power, etc.

[0094] Specifically, in one embodiment, the inference progress correction value corresponding to the current inference position can be determined based on the following formula:

[0095]

[0096] in, This indicates the reasoning progress correction value corresponding to the current reasoning position. This indicates the preset upper limit for inference progress correction. Preset weights, This indicates the preset lower limit for inference progress correction. This indicates the distance between the current inference position and the starting point of the inference.

[0097] Specifically, the farther the current position t is from the starting position, that is, the farther position t is from the maximum acceptable output sequence length set by the user. The closer, The larger the value, the stronger the reduction in the probability of the corresponding token appearing. Therefore, the effect is that in the early stages of inference, the model is relatively encouraged to generate tokens related to thought switching and path verification. As the model's output sequence increases, the penalty for tokens related to thought switching and path verification gradually increases, guiding the model to perform serial inference and obtain the final conclusion.

[0098] Furthermore, in one embodiment, the first penalty coefficient can be determined based on the following formula:

[0099]

[0100] in, Indicates the first penalty coefficient. This indicates the preset first penalty coefficient. The preset logarithmic probability correction constraint also includes a preset first penalty coefficient. Indicates candidate lexical units to be generated The corresponding initial logarithmic probability. The first penalty coefficient is preset to represent the reward / penalty intensity; the larger the value, the greater the severity.

[0101] Furthermore, in one embodiment, the target log-probability of the candidate lexical units to be generated can be determined based on the following formula:

[0102]

[0103] in, Indicates candidate lexical units to be generated The target log odds, This indicates the correction value for the reasoning progress. Indicates the first penalty coefficient. Indicates candidate lexical units to be generated The corresponding initial logarithmic odds.

[0104] It should be noted that when the candidate word w to be generated belongs to the preset approach switching word set, and hour, This indicates a preset standard threshold value. The model has a strong tendency to generate this word. In this case, a multiplicative penalty standard is used to achieve strong correction, that is, a multiplicative penalty, which is stronger for large numbers, is used for punishment.

[0105] Accordingly, in one embodiment, if the absolute value of the initial log odds is less than a preset standard threshold, the initial log odds are determined as a subtraction penalty standard; based on the subtraction penalty standard, the target log odds of the candidate lexical to be generated are determined according to the inference progress correction value, the second penalty coefficient, and the initial log odds.

[0106] The preset logarithmic probability correction constraint also includes a second penalty coefficient.

[0107] It should be noted that when the candidate word w to be generated belongs to the preset approach switching word set, and When the model has a weak tendency to generate this word, a subtraction penalty standard is used to achieve strong correction, that is, a subtraction penalty with stronger penalty for decimals is used.

[0108] Specifically, in one embodiment, the target log-probability of the candidate lexical units to be generated can be determined based on the following formula:

[0109]

[0110] in, Indicates candidate lexical units to be generated The target log odds, This indicates the correction value for the reasoning progress. This represents the second penalty coefficient. , representing the baseline value for logarithmic odds correction. The higher the value, the stronger the reward / punishment. Indicates candidate lexical units to be generated The corresponding initial logarithmic odds.

[0111] Specifically, the comprehensive expression for the target log-probability of the candidate lexical units to be generated is as follows:

[0112]

[0113] in, This represents the preset set of tokens for switching thought processes. Through the above comprehensive expression, the probability of tokens appearing related to thought process switching and path verification is adjusted, achieving an initial reward and later penalty effect. The penalty considers the influence of the logarithmic probability of the token at that position; that is, a stronger multiplicative penalty is applied for larger probabilities, and a stronger subtractive penalty is applied for smaller probabilities. The influence of position t is also considered. When position t is in the early stages of inference (i.e., the distance between the current inference position and the inference start is less than a preset distance threshold), the model is allowed to switch thought processes more flexibly. When a large number of tokens have already been generated before position t (i.e., the distance between the current inference position and the inference start is not less than the preset distance threshold), the penalty is increased, reducing the possibility of switching thought processes again. By changing the position where thought process switching tokens tend to appear through rewards / penalties, this prevents the inability to obtain the corresponding conclusion within the user's required computing resources, while ensuring sufficient initial exploration opportunities for inference and guaranteeing the accuracy of long thought chain inference.

[0114] The model inference method provided in this application obtains a target inference model; during the inference process of the target inference model in response to the user-input inference question, it determines the initial log odds corresponding to multiple candidate words to be generated based on the generated inference process; wherein the generated inference process includes multiple words generated in sequence; for any candidate word to be generated, it determines the target log odds of the candidate word to be generated based on the initial log odds corresponding to the candidate word to be generated, the preset set of word words for switching reasoning approaches, the preset log odds correction constraints, and the distance between the current inference position and the inference start; based on the target log odds of each candidate word to be generated, it determines the target word to be generated, so as to obtain the target inference process of the target inference model. The method provided by the above scheme, because it integrates multiple information during the model inference process and corrects the log odds of candidate words to be generated, avoids the model switching reasoning approaches in the later stages of inference, thus avoiding the inference process being too long due to frequent switching of reasoning approaches, ensuring the accuracy of the model inference results while avoiding the waste of model inference resources. Furthermore, since the target model training samples used for model optimization include prompt words, inference length thresholds, and standard answers, model optimization is constrained in terms of both inference length and answer accuracy. This ensures that the final target inference model can obtain accurate inference answers while also preventing the inference process from becoming too long, thus avoiding the waste of inference resources.

[0115] Furthermore, this paper addresses the uncontrollable reasoning cost problem inherent in long-thinking-chain models due to their path switching and multiple verification characteristics. It defines the correspondence between reasoning in long-thinking-chain models under resource constraints and the length of the reasoning sequence tokens, and based on this, defines a capability evaluation standard for long-thinking-chain models under constrained conditions. This standard considers both reasoning accuracy and efficiency. To further address the reasoning problem of long-thinking-chain models under resource constraints, the paper proposes solutions at both the model training and inference levels. At the training level, a method for modifying training samples is proposed, using fine-tuning to enable the model to perform correct reasoning under the constraint of limited model sequence outputs. Furthermore, a continuous reinforcement learning reward method is defined to guide the model to balance reasoning accuracy and resource constraints, further enhancing its reasoning ability under limited sequence outputs. On the other hand, regarding inference, a reward-and-penalty-based reasoning strategy is proposed, using initial rewards and later penalties to ensure the model completes reasoning under limited output conditions. Through these methods, the model can achieve high-precision reasoning cost control in various reasoning tasks. That is, it can intelligently adjust the thinking chain to adapt to specified resource constraints, i.e., length constraints, without disrupting the reasoning process. This ability to flexibly adjust inference strategies under different resource constraints enables the model to effectively utilize computational resources and improve inference efficiency while ensuring inference accuracy.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0117] Embodiments of this application also provide a model reasoning apparatus for executing the model reasoning method provided in the above embodiments.

[0118] like Figure 3 The diagram shown is a schematic representation of the model inference device provided in an embodiment of this application. The model inference device 30 includes: an acquisition module 301, a first determination module 302, a second determination module 303, and an inference module 304.

[0119] The system comprises the following modules: an acquisition module for acquiring the target inference model; a first determination module for determining the initial log probabilities of multiple candidate words to be generated based on the generated inference process during the inference process of the target inference model responding to the user-input inference question; wherein the generated inference process includes multiple words generated in sequence; a second determination module for determining the target log probabilities of any candidate word to be generated based on the initial log probabilities of the candidate word to be generated, a preset set of word words for switching ideas, preset log probabilities correction constraints, and the distance between the current inference position and the inference start point; and an inference module for determining the target word to be generated based on the target log probabilities of each candidate word to be generated, so as to obtain the target inference process of the target inference model.

[0120] For a description of the features in the embodiment corresponding to the model inference device, please refer to the relevant description of the embodiment corresponding to the model inference method, which will not be repeated here.

[0121] Embodiments of this application also provide an electronic device, such as... Figure 4 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 10 and a memory 20. The memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-described model reasoning method embodiments.

[0122] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model inference method embodiments at runtime.

[0123] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0124] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model reasoning method embodiments.

[0125] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model reasoning method embodiments.

[0126] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] The foregoing has provided a detailed description of the model reasoning method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model reasoning method, characterized in that, include: Obtain the target inference model; During the process of the target reasoning model performing reasoning in response to the user-input reasoning question, the initial log odds corresponding to multiple candidate words to be generated are determined based on the generated reasoning process; wherein, the generated reasoning process includes multiple words that have been generated in sequence; For any of the candidate words to be generated, the target log odds of the candidate words to be generated are determined based on the initial log odds corresponding to the candidate words to be generated, the preset logic switching word set, the preset log odds correction constraints, and the distance between the current reasoning position and the reasoning start. Based on the target log odds of each candidate word to be generated, the target word to be generated is determined, so as to obtain the target reasoning process of the target reasoning model; For any of the candidate words to be generated, the target log odds of the candidate words to be generated are determined based on the initial log odds corresponding to the candidate word, the preset logic switching word set, the preset log odds correction constraints, and the distance between the current inference position and the inference start point, including: For any of the candidate words to be generated, determine whether the candidate word to be generated belongs to the preset idea switching word set; When the candidate word to be generated belongs to the preset logic switching word set, the target log probability of the candidate word to be generated is determined based on the initial log probability corresponding to the candidate word to be generated, the preset log probability correction constraint, and the distance between the current inference position and the inference start.

2. The model reasoning method according to claim 1, characterized in that, The method further includes: If the candidate lexicon to be generated does not belong to the preset idea switching lexicon set, the initial log probability corresponding to the candidate lexicon to be generated is taken as the target log probability.

3. The model reasoning method according to claim 1, characterized in that, The step of determining the target log odds of the candidate words to be generated based on the initial log odds corresponding to the candidate words to be generated, the preset log odds correction constraints, and the distance between the current inference position and the inference start point includes: Based on the preset inference progress correction value constraint and the distance between the current inference position and the inference start, determine the inference progress correction value corresponding to the current inference position; Determine whether the absolute value of the initial logarithmic probability of the candidate lexical to be generated is less than a preset standard threshold value; If the absolute value of the initial logarithmic probability is not less than a preset standard threshold, the initial logarithmic probability is determined to be the multiplication penalty standard. The first penalty coefficient is determined based on the sign of the initial logarithmic probability. Based on the multiplication penalty criterion, the target log odds of the candidate lexical to be generated are determined according to the reasoning progress correction value, the first penalty coefficient, and the initial log odds. The preset logarithmic probability correction constraint includes a preset inference progress correction value constraint and a preset standard definition value.

4. The model reasoning method according to claim 3, characterized in that, The step of determining the inference progress correction value corresponding to the current inference position based on the preset inference progress correction value constraint and the distance between the current inference position and the inference start point includes: The inference progress correction value corresponding to the current inference position is determined based on the preset inference progress correction upper limit, the preset inference progress correction lower limit, the inference length upper limit, and the distance between the current inference position and the inference start. The preset inference progress correction value constraint includes the preset inference progress correction upper limit, the preset inference progress correction lower limit, and the preset inference length upper limit.

5. The model reasoning method according to claim 4, characterized in that, The step of determining the inference progress correction value corresponding to the current inference position based on the preset inference progress correction upper limit, the preset inference progress correction lower limit, the inference length upper limit, and the distance between the current inference position and the inference start point includes: The inference progress correction value corresponding to the current inference position is determined based on the following formula: in, This indicates the reasoning progress correction value corresponding to the current reasoning position. This indicates the preset upper limit value for inference progress correction. Preset weights, This indicates the preset lower limit value for inference progress correction. This indicates the distance between the current inference position and the inference start point.

6. The model reasoning method according to claim 3, characterized in that, The step of determining the first penalty coefficient based on the sign of the initial logarithmic probability includes: The first penalty coefficient is determined based on the following formula: in, Indicates the first penalty coefficient. This indicates the preset first penalty coefficient. The preset logarithmic probability correction constraint also includes the preset first penalty coefficient. Indicates candidate words to be generated The corresponding initial logarithmic odds.

7. The model reasoning method according to claim 3, characterized in that, The determination of the target log-probability of the candidate lexical units to be generated, based on the multiplication penalty criterion, the inference progress correction value, the first penalty coefficient, and the initial log-probability, includes: The target log-probability of the candidate lexical units to be generated is determined based on the following formula: in, Indicates candidate words to be generated The target log odds, This represents the inference progress correction value. This represents the first penalty coefficient. Indicates candidate words to be generated The corresponding initial logarithmic odds.

8. The model reasoning method according to claim 3, characterized in that, The method further includes: If the absolute value of the initial logarithmic probability is less than a preset standard threshold, the initial logarithmic probability is determined to be the subtraction penalty standard. Based on the subtraction penalty standard, the target log odds of the candidate lexical to be generated are determined according to the reasoning progress correction value, the second penalty coefficient, and the initial log odds. The preset logarithmic probability correction constraint also includes the second penalty coefficient.

9. The model reasoning method according to claim 8, characterized in that, The determination of the target log odds of the candidate lexical units to be generated, based on the subtraction penalty criterion, the inference progress correction value, the second penalty coefficient, and the initial log odds, includes: The target log-probability of the candidate lexical units to be generated is determined based on the following formula: in, Indicates candidate words to be generated The target log odds, This represents the inference progress correction value. This represents the second penalty coefficient. Indicates candidate words to be generated The corresponding initial logarithmic odds.

10. The model reasoning method according to claim 1, characterized in that, The acquisition of the target inference model includes: Obtain the training sample sets of the inference model to be optimized and the target model; wherein, each training sample of the target model in the training sample set includes at least a prompt word, an inference length threshold, and a standard answer; Based on the training sample set of the target model, the inference model to be optimized is iteratively optimized until the target inference model is obtained.

11. The model reasoning method according to claim 10, characterized in that, The method further includes: Obtain the original model training sample set; wherein, each original model training sample in the original model training sample set includes at least a prompt word and a standard answer; For any of the original model training samples, the inference length threshold is determined based on the standard inference length and preset fluctuation value corresponding to the standard answer in the original model training sample. The inference length threshold is added to the original model training samples to obtain the target model training samples.

12. The model reasoning method according to claim 10, characterized in that, The step of iteratively optimizing the inference model to be optimized based on the training sample set of the target model until the target inference model is obtained includes: For any target model training sample in the target model training sample set, obtain the prompt words in the target model training sample in response to the reasoning model to be optimized, and determine the sample reasoning process and sample reasoning answer; The current comprehensive reward of the reasoning model to be optimized is determined based on the length of the sample reasoning process, the sample reasoning answer, the reasoning length threshold, and the standard answer. Based on the current comprehensive reward of the inference model to be optimized, the inference model to be optimized is iteratively optimized until the target inference model is obtained.

13. The model reasoning method according to claim 12, characterized in that, The step of determining the current comprehensive reward of the reasoning model to be optimized based on the length of the sample reasoning process, the sample reasoning answer, the reasoning length threshold, and the standard answer includes: The current length reward of the inference model to be optimized is determined based on the deviation between the length of the sample inference process and the inference length threshold. Based on the deviation between the sample reasoning answer and the standard answer, the current correctness reward of the reasoning model to be optimized is determined; The current comprehensive reward of the inference model to be optimized is determined based on the current length reward and the current correctness reward.

14. The model reasoning method according to claim 13, characterized in that, Determining the current length reward of the inference model to be optimized based on the deviation between the length of the sample inference process and the inference length threshold includes: The unverified length reward of the inference model to be optimized is determined based on the deviation between the length of the sample inference process and the inference length threshold. Verify whether the length reward to be verified belongs to a preset value range; If the reward for the length to be verified falls within a preset value range, the reward for the length to be verified will be used as the current length reward.

15. The model reasoning method according to claim 14, characterized in that, The method further includes: If the reward for the length to be verified is lower than the preset value range, the preset lower limit of the preset value range will be used as the current length reward. The preset lower limit value is negatively correlated with the degree of requirement for the correctness of the reasoning answer in the model iterative optimization.

16. The model reasoning method according to claim 14, characterized in that, The method further includes: If the reward for the length to be verified is higher than the preset value range, the preset upper limit of the preset value range will be used as the current length reward. The preset upper limit value is positively correlated with the length requirement of the inference process for model iteration optimization.

17. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model inference method as described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model inference method as described in any one of claims 1 to 16.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the model reasoning method as described in any one of claims 1 to 16.

Images