Model training method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a model training method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: reasoning a sample problem through a reasoning model to obtain a reasoning result; determining a corresponding reasoning length control hyper-parameter according to the difficulty level category of the sample problem; constructing a reasoning length reward function according to the reasoning length control hyper-parameter and the reasoning length of the reasoning result, and constructing a reasoning accuracy reward function according to the reasoning result; and performing model training based on reinforcement learning on the reasoning model according to the reasoning length reward function and the reasoning accuracy reward function. The reasoning model trained by the method can give consideration to reasoning efficiency and accuracy, and a more efficient and accurate reasoning process can be realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing and reasoning tasks, the model's reasoning capabilities have been significantly improved. In particular, in complex reasoning tasks, such as mathematical problem solving and programming, approaches based on Chain-of-Thought (CoT) are widely adopted. This approach gradually unfolds reasoning steps to generate a clear chain of reasoning, thereby improving the accuracy and reliability of problem solving.

[0003] However, while chain thinking effectively enhances reasoning capabilities, it also introduces a significant problem: overthinking. During the reasoning process of long chain thinking, the model often generates too many redundant steps, resulting in inefficient reasoning, significant waste of computing resources, and even reduced reasoning speed. This is particularly true in resource-constrained scenarios, such as real-time systems and mobile devices. This phenomenon not only increases computing costs but can also lead to excessively long reasoning times, affecting the model's responsiveness and limiting its feasibility in practical applications. Summary of the Invention

[0004] Based on this, it is necessary to provide a model training method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems. The trained inference model can take into account both inference efficiency and accuracy, and can achieve a more efficient and accurate inference process.

[0005] In a first aspect, the present application provides a model training method, comprising:

[0006] Reasoning the sample problem through the reasoning model to obtain the reasoning result;

[0007] Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem;

[0008] Controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result;

[0009] According to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning.

[0010] In a second aspect, the present application also provides a model training device, comprising:

[0011] The reasoning module is used to reason about sample problems through the reasoning model to obtain reasoning results;

[0012] A hyperparameter determination module, configured to determine a corresponding inference length control hyperparameter according to the difficulty level of the sample problem;

[0013] a reward function construction module, configured to control a hyperparameter and an inference length of the inference result according to the inference length, construct an inference length reward function, and construct an inference accuracy reward function according to the inference result;

[0014] A model updating module is used to perform reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function.

[0015] In an exemplary embodiment, the reasoning module is further configured to perform multiple reasonings on the same sample problem using the reasoning model to obtain multiple reasoning results; the device further includes:

[0016] A difficulty evaluation module is used to determine the difficulty category of the sample problem based on the multiple inference results.

[0017] In an exemplary embodiment, the difficulty evaluation module is further configured to collect statistics on the accuracy of the multiple inference results; and determine the difficulty category of the sample problem based on the accuracy of the multiple inference results.

[0018] In an exemplary embodiment, the difficulty assessment module is further used to obtain a sample answer corresponding to the sample question; when the inference result and the sample answer corresponding to the sample question meet a consistency condition, the inference result is determined to be a correct inference result; and the accuracy of the multiple inference results is counted based on the number of correct inference results in the multiple inference results and the total number of inferences of the multiple inferences.

[0019] In an exemplary embodiment, the reasoning module is further configured to perform reasoning on the same sample problem using multiple reasoning models to obtain multiple reasoning results; the device further comprises:

[0020] A difficulty evaluation module is used to determine the difficulty category of the sample problem based on the multiple inference results.

[0021] In an exemplary embodiment, the apparatus further comprises:

[0022] The difficulty assessment module is used to search historical reasoning problems similar to the sample problem from historical reasoning records; determine the approximate reasoning length of the sample problem based on the reasoning length of the reasoning results of the similar historical reasoning problems; and determine the difficulty category of the sample problem based on the approximate reasoning length.

[0023] In an exemplary embodiment, the difficulty assessment module is further used to identify the category label of the sample problem when there is no historical reasoning problem similar to the sample problem in the historical reasoning record, and determine the difficulty category of the sample problem based on the average reasoning length corresponding to the category label.

[0024] In an exemplary embodiment, the inference length reward function is negatively correlated with the inference length of the inference result; the hyperparameter determination module is further used to determine that the inference length control hyperparameter corresponding to the sample problem is λ1 when the sample problem belongs to the complex problem category; when the sample problem belongs to the medium difficulty problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ2; when the sample problem belongs to the simple problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ3; wherein, λ1>λ2>λ3>0.

[0025] In an exemplary embodiment, the reward function construction module is further used to accumulate the inference lengths of each of the multiple inference results to obtain a total inference length; control the hyperparameters and the total inference length according to the inference length to construct the inference length reward function, and the inference length reward function is negatively correlated with the total inference length.

[0026] In an exemplary embodiment, the reward function construction module is further used to obtain sample answers corresponding to the sample questions; and construct a reasoning accuracy reward function based on the multiple reasoning results and the sample answers.

[0027] In an exemplary embodiment, the model update module is further used to determine the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem; perform weighted summation of the reasoning length reward function and the reasoning accuracy reward function according to the corresponding reward weights to obtain a multi-objective reward function; and update the model parameters of the reasoning model according to the multi-objective reward function.

[0028] In an exemplary embodiment, the reward weight of the inference accuracy reward function is α1, the reward weight of the inference length reward function is α2, and α1+α2=1; when the sample problem belongs to the complex problem category, set α2>α1>0; when the sample problem belongs to the simple problem category, set α1>α2>0.

[0029] In one embodiment, the reasoning module is further used to obtain an input question, and to reason on the input question using a trained reasoning model to obtain a corresponding reasoning result.

[0030] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Reasoning the sample problem through the reasoning model to obtain the reasoning result;

[0032] Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem;

[0033] Controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result;

[0034] According to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] Reasoning the sample problem through the reasoning model to obtain the reasoning result;

[0037] Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem;

[0038] Controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result;

[0039] According to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning.

[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0041] Reasoning the sample problem through the reasoning model to obtain the reasoning result;

[0042] Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem;

[0043] Controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result;

[0044] According to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning.

[0045] The above-mentioned model training method, apparatus, computer device, computer-readable storage medium, and computer program product use an inference model to infer sample problems to obtain inference results; determine the corresponding inference length control hyperparameter according to the difficulty category of the sample problem; construct an inference length reward function based on the inference length control hyperparameter and the inference length of the inference result, and construct an inference accuracy reward function based on the inference result; and perform reinforcement learning-based model training on the inference model based on the inference length reward function and the inference accuracy reward function. In other words, during the reinforcement learning training process, the inference length control hyperparameter is automatically adjusted according to the difficulty category of the sample problem to enable the inference model to automatically adjust the inference length for problems of different difficulty levels. By introducing the difficulty category of the sample problem, the inference model is encouraged to reduce lengthy and invalid inference steps when generating accurate inference steps, thereby achieving the goal of maintaining inference accuracy while significantly improving inference efficiency, thereby achieving a more efficient and accurate inference process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A diagram showing an application environment of a model training method in one embodiment;

[0048] Figure 2 Schematic diagram of a flow chart of a model training method in one embodiment;

[0049] Figure 3 A schematic flow chart of a model training method in another embodiment;

[0050] Figure 41 is a flow chart of the training steps of an inference model in one embodiment;

[0051] Figure 5 A schematic diagram of a change in inference length in one embodiment;

[0052] Figure 6 1 is a flow chart of the training steps of the inference model in another embodiment;

[0053] Figure 7 A flowchart of a model training method is shown as a detailed embodiment;

[0054] Figure 8 is a structural block diagram of a model training device in one embodiment;

[0055] Figure 9 is a diagram of the internal structure of a computer device in one embodiment;

[0056] Figure 10 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] Although the reasoning models in related technologies have achieved remarkable results in handling complex reasoning tasks, some problems still exist, such as:

[0059] Lengthy and inefficient reasoning: While current reasoning models can generate detailed reasoning chains when handling complex problems, they can also lead to "overthinking." When answering simple questions, reasoning models often generate too many redundant reasoning steps, resulting in a lengthy reasoning process, increased computing resource consumption, and increased reasoning time. This lengthy reasoning output significantly limits the model's applicability, especially in scenarios with limited computing resources.

[0060] Lack of ability to dynamically adjust inference length: The reasoning models used in related technologies fail to adaptively adjust the inference length, or the number of inference steps, based on the difficulty of the problem. For simple problems, the inference models fail to effectively generate a concise reasoning process, instead generating lengthy inference steps. For complex problems, the inference models may not be able to generate a sufficiently long inference chain in response to the difficulty of the problem, thus affecting the final inference results and failing to achieve a good balance between inference efficiency and accuracy.

[0061] Inability to accurately assess problem difficulty: Related technologies are typically designed based on predetermined rules or empirical evidence, lacking effective mechanisms for assessing the difficulty of input problems. Obviously, the difficulty of different problems varies significantly, and related technologies often adopt a unified reasoning strategy, failing to flexibly adjust reasoning based on the difficulty of the problem. This results in reasoning models being inefficient when faced with tasks of varying difficulty, or lacking sufficient reasoning depth for complex tasks, making it impossible to obtain accurate reasoning results.

[0062] Therefore, when dealing with problems of different difficulty levels, the reasoning models of related technologies have problems such as low reasoning efficiency, waste of resources and insufficient reasoning accuracy. These problems are particularly prominent in application scenarios with limited resources or high real-time requirements.

[0063] The model training method provided in the embodiment of the present application uses an inference model to infer sample problems to obtain inference results; determines the corresponding inference length control hyperparameter according to the difficulty category of the sample problem; constructs an inference length reward function based on the inference length control hyperparameter and the inference length of the inference result, and constructs an inference accuracy reward function based on the inference result; and performs reinforcement learning-based model training on the inference model based on the inference length reward function and the inference accuracy reward function. That is, during the reinforcement learning training process, the inference length control hyperparameter is automatically adjusted according to the difficulty category of the sample problem to enable the inference model to automatically adjust the inference length for problems of different difficulty levels. By introducing the difficulty category of the sample problem, the inference model is encouraged to reduce lengthy and invalid inference steps when generating accurate inference steps, thereby achieving the purpose of maintaining inference accuracy while significantly improving inference efficiency, thereby achieving a more efficient and accurate inference process.

[0064] Explain some concepts involved in this application:

[0065] An inference model is a large language model with reasoning capabilities. It generates answers through multi-step reasoning when faced with complex task scenarios. The inference model can be a textual inference model.

[0066] Based on Chain-of-Thought (CoT), it generates a clear chain of reasoning by gradually unfolding the reasoning steps. It imitates a step-by-step thinking process to get the answer.

[0067] Reinforcement Learning (RL) is a machine learning paradigm that allows an agent to learn optimal strategies through interaction with its environment to maximize cumulative rewards. It adjusts strategies through trial and error to increase the probability of high-reward actions.

[0068] Hyperparameters are parameters set before model training. They are directly involved in the calculation of the loss function (or reward function) but do not participate in the update of gradient descent.

[0069] Inference length control hyperparameters, in this application, refer to the hyperparameters used to construct the inference length reward function. Setting appropriate values ​​for the hyperparameters of the inference length reward function can guide the inference model to make decisions at a reasonable inference length. Different difficulty levels have different corresponding inference length control hyperparameters. Rewards are given based on the inference length of the inference result. Longer inference results receive lower rewards, while concise inference results receive higher rewards. This reflects the need to avoid excessive inference in simple problems, while encouraging the generation of sufficient inference steps in complex problems.

[0070] Reward weight, in this embodiment of the application, refers to the loss weight of the reward function. In order to balance different tasks in multi-task learning, different reward weights are set for different reward functions. For example, for sample problems of simple problem categories (for example, simple mathematical operations), the value of the reward weight of the accuracy reward function is larger, so that the accuracy reward function dominates, thereby reducing the lengthy reasoning process. The model will prioritize the accuracy of reasoning and minimize unnecessary reasoning steps; for sample problems of complex problem categories (for example, programming problems or logical reasoning problems), the value of the reward weight of the accuracy reward function is smaller, thereby reducing the impact of the reasoning length reward function and allowing the reasoning model to generate more reasoning steps.

[0071] The model training method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. In some embodiments, the server 104 can obtain sample problems, and use the inference model to infer the sample problems to obtain inference results; according to the difficulty category of the sample problems, the corresponding inference length control hyperparameters are determined; according to the inference length control hyperparameters and the inference length of the inference results, an inference length reward function is constructed, and an inference accuracy reward function is constructed according to the inference results; according to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning to obtain an inference model that can dynamically respond to the complexity of the problem and adjust the inference length. In some embodiments, after obtaining the optimized trained inference model, the server 104 can receive input problems submitted from the terminal 102, and infer the input problems to obtain corresponding inference results.

[0072] Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0073] The inference model obtained by the model training method provided in the embodiment of the present application can be applied in many scenarios, especially in application scenarios with limited resources or high real-time requirements, such as text generation, translation, summary, basic knowledge question and answer, real-time interactive systems, online customer service, mobile devices, edge computing scenarios, etc.

[0074] The following also introduces some specific application scenarios of the reasoning model:

[0075] (1) Problem Solving: In disciplines such as mathematics, physics, and chemistry, reasoning models can handle complex calculations and proofs. For example, when solving difficult problems in calculus and linear algebra in advanced mathematics, reasoning models can gradually demonstrate the problem-solving process, from problem analysis, formula derivation to the calculation of the final result, providing students and researchers with clear problem-solving ideas. In physical problems, such as complex scenario analysis in mechanics and electromagnetism, reasoning models can combine physical laws and formulas to perform multi-step logical reasoning, helping users understand the nature of the problem and the solution.

[0076] (2) Logical reasoning and decision support: In business decision-making, legal analysis, strategic planning, and other fields, reasoning models can process fuzzy information, conduct cross-document reasoning, and make complex decisions. For example, in legal contract analysis, reasoning models can quickly extract key clauses, understand the logical relationships between clauses, and even accurately judge potential risks and impacts when faced with fuzzy information. In business strategic planning, reasoning models can comprehensively consider multiple factors such as market data, competitor information, and company resources, and provide decision makers with reasonable strategic advice and planning solutions through multi-step reasoning and analysis.

[0077] (3) Code generation and debugging: For complex programming tasks, the reasoning model can understand the code logic, generate high-quality code solutions, and perform code review and debugging. For example, when developing large software projects, the reasoning model can help developers quickly generate code frameworks, provide code optimization suggestions, and detect potential errors and vulnerabilities in the code. In multi-file code comparison and review, the reasoning model can accurately identify subtle changes in the code, ensure code quality and security, and improve development efficiency and code quality.

[0078] (4) Cross-modal reasoning and data analysis: When processing complex data involving multiple modalities (such as text, images, audio, etc.), reasoning models can perform cross-modal understanding and reasoning. For example, in image description generation, reasoning models can combine image content and related text information to generate accurate and creative image descriptions. In visual question answering tasks, reasoning models can understand the scenes and objects in the image, combine the semantic information of the question, perform multi-step reasoning, and give accurate answers. In complex data analysis tasks, reasoning models can comprehensively consider the characteristics and relationships of data from different modalities, conduct in-depth data mining and analysis, and provide users with valuable insights and decision support.

[0079] Figure 2 This is a flowchart of a model training method shown in an exemplary embodiment of the present application. The method can be executed by a computer device. The method can also be applied to Figure 1 The application environment shown in FIG. 1 , the computer device executing the method may be Figure 1The server 104 or terminal 102 in the application environment is shown.

[0080] In an exemplary embodiment, Figure 2 As shown, a model training method is provided, including the following steps 202 to 208. Among them:

[0081] Step 202: Inferring the sample problem through the inference model to obtain an inference result.

[0082] The inference model is a large language model with reasoning capabilities. When given a complex task, it generates an answer, or inference result, through multi-step reasoning. This model breaks down complex problems / tasks into parts that closely replicate the human thought process (e.g., trying and verifying different approaches until the optimal solution is found and outputting the result). This approach may increase inference time, but it exponentially increases the success rate and accuracy of understanding and handling complex tasks. The inference model can improve its generalization and robustness to new problems through reinforcement learning across multiple scenarios.

[0083] The computer device may obtain training samples, each of which includes a sample question and a sample answer corresponding to the sample question. In some embodiments, for each training sample, the computer device may input the sample question into an inference model, and use the inference model to perform inference on the sample question to obtain an inference result. In some embodiments, for each training sample, the computer device may input the sample question into the inference model multiple times, and use the inference model to perform multiple inferences on the same sample question to obtain multiple inference results. The inference model may perform inference on the same sample question multiple times, for example, 10 times, to obtain 10 inference results.

[0084] Step 204 : Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem.

[0085] The difficulty category of the sample question is the category to which the difficulty of the sample question belongs, and the difficulty category of the sample question can be one of a complex question category, a medium difficulty question category, and a simple question category.

[0086] To ensure that the reasoning model balances reasoning efficiency and accuracy, a more efficient and accurate reasoning process can be achieved. Specifically, the reasoning model should efficiently respond to simple problems, which require fewer reasoning steps and produce accurate results in a shorter time, improving the response speed in real-time systems. Furthermore, the reasoning model should appropriately reason for complex problems. Although complex problems require more reasoning steps, controlling the inference length can avoid unnecessary lengthy reasoning, reducing computation and processing time. To this end, an effective method for assessing the difficulty of a problem is needed.

[0087] In some embodiments, a computer device repeatedly infers the same sample problem using an inference model. Based on the multiple inference results, the inference accuracy can be calculated to determine the difficulty category of the sample problem. The higher the accuracy, the simpler the sample problem is considered, and conversely, the lower the accuracy, the more complex the sample problem is considered. Specifically, the inference model is used to perform multiple inferences on the same sample problem, obtaining multiple inference results. The accuracy of these multiple inference results is then calculated, and the difficulty category of the sample problem is determined based on the accuracy of these multiple inference results.

[0088] In some embodiments, counting the accuracy of the multiple inference results includes: obtaining a sample answer corresponding to the sample question; determining that the inference result is a correct inference result when the inference result and the sample answer corresponding to the sample question meet a consistency condition; and counting the accuracy of the multiple inference results based on the number of correct inference results in the multiple inference results and the total number of inferences of the multiple inferences.

[0089] For example,

[0090] Among them, Accuracy(q) represents the accuracy of multiple inference results obtained by performing multiple inferences on sample question q, Correct Responses is the number of times the inference model answers correctly, and Total Responses is the total number of inferences performed.

[0091] In some embodiments, the computer device may determine the difficulty category of the sample question based on the accuracy of multiple inference results, specifically by comparing the accuracy of multiple inference results with a set threshold to determine the difficulty category of the sample question. For example:

[0092] When Accuracy(q)≥90%, the sample problem belongs to the simple problem category;

[0093] When Accuracy(q)≤60%, the sample problem belongs to the complex problem category;

[0094] If Accuracy(q) is between 60% and 90%, the sample problem is considered to be of medium difficulty. It is understood that the threshold value can be set according to actual conditions, and the present embodiment does not limit this.

[0095] In some embodiments, a computer device can train a lightweight classification model by extracting text features from sample questions to obtain text features, and performing semantic analysis on the sample questions to obtain semantic analysis features. The trained lightweight classification model then classifies the sample questions by difficulty based on the text features and semantic analysis features, thereby determining the difficulty category to which they belong. Text features may include question length, lexical complexity (e.g., using TF-IDF or word frequency statistics), and syntactic structure. Semantic analysis features may include the subject matter of the question, the type of knowledge required to answer the question, and the type of sample questions. Examples of knowledge types required to answer questions include factual knowledge, analytical knowledge, and creative knowledge. Examples of sample question types include factual, explanatory, analytical, evaluative, and creative. In some embodiments, the computer device can perform supervised training on the classification model using training samples of known difficulty categories, enabling it to accurately classify questions by difficulty.

[0096] In some embodiments, the computer device may also use multiple different reasoning models to reason about the sample problem separately when the sample answer corresponding to the sample problem is known, obtain respective reasoning results, compare the respective reasoning results with the sample answer, calculate the accuracy of the multiple reasoning models, and classify the difficulty of the sample problem according to the accuracy. Specifically, the performance of the multiple different reasoning models varies. If most of the multiple different reasoning models output accurate reasoning results for the sample problem, that is, they unanimously agree that the sample problem is relatively simple, then the difficulty category of the sample problem can be determined to be a simple problem category. If most of the multiple different reasoning models output incorrect reasoning results for the sample problem, that is, they unanimously agree that the sample problem is relatively difficult, then the difficulty category corresponding to the sample problem can be determined to be a complex problem category. For example, for the same sample problem, if the accuracy of the multiple inference models is greater than 75%, the sample problem can be determined to be classified as a simple problem; if the accuracy of the multiple inference models is less than 60%, the sample problem can be determined to be a complex problem; and if the accuracy of the multiple inference models is between 60% and 75%, the sample problem can be considered to be a medium-difficulty problem. Optionally, the multiple different inference models may have different network structures, or the multiple different inference models may have the same network structure but different network parameters.

[0097] In some embodiments, the computer device can also use multiple different reasoning models to perform multiple reasoning on the sample problem, and obtain multiple reasoning results. For each reasoning model, the corresponding multiple reasoning results are compared with the sample answer, and the accuracy of the reasoning model is statistically analyzed. Then, the accuracy of the multiple different reasoning models can be obtained. At the same time, different weights are given to the accuracy of different reasoning models. For example, the better the performance of the reasoning model, the greater the corresponding weight. The accuracy of the multiple different reasoning models is then weighted and summed according to the corresponding weights to obtain a comprehensive accuracy. The difficulty category of the sample problem is determined based on the comprehensive accuracy.

[0098] In some embodiments, the computer device may also establish a database of difficulty questions, which includes questions of known difficulty categories. For a sample question, the computer device may calculate the similarity between the sample question and each question in the database, thereby determining the difficulty category of the sample question based on the difficulty category of the most similar question. The difficulty category of each question in the database may be manually annotated or predicted by a trained lightweight classification model.

[0099] In some embodiments, the computer device may also use a teacher model to reason about sample problems, obtain corresponding reasoning results, and classify the sample problems according to the number of reasoning steps or the length of the reasoning results to determine the difficulty category of the sample problems. The teacher model is a pre-trained large-scale model with a complex and high-performance neural network. The sample problems are classified according to the number of reasoning steps or the length of the reasoning results to determine the difficulty category of the sample problems. Specifically, when the reasoning length is greater than a first set threshold, it can be considered a complex problem category; when the reasoning length is less than a second set threshold, it can be considered a simple problem category; and when the reasoning length is between the first set threshold and the second set threshold, it can be considered a medium difficulty problem category.

[0100] In some embodiments, the computer device can also search for historical reasoning problems similar to the sample problem from historical reasoning records, determine the approximate reasoning length of the sample problem based on the reasoning length of the reasoning results of the similar historical reasoning problems, and then determine the difficulty category of the sample problem based on the approximate reasoning length.

[0101] Optionally, the computer device may extract at least one keyword from the sample question, and search for a historical reasoning question matching the extracted keyword from the historical reasoning record as a historical reasoning question similar to the sample question.

[0102] Optionally, the computer device can also classify historical reasoning records by keywords or category tags. This means that classifying reasoning problems by keywords can improve keyword matching efficiency. Classifying reasoning problems by category tags allows for the calculation of the average reasoning length of corresponding reasoning results for different category tags. Optionally, the average reasoning length for each category tag can be updated periodically.

[0103] Optionally, when there are no historical reasoning questions similar to the sample question in the historical reasoning records (for example, there are no historical reasoning questions that match the keyword), the category label of the sample question can be identified first, and the approximate reasoning length of the sample question can be determined based on the statistical average reasoning length corresponding to the category label, so that the difficulty category of the sample question can be determined based on the approximate reasoning length.

[0104] Optionally, the approximate reasoning length of the sample problem is determined based on the reasoning lengths of the reasoning results of similar historical reasoning problems, and the average of the reasoning lengths of the reasoning results of multiple similar historical reasoning problems can be calculated as the approximate reasoning length.

[0105] In some embodiments, the computer device may also search historical reasoning records for historical reasoning problems similar to the sample problem, determine an approximate reasoning time for the sample problem based on the reasoning time of the similar historical reasoning problems, and then determine the difficulty category of the sample problem based on the approximate reasoning time. Determining the approximate reasoning time for the sample problem based on the reasoning time of the reasoning results of similar historical reasoning problems may involve calculating the average reasoning time of the reasoning results of multiple similar historical reasoning problems as the approximate reasoning time.

[0106] The above methods can all use a unified mechanism to evaluate the difficulty categories of sample questions. Even when the difficulty levels of the questions vary greatly, an effective, accurate and reasonable assessment of the difficulty levels of the sample questions can be achieved.

[0107] The inference length control hyperparameter, in the embodiment of this application, refers to the hyperparameter used to construct the inference length reward function. Setting the appropriate value of the hyperparameter of the inference length reward function can guide the inference model to make decisions at a reasonable inference length. Different difficulty levels have different corresponding inference length control hyperparameters. Rewards are given based on the inference length of the inference result. Longer inference results will receive lower rewards, and concise inference results will receive higher rewards. This reflects the need to avoid excessive inference in simple problems, and encourage the generation of sufficient inference steps in complex problems.

[0108] Different difficulty categories correspond to different inference length control hyperparameters. A correspondence between difficulty categories and inference length control hyperparameters can be pre-set. Based on this correspondence, the corresponding inference length control hyperparameter for the sample problem can be determined. For example, if the sample problem belongs to the complex problem category, the corresponding inference length control hyperparameter is determined to be λ1; if the sample problem belongs to the medium difficulty problem category, the corresponding inference length control hyperparameter is determined to be λ2; if the sample problem belongs to the simple problem category, the corresponding inference length control hyperparameter is determined to be λ3. Each inference length control hyperparameter is a positive number greater than 0. For another example, the inference length control hyperparameter λ = λ0 * f(d), where λ0 is a base parameter whose value can be a set default value, d represents the difficulty category or difficulty score of the sample problem, and f(d) represents a difficulty mapping function. In this way, different inference length control hyperparameters can be assigned to different difficulty categories.

[0109] Step 206 : Control hyperparameters according to the inference length and the inference length of the inference result, construct an inference length reward function, and construct an inference accuracy reward function according to the inference result.

[0110] Among them, the inference length reward function is to dynamically adjust the inference length according to the difficulty of the input problem. For sample problems in the simple problem category, the shorter the inference length, the higher the reward, and the longer the inference length, the lower the reward; for sample problems in the complex problem category, the simpler the inference length, the relatively lower the reward, because the inference length needs to be longer to ensure the generation of detailed inference steps, so longer inference lengths are rewarded. In some embodiments, the inference length reward function is negatively correlated with the inference length of the inference result and negatively correlated with the inference length control hyperparameter, that is, the shorter the inference length of the inference result and the smaller the inference length control hyperparameter, the higher the inference length reward.

[0111] An excessively long inference chain can improve inference accuracy, but it also introduces computational redundancy and time delays. An excessively short inference chain, while improving inference efficiency, can lead to decreased accuracy. To address this, an accuracy reward function is provided within the reinforcement learning process. This function effectively combines inference accuracy with inference length to optimize the inference process, flexibly adjusting the number and quality of inference steps during reasoning for different problems. This ensures inference accuracy while dynamically adjusting inference length to better meet the specific requirements of the task.

[0112] In one embodiment, an inference length reward function is constructed based on the inference length control hyperparameter and the inference length of the inference result. The formula is as follows:

[0113] ;

[0114] The inference length reward function Rlength is negatively correlated with the inference length Lpred of the inference result. Lpred is the inference length of the inference result, and λ is the inference length control hyperparameter. λ>0. Setting an appropriate value of the inference length control hyperparameter can guide the inference model to make decisions at a reasonable inference length.

[0115] In some embodiments, when the sample problem belongs to the complex problem category, the corresponding inference length control hyperparameter is determined to be λ1; when the sample problem belongs to the medium difficulty problem category, the corresponding inference length control hyperparameter is determined to be λ2; and when the sample problem belongs to the simple problem category, the corresponding inference length control hyperparameter is determined to be λ3; where λ1>λ2>λ3>0. In some embodiments, the inference length control hyperparameter λ=λ0*f(d), where λ0 is a base parameter, d represents the difficulty category of the sample problem, and f(d) represents a difficulty mapping function. When d1 is a complex problem category, d2 is a medium difficulty problem category, and d3 is a simple problem category, f(d1)>f(d2)>f(d3)>0. In this embodiment, the inference length control hyperparameter corresponding to the sample problem is obtained by mapping the corresponding difficulty category using a mapping function, which can accurately provide sample data for the model to dynamically adjust the inference length. Specifically, when the computer device determines that a sample problem belongs to the complex problem category, it can obtain the corresponding inference length control hyperparameter λ1 based on the correspondence between the difficulty category of the sample problem and the inference length control hyperparameter to encourage the inference model to generate sufficient inference steps. If the sample problem belongs to the simple problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ3 to prevent the inference model from over-inference. If the sample problem belongs to the medium difficulty problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ2 to encourage the inference model to generate an inference chain that is neither too long nor too short.

[0116] In some embodiments, constructing the reasoning accuracy reward function based on the reasoning result includes: obtaining a sample answer corresponding to the sample question; and constructing the reasoning accuracy reward function based on the reasoning result of the sample question and the sample answer. Specifically, the reasoning accuracy reward function can be obtained by calculating the cross entropy between the reasoning result of the sample question and the sample answer.

[0117] In this embodiment, when constructing the reasoning accuracy reward function, the reward function is calculated based on the reasoning result of the sample question and the sample answer corresponding to the sample question, encouraging the model to output an answer as close to the sample answer as possible to ensure reasoning accuracy.

[0118] In some embodiments, a reasoning accuracy reward function is constructed based on the reasoning result of the sample question and the sample answer, including: using a sentence embedding model to calculate the similarity between the steps of the reasoning result and the steps of the sample answer, and constructing the reasoning accuracy reward function based on the similarity.

[0119] Step 208: Perform reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function.

[0120] In some embodiments, the computer device can respectively obtain the reward weights of the inference length reward function and the inference accuracy reward function, perform weighted summation of the inference length reward function and the inference accuracy reward function according to the corresponding reward weights to obtain a multi-objective reward function, and update the model parameters of the inference model according to the multi-objective reward function.

[0121] Reward weight, in this embodiment of the application, refers to the loss weight of the reward function. In order to balance different tasks in multi-task learning, different reward weights are set for different reward functions. In this embodiment, the reward weights of the inference length reward function and the inference accuracy reward function can be pre-set. For example, the reward weight of the inference accuracy reward function is α, and the reward weight of the inference length reward function is α2, and α1+α2=1.

[0122] In some embodiments, the inference length reward function and the inference accuracy reward function are weighted and summed according to the corresponding reward weights to obtain a multi-objective reward function, which is formulated as follows:

[0123] .

[0124] In some embodiments, the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function are determined according to the difficulty category of the sample problem; the reasoning length reward function and the reasoning accuracy reward function are weightedly summed according to the corresponding reward weights to obtain a multi-objective reward function; and the model parameters of the reasoning model are updated according to the multi-objective reward function.

[0125] In this embodiment, during the reinforcement learning training process, the reward weights of the inference length reward function and the inference accuracy reward function are dynamically adjusted based on the difficulty level of the sample problems. For example, for sample problems in the simple problem category (e.g., simple mathematical operations), the reward weight of the accuracy reward function is larger, making the accuracy reward function dominant, thereby reducing lengthy inference processes. The model will prioritize inference accuracy and minimize unnecessary inference steps. For sample problems in the complex problem category (e.g., programming problems or logical reasoning problems), the reward weight of the accuracy reward function is smaller, thereby reducing the impact of the inference length reward function and allowing the inference model to generate more inference steps.

[0126] In some embodiments, the reward weight of the reasoning accuracy reward function is α1, the reward weight of the reasoning length reward function is α2, and α1+α2=1; determining the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem includes:

[0127] When the sample problem belongs to the complex problem category, set α2>α1>0;

[0128] In the case that the sample question belongs to the simple question category, it is set that α1>α2>0.

[0129] Optionally, when the sample question belongs to the complex question category, set α1=α2, and α1>0.

[0130] Through reinforcement learning training, the model will provide feedback based on a combination of length rewards and accuracy rewards each time it generates an inference chain. The inference model will continuously adjust each step of the inference process, optimize the inference strategy based on the multi-objective reward function, adjust the generated inference steps and inference depth, and gradually approach the optimal solution, thereby achieving efficient and accurate inference.

[0131] In the above model training method, the inference model is used to infer sample problems to obtain inference results. The corresponding inference length control hyperparameter is determined based on the difficulty level of the sample problem. An inference length reward function is constructed based on the inference length control hyperparameter and the inference length of the inference result, and an inference accuracy reward function is constructed based on the inference result. Based on the inference length reward function and the inference accuracy reward function, the inference model is trained using reinforcement learning. On the one hand, the combination of the inference length reward function and the accuracy reward function can guide the inference model to avoid generating redundant inference steps while ensuring accuracy. This dual optimization mechanism not only improves the accuracy of inference, but also improves the efficiency of the inference process, ensuring that the inference process can produce correct results in a short time in resource-constrained scenarios. On the other hand, during the reinforcement learning training process, the inference length control hyperparameters are automatically adjusted according to the difficulty category of the sample problems, so that the inference model can automatically adjust the inference length for problems of different difficulty levels. In other words, it is not limited to punishing overly long reasoning chains, but also introduces a dynamic adjustment mechanism, so that the model can flexibly determine the length of the reasoning steps according to the difficulty of the problem. By introducing the difficulty category of the sample problems, the inference model is encouraged to reduce lengthy and invalid reasoning steps when generating accurate reasoning steps, so as to achieve the purpose of maintaining reasoning accuracy and significantly improving reasoning efficiency, thereby achieving a more efficient and accurate reasoning process. In this way, the optimal balance point can be found when reasoning problems of different complexity, thereby avoiding the situation where the model over-reasons or lacks reasoning.

[0132] Figure 3 This is a flowchart of a model training method shown in an exemplary embodiment of the present application. The method can be executed by a computer device. The method can also be applied to Figure 1 The application environment shown in FIG. 1 , the computer device executing the method may be Figure 1 The server 104 or terminal 102 in the application environment is shown.

[0133] In an exemplary embodiment, Figure 3 As shown, a model training method is provided, including the following steps 302 to 310. In which:

[0134] Step 302: Perform multiple inferences on the same sample problem using the inference model to obtain multiple inference results.

[0135] Step 304: Determine the difficulty level of the sample question based on the multiple inference results.

[0136] In some embodiments, the computer device may calculate the accuracy of the multiple inference results; and determine the difficulty category of the sample question based on the accuracy of the multiple inference results.

[0137] In some embodiments, the computer device can obtain a sample answer corresponding to the sample question; when the inference result and the sample answer corresponding to the sample question meet the consistency condition, the inference result is determined to be a correct inference result; based on the number of correct inference results in the multiple inference results and the total number of inferences of the multiple inferences, the accuracy of the multiple inference results is counted.

[0138] Step 306 : Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem.

[0139] In some embodiments, when the sample problem belongs to the complex problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ1; when the sample problem belongs to the medium difficulty problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ2; when the sample problem belongs to the simple problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ3; wherein, λ1>λ2>λ3>0.

[0140] Step 308 : Control hyperparameters and the inference lengths of multiple inference results according to the inference length, construct an inference length reward function, and construct an inference accuracy reward function according to the multiple inference results.

[0141] In some embodiments, an inference length reward function is constructed based on the inference length control hyperparameter and the inference lengths of the multiple inference results, including: accumulating the inference lengths of the multiple inference results to obtain a total inference length; constructing the inference length reward function based on the inference length control hyperparameter and the total inference length, and the inference length reward function is negatively correlated with the total inference length.

[0142] In one embodiment, the inference length reward function is constructed based on the inference length control hyperparameter and the total inference length, and the formula is as follows:

[0143] ;

[0144] Among them, the reasoning length reward function Rlength is equal to the total reasoning length of n reasoning results is negatively correlated, Lpred_i is the inference length of the i-th inference result, λ is the inference length control hyperparameter, λ>0.

[0145] In some embodiments, constructing the reasoning accuracy reward function based on the multiple reasoning results includes: obtaining a sample answer corresponding to the sample question; and constructing a reasoning accuracy reward function based on the multiple reasoning results and the sample answer respectively.

[0146] Step 310 : Perform reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function.

[0147] In some embodiments, constructing the reasoning accuracy reward function based on the multiple reasoning results includes: obtaining a sample answer corresponding to the sample question; and constructing a reasoning accuracy reward function based on the multiple reasoning results of the sample question and the sample answer respectively.

[0148] For relevant embodiments of the steps in the model training method, please refer to the previous text and the detailed description will not be repeated here.

[0149] In the above model training method, the inference model performs multiple inferences on the same sample problem to obtain multiple inference results. The difficulty category of the sample problem is determined based on the multiple inference results. According to the difficulty category of the sample problem, the corresponding inference length control hyperparameter is determined. Based on the inference length control hyperparameter and the inference length of the multiple inference results, an inference length reward function is constructed, and based on the multiple inference results, an inference accuracy reward function is constructed. Based on the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning. On the one hand, by combining the inference length reward function with the accuracy reward function, the inference model can be guided to avoid generating redundant inference steps while ensuring accuracy. This dual optimization mechanism not only improves the accuracy of inference, but also improves the efficiency of the inference process, ensuring that the inference process can produce correct results in a short time in resource-constrained scenarios. On the other hand, during the reinforcement learning training process, the corresponding inference length control hyperparameters are automatically determined based on the difficulty categories of sample problems determined according to multiple inference results, so that the inference model can automatically adjust the inference length for problems of different difficulty levels. By introducing the difficulty categories of sample problems, the inference model is encouraged to reduce lengthy and invalid inference steps when generating accurate inference steps, so as to achieve the purpose of maintaining inference accuracy and significantly improving inference efficiency, thereby realizing a more efficient and accurate inference process.

[0150] In an exemplary embodiment, the method further includes: obtaining an input question; and reasoning the input question using a trained reasoning model to obtain a corresponding reasoning result.

[0151] In this embodiment, after the trained inference model is obtained through reinforcement learning training, the inference model can automatically control the length of the inference output for the input problem while ensuring the accuracy of the inference, avoiding problems such as low inference efficiency, waste of computing resources, and lengthy inference process in the inference model.

[0152] For example, consider a simple math problem: "2 + 3 = ?" After reinforcement learning, the reasoning model automatically determines the difficulty of the problem. Clearly, this problem falls into the simple category, so the length of the reasoning is strictly controlled. The model minimizes redundant reasoning steps and quickly provides an answer in just a few steps.

[0153] For example, if the input problem is a complex problem, such as the programming task "implement a recursive function to solve the Fibonacci sequence", after reinforcement learning, the reasoning model has the ability to automatically judge the difficulty of the input problem. Obviously, the input problem belongs to the category of complex problems, so the length of the reasoning process is allowed to increase appropriately, and the reasoning model will generate more reasoning steps to ensure the final correct answer.

[0154] In some embodiments, the computer device can also use a lightweight classification model to predict the difficulty category of the input question, and use the predicted difficulty category and the input question as the input of the reasoning model to prompt the reasoning model to reason about the input question according to the difficulty category to obtain the corresponding reasoning result. In this way, the input question can automatically control the length of the reasoning output while ensuring the accuracy of the reasoning, avoiding problems such as low reasoning efficiency, waste of computing resources and lengthy reasoning process in the reasoning model.

[0155] like Figure 4 As shown, in an exemplary embodiment, the training steps of the inference model include:

[0156] Step 401: Input the same sample question into the inference model multiple times to obtain multiple inference results.

[0157] Step 402: Evaluate the difficulty level of the sample question based on multiple inference results.

[0158] Step 403: Determine the corresponding inference length control hyperparameter based on the difficulty category of the sample question, construct an inference length reward function based on the inference length control hyperparameter and the inference lengths of the multiple inference results, and construct an inference accuracy reward function based on the multiple inference results and the sample answer.

[0159] Step 404: Determine the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem, and perform a weighted summation of the reasoning length reward function and the reasoning accuracy reward function according to the corresponding reward weights to obtain a multi-objective reward function.

[0160] Step 405: Adjust model parameters according to the multi-objective reward function.

[0161] Step 406: Iterate training to optimize the inference process.

[0162] When applying the reasoning model, the reasoning model can automatically determine the difficulty of the input problem and dynamically adjust the generation parameters of the reasoning steps, such as the maximum length of the generated text max_length or the beam search width num_beams, etc. That is, for simple problems, the reasoning model generates concise reasoning steps and quickly outputs the answer. For complex problems, the reasoning model generates more detailed reasoning steps to ensure sufficient reasoning depth and obtain the correct answer.

[0163] Figure 5 This is a schematic diagram of the change in inference length in an example, refer to Figure 5 The inference model provided by the embodiment of the present application achieves the following effects as the training process continues: for complex sample problems ( Figure 5 The broken line in the middle upper part) increases the number of reasoning steps, and the model can generate a longer reasoning chain, ensuring the meticulousness and accuracy of the reasoning process and avoiding simple reasoning errors. For simple sample problems ( Figure 5 The model generates a more concise reasoning chain, avoiding excessive reasoning or irrelevant steps (the broken line in the lower middle section). This flexible control of inference length enables the model to maintain high inference accuracy across a wide range of problem types, avoids overthinking, and ensures the correctness of problem answers.

[0164] Figure 6 For an exemplary embodiment, the detailed training steps of the inference model are shown in FIG. Figure 6 , including the following steps:

[0165] Step 601: Receive an input reasoning question.

[0166] Step 602: Determine the difficulty level of the reasoning problem based on the accuracy of the reasoning results of multiple reasonings of the same reasoning problem by the reasoning model.

[0167] The difficulty levels are: complex question category, medium difficulty question category and simple question category.

[0168] Step 603: The reasoning model automatically determines the reasoning length based on the difficulty level of the reasoning problem.

[0169] For simple types of reasoning problems, short reasoning chains are generated; for medium-difficulty reasoning problems, medium-length reasoning chains are generated; and for complex types of reasoning problems, longer reasoning chains are generated.

[0170] Step 604: Calculate the inference accuracy reward function.

[0171] The computer device calculates an inference accuracy reward function based on the inference result of the inference model and the answer to the inference question.

[0172] Step 605: Calculate the inference length reward function.

[0173] The reward function is calculated based on the length of the reasoning chain to avoid excessive model thinking.

[0174] Step 606: Calculate the multi-objective reward function.

[0175] A multi-objective reward function is calculated based on the accuracy reward function and the inference length reward function, and the model parameters are optimized accordingly.

[0176] Step 607: Update model parameters.

[0177] Based on the feedback of the multi-objective reward function, the model parameters are optimized, thereby optimizing the model reasoning strategy.

[0178] Step 608: After optimization, a complete reasoning result is output for the input problem, and the reasoning process ends.

[0179] Figure 7 This is a flowchart of a model training method shown in a detailed embodiment of the present application. The method can be executed by a computer device. The method can also be applied to Figure 1 The application environment shown in FIG. 1 , the computer device executing the method may be Figure 1 The server 104 or terminal 102 in the application environment is shown.

[0180] In an exemplary embodiment, Figure 7 As shown, a model training method is provided, including the following steps 702 to 710. Among them:

[0181] Step 702: Perform multiple inferences on the same sample problem through the inference model to obtain multiple inference results, obtain a sample answer corresponding to the sample problem, and determine that the inference result is a correct inference result when the inference result and the sample answer corresponding to the sample problem meet the consistency condition. According to the number of correct inference results in the multiple inference results and the total number of inferences in the multiple inferences, the accuracy of the multiple inference results is counted, and the difficulty category of the sample problem is determined based on the accuracy of the multiple inference results.

[0182] Step 704: If the sample problem belongs to the complex problem category, determine the inference length control hyperparameter corresponding to the sample problem as λ1; if the sample problem belongs to the medium difficulty problem category, determine the inference length control hyperparameter corresponding to the sample problem as λ2; if the sample problem belongs to the simple problem category, determine the inference length control hyperparameter corresponding to the sample problem as λ3; wherein λ1>λ2>λ3>0, and the inference length reward function is negatively correlated with the inference length of the inference result.

[0183] Step 706: Accumulate the inference lengths of the multiple inference results to obtain a total inference length. Control the hyperparameters and the total inference length based on the inference length to construct an inference length reward function. The inference length reward function is negatively correlated with the total inference length.

[0184] Step 708: Obtain a sample answer corresponding to the sample question; and construct an inference accuracy reward function based on the multiple inference results and the sample answer.

[0185] Step 710: Determine the reward weight of the inference length reward function and the reward weight of the inference accuracy reward function according to the difficulty category of the sample problem; perform weighted summation of the inference length reward function and the inference accuracy reward function according to the corresponding reward weights to obtain a multi-objective reward function; and update the model parameters of the inference model according to the multi-objective reward function.

[0186] Among them, the reward weight of the inference accuracy reward function is α1, the reward weight of the inference length reward function is α2, and α1+α2=1; when the sample problem belongs to the complex problem category, set α2>α1>0; when the sample problem belongs to the simple problem category, set α1>α2>0.

[0187] The model training method provided in this embodiment solves the problems of low reasoning efficiency, waste of computing resources, and lengthy reasoning processes in inference models in related technologies. By precisely controlling the length of the inference output, the inference efficiency is improved while ensuring the accuracy of the inference, and unnecessary computing consumption is effectively reduced. The specific beneficial effects are as follows:

[0188] 1. Improve inference efficiency

[0189] By combining a length reward function with an accuracy reward function, the reasoning process can adaptively adjust the length of inference steps based on the complexity of the input problem. For complex problems, the model can generate long inference chains, ensuring detailed and accurate reasoning; while for simple problems, the model generates concise inference chains, significantly reducing redundant computation. This makes the reasoning process more efficient and significantly improves response speed.

[0190] 2. Save computing resources

[0191] By reducing redundant reasoning steps and the generation of irrelevant information, computing resource consumption during the reasoning process is significantly reduced. This effect is particularly evident in environments with limited computing resources, such as mobile devices and embedded systems. Reducing unnecessary reasoning chains and computational complexity enables large language models to be applied in a wider range of scenarios, especially in environments that are sensitive to latency or have limited computing resources.

[0192] 3. Ensure the accuracy of reasoning

[0193] Through reinforcement learning, the model can be guided to adjust its reasoning strategy when faced with problems of varying difficulty, thereby improving reasoning accuracy. Especially for complex problems, the model can generate detailed reasoning steps, ensuring the depth of reasoning and the comprehensiveness of the solution. At the same time, the reasoning steps for simple problems are compressed, avoiding errors or unnecessary calculations caused by lengthy reasoning.

[0194] 4. Dynamic inference length control

[0195] Unlike fixed inference length control methods, this approach achieves more flexible and effective inference length adjustment by dynamically evaluating problem difficulty and combining a dual reward mechanism based on both inference accuracy and length. This allows the model to adaptively adjust inference length based on different input tasks, maximizing inference effectiveness and efficiency.

[0196] It should be understood that, although the various steps in the flowcharts involved in the various embodiments as above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments as above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0197] Based on the same inventive concept, the present application also provides a model training device for implementing the aforementioned model training method. The solution to the problem provided by the device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following model training device embodiments can be found in the above-mentioned limitations on the model training method, and will not be repeated here.

[0198] In an exemplary embodiment, Figure 8 As shown, a model training device 800 is provided, comprising: an inference module 802, a hyperparameter determination module 804, a reward function construction module 806 and a model update module 808, wherein:

[0199] The reasoning module 802 is used to reason about the sample problem through the reasoning model to obtain the reasoning result;

[0200] A hyperparameter determination module 804 is configured to determine a corresponding inference length control hyperparameter based on the difficulty level of the sample problem;

[0201] a reward function construction module 806 for controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result;

[0202] The model updating module 808 is configured to perform reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function.

[0203] In an exemplary embodiment, the reasoning module 802 is further configured to perform multiple reasonings on the same sample problem using the reasoning model to obtain multiple reasoning results; the model training device 800 further includes:

[0204] The difficulty evaluation module is used to determine the difficulty category of the sample problem according to the multiple reasoning results.

[0205] In an exemplary embodiment, the difficulty evaluation module is further configured to calculate the accuracy of the multiple inference results; and determine the difficulty category of the sample question based on the accuracy of the multiple inference results.

[0206] In an exemplary embodiment, the difficulty assessment module is also used to obtain a sample answer corresponding to the sample question; when the inference result and the sample answer corresponding to the sample question meet the consistency condition, the inference result is determined to be a correct inference result; based on the number of correct inference results in the multiple inference results and the total number of inferences of the multiple inferences, the accuracy of the multiple inference results is counted.

[0207] In an exemplary embodiment, the inference length reward function is negatively correlated with the inference length of the inference result; the hyperparameter determination module 804 is further used to determine that the inference length control hyperparameter corresponding to the sample problem is λ1 when the sample problem belongs to the complex problem category; when the sample problem belongs to the medium difficulty problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ2; when the sample problem belongs to the simple problem category, the inference length control hyperparameter corresponding to the sample problem is determined to be λ3; wherein, λ1>λ2>λ3>0.

[0208] In an exemplary embodiment, the reward function construction module 806 is further used to accumulate the inference lengths of the multiple inference results to obtain a total inference length; control the hyperparameters and the total inference length according to the inference length to construct the inference length reward function, and the inference length reward function is negatively correlated with the total inference length.

[0209] In an exemplary embodiment, the reward function construction module 806 is further configured to obtain a sample answer corresponding to the sample question; and construct a reasoning accuracy reward function based on the multiple reasoning results and the sample answer.

[0210] In an exemplary embodiment, the model update module 808 is further used to determine the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem; perform weighted summation of the reasoning length reward function and the reasoning accuracy reward function according to the corresponding reward weights to obtain a multi-objective reward function; and update the model parameters of the reasoning model according to the multi-objective reward function.

[0211] In an exemplary embodiment, the reward weight of the inference accuracy reward function is α1, the reward weight of the inference length reward function is α2, and α1+α2=1; when the sample problem belongs to the complex problem category, set α2>α1>0; when the sample problem belongs to the simple problem category, set α1>α2>0.

[0212] In one embodiment, the reasoning module 802 is further configured to obtain an input question, and perform reasoning on the input question using a trained reasoning model to obtain a corresponding reasoning result.

[0213] The model training device 800 uses an inference model to infer sample problems to obtain inference results; determines the corresponding inference length control hyperparameter based on the difficulty category of the sample problem; constructs an inference length reward function based on the inference length control hyperparameter and the inference length of the inference result, and constructs an inference accuracy reward function based on the inference result; and performs reinforcement learning-based model training on the inference model based on the inference length reward function and the inference accuracy reward function. In other words, during the reinforcement learning training process, the inference length control hyperparameter is automatically adjusted based on the difficulty category of the sample problem to enable the inference model to automatically adjust the inference length for problems of different difficulty levels. By introducing the difficulty category of the sample problem, the inference model is encouraged to reduce lengthy and invalid inference steps when generating accurate inference steps, thereby achieving the goal of maintaining inference accuracy while significantly improving inference efficiency, thereby achieving a more efficient and accurate inference process.

[0214] Each module in the above-mentioned model training device 800 can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0215] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model training method is implemented.

[0216] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, a model training method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0217] Those skilled in the art will understand that Figure 9 、 Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0218] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the model training method provided in any one or more embodiments of the present application are implemented.

[0219] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method provided in any one or more embodiments of the present application are implemented.

[0220] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the model training method provided in any one or more embodiments of the present application.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLCs), artificial intelligence (AI) processors, and the like.

[0223] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0224] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A model training method, characterized in that: The method comprises: Reasoning the sample problem through the reasoning model to obtain the reasoning result; Determine the corresponding inference length control hyperparameter according to the difficulty level of the sample problem; Controlling a hyperparameter and an inference length of the inference result according to the inference length, constructing an inference length reward function, and constructing an inference accuracy reward function according to the inference result; According to the inference length reward function and the inference accuracy reward function, the inference model is trained based on reinforcement learning.

2. The method according to claim 1, characterized in that The method of reasoning the sample problem using the reasoning model to obtain the reasoning result includes: Through the reasoning model, multiple reasoning results are obtained for the same sample problem; The method further comprises: The difficulty level category of the sample question is determined according to the multiple inference results.

3. The method according to claim 1, characterized in that The method of reasoning the sample problem using the reasoning model to obtain the reasoning result includes: Multiple reasoning models are used to reason about the same sample problem and obtain multiple reasoning results; The method further comprises: The difficulty level category of the sample question is determined according to the multiple inference results.

4. The method according to claim 1, wherein The method further comprises: Searching historical reasoning questions similar to the sample question from historical reasoning records; An approximate reasoning length of a sample problem is determined based on the reasoning length of the reasoning results of the similar historical reasoning problem, and a difficulty category of the sample problem is determined based on the approximate reasoning length.

5. The method according to claim 4, characterized in that The method further comprises: When there is no historical reasoning problem similar to the sample problem in the historical reasoning record, the category label of the sample problem is identified, and the difficulty category of the sample problem is determined based on the average reasoning length corresponding to the category label.

6. The method according to claim 1, characterized in that The inference length reward function is negatively correlated with the inference length of the inference result; Determining the corresponding inference length control hyperparameter according to the difficulty level of the sample problem includes: When the sample problem belongs to the complex problem category, determining the inference length control hyperparameter corresponding to the sample problem to be λ1; When the sample problem belongs to a medium-difficulty problem category, determining that the inference length control hyperparameter corresponding to the sample problem is λ2; When the sample problem belongs to the simple problem category, determining the inference length control hyperparameter corresponding to the sample problem to be λ3; Among them, λ1>λ2>λ3>0.

7. The method according to claim 2, characterized in that The constructing of an inference length reward function by controlling a hyperparameter and an inference length of the inference result according to the inference length includes: Accumulating the inference lengths of the plurality of inference results to obtain a total inference length; The inference length reward function is constructed according to the inference length control hyperparameter and the total inference length, and the inference length reward function is negatively correlated with the total inference length.

8. The method according to claim 2, characterized in that The constructing of the inference accuracy reward function according to the inference result includes: Obtaining sample answers corresponding to the sample questions; Construct an inference accuracy reward function based on the multiple inference results and the sample answers.

9. The method according to any one of claims 1 to 8, characterized in that The performing reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function includes: Determining a reward weight of the reasoning length reward function and a reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem; Performing a weighted summation of the inference length reward function and the inference accuracy reward function according to corresponding reward weights to obtain a multi-objective reward function; The model parameters of the inference model are updated according to the multi-objective reward function.

10. The method according to claim 9, characterized in that The reward weight of the inference accuracy reward function is α1, the reward weight of the inference length reward function is α2, and α1+α2=1; Determining the reward weight of the reasoning length reward function and the reward weight of the reasoning accuracy reward function according to the difficulty category of the sample problem includes: When the sample problem belongs to the complex problem category, set α2>α1>0; In the case that the sample question belongs to the simple question category, it is set that α1>α2>0.

11. A model training device, characterized in that: The device comprises: The reasoning module is used to reason about sample problems through the reasoning model to obtain reasoning results; A hyperparameter determination module, configured to determine a corresponding inference length control hyperparameter according to the difficulty level of the sample problem; a reward function construction module, configured to control a hyperparameter and an inference length of the inference result according to the inference length, construct an inference length reward function, and construct an inference accuracy reward function according to the inference result; A model updating module is used to perform reinforcement learning-based model training on the inference model according to the inference length reward function and the inference accuracy reward function.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images