A concise reasoning model training method based on difficulty self-adaption

Abstract

The application provides a concise reasoning model training method based on difficulty self-adaptation, and relates to the technical field of image data processing, comprising: using a plurality of prompt words to drive a starting point model based on original data to generate a plurality of responses for each input question to form a response set; for each question, calculating a token length budget based on the response set, and calculating a reward value of each response according to the token length budget and response correctness; using a plurality of pairing algorithms to construct a preference data pair based on the reward value, and filtering through a reward difference threshold value to obtain a training preference data pair, wherein the pairing algorithms include a head-tail method, a polling method and a shortest method; based on the preference data pair, training an optimized reasoning model through a length-sensitive preference training algorithm to generate an optimized model with concise responses; and performing performance evaluation and efficiency evaluation on the optimized model. The application enables the model to autonomously adjust the length of the thinking chain according to the difficulty of the question, while keeping the reasoning performance unchanged and greatly improving the reasoning efficiency.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a concise inference model training method based on difficulty adaptation. Background Technology

[0002] With the rapid development of artificial intelligence, the Slow Thinking Model, as an emerging cognitive computing paradigm, is demonstrating remarkable performance advantages in various complex reasoning tasks. Unlike traditional fast-response systems, this model simulates the cognitive process of deep human thinking, exhibiting higher accuracy and robustness in tasks requiring logical reasoning, multi-step verification, and comprehensive judgment. It is particularly outstanding in professional fields such as mathematical proof, legal reasoning, and medical diagnosis, effectively avoiding systematic biases caused by rapid judgments and providing new ideas and methodologies for solving complex problems with artificial intelligence. The Slow Thinking Reasoning Model, representing a new generation of cognitively enhanced artificial intelligence, focuses on simulating the neural mechanisms and decision-making logic of deep human thinking. Its training methodology employs a three-stage paradigm: "cognitive warm-up – meta-learning reinforcement – ​​preference alignment." First, a basic reasoning framework is built using massive amounts of expert decision-making data. Then, a dynamic computational graph simulates the human brain's resource allocation process. Finally, an ethical constraint module is introduced to ensure decisions conform to socially rational standards. This model has already demonstrated transformative application value in high-risk fields—reducing misdiagnosis rates by 40% in medical diagnosis through delayed verification mechanisms and reducing implicit bias by 25% in judicial assistance systems. Compared to traditional AI, the reasoning model performs step-by-step analysis and logical deduction through internally generated thought chains, excelling particularly in mathematical problem-solving, code generation, and multi-step decision-making tasks. This technology is driving a paradigm shift in artificial intelligence from "rapid response" to "deep thinking."

[0003] The aforementioned technical solutions constitute the mainstream methodology for training large-scale "deep thinking" reasoning models. The core idea is to warm up and launch the model using high-quality, multi-step inference datasets, and then, leveraging reinforcement learning and a reasonable reward mechanism, guide the model to autonomously generate realistic decision-making processes, gradually endowing it with the ability to evolve from simple pattern recognition to complex logical reasoning. With the continuous upgrading of computing hardware and the constant emergence of algorithmic innovations, this technical system continues to iterate and develop rapidly.

[0004] In recent years, slow-thinking reasoning models have demonstrated superior performance in complex reasoning tasks. However, these models often exhibit overthinking (generating redundant reasoning steps for simple problems), leading to excessive consumption of computational resources. In fast-paced decision-making scenarios, while this "slow-thinking" mode excels in rational analysis, it often falls into the "overthinking trap": forcing complex cognitive processes into simple problems that could be handled intuitively. This phenomenon not only wastes time and resources but can also trigger decision anxiety—for example, consumers repeatedly comparing prices when purchasing daily necessities, or programmers performing unnecessary refactoring of basic code. This cognitive efficiency loss suggests that the optimization direction of slow thinking should not be simply to enhance depth, but rather to establish a "thinking energy-saving" mechanism, finding a dynamic balance between the necessity of analysis and execution efficiency.

[0005] In other words, current slow-thinking reasoning models have problems in reasoning tasks, including: High latency response: Currently, large AI models with billions of parameters are showing amazing potential in scenarios such as text generation and code writing, but their inference process often requires a response time of 30-100 seconds. This high latency characteristic leads to three levels of obstacles to implementation: First, in real-time interactive scenarios, such as customer service dialogue systems that require a response within 500 milliseconds, the thinking time of large models will directly disrupt the smoothness of the user experience; second, in the field of industrial control, such as autonomous driving which needs to complete decisions within 100 milliseconds, latency may cause safety accidents.

[0006] High inference cost: In the field of artificial intelligence, the inference cost of large language models has become a key bottleneck restricting the commercialization of the technology. This high cost is mainly reflected in three dimensions: First, the model needs to construct complex thought chains, similar to the "scratch paper" process when humans solve problems. The system needs to perform multiple intermediate calculations in the implicit space. Although these calculations are not directly presented to the user, they consume an equal amount of computing resources. Second, iterative thinking mechanisms exacerbate resource consumption. Modern large models generally adopt "Mind Tree" (ToT) or "Mind Chain" (CoT) techniques to improve output quality through multiple trials and verifications. For example, when dealing with mathematical proof problems, the model may need to try 5-8 different solution paths to determine the optimal solution. Although this mechanism improves accuracy, each attempt requires the complete execution of forward calculations. Finally, long context windows further amplify the cost pressure. To maintain dialogue coherence, modern models generally support 128K or even longer contexts, and each inference requires attention calculations on the entire dialogue history. Although this design improves the user experience, it causes the computational complexity to jump from O(1) to O(n²).

[0007] The problem of complicating simple questions: Current mainstream large language models generally suffer from a "cognitive resource mismatch" problem. The core contradiction lies in the failure of the system architecture to establish an effective difficulty-level response mechanism. When a user asks a basic query like "What's the weather like today?", the model still initiates the entire process of semantic parsing, knowledge retrieval, and logical deduction. This processing mode is akin to running a calculator program on a supercomputer. Specifically, it automatically loads redundant context analysis modules when faced with simple instructions, still explains arithmetic principles when answering questions like "What is 1+1?", and even generates lengthy arguments analyzing the pros and cons when dealing with clear true / false questions.

[0008] The three shortcomings mentioned above have become bottlenecks for the commercialization of current mainstream large-scale inference models. There is an urgent need to provide a simplified inference model training method to solve the problems of high latency, high cost, and coarse-grained input processing. This would be a key breakthrough in enabling large-scale inference models to truly adapt to commercial applications and achieve the widespread adoption of AI. Summary of the Invention

[0009] To address the problems in the background technology, this invention provides a concise reasoning model training method based on difficulty adaptation. By enabling the model to autonomously adjust the length of the thought chain according to the difficulty of the problem, the reasoning efficiency is greatly improved without changing the task performance of the reasoning model, while solving the problems of high latency, high cost, and coarse input processing.

[0010] To achieve the above objectives, this invention provides a concise inference model training method based on difficulty adaptation, comprising: Data preparation involves using multiple prompt words to drive the starting point model based on the raw data to generate multiple responses for each input question, forming a response set. For each problem, a tag length budget is calculated based on the response set, wherein the tag length budget integrates response accuracy and length distribution characteristics, and a reward value for each response is calculated based on the tag length budget and response correctness; Preference data pairs are constructed based on the reward values ​​using multiple pairing algorithms and filtered by a reward difference threshold to obtain preference data pairs for training. The pairing algorithms include head-tail method, round-robin method, and shortest method. Based on the aforementioned preference data pairs, a length-sensitive preference training algorithm is used to train the inference model to be optimized, generating an optimized model with a concise response. The optimization model is then evaluated for performance and efficiency.

[0011] As a further improvement of the present invention, the starting model is a pre-trained inference model, and the dataset includes a multi-step mathematical problem dataset and a complex logic proof dataset. The multi-step mathematical problem dataset includes GSM8K and MATH, and the complex logic proof dataset includes Openr1_default and Lightr1.

[0012] As a further improvement of the present invention, the multiple prompt words include normal reasoning prompt words, simple thinking prompt words, deep thinking prompt words, and Few-Shot prompt words, each of which is used to generate responses of different lengths to ensure response diversity.

[0013] As a further improvement of the present invention, the mark length budget L budget The calculation formula is: Where c represents the number of correct responses in the response set, and N represents the total number of responses in the response set. L represents the average length of a correct response. max This indicates the maximum length of the model output; The formula for calculating the reward value is as follows: Among them, L i This represents the length of the i-th response; select the corresponding reward value calculation formula for correct and incorrect responses.

[0014] As a further improvement of the present invention, multiple pairing algorithms are used to construct preference data pairs based on the reward values, and these pairs are filtered through a reward difference threshold to obtain preference data pairs for training, including: The response set is sorted in descending order of the reward value, and correct-correct response pairs and correct-incorrect response pairs are constructed as preference data pairs. For correct-correct response pairs, the head-tail method or round-robin method is used for pairing, and the pairing is filtered by the reward difference threshold. For correct-incorrect response pairs, the shortest method is used for pairing.

[0015] As a further improvement of the present invention, the head-tail method includes pairing the longest and shortest correct responses in the response set, and the polling method includes polling the shortest top 3 and the longest top 3 responses.

[0016] As a further improvement of the present invention, the length-sensitive preference training algorithm is the SimPO algorithm, wherein the SimPO algorithm uses the average log probability of the sequence as an implicit reward and introduces a target reward margin to optimize training efficiency.

[0017] As a further improvement of the present invention, the efficiency evaluation is based on the response length index, including the average length LEN of all responses and the average length C_LEN of correct responses, calculated as follows: Among them, res i Represents the i-th response, res i=T This indicates a correct response.

[0018] As a further improvement of the present invention, the starting model adopts the Jiusi-R1-V1.0.0-32B model, and the dataset includes GSM8K, MATH, Openr1_default and Lightr1.

[0019] As a further improvement of the present invention, performance evaluation is performed based on the average accuracy metric of multiple inferences. For each question, the optimization model is used to perform multiple inferences, and the accuracy of each inference is judged. Correct answers are scored 1, and incorrect answers are scored 0. The average score is the score for that question. The average score of all questions in the evaluation set is the evaluation score of that evaluation set on the optimization model.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention successfully addresses the core shortcomings of existing slow-thinking inference models, such as overthinking, high latency response, high inference cost, and the overcomplication of simple problems, by introducing a difficulty-adaptive mechanism. Specifically, while maintaining essentially no decrease in inference performance (an average performance reduction of only 0.37% on five authoritative evaluation sets), this invention achieves a significant improvement in inference efficiency (an average efficiency improvement of 32.67%, with a maximum of 36.56%), thereby effectively reducing computational resource consumption and response latency. This makes the model more suitable for real-time interaction and commercial applications, promoting the democratization of AI. These effects stem directly from key innovations such as self-training-driven data generation, budget-aware reward design, appropriate pairing algorithms, and length-sensitive preference training, ensuring that the model can autonomously adjust the length of the thought chain according to the complexity of the problem, avoiding resource misallocation.

[0021] This invention achieves "energy-saving thinking" through adaptive difficulty, finding a dynamic balance between the necessity of analysis and execution efficiency; secondly, the training method is low-cost, avoiding dependence on external models due to self-training; and thirdly, it has strong generalization ability, applicable to various complex task scenarios. Overall, this invention not only improves the economy and practicality of the model but also provides a feasible path for the paradigm shift in artificial intelligence from "rapid response" to "deep thinking." Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall scheme for generating training data for an inference model disclosed in one embodiment of the present invention; Figure 2 This is a schematic diagram of different pairing algorithms disclosed in one embodiment of the present invention; Figure 3 This is a response length distribution diagram of preference pairs disclosed in one embodiment of the present invention; Figure 4 This is a schematic diagram of Self-Training disclosed in one embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the superiority of the SimPO algorithm according to an embodiment of the present invention; Figure 6 This is a bar chart of the model performance evaluation results index disclosed in one embodiment of the present invention; Figure 7 This is a bar chart of the model efficiency evaluation results disclosed in one embodiment of the present invention; Figure 8 This is a loss monitoring curve of the model training process disclosed in one embodiment of the present invention; Figure 9 This is a graph showing the reward and accuracy monitoring during the model training process, as disclosed in one embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, the present invention provides a concise inference model training method based on difficulty adaptation, comprising: Phase 1: Data preparation, such as Figure 1 As shown, it includes: S1. Data preparation: Based on the raw data, use multiple prompt words to drive the starting point model to generate multiple responses for each input question to form a response set; in, The starting model is a pre-trained inference model. The datasets include multi-step mathematical problem datasets and complex logic proof datasets. The multi-step mathematical problem datasets include GSM8K and MATH, and the complex logic proof datasets include Openr1_default and Lightr1.

[0025] Multiple cue words are included, such as normal reasoning cue words, simple thinking cue words, deep thinking cue words, and Few-Shot cue words. Each cue word is used to generate responses of different lengths to ensure response diversity.

[0026] Furthermore, the starting model adopts the Jiusi-R1-V1.0.0-32B model, and the dataset includes GSM8K, MATH, Openr1_default, and Lightr1.

[0027] Specifically, Constructing preference pairs is crucial for training a concise, large-scale inference model. In this step, we adopted a self-training approach. Instead of using ultra-large-scale models like Deepseek to generate data, we used version 32B of the Jiusi-R1-V1.0.0 model to generate the responses, thus obtaining the trained preference pairs. Self-training is a semi-supervised learning method, such as... Figure 4 As shown, the core idea is to augment the training data using the model's own predictions. In training concise inference models, self-training generates pairs of long and short data preferences that the model can better distinguish, while avoiding the high costs of using extremely large-scale models. An overview of data preparation is as follows: Figure 1 As shown.

[0028] Determining the Base Model and Dataset: Optimizing and iterating the large inference model requires a starting point for both the model and the data. Specifically, it involves determining which model to optimize and iterate on, and what data to use for the iteration. For the model starting point, we chose our predecessor product, Jiusi-R1-V1.0.0-32B. For the data starting point, we selected two sets of data: one set consists of relatively simple multi-step mathematical problem datasets, such as GSM8K and MATH; the other set consists of complex and challenging multi-step logic proof and mathematical problem datasets, such as Openr1_default and Lightr1.

[0029] Cue word design for different generation methods: Four types of cue words were designed to guide the model to generate responses of different lengths, thereby ensuring response diversity and that the length distribution of the responses conforms to expectations. The four types of cue words are: normal reasoning cue words, simple thinking cue words, deep thinking cue words, and Few-Shot cue words.

[0030] S2. For each problem, calculate the tag length budget based on the response set, where the tag length budget integrates the response accuracy and length distribution characteristics, and calculate the reward value for each response based on the tag length budget and response correctness to achieve difficulty adaptation and budget awareness. Among them, the marker length budget L budget The calculation formula is: In the formula, c represents the number of correct responses in the response set, and N represents the total number of responses in the response set. L represents the average length of a correct response. max This indicates the maximum length of the model output; The formula for calculating reward value is: In the formula, L i This represents the length of the i-th response; select the corresponding reward value calculation formula for correct and incorrect responses.

[0031] Specifically, for a specific mathematical problem, its difficulty can be estimated by the length of the response to complete it. Therefore, we designed a token length budget to calculate the budget (token length) required to answer the problem using the lengths of the responses in the problem's response set.

[0032] Once a question's tagged budget is obtained, the reward value for each response can be calculated based on that budget.

[0033] S3. Use multiple pairing algorithms to construct preference data pairs based on reward values, and filter them through reward difference thresholds to obtain preference data pairs for training. The pairing algorithms include head-tail method, round-robin method and shortest method. Based on the correctness of the preference-match response, there are four cases, as shown in Table 1: Table 1 Based on training performance requirements, we do not adopt the wrong-right or wrong-wrong scenarios, and only design three pairing algorithms based on the first two scenarios: Furthermore, multiple pairing algorithms are used to construct preference data pairs based on reward values, including: Arrange the response sets in descending order of reward value, and construct correct-correct response pairs and correct-incorrect response pairs as preference data pairs; For correct-correct response pairs, the head-tail method or round-robin method is used for pairing, and the pairing is filtered by the reward difference threshold. For correct-incorrect response pairs, the shortest method is used for pairing.

[0034] The head-and-tail method involves pairing the longest and shortest correct responses in the response set, while the round-robin method involves pairing the shortest top 3 and the longest top 3 responses in a round-robin fashion.

[0035] Specifically, such as Figure 2 As shown, Regarding the "correct-correct" situation: Algorithm 1 (Head and Tail Method): Pair the longest and shortest correct responses, the second longest and the second shortest correct responses, and the third longest and the third shortest correct responses. Set a threshold for the reward difference. Only when the reward difference between two pairs of responses is greater than the threshold will they be added to the training set.

[0036] Algorithm 2 (Round-robin method): Take the shortest top 3 and the longest top 3, pair them up in a round-robin fashion, and then filter using a threshold.

[0037] Regarding the [True-False] situation: Algorithm 3 (Shortest Method): Take the shortest of the incorrect answers, and take the shortest of the correct answers whose length is greater than that of the incorrect answer.

[0038] Based on the pairing algorithm described above, we generated a different number of preference pairs for each dataset, such as... Figure 3 As shown, the number of specific preference data pairs depends on the pairing algorithm and the corresponding filtering threshold. The basic construction principle is to ensure that the number of preference pairs is roughly consistent with the number of questions.

[0039] Phase Two: Model Training, such as Figure 4 As shown, it includes: S4. Based on preference data pairs, train the inference model to be optimized using a length-sensitive preference training algorithm to generate an optimized model with a concise response. Among them, the inference model to be optimized is the starting model Jiusi-R1-V1.0.0-32B, and the length-sensitive preference training algorithm is the SimPO algorithm, such as... Figure 5 As shown, the SimPO algorithm uses the average log probability of the sequence as an implicit reward and introduces a target reward margin to optimize training efficiency.

[0040] Phase 3: Model evaluation, such as Figure 6 , 7 As shown, it includes: S5. Evaluate the performance and efficiency of the optimization model.

[0041] The model inference is based on the VLLM framework for model loading, weight loading, and quantization capability deployment, and also supports multi-GPU and concurrent inference.

[0042] Furthermore, Efficiency is evaluated based on response length metrics, including the average length of all responses (LEN) and the average length of correct responses (C_LEN). The calculation formula is as follows: In the formula, res i Represents the i-th response, res i=T This indicates a correct response.

[0043] Performance evaluation is based on the average accuracy metric from multiple inferences. For each question, the model is optimized and reasoned multiple times. The accuracy of each reasoning is judged. Correct answers are scored 1, and incorrect answers are scored 0. The average score is the score for that question. The average score of all questions in the evaluation set is the evaluation score of that evaluation set on the optimization model.

[0044] Specifically, To evaluate the model's inference performance, we perform 64 inferences for each question in the evaluation set and assess the accuracy of these 64 responses, awarding 1 for correct responses and 0 for incorrect responses. The average score of the 64 responses is the score for that evaluation data point, and the average score of all data points in the evaluation set is the final evaluation score for that dataset on that specific model.

[0045] Efficiency evaluation involves taking the length of each response (tokens) based on the performance evaluation results and setting efficiency metrics LEN and C_LEN, where LEN represents the average length of all responses and C_LEN represents the average length of all correct responses. Smaller metrics indicate higher inference efficiency.

[0046] The effects of this invention include: (1) Reasoning performance remains unchanged: Compared to version V1, Jiusi-R1's V2 version maintains essentially the same inference performance. Tests were conducted on five authoritative evaluation sets, showing an average performance decrease of 0.37% and a maximum performance increase of 0.24%, as shown in Table 2. Table 2 (2) Reasoning efficiency is greatly improved: Compared to version V1, Jiusi-R1's V2 version has significantly improved inference efficiency. It was tested on five authoritative evaluation sets, and the response length was significantly reduced, with an average efficiency improvement of 32.67% and a maximum efficiency improvement of 36.56%.

[0047] The inference performance on the AIME24, AIME25, and MATH_500 datasets is shown in Table 3: Table 3 The inference performance on the GPQA and GSM8K datasets is shown in Table 4: Table 4 Before and after comparison images are shown. Figure 6 As shown.

[0048] Example 1 This invention provides a concise inference training scheme example for a 32B model with adaptive difficulty, including the following specific implementation steps: 1) The 32B version of Jiusi-R1-V1.0.0 is used as the starting model, which is also a self-trained iterative model.

[0049] 2) The Openr1_default, Lightr1, GSM8K, and MATH datasets were used as the raw materials for data generation, as shown in Table 5: Table 5 3) Four types of prompts are used: normal reasoning prompts, simple thinking prompts, deep thinking prompts, and Few-Shot prompts. Each input math problem is driven by one of the four types of prompts to generate a response, with each type of prompt generating eight responses, for a total of 32 responses per input problem.

[0050] 4) Based on the designed reward formula and pairing algorithm, and after threshold filtering (reward score threshold 0.5, response length threshold 800), the following preference data pairs were finally generated, as shown in Table 6: Table 6 The length distribution of response pairs is as follows Figure 3 As shown.

[0051] Model training The training model uses the LLaMA_Factory project, and the key hyperparameter configurations are shown in Table 7. Table 7 Training process monitoring curves as follows Figure 8 , 9 As shown.

[0052] 6) Performance and efficiency evaluation of the trained Jiusi-R1-V2.0.0-32B model are as follows: Figure 6 , 7 As shown.

[0053] Advantages of this invention: This invention focuses on a self-training approach, employing self-training to drive budget-sensitive data generation to unlock and optimize the potential refined inference capabilities observed in current large models. Self-training not only eliminates external dependencies but also helps maintain inference capabilities because the training data originates from the model's own distribution characteristics. Furthermore, guided by different types of cue words, the self-training method can generate highly discriminative and length-sensitive preference responses based on the model's own characteristics.

[0054] This invention addresses the inherent need for varying inference depths across tasks of different difficulty levels. It establishes a mapping between problem complexity and target response length, achieving difficulty adaptation and a budget-aware mechanism. By comparing the current response length with the target response length, it determines whether additional rewards or penalties should be applied to the current answer. First, it introduces a difficulty quantification metric called "Label Length Budget" (TLB), which integrates the accuracy of sampled responses and their length distribution characteristics. This metric effectively combines problem difficulty features with label length information. A budget-aware reward shaping mechanism is employed for multiple generated response samples. Responses exceeding the TLB for simple problems receive a penalty signal, while responses close to the TLB for complex problems receive positive incentives. This mechanism allows us to construct budget-biased training data, which inherently encodes the relationship between problem difficulty and target response length.

[0055] This invention employs a combination of multiple pairing algorithms and uses threshold filtering to construct the final training preference pair dataset. When the pairing format and algorithms are appropriately selected and designed, the resulting preference pairs accurately reflect differences in inference efficiency and guide the model towards the selected direction for optimization.

[0056] This invention employs length-sensitive preference training using the Simpo algorithm, which uses the average log probability of the sequence as an implicit reward. This reward format aligns better with model generation and eliminates the need for a reference model, making it more computationally and memory efficient. Furthermore, a target reward margin is introduced into the optimization objective to encourage larger margins between winning and losing responses, further improving the algorithm's performance. The inclusion of a length-normalized implicit reward in the optimization method results in shorter sequence lengths relative to the DPO (Difference Probability of Positive Prediction).

[0057] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A simplified inference model training method based on difficulty adaptation, characterized in that, include: Data preparation involves using multiple prompt words to drive the starting point model based on the raw data to generate multiple responses for each input question, forming a response set. For each problem, a tag length budget is calculated based on the response set, wherein the tag length budget integrates response accuracy and length distribution characteristics, and a reward value for each response is calculated based on the tag length budget and response correctness; Preference data pairs are constructed based on the reward values ​​using multiple pairing algorithms and filtered by a reward difference threshold to obtain preference data pairs for training. The pairing algorithms include head-tail method, round-robin method, and shortest method. Based on the aforementioned preference data pairs, a length-sensitive preference training algorithm is used to train the inference model to be optimized, generating an optimized model with a concise response. The optimization model is then evaluated for performance and efficiency.

2. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: The starting model is a pre-trained inference model. The dataset includes a multi-step mathematical problem dataset and a complex logic proof dataset. The multi-step mathematical problem dataset includes GSM8K and MATH, and the complex logic proof dataset includes Openr1_default and Lightr1.

3. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: The various cue words include normal reasoning cue words, simple thinking cue words, deep thinking cue words, and Few-Shot cue words. Each cue word is used to generate responses of different lengths to ensure response diversity.

4. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that, The mark length budget L budget The calculation formula is: Where c represents the number of correct responses in the response set, and N represents the total number of responses in the response set. L represents the average length of a correct response. max This indicates the maximum length of the model output; The formula for calculating the reward value is as follows: Among them, L i This represents the length of the i-th response; select the corresponding reward value calculation formula for correct and incorrect responses.

5. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that, Preference data pairs are constructed based on the reward values ​​using multiple pairing algorithms and filtered through a reward difference threshold to obtain preference data pairs for training, including: The response set is sorted in descending order of the reward value, and correct-correct response pairs and correct-incorrect response pairs are constructed as preference data pairs. For correct-correct response pairs, the head-tail method or round-robin method is used for pairing, and the pairing is filtered by the reward difference threshold. For correct-incorrect response pairs, the shortest method is used for pairing.

6. The method for training a concise inference model based on difficulty adaptation according to claim 5, characterized in that: The head-and-tail method includes pairing the longest and shortest correct responses in the response set, and the round-robin method includes pairing the shortest top 3 and the longest top 3 responses in a round-robin fashion.

7. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: The length-sensitive preference training algorithm is the SimPO algorithm, which uses the average log probability of the sequence as an implicit reward and introduces a target reward margin to optimize training efficiency.

8. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: The efficiency evaluation is based on the response length metric, including the average length of all responses (LEN) and the average length of correct responses (C_LEN), calculated using the following formula: Among them, res i Represents the i-th response, res i=T This indicates a correct response.

9. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: The starting model adopts the Jiusi-R1-V1.0.0-32B model, and the dataset includes GSM8K, MATH, Openr1_default, and Lightr1.

10. The method for training a concise inference model based on difficulty adaptation according to claim 1, characterized in that: Performance evaluation is based on the average accuracy metric from multiple inferences. For each question, the optimization model is used to perform multiple inferences, and the accuracy of each inference is judged. Correct answers are scored 1, and incorrect answers are scored 0. The average score is the score for that question. The average score of all questions in the evaluation set is the evaluation score of that evaluation set on the optimization model.

Images