Inference big language model-oriented difficult problem data synthesis method and system

Abstract

The invention discloses a difficult problem data synthesis method and system for an inference large model, and the method comprises the steps: carrying out the concept extraction of an existing mathematical data set, and obtaining a concept entity containing a mathematical concept, an application scene and an example; performing difficult problem synthesis on the randomly extracted concept group based on an existing reasoning big language model, generating an answer with a long reasoning chain form, and establishing a candidate difficult problem data set; and a method based on rules and large language model verification is adopted, correct difficult problems are screened out from the candidate data set, and a final difficult problem data set is obtained through long reasoning chain answering and reference answers. The invention provides an effective data synthesis method, a mathematical problem with high quality and high difficulty can be constructed, training and evaluation of a large language model are effectively supported during long reasoning chain solution, and a solid data basis and a technical path are provided for improving the model capability in a complex reasoning task in the future.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and system for synthesizing challenging data for large language models of reasoning. Background Technology

[0002] Recently, large language models have made significant progress in mathematics, programming, science, and general domains. Representative works include ChatGPT o1, DeepSeek-R1, and Google Flash Thinking. These models first "think" in the form of long inference chains before outputting the answer. This type of large model, which uses long inference chains as additional reasoning costs to improve the quality of the final response, is called a large inference model.

[0003] Training large-scale reasoning models relies on challenging problems, as these facilitate the generation of more complex and longer reasoning chains, a key factor in training high-level reasoning models. However, challenging problems are an extremely scarce resource. Given their importance and scarcity, synthesizing challenging problems using methods based on large language models is a natural approach. However, creating challenging problems is itself a very difficult task. Existing data synthesis methods, often based on traditional non-reasoning models, struggle to generate sufficiently challenging problems and lack guarantees on problem diversity and quality. Therefore, there is an urgent need for an efficient and scalable method to automatically construct high-quality challenging problem datasets to improve the complex reasoning capabilities of models. Summary of the Invention

[0004] The present invention aims to provide a method and system for synthesizing challenging data for large reasoning models, so as to improve the performance of models in complex reasoning tasks.

[0005] The technical solution to achieve the objective of this invention is: a method for synthesizing challenging data for large reasoning language models, comprising the following steps:

[0006] Concept extraction is performed on the original dataset of mathematical problems to obtain conceptual entities containing mathematical concepts, application scenarios, and examples, and a mathematical concept dataset is constructed.

[0007] Concept sampling is performed on the constructed mathematical concept dataset;

[0008] Based on the reasoning big language model, difficult problems are synthesized from groups of mathematical concepts, and solutions with long reasoning chains are generated to establish a candidate difficult problem dataset.

[0009] By employing a rule-based and large language model-based verification method, correct challenging problems, long reasoning chain solutions, and reference answers are selected from the candidate dataset to obtain the final mathematical challenging problem dataset.

[0010] Furthermore, concept extraction includes: classifying the difficulty of each mathematical problem in the original dataset of mathematical problems, using structured prompting engineering methods to guide the large inference model to extract mathematical concepts, including mathematical concepts, application scenarios, and relevant examples.

[0011] Furthermore, the difficulty grading includes: using a difficulty grading method based on the problem difficulty grading method defined by AoPS, and using AoPS difficulty grading method to score the mathematical problems in the original dataset from difficulty to 10; then, constructing a mathematical concept dataset, extracting 2 core mathematical concepts from mathematical problems with difficulty ≥4 in the original dataset, and extracting 1 core mathematical concept from mathematical problems with difficulty ≤3.

[0012] Furthermore, the structured suggestion engineering method includes: Based on the problem, solution process, and difficulty score, the method extracts core mathematical concepts according to rules; the concept extraction process follows these rules: first, the number of extracted concepts is controlled according to the difficulty of the problem; second, the extracted concepts must be self-consistent, independent, and independently understandable; finally, the extraction form of the concepts is specified, and for each mathematical problem and its solution, a standardized conceptual entity containing the following elements will be output:

[0013] Mathematical concepts: First, state the name of the concept, then provide a detailed explanation of it;

[0014] Example: Provide a minimal example to demonstrate the application of the concept;

[0015] Application scenarios: Provide two differentiated application scenarios to illustrate the scenarios in which the concept can be applied.

[0016] Furthermore, the concept sampling includes: since mathematical problems of different difficulties involve different levels of knowledge points, the difficulty of the problem needs to be fully considered when sampling, and a sampling strategy is used to control the difficulty.

[0017] The sampling strategy includes: firstly, constructing a difficulty dictionary, classifying mathematical concepts according to the difficulty of the original problem, and forming a dictionary grouped by difficulty; ensuring that concepts of specific difficulty are quickly obtained during sampling to improve sampling efficiency;

[0018] Secondly, the sampling range was set, and the experiment selected concepts whose original problem difficulty was between 3 and 7.

[0019] Finally, random sampling is performed, randomly selecting four concepts from the concept set of the selected difficulty level as a group, allowing the model to explore diverse combinations of concepts.

[0020] Furthermore, the problem synthesis includes: using a set of strict problem generation constraints, generating high-quality and difficult problems and corresponding long reasoning chain solutions for the extracted concept groups through a large reasoning model, and combining the generated problems and solutions to form a candidate problem dataset; the long reasoning chain form includes: consisting of multiple interrelated intermediate reasoning steps, each of which is an indispensable logical reasoning unit in the problem-solving process.

[0021] Furthermore, the verification methods include: using a rule-based approach to initially screen each problem in the candidate problem dataset to ensure that the problems conform to mathematical logic and grammatical rules; and using a large language model to further verify the problems after the initial screening to ensure the correctness of the problem-solving process and the final answer.

[0022] Furthermore, the verification methods include: the experiment uses a large inference model as the verification model, samples the generated questions in multiple rounds, and uses a comparison strategy to filter questions that meet the standards; for each generated mathematical question, the experiment uses the verification model to solve it independently in multiple rounds, obtaining multiple solution results; the temperature parameter is set to 0.7 to ensure that the model has a certain degree of randomness in the inference process, thereby detecting the stability of the questions; after obtaining multiple rounds of solutions, the experiment compares the different solution results, selects questions with completely consistent solutions (i.e., complete matching), and if the final answer obtained from all multiple rounds of inference is consistent, then the question is considered to have passed the verification and enters the final mathematical problem dataset.

[0023] Based on the same inventive concept, the present invention also provides a problem data synthesis system for large reasoning language models, used to implement the method described in any one of claims 1 to 9, the system comprising:

[0024] The first module is used to extract concepts from the original dataset of mathematical problems, obtain conceptual entities containing mathematical concepts, application scenarios and examples, and construct a mathematical concept dataset.

[0025] The second module is used for concept sampling of the constructed mathematical concept dataset;

[0026] The third module is used to synthesize challenging problems from groups of mathematical concepts based on a large reasoning language model, generate solutions with long reasoning chains, and establish a candidate challenging problem dataset.

[0027] The fourth module is used to select the correct challenging problems, long reasoning chain solutions, and reference answers from the candidate dataset using rule-based and large language model verification methods, thus obtaining the final mathematical challenging problem dataset.

[0028] Compared with the prior art, the significant advantages of this invention are: (1) This invention ensures that the generated problem dataset is of high quality and high difficulty through multiple steps such as concept extraction, concept sampling, problem synthesis and problem verification; (2) This invention is based on a wide range of mathematical concept datasets and random sampling strategies to ensure the diversity of generated problems; (3) The problems generated by this invention and their long reasoning chain solutions can effectively support the training and evaluation of large reasoning models; using the mathematical problem dataset generated by the method of this invention to fine-tune the model can significantly improve the model's performance on multiple tasks. Attached Figure Description

[0029] Figure 1 The overall structure of the data synthesis method for difficult problems.

[0030] Figure 2 Provide hints for extracting mathematical concepts.

[0031] Figure 3 Examples of extracted mathematical concepts.

[0032] Figure 4 Hints for constructing mathematical problems.

[0033] Figure 5 Examples of generated mathematical problems.

[0034] Figure 6 This section explains the parameters used for model fine-tuning.

[0035] Figure 7 Evaluation results of the model fine-tuned on the datasets created for this method and benchmark datasets. Detailed Implementation

[0036] In training large-scale reasoning language models, challenging problems play a crucial role. These problems typically contain more complex semantic relationships and longer reasoning paths, effectively guiding the model to learn multi-step reasoning logic and are a core factor in improving reasoning performance. However, high-quality challenging problems are extremely limited and costly to obtain, hindering further improvements in reasoning model capabilities.

[0037] Automatic generation of challenging problems based on large language models is gradually becoming an important research direction due to its high generation efficiency and strong scalability. For example, WizardLM-Math proposes the Evol-Instruct synthesis method, which generates mathematical instruction data of different difficulties and complexities through downward and upward evolution; MetaMath uses methods such as answer enhancement, restatement, backward reasoning, self-verification, and FOBAR to synthesize data, thereby expanding the diversity and complexity of problems. Although the above methods have improved the quality of synthesized data to some extent, they are still mostly based on traditional model construction lacking deep reasoning capabilities, and cannot yet generate truly complex problems with high cognitive challenge. The creation of problems is itself a highly complex task, requiring extremely high levels of problem structure and logical coherence. Existing methods have not yet fully met the data quality and difficulty levels required for training large reasoning models.

[0038] To address the aforementioned problems, this invention proposes a method for synthesizing challenging problem data for large-scale reasoning models, comprising the following steps: Step 1, extracting concepts from the original mathematical problem dataset to construct a mathematical concept dataset; Step 2, sampling concepts from the mathematical concept dataset; Step 3, synthesizing challenging problems from the mathematical concept groups based on a large-scale reasoning language model, generating solutions with long reasoning chains, and establishing a candidate challenging problem dataset; Step 4, using rule-based and large-scale language model verification methods, selecting correct challenging problems, long reasoning chain solutions, and reference answers from the candidate dataset to obtain the final mathematical challenging problem dataset.

[0039] The specific steps of the present invention’s method for synthesizing challenging data for large-scale reasoning models will be described in detail below.

[0040] like Figure 1 As shown, the method includes the following steps:

[0041] Step 1: Extract concepts from the original dataset of mathematical problems to construct a mathematical concept dataset.

[0042] The difficulty rating method used is based on the problem difficulty rating method defined by the Art of Problem Solving (AoPS). The original dataset's math problems are rated from 1 to 10 using AoPS's difficulty rating method. Then, a math concept dataset is constructed, extracting two core math concepts from math problems with a difficulty ≥4 and one core math concept from math problems with a difficulty ≤3 in the original dataset.

[0043] Concept extraction methods are based on questions, solution processes, and difficulty scores, and extract core mathematical concepts according to rules, such as... Figure 2As shown. The concept extraction process follows these rules: First, the number of concepts extracted is controlled according to the difficulty of the problem. Second, the extracted concepts must be self-consistent, independent, and independently understandable. Finally, strict requirements are placed on the form of concept extraction. For each mathematical problem and its solution, a standardized conceptual entity containing the following elements will be output:

[0044] Mathematical concepts: First, state the name of the concept, and then provide a detailed explanation of the concept.

[0045] Example: Provide a minimal example to demonstrate the application of the concept.

[0046] Application scenarios: Provide two differentiated application scenarios to illustrate the scenarios in which the concept can be applied.

[0047] Finally, all extracted mathematical concepts are merged to construct a mathematical concept dataset, such as... Figure 3 As shown.

[0048] Step 2: Perform concept sampling on the mathematical concept dataset

[0049] Since mathematical problems of varying difficulty involve different levels of knowledge, this study fully considered the difficulty of the problems during concept sampling and adopted the following strategies: First, a difficulty dictionary was constructed, classifying mathematical concepts according to the difficulty of the original problem, forming a dictionary grouped by difficulty. This ensured the rapid acquisition of concepts of specific difficulty during sampling, improving sampling efficiency. Second, a sampling range was set. To ensure that the selected concepts had a certain level of complexity, while avoiding interference from overly simple or extremely difficult knowledge points on the synthesis task, the experiment selected concepts with an original problem difficulty between 3 and 7. This helped generate challenging but still solvable problems while covering a wide range of mathematical topics. Finally, random sampling was performed, randomly selecting four concepts from the concept set of the selected difficulty level as a group, allowing the model to explore diverse concept combinations, which helped increase the difficulty of the synthesized problems.

[0050] Step 3: Based on the reasoning big language model, synthesize difficult problems from groups of mathematical concepts and generate solutions with long reasoning chains to establish a candidate difficult problem dataset.

[0051] After concept sampling, a large-scale reasoning model is used to synthesize the sampled concept groups to generate new questions and solutions with long reasoning chains. The long reasoning chain consists of multiple interconnected intermediate reasoning steps, each of which is an indispensable logical reasoning unit in the problem-solving process. To ensure the success rate and quality of the synthesized questions, a set of strict constraints on question generation is designed, such as... Figure 4 It employs various construction techniques to ensure that the generated problems have logical depth and are not merely a simple superposition of computational steps.

[0052] First, set a clear goal: to create a mathematical problem that deeply integrates multiple concepts and involves a complex logical reasoning process.

[0053] Secondly, six design requirements were established: (1) Problem singularity: The problem should be a single problem and should not be broken down into multiple subproblems; (2) Deep integration of concepts: The problem should involve at least two different categories of mathematical concepts, and the concepts used should interact naturally rather than be isolated calculations; (3) Non-trivial solution path: The solution process should involve no less than three steps of logical reasoning; (4) Bottleneck of logical depth: A key step is introduced where the solver must pause and think deeply before continuing, avoiding problems where the difficulty lies only in more steps rather than requiring deeper reasoning; (5) Hidden constraints and self-consistency: The problem contains constraints that are not explicitly stated but can be deduced from the context, ensuring that different parts of the problem restrict each other. This approach ensures that the problem has only one solution, allowing one part of the problem to naturally control the form of another, thereby creating a tight logical structure; (6) Reverse design method: Instead of simply stacking steps, the problem is built from a requirement, ensuring that the problem is coherent and elegant, rather than just a series of disjointed calculations.

[0054] Based on this, four construction techniques are provided to help construct the problem: (1) Hide explicit instructions: Instead of directly stating an instruction, state a condition to force the solver to discover how these steps are related; (2) Cleverly embed constraints: Express logical or mathematical constraints in embedded sentences instead of listing them directly; (3) Effectively interweave concepts: Alternate multiple related knowledge points in the problem description to create interference in the solution; (4) Introduce unexpected twists: Design questions that seem conventional but have a twist in the middle or later part to guide the solver to re-examine the conditions.

[0055] Finally, the output format of the problem is defined, including the problem statement and the solution. To ensure the validity of the problem, the format of the synthesized model is validated to ensure that it is a valid mathematical problem. Using the above methods, highly difficult mathematical problems are successfully synthesized and corresponding long reasoning chain solutions are generated, establishing a candidate problem dataset, such as... Figure 5 As shown.

[0056] Step 4: Based on the method of rule-based and large language model verification, the correct problem, long reasoning chain solution and reference answer are selected from the candidate dataset to obtain the final mathematical problem dataset.

[0057] Since the newly generated questions are derived through concept synthesis without manual annotation, they may contain logical flaws or be unsolvable. Validation can filter out illogical or erroneous questions. To ensure the quality of the final training dataset, the generated questions need to be rigorously screened.

[0058] To achieve the above objectives, a powerful inference model is used as the validation model. The generated questions are sampled multiple times, and a comparison strategy is used to filter questions that meet the criteria. For each generated mathematical problem, the validation model performs multiple independent solutions, obtaining multiple solution results. A temperature parameter of 0.7 is set to ensure a certain degree of randomness in the model's inference process, thereby testing the stability of the questions. After obtaining multiple solutions, the different solution results are compared. Questions where all solutions are completely identical (i.e., a perfect match) are strictly selected. If the final answer obtained from all multiple rounds of inference is consistent, the question is considered to have passed the validation and is included in the final dataset.

[0059] This invention innovatively proposes challenging problem data synthesis as a training objective for large-scale reasoning models, defines a corresponding synthesis method, solves the problem of challenging problem data synthesis for large-scale reasoning models, and forms a method to support model training. It also innovatively proposes a novel challenging problem synthesis strategy based on concept sampling, forming a challenging problem data synthesis method.

[0060] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0061] Example

[0062] The embodiments of this invention aim to illustrate a method for generating high-quality mathematical problems. This embodiment will describe in detail the steps of dataset construction, the model used, and the specific parameter settings.

[0063] This embodiment extracts 7500 questions from the Sky-T1-32B training dataset of the inference model to construct the original dataset of mathematical questions. Mathematical concepts are extracted using structured prompting engineering methods to obtain concept entities containing mathematical concepts, application scenarios, and examples, ultimately yielding 9054 valid mathematical concepts for constructing the mathematical concept dataset. From these 9054 mathematical concepts, concepts with a difficulty level between 3 and 7 are selected. Four concepts are randomly selected from each difficulty level as a group, with 1000 groups selected for each difficulty level, for a total of 5000 concept groups. These 5000 concept groups are synthesized using the QwQ-32B inference model. After format validation, 4269 mathematical questions are successfully synthesized, and corresponding long inference chain solutions are generated. Next, the inference model is used to validate the synthesized 4269 questions; 1000 questions meet the requirements, and these 1000 mathematical questions and their corresponding long inference chain solutions are merged. Finally, this study obtains an SD1k dataset containing 1000 high-quality mathematical questions and long inference chain solutions.

[0064] Further model training and evaluation were performed on the constructed SD1k dataset. For example... Figure 6 As shown, the experiment uses Qwen2.5-32B-Instruct as the base model, and performs LORA fine-tuning using the benchmark dataset and the SD1k dataset. Specific hyperparameters are set as follows: batch size per device is 1, gradient accumulation steps are 2, learning rate is 2.0e-4, training epochs are 4, a cosine learning rate scheduler is used, warm-up ratio is 0.05, weight decay is 1.0e-4, and the beta1 and beta2 parameters of the Adam optimizer are 0.9 and 0.95, respectively. To comprehensively evaluate the model's inference ability, this embodiment uses three evaluation datasets: MATH500, AIME2024, and GPQADiamond. MATH500 contains 500 high school math problems, AIME2024 contains 30 challenging high school math competition problems, and GPQADiamond contains 198 doctoral-level scientific problems. The results are as follows... Figure 7As shown, the model fine-tuned on SD1k achieves significant advantages on all three evaluation tasks, with accuracy improvements of 6.5%, 16.6%, and 12.1% compared to the original base model, respectively; improvements of 3.2%, 10%, and 8.6% compared to the MetaMathQA benchmark dataset; and improvements of 3.2%, 3.3%, and 6.6% compared to the best benchmark. This result fully demonstrates that the SD1k dataset effectively enhances the ability of large language models to handle challenging tasks. It not only significantly improves the model's performance in complex mathematical reasoning but also exhibits excellent cross-task generalization ability in the GPQA Diamond evaluation, validating the effectiveness of the data synthesis method proposed in this embodiment in generating high-quality challenging questions. It also reflects the important role of challenging data in improving the reasoning ability of large language models. The SD1k dataset not only provides scarce, high-quality challenging question resources for training large reasoning models but also offers a scalable challenging data synthesis method for subsequent research.

[0065] The above description is merely for illustrating the technical solutions and implementation methods of the present invention, aiming to help understand the core ideas and concepts of the present invention, and is not intended to limit the scope of protection of the present invention. Although this specification describes the technical features of each embodiment in detail, all possible combinations of these technical features, unless listed one by one, should be considered part of the present invention as long as there is no logical or technical conflict. For those skilled in the art, any changes, substitutions, improvements, or functionally equivalent extensions made to the above embodiments without departing from the core ideas of the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A method for synthesizing challenging problem data for large reasoning language models, characterized in that, Includes the following steps: Concept extraction is performed on the original dataset of mathematical problems to obtain conceptual entities containing mathematical concepts, application scenarios, and examples, and a mathematical concept dataset is constructed. Concept sampling is performed on the constructed mathematical concept dataset; Based on the reasoning big language model, difficult problems are synthesized from groups of mathematical concepts, and solutions with long reasoning chains are generated to establish a candidate difficult problem dataset. By employing a rule-based and large language model-based verification method, correct challenging problems, long reasoning chain solutions, and reference answers are selected from the candidate dataset to obtain the final mathematical challenging problem dataset.

2. The method according to claim 1, characterized in that, The concept extraction includes: classifying the difficulty of each mathematical problem in the original dataset of mathematical problems, and using structured prompting engineering methods to guide the large inference model to extract mathematical concepts. The extracted mathematical concepts include mathematical concepts, application scenarios, and related examples.

3. The method according to claim 2, characterized in that, The difficulty grading includes: using a difficulty grading method based on the problem difficulty grading method defined by AoPS, and using AoPS difficulty grading method to score the mathematical problems in the original dataset from difficulty from 1 to 10; then, constructing a mathematical concept dataset, extracting 2 core mathematical concepts from mathematical problems with difficulty ≥4 in the original dataset, and extracting 1 core mathematical concept from mathematical problems with difficulty ≤3.

4. The method according to claim 2, characterized in that, The structured suggestion engineering method includes: Based on the problem, solution process, and difficulty score, the method extracts core mathematical concepts according to rules. The concept extraction process follows these rules: First, the number of extracted concepts is controlled according to the difficulty of the problem; second, the extracted concepts must be self-consistent, independent, and independently understandable; finally, the extraction form is specified, and for each mathematical problem and its solution, a standardized conceptual entity containing the following elements will be output: Mathematical concepts: First, state the name of the concept, then provide a detailed explanation of it; Example: Provide a minimal example to demonstrate the application of the concept; Application scenarios: Provide two differentiated application scenarios to illustrate the scenarios in which the concept can be applied.

5. The method according to claim 1, characterized in that, The concept sampling includes: since mathematical problems of different difficulties involve different levels of knowledge points, the difficulty of the problem needs to be considered when sampling, and a sampling strategy is used to control the difficulty.

6. The method according to claim 5, characterized in that, The sampling strategy includes: firstly, constructing a difficulty dictionary, classifying mathematical concepts according to the original problem difficulty of the concepts, and forming a dictionary grouped by difficulty; ensuring that concepts of specific difficulty are quickly obtained during sampling; Secondly, the sampling range was set, and the experiment selected concepts whose original problem difficulty was between 3 and 7. Finally, random sampling is performed, and four concepts are randomly selected from the set of concepts at the selected difficulty level as a group.

7. The method according to claim 1, characterized in that, The problem synthesis includes: using a set of problem generation constraints, generating problems and corresponding long reasoning chain solutions for the extracted concept groups through a large reasoning model, and combining the generated problems and solutions to form a candidate problem dataset; the long reasoning chain form includes: consisting of multiple interrelated intermediate reasoning steps, each of which is an indispensable logical reasoning unit in the problem-solving process.

8. The method according to claim 1, characterized in that, The verification method includes: using a rule-based approach to initially screen each problem in the candidate problem dataset to ensure that the problems conform to mathematical logic and grammatical rules; and using a large language model to verify the problems after initial screening to ensure the correctness of the solution process and the final answer.

9. The method according to claim 8, characterized in that, The verification method includes: using a large inference model as the verification model, sampling the generated questions in multiple rounds, and using a comparison strategy to filter questions that meet the standards; for each generated mathematical question, the experiment uses the verification model to solve it independently in multiple rounds, obtaining multiple solution results; setting the temperature parameter to 0.7 to ensure that the model has a certain degree of randomness in the inference process, thereby detecting the stability of the questions; after obtaining multiple rounds of solutions, the experiment compares the different solution results, selects questions with completely consistent solutions (i.e., complete matching), and if the final answer obtained from all multiple rounds of inference is consistent, then the question is considered to have passed the verification and enters the final mathematical problem dataset.

10. A problem data synthesis system for large reasoning language models, characterized in that, The system for implementing the method according to any one of claims 1 to 9 comprises: The first module is used to extract concepts from the original dataset of mathematical problems, obtain conceptual entities containing mathematical concepts, application scenarios and examples, and construct a mathematical concept dataset. The second module is used for concept sampling of the constructed mathematical concept dataset; The third module is used to synthesize challenging problems from groups of mathematical concepts based on a large reasoning language model, generate solutions with long reasoning chains, and establish a candidate challenging problem dataset. The fourth module is used to select the correct challenging problems, long reasoning chain solutions, and reference answers from the candidate dataset using rule-based and large language model verification methods, thus obtaining the final mathematical challenging problem dataset.

Images