Method and system for synthesizing reasoning data

Abstract

The invention relates to a reasoning data synthesis method and system, and the method comprises the steps: obtaining a mathematical big language model, and carrying out the question fine adjustment of the mathematical big language model through a question fine adjustment method by employing a preset data set in a model training process, and obtaining the mathematical big language model processed through the question fine adjustment method; generating a sample problem through the mathematical big language model processed by the problem fine tuning method; the sample problem is optimized through a problem preference optimization method, and then the optimized sample problem is used for training the mathematical big language model processed through the problem fine adjustment method; taking the mathematical big language model processed by the problem fine tuning method and the problem preference optimization method as a problem generator, and filtering problems generated by the problem generator; a corresponding answer is generated for each filtered question to synthesize inference data. The reasoning data synthesis method not only can ensure the data quality, but also can reduce the generation cost and improve the generation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of inference data synthesis, and in particular to an inference data synthesis method and system. Background Art

[0002] In recent years, large language models (LLMs) have seen rapid development in the field of deep learning, particularly in natural language processing. These models have not only demonstrated exceptional performance and diverse application potential, but have also significantly accelerated the transition of AI technology from theoretical research to practical application, driving the rapid implementation of numerous innovative products and services. Existing research has widely demonstrated that the quantity and quality of data are crucial for the performance of large language models. However, with the increasing development and utilization of high-quality, publicly available data on the internet, coupled with increasingly stringent privacy regulations and policy restrictions, the supply of high-quality, real-world data may be at risk of exhaustion in the near future. To address this potential data scarcity crisis, academia and industry are actively exploring alternatives, with the research and application of synthetic data receiving increasing attention. This approach has the potential to not only alleviate data scarcity but also provide a more diverse and controllable data source for model training, thereby driving continued innovation and development in large language model technology.

[0003] With the rapid development of large language model technology, both commercial closed-source and academic open-source models have demonstrated remarkable improvements in capabilities. These models not only excel in instruction following, accurately understanding and executing a variety of complex tasks, but have also made significant progress in reasoning capabilities, enabling them to handle scenarios requiring logical thinking and problem-solving. This rapid improvement in capabilities is largely attributed to training with high-quality instruction and reasoning data. Of particular note, mathematical reasoning, as one of the representative tasks of reasoning, has recently become a research hotspot. Existing methods for synthesizing reasoning data primarily fall into two technical approaches: problem-driven and knowledge-driven. Problem-driven methods include, but are not limited to, question rewriting, instruction evolution, and question back-translation. These methods suffer from limited data diversity. This is because they rely on seed questions, and the generated new questions are highly similar to the seed questions, with only minor differences in terms or values. This lack of diversity limits their scalability. To address these technical issues, knowledge-driven methods have emerged. These methods generate new questions by constructing a knowledge base or concept graph and extracting key information from it, thereby increasing the diversity of questions and expanding the scale of question synthesis. However, existing problem-driven methods and knowledge-driven methods all rely on high-performance language models (such as GPT-4) to synthesize new problems. Due to the high cost of using the API of such models, it is difficult to achieve large-scale data generation, which limits the further development of reasoning data synthesis technology. In addition, some methods use the autoregressive properties of large language models, do not require a preset seed set or prompts, and directly guide the large language model to synthesize related problems, which has good scalability. However, most of the problems generated by this method are relatively simple or have certain logical errors, and the quality is relatively low, which cannot effectively improve the reasoning ability of the large language model. Therefore, this field urgently needs a reasoning data synthesis method that can significantly reduce generation costs and improve generation efficiency while ensuring data quality. Summary of the Invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem that the inference data synthesis method in the prior art cannot simultaneously guarantee data quality, generation cost and generation efficiency.

[0005] To solve the above technical problems, the present invention provides a method for synthesizing inference data, comprising:

[0006] Step S1: obtaining a mathematical language model, and during the model training process, fine-tuning the mathematical language model using a preset data set using a question fine-tuning method to obtain a mathematical language model processed by the question fine-tuning method;

[0007] Step S2: Generate sample questions using the large mathematical language model processed by the question fine-tuning method;

[0008] Step S3: Optimizing the sample questions using the question preference optimization method, and then training the large mathematical language model processed by the question fine-tuning method using the optimized sample questions;

[0009] Step S4: using the large mathematical language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filtering the questions generated by the question generator;

[0010] Step S5: Generate corresponding answers for each filtered question to synthesize reasoning data.

[0011] In one embodiment of the present invention, step S1, during the model training process, performs question fine-tuning on the mathematical language model using a preset data set by a question fine-tuning method, specifically comprising:

[0012] During the model training process, question fine-tuning is performed on the total dataset synthesized by two datasets with different characteristics in the mathematical language model through the question fine-tuning method, and only one round of fine-tuning is performed, wherein the mathematical language model is eepSeek-Math-7B-RL or Qwen2-Math-7B-Instruc; the two datasets with different characteristics differ in difficulty distribution and question types.

[0013] In one embodiment of the present invention, during the model training process, the loss function used when the mathematical large language model is fine-tuned using a preset data set through the question fine-tuning method is:

[0014] ;

[0015] in, Indicates that from the distribution The problem of sampling Take the expected value, eos represents the word that marks the end of the question, and Represents contextual prompts and questions respectively, Indicates the position of the word in the question, Indicates the total length of the question, Indicates the first word, Indicates the front of the question word, Indicates that the parameter is A mathematical language model.

[0016] In one embodiment of the present invention, the step S3 optimizes the sample questions using a question preference optimization method, and then uses the optimized sample questions to train the large mathematical language model processed by the question fine-tuning method, specifically including:

[0017] Optimizing the sample questions using a first large language model, and randomly optimizing the solvability or difficulty of each sample question by randomly selecting an optimization direction strategy during the optimization process, wherein the solvability refers to whether the question can be solved;

[0018] The optimized sample problem is then used to train the mathematical language model processed by the problem fine-tuning method. During the training process, the optimized sample problem is regarded as the preferred problem, and the original sample problem before optimization is regarded as the non-preferred problem. The loss function of the problem preference optimization method in the optimization process is constructed based on the preferred problem and the non-preferred problem. The formula is:

[0019] ;

[0020] in, Indicates the mathematical language model currently being optimized. represents the reference mathematical large language model, Expressing preference problems in datasets expectations, represents a dataset of preference question pairs consisting of questions before and after optimization of the first large language model, represents the optimized preferred problem, represents a non-preferred problem before optimization, express function, Represents the temperature parameter, which is used to control the degree of optimization.

[0021] In one embodiment of the present invention, the method of filtering the questions generated by the question generator in step S4 includes:

[0022] Language filtering: used to identify characters in non-preset languages ​​and filter questions in non-preset languages;

[0023] Solvability filtering: used to filter unsolvable questions using a preset language model to ensure that the generated questions are semantically coherent and have sufficient conditions.

[0024] Difficulty filtering: used to calculate the difficulty score of each question. If the difficulty score is outside the preset range, the question is filtered.

[0025] In one embodiment of the present invention, the method for calculating the difficulty score of each question in the difficulty filtering includes:

[0026] A difficulty scorer with a classification head is constructed. The classification head is used to calculate the difficulty score of each question. The calculation formula of the difficulty score is:

[0027] ;

[0028] The loss function of the difficulty scorer is constructed based on the difficulty score, and the formula is:

[0029] ;

[0030] in, and are the weight and bias of the classification head, represents the last hidden state of the vectorized representation of the input question sequence, is the loss function of the difficulty scorer, It is The predicted difficulty score of the question, Indicates the The actual difficulty score of the question.

[0031] In one embodiment of the present invention, in step S5, generating a corresponding answer for each filtered question to synthesize inference data, the method includes:

[0032] A second large language model is used to generate multiple answers for each filtered question, and Internlm2-7b-reward is used as a reward model to evaluate the score of each answer. The answer with the highest score from the reward model is selected as the preferred answer, and each filtered question and the corresponding preferred answer are synthesized into reasoning data.

[0033] To solve the above technical problems, the present invention provides an inference data synthesis system, comprising:

[0034] The first optimization module is used to obtain a mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset data set using a problem fine-tuning method to obtain a mathematical language model processed by the problem fine-tuning method.

[0035] Question generation module: used to generate sample questions using the large mathematical language model processed by the question fine-tuning method;

[0036] The second optimization module is used to optimize the sample problem by using the problem preference optimization method, and then train the large mathematical language model processed by the problem fine-tuning method with the optimized sample problem;

[0037] Question filtering module: used to use the large mathematical language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filter the questions generated by the question generator;

[0038] Data synthesis module: used to generate corresponding answers to each filtered question to synthesize inference data.

[0039] To solve the above technical problems, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned inference data synthesis method when executing the computer program.

[0040] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned inference data synthesis method are implemented.

[0041] The above technical solution of the present invention has the following advantages over the prior art:

[0042] The reasoning data synthesis method of the present invention includes four key steps: question fine-tuning, question preference optimization, question filtering, and answer generation. The question fine-tuning process activates the model's question generation capability by selecting datasets with different features to train the model, enabling it to generate diverse and cross-language questions. The question preference optimization stage introduces an improved direct preference optimization loss function, which effectively improves the solvability and difficulty of generated questions. The question filtering link adopts a multi-level filtering mechanism, including language filtering, solvability filtering, and difficulty filtering, to ensure the high quality of generated questions. The answer generation stage uses a large language model to generate multiple reasoning paths, and introduces a reward model score as a quality evaluation indicator to select the optimal answer. The series of technical innovations of the present invention significantly improve the quality and reliability of generated data, providing high-quality training data for complex reasoning tasks.

[0043] The method of the present invention not only solves the problems of low data generation quality and high cost in the existing technology, but also provides key support for breakthroughs in artificial intelligence reasoning capabilities. It has broad application prospects and important technical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0045] Figure 1 is a flow chart of the method of the present invention;

[0046] Figure 2 2 is an experimental analysis comparison diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0048] Example 1

[0049] Reference Figure 1 As shown, the present invention relates to a method for synthesizing inference data, comprising:

[0050] Step S1: obtaining a mathematical large language model, and during the model training process, fine-tuning the mathematical large language model using a preset data set using a question fine-tuning method to obtain two mathematical large language models processed by the question fine-tuning method;

[0051] Step S2: Generate sample questions using the large mathematical language model processed by the question fine-tuning method;

[0052] Step S3: Optimizing the sample questions using the question preference optimization method, and then training the large mathematical language model processed by the question fine-tuning method using the optimized sample questions;

[0053] Step S4: using the large mathematical language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filtering the questions generated by the question generator;

[0054] Step S5: Generate corresponding answers for each filtered question to synthesize reasoning data.

[0055] The following is a detailed introduction to this embodiment:

[0056] The data synthesis method of this embodiment mainly covers the following aspects: 1) training a basic question generator based on the question fine-tuning method; 2) optimizing the question generator in the first step based on the question preference optimization method to obtain an advanced question generator; 3) synthesizing a large amount of data through the question generator obtained in the previous step, and screening the questions based on language clarity, solvability, and appropriate difficulty; 4) generating multiple corresponding answers for the synthetic questions filtered in the previous step, and performing rejection sampling on the multiple answers generated for each question based on the reward score of the reward model, and selecting the answer with the highest reward score to form the final synthetic data set.

[0057] In this embodiment, the question synthesis process only involves providing a small number of prefix tokens in the instruction template (e.g., "<|begin_of_sentence|>User:") to guide the model to generate questions. This embodiment utilizes a fine-tuned causal language model that has learned to generate answers based on question-answer pairs (e.g., {<|begin_of_sentence|> User: {question}. Assistant: {answer}"). Such a model can potentially be used directly to generate questions. This is because during the instruction tuning process, the model is trained using causal masks, where each token only focuses on the previous token. This ensures that the hidden state evolves only based on past context and is not affected by future tokens. Therefore, the method of the present invention requires activating the model's ability to synthesize questions.

[0058] (1) Question Fine-Tuning Method: In this embodiment, the primary purpose of the Question Fine-Tuning (QFT) process is to activate the model's question generation capabilities, rather than forcing the model to memorize specific questions. To verify this hypothesis, this embodiment selected two datasets with different characteristics to form a total dataset for model training. These two datasets differ significantly in difficulty distribution and question types, ensuring that the model can learn diverse question generation patterns and serve as in-domain data for evaluating model performance. Specifically, this embodiment used the above total dataset to perform question fine-tuning on two large mathematical language models: DeepSeek-Math-7B-RL and Qwen2-Math-7B-Instruct. It should be noted that this embodiment uses two models for simultaneous training because the data generated by multiple models is more diverse and easier to analyze later. In other embodiments, a single model can also be used. Considering that excessive rounds of fine-tuning may lead to model overfitting and negatively impact the diversity of generated questions, this embodiment only performs one round of fine-tuning. In addition, this embodiment also uses sequence packing to accelerate training. To evaluate the distribution of generated questions. A large number of sample questions (e.g., 10,000) were generated for each model after question fine-tuning (QFT).

[0059] This embodiment also constructs a loss function for the problem fine-tuning stage, and the formula is:

[0060] ;

[0061] in, Indicates that from the distribution The problem of sampling Take the expected value, eos represents the word that marks the end of the question, and Represents contextual prompts and questions respectively, Indicates the position of the word in the question, Indicates the total length of the question, Indicates the first word, Indicates the front of the question word, Indicates that the parameter is A mathematical language model.

[0062] By comparing the difficulty distribution of these generated questions with the distribution of the original training dataset, the present invention found that the distribution of the generated questions differs from both original datasets, and despite being trained using different datasets, the distributions of questions generated by the two models that have undergone question fine-tuning (QFT) (referred to as QFT models) tend to be similar.

[0063] Notably, the QFT model, trained only with English questions, demonstrated the ability to generate questions in other languages. This phenomenon further demonstrates that the QFT process enhances the model's question generation capabilities, rather than simply overfitting the training data. Through the above experiments and analysis, the present invention confirms that the QFT process indeed improves the model's question generation capabilities, rather than simply memorizing the training data. This approach enables the model to generate diverse, moderately difficult, and cross-lingual questions, laying the foundation for the synthesis of large-scale, high-quality reasoning data.

[0064] (2) Question Preference Optimization Method: In this example, the Question Preference Optimization (QPO) process aims to further improve the quality of questions generated after Question Fine-tuning (QFT). Although the QFT process enables the model to generate meaningful and diverse questions, there is still room for improvement in terms of solvability and difficulty.

[0065] To solve these two key problems, this embodiment adopts the problem preference optimization (QPO) technology. First, a large number of sample problems are generated using a model processed by QFT. Subsequently, 10,000 of these sample problems are optimized by an external large language model, such as GPT-4o-mini, with a focus on solvability and difficulty. Since optimizing these two aspects at the same time is challenging for large language models, this embodiment adopts a strategy of randomly selecting the optimization direction, that is, randomly optimizing the solvability or difficulty of each sample problem. The optimized sample problems are then trained and optimized on the mathematical large language model processed by the problem fine-tuning method. During the optimization process, the problem to be optimized is used as the preferred data, and the original problem before optimization is used as the non-preferred data. This embodiment optimizes the loss function of Direct Preference Optimization (DPO) to adapt to this method, and obtains the loss function of Problem Preference Optimization (QPO), which is formulated as follows:

[0066] ;

[0067] in, Indicates the mathematical language model currently being optimized (the model after QFT), Indicates the reference mathematical large language model (the model after QFT), Expressing preference problems in datasets expectations, represents a dataset containing preference question pairs (preference question pairs composed of questions before and after GPT-4o-mini optimization), represents the optimized preferred problem, represents a non-preferred problem before optimization, express function, Represents the temperature parameter, which is used to control the degree of optimization. It should be noted that and The same at the beginning of training, but as the number of training rounds increases, Still remains the same, but It is changing.

[0068] The aforementioned problem optimization process places high demands on the model's ability to follow complex instructions. During the QPO phase, this example experiment set a learning rate of 5e-7 and a batch size of 128. To evaluate improvements in solvability and difficulty, this example employed a specialized evaluation model and prompts. Experimental results demonstrate that the QPO process significantly improves the quality of generated problems, particularly in terms of solvability and difficulty. This approach not only ensures the solvability of generated problems but also increases their overall difficulty, thereby providing higher-quality training data for subsequent reasoning tasks.

[0069] (3) Question Generation and Filtering: In this example, question filtering is a key step in ensuring the quality of generated questions. After question fine-tuning (QFT) and question preference optimization (QPO), this example obtained two question generators (based on DeepSeek-Math-7B-RL and Qwen2-Math-7B-Instruct). This example then used these two question generators to synthesize 1M pieces of data, for a total of 2M questions. In terms of parameter settings, this example set the maximum length of generated questions to 512, the temperature to 1.0, and the top-p value to 0.99. However, the synthesized questions still had some subtle issues, primarily related to language, solvability, and difficulty. To address these challenges, this example employed the following filtering steps: 1) Language Filtering. Although this invention primarily focuses on English math problems, the question generator still generates approximately 20% of questions in other languages. To this end, this example effectively removes non-English questions by identifying questions containing non-English characters and filtering out these sample questions. 2) Solvability Filtering Although the QPO process significantly improves the solvability of generated questions, some questions still have semantic incoherence or insufficient conditions. This is mainly due to improper condition setting (such as missing conditions, redundant conditions, or logical inconsistency) and the inability of the questions to produce meaningful results (such as the answer to the number of people involved should be a non-negative integer). To filter out such sample questions, this embodiment uses a special language model (such as the Qwen2-Math-Instruct model) to evaluate the meaningfulness of the questions and the sufficiency of the conditions. 3) Difficulty sampling. This embodiment uses the failure rate to measure the difficulty of the question, that is, the proportion of incorrect answers when sampling n answers to a given question. This indicator is consistent with intuition, that is, more difficult questions usually lead to fewer correct answers. Specifically, this embodiment uses a special model (DeepseekMath-7B) to evaluate the difficulty of each question in the training set, and uses this to train a difficulty scorer. The difficulty scorer is built based on the existing language model and calculates the difficulty score by adding a classification head. The formula for calculating the difficulty score is as follows:

[0070] ;

[0071] The loss function of the difficulty scorer is constructed based on the difficulty score, and the formula is:

[0072] ;

[0073] in, and are the weight and bias of the classification head, represents the last hidden state of the vectorized representation of the input question sequence, is the loss function of the difficulty scorer, It is The predicted difficulty score of the question, Indicates the The actual difficulty score of the question.

[0074] Through the above method, the present invention can predict the difficulty score of each generated question, and if the difficulty score is outside the preset range, the question is filtered. Of course, this embodiment can also sample questions based on difficulty: specifically, for certain question generators that generate overly simple questions, this embodiment will filter them; and for question generators with a more balanced difficulty distribution, no additional sampling is required. This multi-level question filtering method ensures that the final generated set of questions meets high standards in terms of language, solvability, and difficulty, providing high-quality training data for subsequent reasoning tasks. Through this innovative filtering mechanism, the present invention significantly improves the quality of generated questions.

[0075] (IV) Answer Generation: In this embodiment, the answer generation phase employs a reward model-based approach to ensure the high quality of the final dataset. This embodiment utilizes a large language model to generate multiple reasoning paths for each question, increasing the likelihood of obtaining high-quality answers. To assess answer quality, this embodiment introduces a reward model score as a core metric. Specifically, this embodiment sets the maximum generation length to 2048, the temperature to 0.7, and the top-p value to 0.99. Furthermore, this embodiment employs a thought chaining approach to synthesize answers, and uses the vLLM and Ray frameworks for distributed reasoning to accelerate answer generation. Internlm2-7b-reward is used as the reward model. This approach has broad applicability and is particularly well-suited for complex reasoning tasks where there is no single correct answer. Specifically, this embodiment generates five solutions for each question and selects the solution with the highest reward model score as the preferred option. The optimal solution selected and the corresponding question are then combined into reasoning data.

[0076] This demonstrates that, through this innovative answer generation and screening mechanism, the present invention can provide high-quality answers to each question, ensuring the overall quality of the final dataset and providing a reliable reference standard for a variety of complex reasoning tasks. This approach significantly improves the quality and reliability of generated data, providing key technical support for the synthesis of large-scale, high-quality reasoning data.

[0077] The experiments and analysis are as follows:

[0078] The inference data synthesis method proposed in this paper has significant technical effects and advantages. By training a model on a synthesized, high-quality inference dataset, the present invention effectively verified the superiority of the generated data. During the verification process, the present invention employed a variety of advanced base models for instruction fine-tuning experiments, including general-purpose large language models (such as Mistral-7B and Llama3-8B) and models specifically optimized for the mathematical domain (such as DeepSeekMath-7B and Qwen2-Math-7B). This multi-model verification approach not only comprehensively evaluated the quality of the synthesized data but also demonstrated the universality and effectiveness of the present method.

[0079] The training method employed in this paper is highly efficient, achieving significant results in just three training cycles. This significantly shortens model optimization time and improves overall R&D efficiency. Furthermore, a carefully designed linear learning rate scheduling strategy is employed, including a 3% warm-up ratio, and differentiated peak learning rates for different model types (5e-5 for Llama3 and DeepSeekMath, and 1e-5 for other models). Subsequently, a cosine decay to zero is applied to ensure training stability and optimize final performance. This optimization strategy not only improves training efficiency but also enhances model performance on inference tasks, demonstrating the high quality of the synthesized dataset. Validation on models of various architectures and sizes demonstrates the robust adaptability and scalability of this method for various language models. This universality allows the method to be easily extended to a wider range of application scenarios and model types. More importantly, by synthesizing high-quality datasets, the proposed method effectively reduces the cost of acquiring large-scale annotated data. Furthermore, the optimized training strategy reduces computing resource consumption, further improving the overall cost-effectiveness.

[0080] This paper further validates the effectiveness and superiority of a low-cost, scalable data synthesis method based on a large language model through rigorous evaluation methods and diverse metrics. The evaluation process covers multiple difficulty levels, from basic to advanced, fully demonstrating the comprehensiveness and adaptability of the proposed method. Specifically, the paper evaluates model performance using four datasets of increasing difficulty. These include the widely used GSM8K (elementary school level) and MATH (competition level), as well as the more challenging College Math (university level) and Olympiad Bench (Olympiad level) benchmarks. This comprehensive evaluation system not only tests the model's performance on basic mathematical problems but also assesses its capabilities in advanced mathematical and complex reasoning tasks, thereby fully verifying the quality of the synthesized data and the effectiveness of model training. During the evaluation process, the correctness of the final answer is determined by extracting the final answer and comparing the answer equivalence. This evaluation method ensures the objectivity and reliability of the results. Notably, the generated outputs all employ natural language thought chain reasoning, achieved through greedy decoding, and do not rely on any external tool integration. This method not only demonstrates the model's reasoning ability, but also reflects the outstanding contribution of the present invention in improving the model's autonomous thinking and problem-solving capabilities. The present invention uses zero-sample pass rate as the main performance indicator, which can directly reflect the model's performance when faced with new, untrained problems. Through zero-sample testing, the present invention fully demonstrates the advantages of models trained based on synthetic data in generalization ability and practical application value, and further proves the remarkable effectiveness of the present invention's method in improving the overall performance of large language models.

[0081] The inference data synthesis method constructed in this paper has demonstrated significant performance advantages and wide applicability in practical applications. Comprehensive experimental evaluations have shown that this method not only significantly outperforms existing synthesis methods but also demonstrates excellent scalability in data scalability, providing a significant breakthrough in the field of artificial intelligence.

[0082] In terms of performance, the present invention has achieved significant improvements over the previous state-of-the-art technologies in both general base models and base models optimized specifically for mathematical domains (e.g. Figure 2), with average performance improvements ranging from 5.6% to 11.5%. It's particularly noteworthy that the Qwen2-Math-7B model trained with our invention achieved a zero-shot pass rate of 73.4% on the MATH benchmark, a score comparable to the GPT-4-Turbo model. Even more impressively, the model even surpassed its teacher model, Qwen2-Math-7B-Instruct, on out-of-domain tasks, achieving excellent scores of 89.7, 73.4, and 38.5 on the GSM8K, MATH, and Olympic benchmarks, respectively. Considering that the Qwen2-Math-7B-Instruct model undergoes Group Relative Policy Optimization (GRPO) and utilizes the powerful reward model Qwen2-Math-RM-72B, the fact that our invention achieves such remarkable results solely through instruction fine-tuning further underscores its innovativeness and effectiveness.

[0083] In terms of scalability, this method demonstrates excellent data scalability. By combining the dataset constructed using this method with publicly available datasets (such as MetaMath, DART-Math, and Numina-Math), the method trained the Llama3-8B model and observed how its performance changed with increasing data size. The results demonstrated that in in-domain evaluations (MATH), this method demonstrated extremely high data efficiency, achieving superior results with the same amount of data. In out-of-domain evaluations (Olympic benchmarks), this method also demonstrated strong scalability, achieving continued improvement even when other datasets reached their performance ceilings.

[0084] Our approach successfully overcomes the performance bottleneck caused by a limited set of questions. For example, the DART-Math dataset's performance improvement was limited by its reliance on a small number of questions and its high number of correct answers generated through rejection sampling. In contrast, our approach supports sustained performance growth by generating a diverse set of questions, demonstrating the importance of generating a broad and diverse set of questions.

[0085] Example 2

[0086] This embodiment provides an inference data synthesis system, including:

[0087] The first optimization module is used to obtain a mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset data set using a problem fine-tuning method to obtain a mathematical language model processed by the problem fine-tuning method.

[0088] Question generation module: used to generate sample questions using the large mathematical language model processed by the question fine-tuning method;

[0089] The second optimization module is used to optimize the sample problem by using the problem preference optimization method, and then train the large mathematical language model processed by the problem fine-tuning method with the optimized sample problem;

[0090] Question filtering module: used to use the large mathematical language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filter the questions generated by the question generator;

[0091] Data synthesis module: used to generate corresponding answers to each filtered question to synthesize inference data.

[0092] Example 3

[0093] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the inference data synthesis method described in the first embodiment are implemented.

[0094] Example 4

[0095] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the inference data synthesis method described in the first embodiment are implemented.

[0096] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application may be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0097] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0100] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0101] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for synthesizing inference data, characterized in that: include: Step S1: obtaining a mathematical large language model, and in the process of model training, performing question fine-tuning on the mathematical large language model using a preset data set by a question fine-tuning method to obtain a mathematical large language model processed by the question fine-tuning method; Step S2: Generate sample questions using the mathematical large language model processed by the question fine-tuning method; Step S3: Optimizing the sample questions by using a question preference optimization method, and then training the mathematical large language model processed by the question fine-tuning method with the optimized sample questions; Step S4: using the mathematical large language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filtering the questions generated by the question generator; Step S5: Generate a corresponding answer for each filtered question to synthesize reasoning data.

2. The method for synthesizing inference data according to claim 1, characterized in that: In the process of model training, step S1 uses a problem fine-tuning method to perform problem fine-tuning on the mathematical language model using a preset data set, specifically including: During the model training process, question fine-tuning is performed on the total data set synthesized by using two data sets with different characteristics in the large mathematical language model through the question fine-tuning method, and only one round of fine-tuning is performed, wherein the large mathematical language model is DeepSeek-Math-7B-RL or Qwen2-Math-7B-Instruc; the two data sets with different characteristics differ in difficulty distribution and question types.

3. The method for synthesizing inference data according to claim 1, characterized in that: In the model training process, the loss function used when the mathematical large language model is fine-tuned using the preset data set through the problem fine-tuning method is: ； in, Indicates that from the distribution The problem of sampling Take the expected value, eos represents the word that marks the end of the question, and Represent contextual hints and questions, respectively. Indicates the position of the word in the question, Indicates the total length of the question, Indicates the problem word, Indicates the previous word, Indicates that the parameter is The mathematical language model of .

4. The method for synthesizing inference data according to claim 1, characterized in that: In step S3, the sample question is optimized by the question preference optimization method, and then the optimized sample question is used to train the mathematical large language model processed by the question fine-tuning method, which specifically includes: Optimizing the sample questions by using a first large language model, and randomly optimizing the solvability or difficulty of each sample question by randomly selecting an optimization direction strategy during the optimization process, wherein the solvability refers to whether the question can be solved; The optimized sample problems are then used to train the mathematical large language model processed by the problem fine-tuning method. During the training process, the optimized sample problems are regarded as the preferred problems, and the original sample problems before optimization are regarded as non-preferred problems. The loss function of the problem preference optimization method in the optimization process is constructed based on the preferred problems and non-preferred problems. The formula is: ； in, Indicates the mathematical language model currently being optimized. represents the reference mathematical large language model, The problem of expressing preference is in the dataset expectations, represents a dataset containing preference question pairs consisting of questions before and after optimization of the first large language model, represents the optimized preferred problem, represents a non-preferred problem before optimization, express function, Represents the temperature parameter, which is used to control the degree of optimization.

5. The method for synthesizing inference data according to claim 1, characterized in that: The method of filtering the questions generated by the question generator in step S4 includes: Language filtering: used to identify characters in non-preset languages ​​and filter questions in non-preset languages; Solvability filtering: used to filter unsolvable questions through a preset language model to ensure that the generated questions are semantically coherent and have sufficient conditions; Difficulty filtering: used to calculate the difficulty score of each question. If the difficulty score is outside the preset range, the question is filtered.

6. The method for synthesizing inference data according to claim 5, characterized in that: The method for calculating the difficulty score of each question in the difficulty filtering includes: A difficulty scorer with a classification head is constructed, and the classification head is used to calculate the difficulty score of each question. The calculation formula of the difficulty score is: ； The loss function of the difficulty scorer is constructed based on the difficulty score, and the formula is: ； in, and are the weight and bias of the classification head, respectively. represents the last hidden state of the vectorized representation of the input question sequence, is the loss function of the difficulty scorer, It is The predicted difficulty score of the question, Indicates The actual difficulty score of the question.

7. The method for synthesizing inference data according to claim 1, characterized in that: In step S5, a corresponding answer is generated for each filtered question to synthesize reasoning data, and the method includes: A second large language model is used to generate multiple answers for each filtered question, and Internlm2-7b-reward is used as a reward model to evaluate the score of each answer. The answer with the highest score of the reward model is selected as the preferred answer, and each filtered question and the corresponding preferred answer are synthesized into reasoning data.

8. A reasoning data synthesis system, characterized in that: include: The first optimization module is used to obtain a mathematical large language model. During the model training process, the mathematical large language model is fine-tuned using a preset data set by a problem fine-tuning method to obtain a mathematical large language model processed by the problem fine-tuning method. Question generation module: used to generate sample questions through the mathematical large language model processed by the question fine-tuning method; The second optimization module is used to optimize the sample problem by using the problem preference optimization method, and then train the mathematical large language model processed by the problem fine-tuning method with the optimized sample problem; Question filtering module: used to use the mathematical large language model processed by the question fine-tuning method and the question preference optimization method as a question generator, and filter the questions generated by the question generator; Data synthesis module: used to generate corresponding answers for each filtered question to synthesize reasoning data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the inference data synthesis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the inference data synthesis method according to any one of claims 1 to 7 are implemented.

Images