A method and system for inductive data synthesis

Abstract

The present application relates to a kind of reasoning data synthesis method and system, wherein, method includes: obtaining mathematical large language model, in the model training process, by question fine-tuning method in mathematical large language model utilizes preset data set and carries out question fine-tuning, obtains the mathematical large language model after being processed by question fine-tuning method;Sample question is generated by the mathematical large language model after being processed by question fine-tuning method;Sample question is optimized by question preference optimization method, and the mathematical large language model after being processed by question fine-tuning method is trained again to the optimized sample question;The mathematical large language model after being processed by question fine-tuning method and question preference optimization method is used as question generator, and the question generated by the question generator is filtered;The corresponding answer is generated for each filtered question to synthesize reasoning data.The reasoning data synthesis method of the present application can not only guarantee data quality, but also reduce generation cost, improve generation efficiency.

Description

A method and system for inference data synthesis Technical Field

[0001] This invention relates to the field of reasoning data synthesis technology, and in particular to a reasoning data synthesis method and system. Background Technology

[0002] In recent years, Large Language Models (LLMs) have seen rapid development in the field of deep learning, especially in natural language processing. These models have not only demonstrated superior performance and diverse application potential, but have also significantly accelerated the transformation of artificial intelligence technology from theoretical research to practical applications, driving the rapid deployment of numerous innovative products and services. Existing research widely indicates that the quantity and quality of data play a decisive role in the performance of large language models. However, with the increasing exploitation of high-quality publicly available data on the internet, coupled with increasingly stringent privacy regulations and policies, high-quality real-world data may face the risk of depletion in the near future. To address this potential data scarcity crisis, academia and industry are actively exploring alternatives, with research and applications of synthetic data receiving increasing attention. This approach not only promises to alleviate the data scarcity problem but may also provide more diverse and controllable data sources for model training, thereby promoting the continuous innovation and development of large language model technology.

[0003] With the rapid development of large language model technology, both commercial closed-source models and academic open-source models have demonstrated remarkable improvements in capabilities. These models not only excel in instruction following, accurately understanding and executing various complex tasks, but have also made significant progress in reasoning abilities, handling scenarios requiring logical thinking and problem-solving. This leap in capabilities is largely attributed to training with high-quality instruction and reasoning data. Of particular note is mathematical reasoning, as one of the representative tasks of reasoning ability, which has recently become a research hotspot. Existing methods for synthesizing reasoning data mainly include two technical routes: problem-driven methods and knowledge-driven methods. Problem-driven methods include, but are not limited to, problem rewriting, instruction evolution, and problem reverse translation. This method suffers from limited data diversity because it relies on a seed problem, and the generated new problems are highly similar to the seed problem, differing only slightly in conditions or values. This lack of diversity limits the method's scalability. To address these technical issues, knowledge-driven methods have emerged. This method generates new problems by constructing a knowledge base or concept graph and extracting key information from it, thereby increasing the diversity of problems and expanding the scale of problem synthesis. However, existing problem-driven and knowledge-driven methods rely on high-performance language models (such as GPT-4) to synthesize new questions. The high API cost of such models hinders large-scale data generation, thus limiting the further development of inference data synthesis techniques. Furthermore, some methods leverage the autoregressive nature of large language models, directly guiding them to synthesize relevant questions without pre-set seed sets or hints, demonstrating excellent scalability. However, the questions generated by these methods are often simple or contain logical errors, resulting in low quality and failing to effectively improve the inference capabilities of large language models. Therefore, there is an urgent need in this field for an inference data synthesis method that can significantly reduce generation costs and improve generation efficiency while ensuring data quality. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing reasoning data synthesis methods cannot simultaneously guarantee data quality, generation cost, and generation efficiency.

[0005] To address the aforementioned technical problems, this invention provides a method for synthesizing inference data, comprising:

[0006] Step S1: Obtain the mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset dataset through a problem fine-tuning method to obtain the mathematical language model after the problem fine-tuning method.

[0007] Step S2: Generate sample questions using the mathematical language model processed by the problem fine-tuning method;

[0008] Step S3: Optimize the sample problem using the problem preference optimization method, and then train the mathematical language model after the problem fine-tuning method with the optimized sample problem;

[0009] Step S4: Use the mathematical language model processed by the problem fine-tuning method and the problem preference optimization method as a problem generator, and filter the problems generated by the problem generator;

[0010] Step S5: Generate corresponding answers for each filtered question to synthesize inference data.

[0011] In one embodiment of the present invention, step S1, during model training, involves fine-tuning the mathematical language model using a preset dataset using a problem fine-tuning method, specifically including:

[0012] During model training, the mathematical language model is fine-tuned using a problem-based fine-tuning method on a total dataset synthesized from two datasets with different features. This fine-tuning is performed in only one round. The mathematical language model is either eepSeek-Math-7B-RL or Qwen2-Math-7B-Instruc. The two datasets with different features differ in difficulty distribution and problem type.

[0013] In one embodiment of the present invention, the loss function used in the mathematical language model for problem fine-tuning using a preset dataset during model training is as follows:

[0014] ;

[0015] in, Indicates from distribution Problems with sampling Take the expected value, where eos represents the term that marks the end of the question. and These respectively represent contextual hints and questions. Indicates the position of a word in the question. Indicates the total length of the problem. Indicates the first in the problem Each word element, Indicate the preceding part of the problem Each word element, The parameter is A mathematical language model.

[0016] In one embodiment of the present invention, step S3 involves optimizing the sample problem using a problem preference optimization method, and then training the optimized sample problem on the mathematical language model processed by the problem fine-tuning method. Specifically, this includes:

[0017] The sample problem is optimized using a first large-scale language model. During the optimization process, the solvability or difficulty of each sample problem is randomly optimized by randomly selecting an optimization direction strategy. Solvability refers to whether the problem can be solved.

[0018] The optimized sample problems are then used to train the mathematical language model processed by the problem fine-tuning method. During training, the optimized sample problems are considered preferred problems, while the original sample problems before optimization are considered non-preferred problems. A loss function for the problem preference optimization method is constructed based on the preferred and non-preferred problems, with the following formula:

[0019] ;

[0020] in, This represents the mathematical language model that is currently being optimized. This indicates a reference to the large mathematical language model. The preference problem represents the data set. Expectations This represents a dataset containing preference question pairs consisting of questions before and after optimization of the first large language model. This indicates the preferred problem after optimization. This indicates a non-preferred problem before optimization. express function, This represents the temperature parameter, used to control the degree of optimization.

[0021] In one embodiment of the present invention, the method for filtering the questions generated by the question generator in step S4 includes:

[0022] Language filtering: Used to identify characters in non-preset languages ​​and filter out non-preset language issues;

[0023] Solvability filtering: Used to filter unsolvable problems through a pre-defined language model to ensure that the generated problems are semantically fluent and have sufficient conditions;

[0024] Difficulty Filtering: This function calculates the difficulty score for each question. If the difficulty score is outside a preset range, the question is filtered out.

[0025] In one embodiment of the present invention, the method for calculating the difficulty score of each question in the difficulty filtering includes:

[0026] Construct a difficulty scorer with a classification head, which is used to calculate the difficulty score for each question. The formula for calculating the difficulty score is as follows:

[0027] ;

[0028] The loss function for constructing the difficulty scorer is given by the difficulty score, and the formula is as follows:

[0029] ;

[0030] in, and These are the weights and biases of the classification heads, respectively. This represents the final hidden state of the vectorized representation of the input problem sequence. Let the loss function be the difficulty scorer. It is the first The predicted difficulty score for each question Indicates the first The true difficulty score for each question.

[0031] In one embodiment of the present invention, step S5 involves generating a corresponding answer for each filtered question to synthesize reasoning data. The method includes:

[0032] Multiple answers are generated for each filtered question using a second large-scale language model. Internlm2-7b-reward is used as a reward model to evaluate the score of each answer, and the answer with the highest score of the reward model is selected as the preferred answer. Each filtered question and its corresponding preferred answer are then combined to form inference data.

[0033] To address the aforementioned technical problems, this invention provides a reasoning data synthesis system, comprising:

[0034] The first optimization module is used to obtain a mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset dataset through a problem fine-tuning method to obtain a mathematical language model after the problem fine-tuning method.

[0035] Problem generation module: used to generate sample problems using the mathematical language model processed by the problem fine-tuning method;

[0036] The second optimization module is used to optimize the sample problem using a problem preference optimization method, and then use the optimized sample problem to train the mathematical language model after it has been processed by a problem fine-tuning method.

[0037] Problem filtering module: used to use the mathematical language model processed by the problem fine-tuning method and the problem preference optimization method as a problem generator, and to filter the problems generated by the problem generator;

[0038] Data synthesis module: used to generate corresponding answers for each filtered question to synthesize reasoning data.

[0039] To address the aforementioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the inference data synthesis method described above.

[0040] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the inference data synthesis method described above.

[0041] The technical solution of the present invention has the following advantages compared with the prior art:

[0042] The reasoning data synthesis method of this invention includes four key steps: question fine-tuning, question preference optimization, question filtering, and answer generation. The question fine-tuning process activates the model's question generation capability by training the model with datasets featuring different characteristics, enabling it to generate diverse, cross-lingual questions. The question preference optimization stage introduces an improved direct preference optimization loss function, effectively improving the solvability and difficulty of the generated questions. The question filtering stage employs a multi-level filtering mechanism, including language filtering, solvability filtering, and difficulty filtering, ensuring high-quality generated questions. The answer generation stage utilizes a large-scale language model to generate multiple reasoning paths and introduces a reward model score as a quality evaluation indicator to select the optimal solution. These technological innovations of this invention significantly improve the quality and reliability of the generated data, providing high-quality training data for complex reasoning tasks.

[0043] The method of this invention not only solves the problems of low data generation quality and high cost in the prior art, but also provides key support for breakthroughs in reasoning ability of artificial intelligence, and has broad application prospects and important technical value. Attached Figure Description

[0044] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0045] Figure 1 is a flowchart of the method of the present invention;

[0046] Figure 2 is a comparative diagram of experimental analysis in the embodiments of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0048] Example 1

[0049] Referring to Figure 1, the present invention relates to a method for synthesizing inference data, comprising:

[0050] Step S1: Obtain the mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset dataset through a problem fine-tuning method to obtain two mathematical language models after the problem fine-tuning method.

[0051] Step S2: Generate sample questions using the mathematical language model processed by the problem fine-tuning method;

[0052] Step S3: Optimize the sample problem using the problem preference optimization method, and then train the mathematical language model after the problem fine-tuning method with the optimized sample problem;

[0053] Step S4: Use the mathematical language model processed by the problem fine-tuning method and the problem preference optimization method as a problem generator, and filter the problems generated by the problem generator;

[0054] Step S5: Generate corresponding answers for each filtered question to synthesize inference data.

[0055] The following is a detailed description of this embodiment:

[0056] The data synthesis method in this embodiment mainly covers the following aspects: 1) Training a basic question generator based on the question fine-tuning method; 2) Optimizing the question generator in the first step based on the question preference optimization method to obtain an advanced question generator; 3) Synthesizing a large amount of data using the question generator obtained in the previous step, and filtering questions based on language clarity, solvability, and appropriate difficulty; 4) Generating multiple corresponding answers for the synthesized questions filtered in the previous step, and rejecting multiple answers generated for each question based on the reward score of the reward model, selecting the answer with the highest reward score to form the final synthesized dataset.

[0057] In this embodiment, the question synthesis process involves providing only a small number of prefix tags (e.g., "<|begin_of_sentence|>User:") from the instruction template to guide the model in question generation. This embodiment utilizes a fine-tuned causal language model that has learned to generate answers based on question-answer pairs (e.g., {<|begin_of_sentence|> User: {question}. Assistant: {answer}"). This model can potentially be used directly to generate questions. This is because during instruction tuning, the model is trained using a causal mask, where each tag only focuses on the preceding tag. This ensures that the hidden state is based solely on past context evolution and is not affected by future tags. Therefore, the method of this invention needs to activate the model's ability to synthesize questions.

[0058] (I) Question Fine-Tuning Method: In this embodiment, the main purpose of the Question Fine-Tuning (QFT) process is to activate the model's question generation ability, rather than to make the model memorize specific questions. To verify this hypothesis, this embodiment selects two datasets with different characteristics to form a total dataset for training the model. These two datasets have significant differences in difficulty distribution and question types to ensure that the model can learn diverse question generation patterns and to evaluate the model's performance as in-domain data. Specifically, this embodiment uses the above-mentioned total dataset for question fine-tuning on two mathematical language models, DeepSeek-Math-7B-RL and Qwen2-Math-7B-Instruct. It should be noted that the reason for using two models to train simultaneously in this embodiment is that the data generated by multiple models is more diverse and easier to analyze later. In other embodiments, a single model can also be used. Considering that too many rounds of fine-tuning may lead to overfitting and negatively affect the diversity of generated questions, this embodiment only fine-tunes one round. In addition, this embodiment also uses sequence packing to accelerate training. To evaluate the distribution of generated questions. The models that underwent Question Fine-Tune (QFT) processing generated a large number of sample questions (e.g., 10,000).

[0059] This embodiment also constructs a loss function for the problem fine-tuning stage, the formula of which is:

[0060] ;

[0061] in, Indicates from distribution Problems with sampling Take the expected value, where eos represents the term that marks the end of the question. and These respectively represent contextual hints and questions. Indicates the position of a word in the question. Indicates the total length of the problem. Indicates the first in the problem Each word element, Indicate the preceding part of the problem Each word element, The parameter is A mathematical language model.

[0062] By comparing the difficulty distribution of these generated questions with the distribution of the original training dataset, this invention found that the distribution of the generated questions differs from both original datasets, and that despite being trained on different datasets, the distribution of questions generated by the two models that have undergone Question Fine-tuning (QFT) (referred to as QFT models) tends to be similar.

[0063] It is noteworthy that the QFT model trained solely on English questions demonstrated the ability to generate questions in other languages. This phenomenon further proves that the QFT process enhances the model's question generation ability, rather than simply overfitting the training data. Through the above experiments and analysis, this invention confirms that the QFT process does indeed improve the model's question generation ability, rather than merely memorizing training data. This method enables the model to generate diverse, moderately difficult, and cross-linguistic questions, laying the foundation for the synthesis of large-scale, high-quality inference data.

[0064] (ii) Question Preference Optimization Method: In this embodiment, the Question Preference Optimization (QPO) process aims to further improve the quality of questions generated after Question Fine-tuning (QFT). Although the QFT process enables the model to generate meaningful and diverse questions, there is still room for improvement in terms of solvability and difficulty.

[0065] To address these two key issues, this embodiment employs Question Preference Optimization (QPO) technology. First, a large number of sample questions are generated using a model processed by QFT. Then, 10,000 of these sample questions are optimized using an external large language model, such as GPT-4o-mini, focusing on solvability and difficulty. Since simultaneously optimizing both aspects is challenging for large language models, this embodiment adopts a strategy of randomly selecting the optimization direction, i.e., randomly optimizing the solvability or difficulty of each sample question. The optimized sample questions are then used to train and optimize a large mathematical language model processed by the question fine-tuning method. During the optimization process, the optimized questions are used as preferred data, while the original questions before optimization are used as non-preferred data. This embodiment optimizes the loss function of Direct Preference Optimization (DPO) to adapt to this method, resulting in the QPO loss function, as follows:

[0066] ;

[0067] in, This represents the mathematical language model that is currently being optimized (the model after QFT). This refers to the reference mathematical language model (the model after QFT). The preference problem represents the data set. Expectations This represents a dataset containing preference question pairs (pairs of preference questions composed of questions before and after GPT-4o-mini optimization). This indicates the preferred problem after optimization. This indicates a non-preferred problem before optimization. express function, This represents the temperature parameter, used to control the degree of optimization. It should be noted that... and The values ​​are the same at the beginning of training, but as the number of training rounds increases, It remains unchanged, but It is subject to change.

[0068] The optimization process described above places high demands on the model's ability to follow complex instructions. In the QPO phase, this embodiment uses a learning rate of 5e-7 and a batch size of 128. To evaluate the improvements in solvability and difficulty, a specialized evaluation model and evaluation hints are employed. Experimental results show that the QPO process significantly improves the quality of generated problems, particularly in terms of solvability and difficulty. This method not only ensures the solvability of generated problems but also increases the overall difficulty, thus providing higher-quality training data for subsequent inference tasks.

[0069] (III) Problem Generation and Filtering: In this embodiment, problem filtering is a crucial step in ensuring the quality of generated problems. After problem fine-tuning (QFT) and problem preference optimization (QPO), this embodiment obtains two problem generators (based on DeepSeek-Math-7B-RL and Qwen2-Math-7B-Instruct). Then, this embodiment uses these two problem generators to synthesize 1M data points each, for a total of 2M problems. Regarding parameter settings, this embodiment sets the maximum generated length to 512, the temperature to 1.0, and the top-p value to 0.99. However, the synthesized problems still have some minor issues, mainly concerning language, solvability, and difficulty. To address these challenges, this embodiment employs the following filtering steps: 1) Language filtering. Although this invention primarily focuses on English mathematical problems, the problem generator still generates approximately 20% of other language problems. Therefore, this embodiment effectively removes non-English problems by identifying and filtering out problems containing non-English characters. 2) Solvability filtering. While the QPO process significantly improves the solvability of generated questions, some questions still suffer from semantic inconsistencies or insufficient conditions. This is mainly due to improper condition setting (such as missing conditions, redundant conditions, or logical inconsistencies) and questions that cannot yield meaningful results (e.g., answers involving the number of people should be non-negative integers). To filter out such sample questions, this embodiment uses a specialized language model (e.g., the Qwen2-Math-Instruct model) to evaluate the meaning and sufficiency of the questions. 3) Difficulty Sampling. This embodiment uses the failure rate to measure the difficulty of a question, which is the proportion of incorrect answers when n answers are sampled for a given question. This metric is intuitive, as more difficult questions generally lead to fewer correct answers. Specifically, this embodiment uses a specialized model (DeepseekMath-7B) to evaluate the difficulty of each question in the training set and trains a difficulty scorer based on this model. The difficulty scorer is built on an existing language model and calculates the difficulty score by adding a classification head. The formula for calculating the difficulty score is as follows:

[0070] ;

[0071] The loss function for constructing the difficulty scorer is given by the difficulty score, and the formula is as follows:

[0072] ;

[0073] in, and These are the weights and biases of the classification heads, respectively. This represents the final hidden state of the vectorized representation of the input problem sequence. Let the loss function be the difficulty scorer. It is the first The predicted difficulty score for each question Indicates the first The true difficulty score for each question.

[0074] Using the above method, this invention can predict the difficulty score of each generated question. If the difficulty score is outside a preset range, the question is filtered. Of course, this embodiment can also sample questions based on difficulty: specifically, for questions generated by certain question generators that are too simple, this embodiment will filter them; while for question generators with a more balanced difficulty distribution, no additional sampling is needed. This multi-layered question filtering method ensures that the final generated question set meets high standards in terms of language, solvability, and difficulty, providing high-quality training data for subsequent reasoning tasks. Through this innovative filtering mechanism, this invention significantly improves the quality of generated questions.

[0075] (iv) Response Generation: In this embodiment, a reward model-based approach is used in the response generation stage to ensure the high quality of the final dataset. This embodiment utilizes a large language model to generate multiple reasoning paths for each question, increasing the likelihood of obtaining high-quality answers. To evaluate response quality, this embodiment introduces a reward model score as a core indicator. Specifically, this embodiment sets the maximum generation length to 2048, the temperature to 0.7, and the top-p value to 0.99. Furthermore, this embodiment uses a thought chain method to synthesize responses and employs vLLM and Ray frameworks for distributed reasoning to accelerate response generation, using Internlm2-7b-reward as the reward model. This method has broad applicability and is particularly suitable for complex reasoning tasks without a single correct answer. Specifically, this embodiment generates 5 solutions for each question and selects the solution with the highest reward model score as the preferred solution, synthesizing the selected optimal solution and the corresponding question into reasoning data.

[0076] Therefore, through the innovative answer generation and filtering mechanism described above, this invention can provide high-quality solutions for each question, ensuring the overall quality of the final dataset and providing a reliable reference standard for various complex reasoning tasks. This method significantly improves the quality and reliability of the generated data, providing key technical support for the synthesis of large-scale, high-quality reasoning data.

[0077] The experiment and analysis are as follows:

[0078] The inference data synthesis method proposed in this invention has significant technical effects and advantages. By training a model on the synthesized high-quality inference dataset, this invention effectively verifies the superiority of the generated data. During the verification process, this invention employs various advanced basic models for instruction fine-tuning experiments, including general-purpose large language models (such as Mistral-7B and Llama3-8B) and models specifically optimized for the mathematical domain (such as DeepSeekMath-7B and Qwen2-Math-7B). This multi-model verification method not only comprehensively evaluates the quality of the synthesized data but also demonstrates the universality and effectiveness of the method proposed in this invention.

[0079] The training method employed in this invention is highly efficient, achieving significant results in just three training cycles, greatly shortening model optimization time and improving overall R&D efficiency. Simultaneously, a carefully designed linear learning rate scheduling strategy is used, including a 3% warm-up ratio, and differentiated peak learning rates are applied for different model types (5e-5 for Llama3 and DeepSeekMath, and 1e-5 for other models). Cosine decay to zero is then applied to ensure the stability of the training process and optimize the final performance. This optimization strategy not only improves training efficiency but also enhances the model's performance on inference tasks, demonstrating the high quality of the synthesized dataset. Validation on models of various architectures and scales demonstrates the strong adaptability and scalability of this method when handling different types of language models. This universality allows the method to be easily extended to a wider range of application scenarios and model types. More importantly, the proposed method effectively reduces the cost of acquiring large-scale labeled data by synthesizing high-quality datasets. Furthermore, the optimized training strategy reduces computational resource consumption, further improving the overall cost-effectiveness.

[0080] This invention further validates the effectiveness and superiority of a low-cost, scalable data synthesis method based on a large language model through rigorous evaluation methods and diverse evaluation metrics. The evaluation process covers multiple difficulty levels from basic to advanced, fully demonstrating the comprehensiveness and adaptability of the method. Specifically, this invention uses four datasets of increasing difficulty for model performance evaluation. These datasets include the widely used GSM8K (primary school level) and MATH (competition level), as well as the more challenging College Math (university level) and Olympiad Bench (Olympiad level) benchmark tests. Through this comprehensive evaluation system, this invention can not only test the model's performance on basic mathematical problems, but also evaluate its ability in advanced mathematics and complex reasoning tasks, thereby comprehensively validating the quality of the synthesized data and the effectiveness of model training. During the evaluation process, correctness is determined by extracting the final answer and comparing answer equivalence. This evaluation method ensures the objectivity and reliability of the results. It is worth noting that the generated outputs all adopt the form of natural language thought chain reasoning, implemented through greedy decoding, without relying on any external tool integration. This method not only demonstrates the model's reasoning ability but also reflects the invention's significant contribution to enhancing the model's autonomous thinking and problem-solving capabilities. The invention uses zero-shot pass rate as the primary performance indicator, which directly reflects the model's performance when faced with novel, untrained problems. Through zero-shot testing, the invention fully demonstrates the advantages of models trained on synthetic data in terms of generalization ability and practical application value, further proving the significant effectiveness of the invention's method in improving the overall performance of large language models.

[0081] The inference data synthesis method constructed in this invention has demonstrated significant performance advantages and wide applicability in practical applications. Through comprehensive experimental evaluation, this invention not only significantly surpasses existing synthesis methods in performance but also excels in data scalability, providing an important breakthrough for the development of the field of artificial intelligence.

[0082] In terms of performance, this invention achieves significant improvements over previous state-of-the-art technologies in both the general-purpose base model and the base model specifically optimized for the mathematical domain (as shown in Figure 2), with an average performance improvement of 5.6% to 11.5%. Particularly noteworthy is that the Qwen2-Math-7B model trained with this invention achieved a zero-sample pass rate of 73.4 on the MATH benchmark, a performance comparable to the GPT-4-Turbo model. Even more remarkably, this model outperforms its teacher model, Qwen2-Math-7B-Instruct, on out-of-domain tasks, achieving excellent scores of 89.7, 73.4, and 38.5 on the GSM8K, MATH, and Olympiad benchmarks, respectively. Considering that the Qwen2-Math-7B-Instruct model undergoes Group Relative Policy Optimization (GRPO) and uses the powerful reward model Qwen2-Math-RM-72B, while this invention achieves such outstanding results through only instruction fine-tuning, further highlighting the innovation and effectiveness of this invention.

[0083] In terms of scalability, this invention demonstrates superior data scaling capabilities. By combining the dataset constructed using this method with publicly available datasets (such as MetaMath, DART-Math, and Numina-Math), this invention trained the Llama3-8B model and observed how its performance changes with increasing data size. The results show that in in-domain evaluation (MATH), this invention exhibits extremely high data efficiency, achieving superior results with the same amount of data. In out-of-domain evaluation (Olympic benchmarks), this invention also demonstrates strong scalability, continuously improving even when other datasets reach their performance limits.

[0084] The method of this invention successfully overcomes the performance bottleneck caused by a limited set of questions. For example, the performance improvement of the DART-Math dataset is limited because it relies on a small number of questions and generates a large number of correct answers through rejection sampling. In contrast, this method supports continuous performance growth by generating diverse questions, fully demonstrating the importance of generating a wide range of diverse questions.

[0085] Example 2

[0086] This embodiment provides a reasoning data synthesis system, including:

[0087] The first optimization module is used to obtain a mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset dataset through a problem fine-tuning method to obtain a mathematical language model after the problem fine-tuning method.

[0088] Problem generation module: used to generate sample problems using the mathematical language model processed by the problem fine-tuning method;

[0089] The second optimization module is used to optimize the sample problem using a problem preference optimization method, and then use the optimized sample problem to train the mathematical language model after it has been processed by a problem fine-tuning method.

[0090] Problem filtering module: used to use the mathematical language model processed by the problem fine-tuning method and the problem preference optimization method as a problem generator, and to filter the problems generated by the problem generator;

[0091] Data synthesis module: used to generate corresponding answers for each filtered question to synthesize reasoning data.

[0092] Example 3

[0093] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the reasoning data synthesis method described in Embodiment 1.

[0094] Example 4

[0095] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the reasoning data synthesis method described in Embodiment 1.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0100] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0101] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for synthesizing inference data, characterized in that: include: Step S1: Obtain the mathematical language model. During model training, fine-tune the mathematical language model using a preset dataset using a question fine-tuning method to obtain the mathematical language model after the question fine-tuning method. Specifically, during model training, fine-tuning the mathematical language model using a preset dataset involves fine-tuning the questions using a combined dataset of two datasets with different features, and only one round of fine-tuning is performed. The mathematical language model is either DeepSeek-Math-7B-RL or Qwen2-Math-7B-Instruc. The two datasets with different features differ in difficulty distribution and question type. Step S2: Generate sample questions using the mathematical language model after the question fine-tuning method. Step S3: Optimize the questions based on question preferences. The sample problem is optimized using a first large-scale language model, and then the optimized sample problem is used to train the mathematical language model processed by the problem fine-tuning method. Step S3 involves optimizing the sample problem using a problem preference optimization method, and then training the mathematical language model processed by the problem fine-tuning method. Specifically, this includes: optimizing the sample problem using a first large-scale language model, and during the optimization process, randomly selecting an optimization direction strategy to randomly optimize the solvability or difficulty of each sample problem, where solvability refers to whether the problem can be solved; then training the mathematical language model processed by the problem fine-tuning method with the optimized sample problem, where the optimized sample problem is considered the preferred problem, and the original sample problem before optimization is considered a non-preferred problem. A loss function for the problem preference optimization method during the optimization process is constructed based on the preferred and non-preferred problems, with the following formula: ;in, This represents the mathematical language model that is currently being optimized. This indicates a reference to the large mathematical language model. The preference problem represents the data set. Expectations This represents a dataset containing preference question pairs consisting of questions before and after optimization of the first large language model. This indicates the preferred problem after optimization. This indicates a non-preferred problem before optimization. express function, The temperature parameter is used to control the degree of optimization; Step S4: The mathematical language model processed by the problem fine-tuning method and the problem preference optimization method is used as a problem generator, and the problems generated by the problem generator are filtered; Step S5: A corresponding answer is generated for each filtered problem to synthesize inference data.

2. The reasoning data synthesis method according to claim 1, characterized in that: The loss function used in the mathematical language model during model training, when fine-tuning the problem using a preset dataset, is as follows: ;in, Indicates from distribution Problems with sampling Take the expected value, where eos represents the term that marks the end of the question. and These respectively represent contextual hints and questions. Indicates the position of a word in the question. Indicates the total length of the problem. Indicates the first in the problem Each word element, Indicate the preceding part of the problem Each word element, The parameter is A mathematical language model.

3. The reasoning data synthesis method according to claim 1, characterized in that: The method of filtering the questions generated by the question generator in step S4 Includes: Language filtering: used to identify characters in non-preset languages ​​and filter out non-preset language issues; Solvability filtering: Used to filter unsolvable problems through a pre-defined language model to ensure that the generated problems are semantically fluent and have sufficient conditions; Difficulty Filtering: This function calculates the difficulty score for each question. If the difficulty score is outside a preset range, the question is filtered out.

4. The reasoning data synthesis method according to claim 3, characterized in that: The method for calculating the difficulty score for each question in the difficulty filtering includes: constructing a difficulty scorer with a classification head, wherein the classification head is used to calculate the difficulty score for each question, and the formula for calculating the difficulty score is: The loss function for constructing the difficulty scorer is based on the aforementioned difficulty score, and the formula is as follows: ;in, and These are the weights and biases of the classification heads, respectively. This represents the final hidden state of the vectorized representation of the input problem sequence. Let the loss function be the difficulty scorer. It is the first The predicted difficulty score for each question Indicates the first The true difficulty score for each question.

5. The reasoning data synthesis method according to claim 1, characterized in that: In step S5, a corresponding answer is generated for each filtered question to synthesize inference data. The method includes: generating multiple answers for each filtered question using a second large-scale language model, using Internlm2-7b-reward as a reward model to evaluate the score of each answer, selecting the answer with the highest score of the reward model as the preferred answer, and synthesizing inference data for each filtered question and the corresponding preferred answer.

6. A reasoning data synthesis system for implementing the reasoning data synthesis method according to any one of claims 1 to 5, characterized in that: include: The first optimization module is used to obtain a mathematical language model. During the model training process, the mathematical language model is fine-tuned using a preset dataset through a problem fine-tuning method to obtain a mathematical language model processed by the problem fine-tuning method. The problem generation module is used to generate sample problems through the mathematical language model processed by the problem fine-tuning method. The second optimization module is used to optimize the sample problem using a problem preference optimization method, and then use the optimized sample problem to train the mathematical language model after it has been processed by a problem fine-tuning method. Question filtering module: used as a question generator to process the mathematical language model after being processed by the question fine-tuning method and the question preference optimization method, and to filter the questions generated by the question generator; Data synthesis module: used to generate corresponding answers for each filtered question to synthesize inference data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the reasoning data synthesis method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the reasoning data synthesis method as described in any one of claims 1 to 5.

Images