Controllable mathematical instruction data synthesis method and device
By leveraging the multi-task processing and structured output format of a large language model, combined with a three-party verification mechanism, the problem of insufficient accuracy of mathematical instruction data in existing technologies has been solved, achieving high-quality and diverse data generation.
Patent Information
- Application Number
- CN202510792607.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-10-31
AI Technical Summary
Existing methods for synthesizing mathematical instruction data using large language models lack a verification process for the original data, making it difficult to guarantee the accuracy and quality of the generated data.
A controllable mathematical instruction data synthesis method is adopted, which generates output data by performing multiple tasks through a large language model, and uses prompt words to guide the structured format of the output results. Combined with a three-party verification mechanism, the accuracy of the data is ensured, including seed data preparation, multi-task processing, code execution, and numerical replacement.
The generated mathematical instruction data undergoes multiple rounds of processing and verification to ensure accuracy and high quality, avoid the generation of erroneous data, and provide diverse data formats.
Smart Images
Figure CN120873582A_ABST
Abstract
Description
Technical Field
[0001] This application relates to a method for generating training data for a Large Language Model (LLM). Background Technology
[0002] In the pre-training and fine-tuning of large language models, a large amount of high-quality training data is essential. Instruction data is one type of training data, usually presented in the form of question-and-answer format. Mathematical instruction data is a more specific type of training data, typically presented in the form of mathematical questions and answers.
[0003] Mathematical instruction data obtained through manual annotation is costly and limited in quantity. Industry research is shifting towards synthesizing mathematical instruction data using large language models or tools, which can then be used as new training data to support the training and testing of large language models.
[0004] Existing methods for synthesizing mathematical instruction data using large language models often lack a verification process for the original data, directly generating new mathematical instruction data from the original data, which is detrimental to the accuracy and high quality of the new mathematical instruction data. Summary of the Invention
[0005] The technical problem this application aims to solve is: how to generate diverse, high-quality mathematical instruction data and make the results controllable.
[0006] To address the aforementioned technical problems, this application proposes a controllable mathematical instruction data synthesis method, comprising the following steps: Step S1: Prepare a seed mathematical instruction dataset, including multiple seed mathematical instruction data, each seed mathematical instruction data consisting of a mathematical problem and an answer. Step S2: Combine each seed mathematical instruction data with a prompt word to obtain an input data; the large language model used to synthesize the mathematical instruction data generates an output data based on each input data. The prompt word guides the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem solution process described in natural language; (2) extracting the answer from the mathematical problem solution process; (3) extracting the answer from the seed mathematical instruction data; (4) extracting numerical values from the seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problem in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; (7) assigning values to variables in the program code and providing the execution method of the program code. The output data of the large language model used to synthesize mathematical instruction data includes the output results of multiple tasks, specifically including: (1) a mathematical problem-solving process described in natural language, referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, referred to as the program code; (7) the variable assignment and program code execution method in the program code. Step S3: Execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model; if the execution is successful, the execution result of the program code is obtained; if the execution fails, the execution result of the program code is set to empty. Step S4: Compare the solution process answer, reference answer, and program code execution result output by the large language model; if the three are equal, it indicates that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and proceed to step S5. Step S5: When the output result of the large language model for a certain seed mathematical instruction data is successfully verified, the parameters in the output data of the large language model for the seed mathematical instruction data are numerically replaced. The new values of the parameters are combined with the general mathematical problem template to form a new mathematical problem, that is, a new mathematical instruction data is obtained.
[0007] Furthermore, in step S2, the first item performed by the large language model used to synthesize mathematical instruction data belongs to the reasoning task, the second and third items belong to the information extraction task, the fourth and fifth items belong to the text summarization task, and the sixth and seventh items belong to the code generation task.
[0008] Furthermore, in step S2, the prompt word also instructs the output data of the large language model used to synthesize mathematical instruction data to adopt a structured format, with the output results of different tasks placed in different fields.
[0009] Furthermore, in step S2, the parameters in the output data of the large language model used to synthesize mathematical instruction data correspond one-to-one with the variables in the program code.
[0010] Furthermore, in step S4, if the three are not equal, it indicates that the large language model has failed to verify the output result of a certain seed mathematical instruction data, and the process returns to step S2 and another seed mathematical instruction data is used.
[0011] Furthermore, in step S4, a large language model is used to verify whether the three data items are equal; prompt words are used to guide the large language model used to compare whether the three data items are equal in a mathematical sense, ignoring differences in format, multiple-choice question format, coordinate pairs or answer order, equivalent mathematical expressions or symbolic representations.
[0012] Further, step S5 includes the following sub-steps: Step S51: Select the four parts from the output data of "the large language model successfully verified the output result of a certain seed mathematical instruction data": general mathematical problem template, parameters, program code, and variable assignment and execution method in the program code. Step S52: Replace the variables in the program code in "variable assignment and execution method in program code" with the same type of numerical value to obtain new numerical values for the variables in the program code. Step S53: Execute the program code output by the large language model according to the new numerical values of the variables in the program code and the program code execution method; if the execution is successful, obtain a new execution result of the program code; if the execution fails, set the new execution result of the program code to empty. Step S54: If the new execution result of the program code is not empty and has the same numerical type as the original execution result of the program code, then replace the parameters output by the large language model according to the new numerical values of the variables in the program code to obtain new numerical values for the parameters; combine the new numerical values of the parameters with the general mathematical problem template output by the large language model to obtain a new mathematical problem; proceed to step S55. Step S55: Combine the new mathematical problem and the new execution result of the program code to obtain a new mathematical instruction data.
[0013] Furthermore, in step S52, the same type of numerical replacement means replacing integer values with integer values, floating-point values with floating-point values, positive numbers with positive numbers, and negative numbers with negative numbers.
[0014] Furthermore, in step S54, if the new execution result of the program code is not empty and its numerical type is different from the original execution result of the program code, then return to step S52 to perform the same type of numerical replacement again. If the new execution result of the program code is empty, then return to step S52 to perform the same type of numerical replacement again.
[0015] This application also proposes a controllable mathematical instruction data synthesis device, including a seed data preparation unit, a guided synthesis unit, a code execution unit, a third-party verification unit, and a numerical replacement unit. The seed data preparation unit is used to prepare a seed mathematical instruction dataset, including multiple seed mathematical instruction data, each seed mathematical instruction data consisting of a mathematical problem and an answer. The guided synthesis unit is used to combine each seed mathematical instruction data with a prompt word to obtain an input data, and to send the input data to a large language model for synthesizing mathematical instruction data to obtain an output data of the large language model. The prompt word guides the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem solution process described in natural language; (2) extracting the answer from the mathematical problem solution process; (3) extracting the answer from the seed mathematical instruction data; (4) extracting numerical values from the seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problem in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; and (7) assigning values to variables in the program code and providing the execution method of the program code. The output data of the large language model simultaneously includes the output results of multiple tasks, specifically including: (1) a mathematical problem-solving process described in natural language, hereinafter referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, hereinafter referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, hereinafter referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, hereinafter referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, hereinafter referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, hereinafter referred to as the program code; and (7) the variable assignment and program code execution method in the program code. The code execution unit is used to execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model, and obtain the execution result of the program code. The third-party verification unit is used to compare the solution process answer, the reference answer, and the execution result of the program code output by the large language model; if the three are equal, it indicates that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and then proceeds to the numerical replacement unit. The numerical replacement unit is used to replace the parameters in the output data of the large language model for a certain seed mathematical instruction data when the output result of the large language model is successfully verified. The new value of the parameter is combined with the general mathematical problem template to form a new mathematical problem, that is, to obtain a new mathematical instruction data.
[0016] The technical effect achieved by this application is as follows: starting from seed mathematical instruction data, a large language model is used to generate multiple parts, including a problem-solving process, the answer to the problem-solving process, a reference answer, parameters, a general mathematical problem template, program code, and variable assignment and execution code within the program code. The generation of each part follows clear rules and requirements. During the generation process, each stage cooperates and progresses step by step. The newly generated mathematical instruction data undergoes multiple rounds of processing and verification, ensuring the accuracy and high quality of the new mathematical instruction data. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the controllable mathematical instruction data synthesis method proposed in this application.
[0018] Figure 2 yes Figure 1 A detailed flowchart of step S5.
[0019] Figure 3 This is a schematic diagram of the controllable mathematical instruction data synthesis device proposed in this application.
[0020] The attached diagrams are labeled as follows: Seed data preparation unit 1, guide synthesis unit 2, code execution unit 3, third-party verification unit 4, and numerical replacement unit 5. Detailed Implementation
[0021] Please see Figure 1 The controllable mathematical instruction data synthesis method proposed in this application includes the following steps.
[0022] Step S1: Prepare a seed math instruction dataset, which includes multiple seed math instruction data. Each seed math instruction data consists of two parts: a math problem and an answer.
[0023] Step S2: Combine each seed mathematical instruction data with a prompt to obtain one input data item. The Large Language Model (LLM) used to synthesize mathematical instruction data generates one output data item based on each input data item.
[0024] The prompt word guides the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem-solving process described in natural language; (2) extracting the answer from the mathematical problem-solving process; (3) extracting the answer from seed mathematical instruction data; (4) extracting numerical values from seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problems in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; and (7) assigning values to variables in the program code and providing the execution method of the program code. Item (1) above belongs to the reasoning task, items (2) and (3) belong to the information extraction task, items (4) and (5) belong to the text summarization task, and items (6) and (7) belong to the code generation task. Implementing multi-task processing within a single prompt word greatly improves the efficiency and comprehensiveness of data processing, enabling the processing and transformation of mathematical instruction data from multiple dimensions.
[0025] The output data of the large language model used to synthesize mathematical instruction data includes the output results of multiple tasks, specifically: (1) a mathematical problem-solving process described in natural language, hereinafter referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, hereinafter referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, hereinafter referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, hereinafter referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, hereinafter referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, hereinafter referred to as the program code; and (7) the variable assignment and program code execution method in the program code. Obviously, the parameters and the variables in the program code are in one-to-one correspondence.
[0026] Specifically, the prompt words instruct the large language model to output both a problem-solving process described in natural language and problem-solving steps described in program code. The problem-solving process described in natural language is easy for humans to understand, while the problem-solving steps described in program code can be verified and applied by executing the program code. The two complement each other, improving the accuracy and practicality of mathematical instruction data processing.
[0027] Preferably, the prompt words also indicate that the output data of the large language model adopts a clear structured format, with the output results of different tasks placed in different labels. This facilitates the subsequent extraction and use of the output results of different tasks, eliminating the need for complex parsing processes, reducing the difficulty of data processing and the probability of errors, and enabling a more intuitive understanding and analysis of the generated data.
[0028] As an example, the prompt words for the large language model used in this step to synthesize mathematical instruction data include the following seven parts.
[0029] Please follow these steps to process the given mathematical text and place the output in the appropriate tags to facilitate the extraction of different parts.
[0030] (1) The process of generating solutions to mathematical problems.
[0031] Extracting content related to mathematical problems from mathematical texts.
[0032] Solve the extracted mathematical problems step by step, using the parameters provided in the problems.
[0033] The language used in the solution to the problem should be consistent with the language of the given mathematical problem.
[0034] The generated content is placed in <solution> and< / solution> middle.
[0035] (2) Generate the answer to the solution process.
[0036] Extract the answer from the solution generated in step (1) and place it in the solution. <answer> and< / answer> middle.
[0037] (3) The answer to the mathematical text.
[0038] Extract the answer from the seed math instruction data and place it in... <textanswer> and< / textanswer> middle.
[0039] (4) Extract the numbers and replace them with parameters.
[0040] Extract all numbers from the mathematical problem (excluding units such as dollar signs, percentages, meters, etc.).
[0041] Assign a suitable parameter name to each extracted number to replace the original number in the problem.
[0042] Parameter names must be in English.
[0043] The generated content is placed in <parameters> and< / parameters> middle.
[0044] (5) Generate general mathematical problem templates.
[0045] Using the replaced parameter names, the original mathematical problem in the seed mathematical instruction data is converted into a general form mathematical problem template.
[0046] Ensure that each number in the math problem template has been replaced with the corresponding parameter name, while the rest remains consistent with the original math problem.
[0047] The generated content is placed in <problem> and< / problem> middle.
[0048] (6) Write Python functions.
[0049] Create a Python function whose parameters correspond to those extracted in step (4).
[0050] Write logic inside the function to calculate the final answer to the original problem (e.g., calculate a value or perform some operation).
[0051] Add detailed docstrings and comments explaining the function's purpose, its parameters, and its return value.
[0052] The generated content is placed in <function> and< / function> middle.
[0053] (7) Write code for variable assignment and function call.
[0054] Output the extracted parameters as Python variables in the format [variable name = value].
[0055] Output Python code to assign values to the extracted parameters and call the function to print the results.
[0056] The generated content is placed in <execution> and< / execution> middle.
[0057] Given mathematical text: {seed mathematical instruction data}.
[0058] The above are examples of prompt words for a large language model for synthesizing mathematical instruction data proposed in this application.
[0059] Existing large language models typically focus on single tasks, such as text summarization, information extraction, and code generation, in their prompt words. The prompt words for the large language model used to synthesize mathematical instruction data proposed in this application achieve multi-task fusion, completing multiple different types of tasks in a single processing step, thus providing more comprehensive data processing capabilities. When processing mathematical instruction data, existing prompt words can only generate solution flows or extract information, and cannot simultaneously generate multiple aspects such as solution flows, solution flow answers, reference answers, parameters, general mathematical problem templates, program code, and variable assignment and execution methods within the program code, as proposed in this application.
[0060] Existing large language models do not enforce a specific structured output format for their prompts, resulting in often free and loose outputs. The large language model for synthesizing mathematical instruction data proposed in this application specifies a strict, label-based structured output format for its prompts. Each part of the content clearly corresponds to a different label, making the output more standardized and orderly, facilitating subsequent data processing and utilization, and reducing the difficulty of data parsing and integration.
[0061] Existing large language models often exhibit weak interrelationships between tasks when processing data, resulting in shallow processing depth. In contrast, the large language model for synthesizing mathematical instruction data proposed in this application features tightly interconnected tasks within its prompts. From extracting questions and reference answers, solving questions, to generating parameters, general mathematical problem templates, and program code, each step is interdependent and progressive, enabling deeper and more systematic processing of mathematical instruction data. This results in richer, higher-quality, and more relevant data, providing stronger support for subsequent applications such as mathematical instruction data synthesis.
[0062] As an example, the output data of the large language model consists of 7 fields: SOLUTION, ANSWER, TEXTANSWER, PARAMETERS, PROBLEM, FUNCTION, and EXECUTION; which correspond to the solution process, the answer to the solution process, the reference answer, the parameters, the general mathematical problem template, the program code, the variable assignment in the program code, and the execution method, respectively.
[0063] Step S3: Execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model. If the execution is successful, obtain the execution result of the program code. If the execution fails (e.g., error or timeout), set the execution result of the program code to empty.
[0064] Step S4: Compare the solution flow answer, reference answer, and execution result of the program code output by the large language model. If the three are equal, it means that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and proceed to step S5. If the three are not equal, it means that the output result of the large language model for a certain seed mathematical instruction data has failed to be verified, and return to step S2 and change to another seed mathematical instruction data.
[0065] For example, this step uses a string matching algorithm to verify whether the three data items are equal. For the data items 0.5 and 1 / 2, even though they are mathematically equal, the string matching algorithm will consider them to be unequal.
[0066] For example, this step uses a large language model to verify whether the three data items are equal. An example prompt is shown below.
[0067] You are a math answer verifier. You need to compare the answer in the reference solution, the final answer in the model solution, and the final answer in the solution generated by Python to determine whether they are equivalent, even if their formats are different.
[0068] Suggested answer: {TEXTANSWER}.
[0069] The model's solution: {ANSWER}.
[0070] Python's solution: {EXECUTION ANSWER}.
[0071] It focuses solely on comparing the final mathematical answer provided by the model, the answer in the reference solution, and the final answer in the Python-generated solution, while ignoring differences in the following aspects: format (e.g., the difference between \boxed{{}} and plain text); multiple-choice question format (e.g., the difference between option "A" and the complete solution); the order of coordinate pairs or solutions; and changes in equivalent mathematical expressions or symbolic representations.
[0072] The conclusion that they are "equivalent" can only be drawn when the answers in the reference solution, the model solution, and the Python solution are exactly the same.
[0073] If any answer (from the model, reference solution, or Python-generated solution) is meaningless, then "Conclusion: Ambiguous" is returned.
[0074] Begin with a brief comparative description (2 to 3 sentences). Then output your final conclusion in one of the following formats: "Conclusion: Equivalent"; "Conclusion: Different".
[0075] The prompts in the large language model used above for comparing whether three data items are equal can compare whether three data items are equal in a mathematical sense, accurately focusing on the essence of the mathematical answer, ignoring differences such as format, multiple-choice question format, coordinate pairs or solution order, equivalent mathematical expressions or symbolic representation changes, for example, it can identify that 0.5 and 1 / 2 are equal; it establishes strict and clear judgment criteria to enhance credibility; it can handle meaningless answers to ensure rigorous verification; and it is highly versatile, applicable to various mathematical problems and models, reducing costs and improving efficiency.
[0076] Step S5: When the output result of the large language model for a certain seed mathematical instruction data is successfully verified, the parameters in the output data of the large language model for the seed mathematical instruction data are numerically replaced. The new values of the parameters are combined with the general mathematical problem template to form a new mathematical problem, that is, a new mathematical instruction data is obtained.
[0077] Please see Figure 2 Step S5 further includes the following sub-steps.
[0078] Step S51: Select the four parts from the output data of "The output result of the large language model for a certain seed mathematical instruction data has been successfully verified"—general mathematical problem template, parameters, program code, variable assignment and execution method in the program code.
[0079] Step S52: Perform same-type value replacement on the variables in the program code within the "Variable Assignment and Execution Method in Program Code". Same-type value replacement means that integer values can only be replaced with integer values, floating-point values can only be replaced with floating-point values, positive numbers can be replaced with positive numbers, and negative numbers can be replaced with negative numbers. This yields the new values for the variables in the program code.
[0080] Step S53: Execute the program code output by the large language model according to the new values of the variables in the program code and the program code execution method. If the execution is successful, obtain the new execution result of the program code. If the execution fails (e.g., an error or timeout), set the new execution result of the program code to empty.
[0081] Step S54: If the new execution result of the program code is not empty and has the same numerical type as the original execution result of the program code in step S3, then the parameters output by the large language model are replaced accordingly based on the new values of the variables in the program code to obtain new values for the parameters. The new values of the parameters are combined with the general mathematical problem template output by the large language model to obtain a new mathematical problem, and then proceed to step S55.
[0082] If the new execution result of the program code is not empty and the value type is different from the original execution result of the program code in step S3, then return to step S52 to perform the same type of value replacement again.
[0083] If the new execution result of the program code is empty, return to step S52 to perform the same type of value replacement again.
[0084] Step S55: Combine the new mathematical problem and the new execution result of the program code to obtain a new mathematical instruction data.
[0085] In step S5, the variable values in the program code are replaced first, and then the parameter values in the general mathematical problem template are replaced after the program code executes successfully. This approach has three advantages: First, it allows for early verification of the feasibility of the new parameter values at the program code execution level, enabling timely detection and avoidance of errors caused by parameter issues, thus ensuring the accuracy of the final data. Second, the parameters verified through program code execution are used to generate new mathematical problems, ensuring that the program code corresponding to the new mathematical problems is executable and the results are reliable. Third, this method, while achieving data diversification, maintains the logical and computational stability of the new mathematical problems by replacing parameters based on reliable program code execution results.
[0086] For example, consider the following seed mathematical instruction: Question: 1800 fish swim westward, 3200 swim eastward, and 500 swim northward. If fishermen catch two-fifths of the fish that swam eastward and three-quarters of the fish that swam westward, how many fish are left in the sea? Answer: 2870.
[0087] After executing the mathematical instruction data synthesis method of this application, the newly generated mathematical instruction data is as follows. Question: 3200 fish swim westward, 1800 swim eastward, and 100 swim northward. If the fishermen catch 2 / 3 of the fish swimming eastward and 1 / 2 of the fish swimming westward, how many fish are left in the sea? Answer: 2300.
[0088] Please see Figure 3 The controllable mathematical instruction data synthesis device proposed in this application includes a seed data preparation unit 1, a guiding synthesis unit 2, a code execution unit 3, a third-party verification unit 4, and a numerical replacement unit 5. Figure 3 The device shown corresponds to Figure 1 The method shown.
[0089] The seed data preparation unit 1 is used to prepare a seed mathematical instruction dataset, which includes multiple seed mathematical instruction data, each of which consists of a mathematical problem and an answer.
[0090] The guided synthesis unit 2 is used to combine each seed mathematical instruction data with a prompt word to obtain an input data, and to send the input data into a large language model for synthesizing mathematical instruction data to obtain an output data of the large language model.
[0091] The prompt words guide the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem solution process described in natural language; (2) extracting the answer from the mathematical problem solution process; (3) extracting the answer from seed mathematical instruction data; (4) extracting numerical values from seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problems in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; and (7) assigning values to variables in the program code and providing the execution method of the program code.
[0092] The output data of the large language model includes the output results of multiple tasks, specifically: (1) a mathematical problem-solving process described in natural language, hereinafter referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, hereinafter referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, hereinafter referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, hereinafter referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, hereinafter referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, hereinafter referred to as the program code; and (7) the variable assignment and program code execution method in the program code.
[0093] The code execution unit 3 is used to execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model, and obtain the execution result of the program code.
[0094] The third-party verification unit 4 is used to compare the solution process answer, reference answer, and execution result of the program code output by the large language model. If the three are equal, it means that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and then proceeds to the numerical replacement unit 5.
[0095] The numerical replacement unit 5 is used to replace the parameters in the output data of the large language model for a certain seed mathematical instruction data when the output result of the large language model is successfully verified. The new value of the parameter is combined with the general mathematical problem template to form a new mathematical problem, that is, to obtain a new mathematical instruction data.
[0096] Compared with existing mathematical instruction data synthesis methods, this application has the following advantages.
[0097] First, this application verifies and controls the output of the seed mathematical instruction data using a large language model, specifically the three data comparisons in step S4, to determine whether the output of the large language model for the seed mathematical instruction data is correct. If correct, subsequent processing is performed based on the seed mathematical instruction data to generate new mathematical instruction data; if incorrect, the seed mathematical instruction data is not used. Thus, this application ensures the accuracy of the original data from the source, avoids using erroneous data, and ensures the reliability and controllability of the generated new mathematical instruction data.
[0098] Second, the new mathematical instruction data generated by this application based on the new values of the parameters of the general mathematical problem template has a variety of different forms compared with the original seed mathematical instruction data, and is not a single fixed pattern.
[0099] Third, in step S5 of this application, the variable values of the program code are replaced first, and the parameter values of the general mathematical problem template are replaced after the program code is successfully executed. This verification mechanism ensures the high quality of the newly generated mathematical instruction data.
[0100] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A controllable method for synthesizing mathematical instruction data, characterized in that, Includes the following steps; Step S1: Prepare a seed math instruction dataset, which includes multiple seed math instruction data, each consisting of a math problem and an answer; Step S2: Combine each seed mathematical instruction data with the prompt word to obtain one input data; the large language model used to synthesize mathematical instruction data generates one output data based on each input data; The prompt words guide the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem-solving process described in natural language; (2) extracting the answer from the mathematical problem-solving process; (3) extracting the answer from seed mathematical instruction data; (4) extracting numerical values from seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problems in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; and (7) assigning values to variables in the program code and providing the execution method of the program code. The output data of the large language model used to synthesize mathematical instruction data includes the output results of multiple tasks, specifically including: (1) a mathematical problem-solving process described in natural language, referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, referred to as the program code; (7) the variable assignment and program code execution method in the program code; Step S3: Execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model; if the execution is successful, obtain the execution result of the program code; if the execution fails, set the execution result of the program code to empty; Step S4: Compare the solution process answer, reference answer, and execution result of the program code output by the large language model; if the three are equal, it means that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and proceed to step S5; Step S5: When the output result of the large language model for a certain seed mathematical instruction data is successfully verified, the parameters in the output data of the large language model for the seed mathematical instruction data are numerically replaced. The new values of the parameters are combined with the general mathematical problem template to form a new mathematical problem, that is, a new mathematical instruction data is obtained.
2. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, In step S2, the first item executed by the large language model used to synthesize mathematical instruction data belongs to the reasoning task, the second and third items belong to the information extraction task, the fourth and fifth items belong to the text summarization task, and the sixth and seventh items belong to the code generation task.
3. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, In step S2, the prompt word also instructs the output data of the large language model used to synthesize mathematical instruction data to adopt a structured format, with the output results of different tasks placed in different fields.
4. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, In step S2, the parameters in the output data of the large language model used to synthesize mathematical instruction data correspond one-to-one with the variables in the program code.
5. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, In step S4, if the three are not equal, it means that the large language model has failed to verify the output result of a certain seed mathematical instruction data, and the process returns to step S2 and another seed mathematical instruction data is used.
6. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, In step S4, a large language model is used to verify whether the three data items are equal; prompt words are used to guide the large language model used to compare whether the three data items are equal in a mathematical sense, ignoring differences in format, multiple-choice question format, coordinate pairs or answer order, equivalent mathematical expressions or symbolic representations.
7. The controllable mathematical instruction data synthesis method according to claim 1, characterized in that, Step S5 further includes the following sub-steps; Step S51: Select the four parts from the output data of "The output result of the large language model for a certain seed mathematical instruction data has been successfully verified": general mathematical problem template, parameters, program code, variable assignment and execution method in the program code; Step S52: Replace the variables of the same type in the program code in "Variable assignment and execution method in program code" with the new values of the variables in the program code; Step S53: Execute the program code output by the large language model according to the new values of the variables in the program code and the program code execution method; if the execution is successful, obtain the new execution result of the program code; If execution fails, set the new execution result of the program code to empty; Step S54: If the new execution result of the program code is not empty and has the same numerical type as the original execution result of the program code, then the parameters output by the large language model are replaced accordingly based on the new values of the variables in the program code to obtain the new values of the parameters; the new values of the parameters are combined with the general mathematical problem template output by the large language model to obtain a new mathematical problem. Proceed to step S55; Step S55: Combine the new mathematical problem and the new execution result of the program code to obtain a new mathematical instruction data.
8. The controllable mathematical instruction data synthesis method according to claim 7, characterized in that, In step S52, the same type of numerical replacement means replacing integer values with integer values, floating-point values with floating-point values, positive numbers with positive numbers, and negative numbers with negative numbers.
9. The controllable mathematical instruction data synthesis method according to claim 7, characterized in that, In step S54, if the new execution result of the program code is not empty and the numerical type is different from the original execution result of the program code, then return to step S52 to perform the same type of numerical replacement again. If the new execution result of the program code is empty, return to step S52 to perform the same type of value replacement again.
10. A controllable mathematical instruction data synthesis device, characterized in that, It includes a seed data preparation unit, a guide synthesis unit, a code execution unit, a third-party verification unit, and a numerical replacement unit; The seed data preparation unit is used to prepare a seed mathematical instruction dataset, which includes multiple seed mathematical instruction data, each of which consists of a mathematical problem and an answer. The guided synthesis unit is used to combine each seed mathematical instruction data with a prompt word to obtain an input data, and to send the input data into a large language model for synthesizing mathematical instruction data to obtain an output data of the large language model. The prompt words guide the large language model to perform multiple tasks simultaneously, including: (1) generating a mathematical problem-solving process described in natural language; (2) extracting the answer from the mathematical problem-solving process; (3) extracting the answer from seed mathematical instruction data; (4) extracting numerical values from seed mathematical instruction data and replacing them with parameters; (5) generating a general mathematical problem template based on the mathematical problems in the seed mathematical instruction data; (6) creating program code corresponding to the general mathematical problem template; and (7) assigning values to variables in the program code and providing the execution method of the program code. The output data of the large language model includes the output results of multiple tasks, specifically: (1) a mathematical problem-solving process described in natural language, hereinafter referred to as the solution process; (2) the answer extracted from the mathematical problem-solving process, hereinafter referred to as the solution process answer; (3) the answer extracted from the seed mathematical instruction data, hereinafter referred to as the reference answer; (4) the parameters extracted from the seed mathematical instruction data, hereinafter referred to as the parameters; (5) a general mathematical problem template generated based on the mathematical problem in the seed mathematical instruction data, hereinafter referred to as the general mathematical problem template; (6) the program code corresponding to the general mathematical problem template, hereinafter referred to as the program code; (7) the variable assignment and program code execution method in the program code. The code execution unit is used to execute the program code output by the large language model according to the "variable assignment and execution method in the program code" output by the large language model, and obtain the execution result of the program code; The third-party verification unit is used to compare the solution process answer, reference answer, and execution result of the program code output by the large language model; if the three are equal, it means that the output result of the large language model for a certain seed mathematical instruction data has been successfully verified, and then it enters the numerical replacement unit. The numerical replacement unit is used to replace the parameters in the output data of the large language model for a certain seed mathematical instruction data when the output result of the large language model is successfully verified. The new value of the parameter is combined with the general mathematical problem template to form a new mathematical problem, that is, to obtain a new mathematical instruction data.