Formula-based numerical reasoning question and answer implementation method, device and medium

Abstract

A formula-based numerical reasoning question and answer implementation method, device and medium, the question and answer in the numerical reasoning question and answer application scene are obtained through formula knowledge annotation to obtain formula data set and parameter data set, which are respectively used for training of formula generation model and parameter identification model, for new questions in the question and answer application scene, formula information is generated by the formula generation model, the parameter name in the formula information is input into the parameter identification model, the parameter information of the formula is obtained, the formula information and the parameter information are input into the formula calculator, and the answer of the question is output. The construction of the training data can be automatically carried out, the data migration generalization and data expansion are facilitated; the formula calculator of the calculable unit is introduced in the calculation, the problem that the generative language model is not sensitive to the numerical value is avoided; the numerical reasoning is divided into several subtasks, the requirement for the model is reduced, the obtained answer has good explainability, and the modularization decoupling is favorable for model optimization.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, and relates to automated dataset construction technology in numerical reasoning and numerical reasoning technology based on large language models. Specifically, it is a formula-based numerical reasoning question answering method, device and medium. Background Technology

[0002] Numerical reasoning is an important research area in fields such as natural language processing, artificial intelligence, and question-answering systems. Research in numerical reasoning aims to analyze and understand numerical relationships to improve the reasoning capabilities of systems, which is significant for both natural language research and the development of artificial intelligence. However, this research direction is also challenging and requires further investigation to address current problems and challenges.

[0003] Numerical reasoning involves a model understanding the content of text or tables, extracting, comparing, calculating, sorting, or logically deriving numbers to ultimately provide an accurate numerical answer. Early research on numerical reasoning was primarily based on statistics or rule-based symbols, using rules or pre-designed computational templates. With the development of neural networks, deep learning-based numerical reasoning methods became a research focus. This stage of research mainly enhanced the system's understanding and application of numerical values ​​by modifying network structures and strengthening network modules. For example, NC-BERT uses masking and other methods in its network structure to reduce the model's focus on specific numbers, thereby strengthening its focus on reasoning logic and improving numerical reasoning capabilities. Subsequently, some reasoning methods combining numerical and symbolic representations have been developed to further enhance numerical reasoning capabilities, such as using neural networks to generate computational and numerical symbols to construct expressions.

[0004] Since the emergence of large language models, their powerful language understanding and generation capabilities have attracted widespread attention, and they have also demonstrated advantages in the field of numerical reasoning. While large language models essentially still fall under the category of deep neural networks, their outstanding performance and massive parameter count clearly distinguish them from other neural network models. Currently, much numerical reasoning research is based on large language models. Among these, the thought chain method and supervised fine-tuning method are important solutions for numerical reasoning. The thought chain method refers to guiding the model to explicitly generate a series of intermediate reasoning steps before generating the final answer, making the model's reasoning process closer to the step-by-step thinking process of humans. Supervised fine-tuning involves introducing labeled supervisory data, enabling the model to achieve stronger performance and better results in specific tasks or domains.

[0005] Early methods for numerical reasoning problems, such as rule-based or computational template-based classification methods, while highly interpretable, often fall short in solving complex natural language problems and exhibit low generalization. Neural network-based methods offer higher generalization, but training is becoming increasingly complex, and they struggle with the increasingly challenging task of collecting and preparing diverse datasets. Large-model methods have shown significant improvement over other approaches, but numerical reasoning requires models to perform sub-tasks such as abstract reasoning steps, numerical retrieval, and numerical computation. Currently, most methods still rely on a single large model to complete all sub-tasks. While large language models have a large number of parameters, they also suffer from knowledge forgetting, making it difficult to adequately solve all sub-tasks with a single fine-tuned model. Therefore, they place high demands on large models, lack assurance of numerical computation accuracy in numerical reasoning, and do not adequately address knowledge content such as formulas, resulting in insufficient interpretability at the knowledge level.

[0006] Furthermore, the training dataset is particularly important for supervised fine-tuning of large language models. Many current numerical reasoning datasets provide questions and answers, and some datasets (such as GSM8K) provide problem-solving processes in natural language form. However, these datasets do not provide annotations for formula knowledge. In numerical reasoning datasets involving formulas (such as Math23K-F), the proportion of formula annotations is too low, and the formulas lack domain-specificity and specialization, and are mostly common-sense formulas. Summary of the Invention

[0007] The problem this invention aims to solve is that existing numerical reasoning methods struggle to simultaneously meet requirements such as problem-solving ability, model performance, model training ease, reasoning interpretability, and reasoning accuracy. This invention investigates a method that, in the absence of formula knowledge annotations, automatically constructs formula and parameter datasets based on the answers to numerical reasoning questions. These datasets are then used for model training and problem-solving in the numerical reasoning domain, reducing the demands on model performance and training difficulty while enhancing the accuracy and interpretability of reasoning.

[0008] The technical solution of this invention is: a formula-based numerical reasoning question-answering method, comprising:

[0009] The questions and answers in the numerical reasoning question-and-answer application scenario are taken as a question-and-answer pair. A question-and-answer pair dataset is constructed as a training dataset. The question-and-answer pair dataset is labeled with formula knowledge through a large language model to obtain formula information and parameter information related to the questions. Formula dataset and parameter dataset are constructed and used to train the formula generation model and parameter recognition model, respectively.

[0010] During prediction, the formula generation model generates formula information corresponding to the input question. The parameter names in the formula information and the question input parameter identification model are used to predict the parameter information involved in the formula. The formula information and parameter information are input into the formula calculator. Through multi-step iterative calculation, the numerical reasoning result containing units and values ​​in the formula calculation steps is obtained. The three modules work together to complete the solution of the numerical reasoning problem. The formula calculator is a dimensional analysis calculator.

[0011] Furthermore, the formula dataset and parameter dataset are automatically constructed, following these steps:

[0012] Step 1: For the question-answering pair dataset, construct prompt words to specify the formula information and parameter information for each question output by the large language model. The formula information includes multiple formulas, each of which is an operational equation composed of operators and parameter names. The parameter information includes the parameter name, value and unit of each parameter.

[0013] Step Two: Verify data quality using a program. Use regular expressions to extract the information generated under the prompts from Step One. Combine this with a formula calculator to verify whether the formula answers match the answers in the original question-and-answer pairs. Check the format and information of each formula and parameter to ensure they are complete.

[0014] Step 3: For the errors that occurred in Step 2, use a large language model to correct them by constructing prompt words. For the errors that did not occur in Step 2, retain them. Repeat the correction of the errors until all the errors generate information correctly or the expected number of correct information is reached. Extract and save the problem and formula information as the content of the formula dataset, and extract and save the problem and parameter information as the content of the parameter dataset.

[0015] Furthermore, the training of the formula generation model and the parameter recognition model specifically involves: the formula generation model and the parameter recognition model being large language models based on a decoder architecture.

[0016] For the formula generation model, prompt words are constructed to describe the formula generation task that the model needs to perform. The question is used as the input of the task in the prompt words, and the output of the task is the formula information. The formula information in the formula dataset is used as the supervision label of the model output, and supervised learning is performed.

[0017] For the parameter recognition model, prompt words are constructed to describe the parameter recognition task that the model needs to perform. The question and parameter name are used as inputs to the task in the prompt words, and the output of the task is parameter information. The parameter information of the parameter dataset is used as the supervision label of the model output, and supervised learning is performed.

[0018] The present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to realize the above-mentioned formula-based numerical reasoning question answering method.

[0019] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or the program is executed, the above-described formula-based numerical reasoning question-answering method is implemented.

[0020] The beneficial effects of this invention are as follows:

[0021] (1) In this invention, the construction of formula information and parameter information of training data can be automated, requiring almost no manual intervention, which facilitates data migration, generalization and data expansion.

[0022] (2) The present invention introduces a formula calculator that can calculate units in the calculation, which avoids the problem that generative language models are not sensitive to numerical values. Existing pure large language models focus more on numerical calculation in numerical reasoning, and pay less attention to the application of formulas and the calculation of units. Even if a calculator is introduced, it is just an ordinary numerical calculator. On the one hand, it lacks accurate formula generation tasks, and on the other hand, it is limited to numerical calculation of numerical formulas given by large models, and cannot cope with different unit dimensions involved in the problem.

[0023] (3) This invention breaks down numerical reasoning into several sub-tasks. A formula generation model predicts and generates formulas based on the problem text, and a parameter identification model predicts the values ​​and units of the parameters in the formula based on the problem and the formula. The two are then combined and calculated using a formula calculator. The multi-module approach used in this invention reduces the requirements on the model and makes it easier for models with small parameters or poor capabilities to perform numerical reasoning. The design of this invention can better mitigate the "knowledge forgetting" of the model, and is more friendly to models with smaller parameters. Smaller models can remember less, while this invention allows a single model to remember less knowledge.

[0024] Compared with existing LLM large models that directly perform numerical inference, this invention uses a multi-module architecture. Each module uses a different fine-tuning model. Although the formula generation model and parameter identification model used are existing models, multiple fine-tuning models are designed. This allows each model to focus on different sub-tasks. The model focuses on only sub-tasks rather than all tasks, forming a collaboration of multiple models, rather than a large model taking care of all sub-tasks, thereby improving the overall accuracy of the method.

[0025] In other existing methods, the large model may not necessarily explain each parameter and the overall computational logic in the formula. However, the formula generation module in this invention provides specific formulas, more clearly indicating what the model is doing, emphasizing the application of formula knowledge, and improving interpretability. This invention also introduces a formula calculator, which transforms the part that previously required unreliable inference calculations (token generation predicted by the large model is based on probability) into deterministic calculations (the formula calculator in this invention prevents calculation errors).

[0026] (4) This invention surpasses similar technologies in the application of real-world data, and the obtained answers have good interpretability. The modular decoupling design facilitates the optimization of the overall question-answering model of this invention. Similar technologies refer to numerical reasoning techniques based on LLM. The following table lists methods based on LLM models, including existing LLM models: Supervised fine-tuning refers to fine-tuning a single LLM model using only the corresponding data; Data augmentation refers to using additional generated data (large model generation) to increase the training data and perform fine-tuning on the basis of supervised fine-tuning; Calculator assistance refers to adding a formula calculator, but still using a single fine-tuned LLM model to perform all sub-tasks. The answer accuracy results of this invention compared with supervised fine-tuning, data augmentation, and calculator assistance methods on different formula source test sets are as follows:

[0027] Supervision and fine-tuning 65.62 58.91 62.27 Data Augmentation 67.55 62.53 65.04 Calculator Assistant 72.64 65.63 69.14 This invention 74.09 65.63 69.86

[0028] As can be seen from the test results, the test results of the present invention are significantly better than those of the existing methods. Attached Figure Description

[0029] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0030] This invention proposes a multi-module numerical reasoning method and apparatus based on automated formula knowledge annotation. The invention will be further described below with reference to embodiments and flowcharts.

[0031] Example 1: This example applies to a numerical reasoning dataset in the physics field. Solving numerical reasoning problems in this scenario requires extensive use of formulas, and the reasoning and calculation are more challenging than problems relying solely on common sense. The method of this invention effectively addresses the problem of constructing formula knowledge in this field and allows for effective numerical reasoning problem-solving based on these formulas. The problem and its answer in the numerical reasoning problem-solving application scenario are treated as a question-answer pair. The question-answer pair dataset is then annotated with coarse-grained tags to obtain relatively standardized problem-solving steps, followed by fine-grained tags to obtain more detailed formula and parameter information. Regular expressions are used in the annotation process to ensure the correct format of the obtained information, and a formula calculator is used to ensure the calculation results are consistent with the original question-answer pair answers. The obtained formula and parameter information contribute to the overall solution. Figure 1 The datasets shown are the formula dataset and the parameter dataset. These two datasets are used to train a large language model based on a decoder architecture. The formula generation model is trained using the question as input and the formula information as the supervision label. The parameter recognition model is trained using the question and parameter names as input and other content of the parameter information (numerical values ​​and units) as the supervision label. During the inference phase, when predicting the solution, the question is embedded in a prompt word template and input into the formula generation model. This model generates formula information related to the question. After recognizing each formula and parameter using regular expressions, the question and parameters are embedded in the prompt word template and input into the parameter recognition model. This model generates parameter information corresponding to each parameter. After performing regular expression-based information extraction, the formula and parameter information are input into the formula calculator. Through multi-step iterative calculations, the numerical inference results containing units and values ​​in each formula calculation step are continuously obtained. The final calculated result is the answer to the question.

[0032] The implementation process of this embodiment will be described below. First, the automated construction of the formula dataset and parameter dataset will be described:

[0033] Step 1: For the question-answering pair dataset used for training, construct coarse-grained and fine-grained prompt words. Coarse-grained prompt words contain the task description and specific question, prompting the large language model to provide a standardized natural language problem-solving process, generating coarse-grained information. Fine-grained prompt words contain the task description, specific question, and coarse-grained information, i.e., the standardized natural language problem-solving process, prompting the large language model to provide formula and parameter information, generating fine-grained information. The formula information contains multiple formulas, each consisting of parameter names and operators. Parameter names are text names delimited by square brackets, and operators include mathematical operators such as addition (+), subtraction (-), multiplication (*), division ( / ), and exponentiation (^). The parameter information contains multiple parameters, each consisting of a parameter name, value, and unit. The value is a mathematical number, not a numerical value represented by Chinese characters, and the unit is a symbolic unit from the physics domain, such as "N" and "kg," representing the units of force and mass, respectively: "Newton" and "kilogram."

[0034] Step Two: Verify Data Quality Using a Program. Regular expressions are used to extract the answer field from the coarse-grained information. The formula calculator of this invention is a dimensional analysis calculator capable of calculating both units and numbers. In this embodiment, Numbat is used as the formula calculator. The calculator is used to compare the answer in the original question-and-answer pair with this field to see if they match. Regular expressions are used to extract the formula and parameter information from the fine-grained information. The correct format of the formula and the completeness of the parameters are checked, ensuring each parameter has a name, value, and unit. The parameter names in the formula information are replaced with the parameter values ​​and units. The resulting expression is input into the formula calculator to check if the calculation process is correct and if the final result matches the answer in the original question-and-answer pair.

[0035] In addition to using regular expressions to extract information generated by the large language model under prompts in step one, we can also write rules to perform string processing to assist in information extraction. For example, in the parameter information given by the large language model in the previous steps, the units may have some textual representations. Predefined replacement rules can replace the text in the units with unit characters, such as replacing "kilogram" with "kg".

[0036] Step 3: For annotation errors that occurred in Step 2, including incomplete formula or parameter information and calculation or result errors, different prompt words are constructed based on different problems. These prompt words include task description, specific problem, annotation information, and the error in the annotation. These prompt the large language model to correct the errors and output the correct annotation information. Problems that did not have erroneous annotations in Step 2 are retained, and the correction of erroneous problems is repeated until all problems generate information correctly or the expected number of correct information is reached. Finally, the problem and formula information are extracted and saved as the formula dataset content, and the problem and parameter information are extracted and saved as the parameter dataset content, resulting in coarse and fine granular annotations for model training.

[0037] The next part is the model training and problem-solving section:

[0038] Step 4: Use a large language model based on a decoder architecture as the formula generation model. Construct prompt words and supervision labels based on the formula dataset saved in Step 3. The prompt words include task descriptions and questions, and the supervision labels include formula information. Perform supervised learning on the formula generation model based on this data.

[0039] Step 5: Use a large language model based on the decoder architecture as the parameter recognition model. Construct prompt words and supervision labels based on the parameter dataset saved in Step 3. The prompt words include task description, question, and parameter name. The supervision labels include the numerical value and unit of the parameter. Perform supervised learning on the parameter recognition model based on this data. The base model of the parameter recognition model (i.e. the model before training) and the base model in Step 4 can be the same model.

[0040] Step Six: For the new problem in this scenario, construct prompt words from the new problem and task description, input them into the formula generation model, and the formula generation model will output the formula information of the steps required to solve the problem. Use regular expressions to extract the parameter names from the formula information, construct prompt words from the problem and parameter names, input them into the parameter recognition model, and the parameter recognition model will output the parameter information corresponding to each parameter. Replace the parameter names in the formula information with the values ​​and units in the parameter information, and input the resulting expressions into the formula calculator. Calculate the results of each formula in turn and record the known parameters. The result obtained after all formulas have been calculated is the answer to the new problem.

[0041] Example 2 illustrates the implementation of the present invention through a specific question-and-answer process.

[0042] Step 1:

[0043] The training dataset contains questions Q and answers A. Coarse-grained prompts P1 and fine-grained prompts P2 are constructed. The coarse-grained prompts aim to standardize the annotations in the original dataset to a relatively uniform natural language format, while the fine-grained prompts aim to extract formatted parameters and formulas from the natural language. Q and A are embedded into prompt P1, and a large language model is used to obtain the response R1. Similarly, Q and R1 are embedded into prompt P2, and a large language model is used to obtain the response R2.

[0044] In this embodiment, when using a large language model to annotate formula knowledge and automatically construct formula and parameter information for training data, the constructed prompt words all adopt the few-shots prompting method.

[0045] Prompt word P1:

[0046] I need you to revise the original parsing of the question and provide a new parsing in a standardized format, as follows:

[0047] 1. Please think step by step. If there are parts with combined formulas, you need to break them down into basic formulas step by step to solve them.

[0048] 2. Calculation symbols in formulas, such as "+", "-", "×", " / " and "^", cannot be omitted;

[0049] 3. The formula needs to provide both the symbol and the form with a specific meaning, and then substitute the numerical values ​​to calculate the answer;

[0050] 4. For parts involving unit conversions, the specific steps need to be shown;

[0051] 5. Use "[ ]" to mark variables in formulas. Numbers in scientific notation such as "a×10^b" and complex units should be marked with "()".

[0052] 6. If there are formulas in LaTeX format, such as "\frac{Q_{absorb}}{Q_{release}}", they need to be changed to the normal formula form: "Q_{absorb} / Q_{release}".

[0053] 7. Parsing begins. If "(The parsing may contain errors)" is given, it indicates that there may be errors in the given parsing.

[0054] Here are some examples:

[0055] input:

[0056] Question: In a certain test, the resistance experienced by the car while traveling at a constant speed is 0.02 times the weight of the car. It traveled 28km in 0.5h. What is the average speed of the car?

[0057] Analysis: Solution: The vehicle traveled 28km in 0.5h, so the average speed was:

[0058] v = s / t = 28 km / 0.5 h = 56 km / h.

[0059] output:

[0060] untie:

[0061] The original parsing is quite clear; below, it will be converted into a standardized format:

[0062] The first step is to calculate the average velocity: v = s / t

[0063] Average speed = Distance / Time

[0064] The formula is: 28km / 0.5h = 56km / h

[0065] Average speed = 56 km / h

[0066] Answer = 56km / h

[0067] ...

[0068] Here is a new question:

[0069] input: \boxed{INPUT}

[0070] output:

[0071] Where \boxed{} represents a placeholder, filled in according to the example. The Few-shot Prompting method, when interacting with a large language model, provides 2-5 input-output examples in the prompt words, allowing the model to "learn" the desired task format and patterns through these examples, and then provide the correct answer to new input. This embodiment provides one example; others will not be detailed further.

[0072] Prompt word P2:

[0073] I need you to help me extract the formulas used to solve the problem from the given problem and its solution, along with the corresponding parameter table:

[0074] 1. Please think step by step. First, analyze the analysis, then generate the extraction results. If there are formula combinations, you need to break them down into basic formulas step by step for solving.

[0075] 2. All calculation symbols in the formula, such as "+", "-", "×", " / " and "^", cannot be omitted;

[0076] 3. Each variable in the formula needs to be marked with "[ ]", and the variables should be identified with meaningful text, avoiding direct numerical values ​​as much as possible;

[0077] 4. If there are formulas in LaTeX format, such as "\frac{Q_absorb}{Q_release}", they need to be changed into normal formula form: "[Q_absorb] / [Q_release]". The unit conversion part in the formula is not part of the formula and does not need to be extracted.

[0078] 5. The parameters in the parameter table are the parameters used in the formula (the parameter names must be consistent with the parameters in the formula). The table includes: concepts, symbols, values, and units, and uses "|" to separate cells.

[0079] 6. The values ​​and units in the parameter table come from the problem itself and the intermediate results of the analytical calculation. If the parameters have been converted to units, the parameter table should give the original parameter form (if no unit conversion has been performed).

[0080] 7. If a parameter in the parameter table does not have a unit or represents the quantity of a certain object, such as "3 lamps", "spun 8 times", "4 people", etc., then the unit should be written as "null".

[0081] 8. No additional explanatory information is required in the formulas and parameter tables provided in the extraction results.

[0082] Here are some examples:

[0083] input:

[0084] Question: In a certain test, the resistance experienced by the car while traveling at a constant speed is 0.02 times the weight of the car. It traveled 28km in 0.5h. What is the average speed of the car?

[0085] Analysis: Solution: The original analysis is quite clear. Below, we convert it to a standard format:

[0086] The first step is to calculate the average velocity: v = s / t

[0087] Average speed = Distance / Time

[0088] The formula is: 28km / 0.5h = 56km / h

[0089] Average speed = 56 km / h

[0090] Answer = 56km / h

[0091] output:

[0092] Extraction and analysis:

[0093] 1. The formula format in the analysis meets the requirements, the calculation symbols are not omitted, the variables are marked with "[ ]", and the variables are identified with meaningful text.

[0094] 2. No unit conversion was performed during the calculation process; the parameters can be extracted directly.

[0095] The extraction results are shown below (unit conversions are not part of the formula and do not need to be extracted; no additional explanation is needed in the parameter table).

[0096] Extraction results:

[0097] 1. First calculate the average speed, based on the relationship between distance and time:

[0098] Average speed = Distance / Time

[0099] Parameter table:

[0100] Concepts | Symbols | Numerical Values ​​| Units

[0101] Distance |s|28|km

[0102] Time |t|0.5|h

[0103] Average speed |v|56|km / h

[0104] ...

[0105] Here is a new question:

[0106] input: \boxed{INPUT}

[0107] output:

[0108] After the prompt words P1 and P2 are constructed, formula knowledge annotation is performed on the questions Q and answers A of the dataset based on the large language model question answering.

[0109] Q: A red-hot iron nail has a temperature of 800℃ and a mass of 2g. How much heat will be released when its temperature is reduced to 20℃?

[0110] A: Solution: Given: t_iron = 800℃, m = 2g = 0.002K, gt = 20℃

[0111] Δt = 800℃ − 20℃ = 780℃

[0112] c_iron = 0.46 × 10^3 J / (Kg·℃)

[0113] Request: Q_release

[0114] Solution: According to Q_released = C_iron mΔt = 0.46 × 10^3 J / (Kg·℃) × 0.002Kg × 780℃ = 717.6 J.

[0115] A: It will release 717.6J of heat.

[0116] Reply to R1: Solution:

[0117] The first step is to calculate the temperature change of the nail:

[0118] Δt=t_0-t

[0119] [Temperature Change] = [Initial Temperature] - [Final Temperature]

[0120] Calculation = 800℃ - 20℃ = 780℃

[0121] Temperature change = 780℃

[0122] The second step is to calculate the heat released:

[0123] Q=C*m*Δt

[0124] [Heat released] = [Specific heat capacity] * [Mass] * [Temperature change]

[0125] The formula is: (0.46 × 10^3) ​​J / (Kg·℃) * (0.002) Kg * 780℃ = 717.6 J

[0126] The heat released = 717.6 J

[0127] Answer = 717.6J

[0128] Reply to R2:

[0129] 1. Calculate the temperature change of the iron nail:

[0130] [Temperature Change] = [Initial Temperature] - [Final Temperature]

[0131] 2. Calculate the heat released:

[0132] [Heat released] = [Specific heat capacity] * [Mass] * [Temperature change]

[0133] Parameter table:

[0134] Concepts | Symbols | Numerical Values ​​| Units

[0135] Initial temperature |t_0|800|℃

[0136] Final temperature |t|20|℃

[0137] Temperature change |Δt|780|℃

[0138] Specific heat capacity |C| 0.46×10^3| J / (Kg·℃)

[0139] Mass | m | 0.002 | Kg

[0140] Heat released |Q| 717.6| J

[0141] Step Two:

[0142] Use regular expressions and other methods to obtain the formula information in R2, such as: [Temperature Change] = [Initial Temperature] - [Final Temperature], and parameter information, such as: {"Name": "Final Temperature" "Symbol": "t", "Value": "20" "Unit": "℃"}.

[0143] Use regular expressions and programs to verify whether the parameters can be obtained from the original question or calculated by the formula. Substitute the parameter information into the formula to obtain the expression, such as 800℃-20℃. Use the Numbat calculator to calculate the result, such as 780℃. Use the Numbat calculator to verify and compare the final calculated result 717.6J with the corresponding answer 717.6J in A to check whether it is correct.

[0144] Step 3:

[0145] To correct errors in step two, for example, if the required parameters cannot be found in the question, construct a prompt word P3 to prompt the large language model to correct the error and output the correct annotation information. Embed Q and the parameters into the prompt word P3 and use the large language model for further correction.

[0146] Prompt word P3:

[0147] I need your help to resolve a missing parameter error. I will provide the problem and the missing parameter. Below are the requirements for error correction:

[0148] 1. You need to first perform error analysis, analyze how to modify and correct the errors, and then provide the error correction section to rectify the errors within it;

[0149] 2. No additional explanation is required for the error correction section. The format of the error correction section is: "Content: Content before modification → Content after modification". When adding content, "Content before modification" is null, and when deleting content, "Content after modification" is null.

[0150] 3. If the question does not contain any missing parameters, then add the missing parameters to the question; if the parameters in the question have the same meaning as the missing parameters but have different formats, then add the original parameters from the question to the parameter table.

[0151] 4. If the missing parameter can be calculated from other parameters in the problem, then provide the corresponding formula and parameter table; no modification to the problem is required.

[0152] 5. The formula format is “[missing parameter name]=[problem parameter 1](+|-|×| / |^)[problem parameter 2]”; the parameter table format is “concept|symbol|value|unit”.

[0153] Here are some examples:

[0154] input:

[0155] Problem: A red-hot iron nail, with a temperature of 820℃ and a mass of 2×10⁻³ kg, is immersed in a liquid with an initial temperature of 10℃ and a mass of 36.8 g. The temperatures of both the iron and the liquid reach 20℃ and then remain constant. Ignoring the influence of the surrounding environment and heat loss, the specific heat capacity of iron is 0.46×10³ J / (kg•℃).

[0156] Find: the heat released by the iron nail.

[0157] Error: The problem is missing the parameter: Specific heat capacity of iron = 0.46 × 10^3 J / (kg·℃);

[0158] The mass of the nail = 2 × 10^-3 kg;

[0159] output:

[0160] Error Analysis:

[0161] 1. Searching for information related to "the specific heat capacity of iron = 0.46 × 10^3 J / (kg·℃)" in the question, we find "the specific heat capacity of iron is 0.46 × 10^3 J / (kg·℃)". However, the scientific notation format of "0.46 × 10^3 J / (kg·℃)" is incorrect; it is missing a "^". It should be changed to "the specific heat capacity of iron is 0.46 × 10^3 J / (kg·℃)".

[0162] 2. Searching the problem for information related to "the mass of the nail = 2 × 10⁻³ kg", we find "the mass is 2 × 10⁻³ kg". However, the scientific notation format is incorrect; it's missing the '^' symbol and needs to be corrected to "the mass is 2 × 10⁻³ kg".

[0163] Error correction:

[0164] Question: The specific heat capacity of iron is 0.46 × 10³ J / (kg•℃) → 0.46 × 10³ J / (kg·℃)

[0165] Question: The mass is 2 × 10⁻³ kg → The mass is 2 × 10⁻³ kg

[0166] ...

[0167] Here is a new question:

[0168] input: \boxed{INPUT}

[0169] output:

[0170] After correction by prompt P3, until all questions generate correct information or the expected number of correct information is reached, save the formula information in Q and R2 as a formula dataset, with each Q corresponding to several formulas; save the parameter information in Q and R2 as a parameter dataset, with each Q corresponding to several parameters. In the parameter dataset, only parameters appearing in the questions are retained, excluding parameters calculated using formulas. For example, in the above example, "temperature change" is calculated using formulas and is not retained in the parameter dataset.

[0171] Step Four:

[0172] Based on the numerical reasoning application scenario, a prompt word P4 is constructed for the formula generation model. Q and formulas from the formula dataset are embedded in the prompt word P4 as data for training the formula generation model. The large language model, such as Qwen2.5-Math-7B-Instruct, is fine-tuned using a fully supervised fine-tuning method to train the formula generation model.

[0173] As an example, prompt word P4:

[0174] This is a middle school physics problem. The calculation process should be given based on the question, expressed using formulas.

[0175] During model training, input question Q to generate formulas, and use the formula dataset for supervised fine-tuning training.

[0176] Question: \boxed{INPUT}

[0177] Answer: \boxed{OUTPUT}

[0178] Step 5:

[0179] Similarly, construct the prompt word P5 for the parameter recognition model, embed Q and parameters from the parameter dataset into the prompt word P5, and use it as the data for training the parameter recognition model; use full-supervised fine-tuning to fine-tune a large language model, such as Qwen2.5-Math-7B-Instruct, to train the parameter recognition model.

[0180] As an example, prompt word P5:

[0181] This is a middle school physics question. Find the numerical value and unit of the corresponding concept.

[0182] Question: \boxed{INPUT}

[0183] Concept: \boxed{INPUT}

[0184] Value:

[0185] \boxed{OUTPUT}, Unit: \boxed{OUTPUT}

[0186] Step Six:

[0187] After the model fine-tuning training is completed, for a new problem Q' in this numerical reasoning scenario, it is input into the formula generation model to obtain formula information in the same format as during training. Regular expressions are used to extract parameter names from the formula string, and Q' and the extracted parameter names are input into the parameter recognition model to obtain parameter information in the same format as during training. The formula information and parameter information are then input into the formula calculator for calculation. For each new problem, the parameters of each formula are calculated sequentially according to the order of the formulas in the formula information, and the known parameters are updated. Finally, the answer to the new problem is obtained.

[0188] Q: An electric water heater has 50 kg of water in its tank, which is heated from 20℃ to 40℃ by electric heating. The specific heat capacity of water is C_water = 4.2 × 10^3 J / (kg·℃). If the total electrical energy consumed during the heating process is 1 × 10^7 J, what is the thermal efficiency of the water heater?

[0189] The formula is obtained by generating the model from Q':

[0190] [Water temperature rise] = [Final temperature] - [Initial temperature]

[0191] [Heat absorbed by water] = [Mass of water] * [Specific heat capacity of water] * [Increase in temperature of water]

[0192] Heater efficiency = Heat absorbed by water / Total electrical energy consumed

[0193] Q' and parameters are obtained through a parameter identification model:

[0194] Final temperature = 40℃

[0195] Initial temperature = 20℃

[0196] The mass of water = 50 kg

[0197] The specific heat capacity of water = 4.2 × 10^3 J / (kg·℃)

[0198] Total electrical energy consumed = 1 × 10^7 J

[0199] Substitute the values ​​into the Numbat calculator to perform the calculation and obtain the answer:

[0200] Efficiency is 42%

[0201] This invention is based on a computer program. Based on this, the invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the above-described formula-based numerical reasoning question-answering method. A computer-readable storage medium is also provided, storing at least one instruction or at least one program segment. When the at least one instruction or program segment is executed, it implements the above-described formula-based numerical reasoning question-answering method, constructing a question-answering model. The instructions or program code used to implement the method of this invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on a data processor, partially on a processor, partially on a remote device as a standalone software package, or entirely on a remote device or server.

Claims

1. A formula-based numerical reasoning question-answering implementation method, characterized by: include: The questions and answers in the numerical reasoning question-and-answer application scenario are taken as a question-and-answer pair. A question-and-answer pair dataset is constructed as a training dataset. The question-and-answer pair dataset is labeled with formula knowledge through a large language model to obtain formula information and parameter information related to the questions. Formula dataset and parameter dataset are constructed and used to train the formula generation model and parameter recognition model, respectively. During prediction, the formula generation model generates formula information corresponding to the input question. The parameter names in the formula information and the question input parameter identification model are used to predict the parameter information involved in the formula. The formula information and parameter information are input into the formula calculator. Through multi-step iterative calculation, the numerical reasoning result containing units and values ​​in the formula calculation steps is obtained. The three modules work together to complete the solution of the numerical reasoning problem. The formula calculator mentioned is a dimensional analysis calculator.

2. The formula-based numerical reasoning question-answering method according to claim 1, characterized in that: The formula and parameter datasets are automatically constructed using the following steps: Step 1: For the question-answering pair dataset, construct prompt words to specify the formula information and parameter information for each question output by the large language model. The formula information includes multiple formulas, each of which is an operational equation composed of operators and parameter names. The parameter information includes the parameter name, value and unit of each parameter. Step Two: Verify data quality using a program. Use regular expressions to extract the information generated under the prompts from Step One. Combine this with a formula calculator to verify whether the formula answers match the answers in the original question-and-answer pairs. Check the format and information of each formula and parameter to ensure they are complete. Step 3: For the errors that occurred in Step 2, use a large language model to correct them by constructing prompt words. For the errors that did not occur in Step 2, retain them. Repeat the correction of the errors until all the errors generate information correctly or the expected number of correct information is reached. Extract and save the problem and formula information as the content of the formula dataset, and extract and save the problem and parameter information as the content of the parameter dataset.

3. The formula-based numerical reasoning question-answering method according to claim 2, characterized in that: In step three, for problems that result in errors, different prompt words are constructed based on different problems, including task description, specific problem, annotation information, and errors in the annotation, prompting the large language model to correct the errors and output the correct annotation information.

4. A formula-based numerical reasoning question-answering implementation method according to claim 2 or 3, characterized in that: In step one, coarse-grained prompt words and fine-grained prompt words are constructed. The coarse-grained prompt words contain the task description and specific questions, prompting the large language model to provide a standard natural language problem-solving process and generating coarse-grained information. The fine-grained prompt words contain the task description, specific questions, and coarse-grained information, i.e., the standard natural language problem-solving process, prompting the large language model to provide formula information and parameter information and generating fine-grained information. In step two, regular expressions are used to extract the answer field from the coarse-grained information. A formula calculator is used to compare the answers and answer fields in the original question-and-answer pair to see if they match. Regular expressions are then used to extract the formula and parameter information from the fine-grained information. The formula is checked for correct format and complete parameters. Each parameter is checked for a name, value, and unit. The parameter names in the formula information are replaced with the parameter values ​​and units. The resulting expression is then entered into the formula calculator to check if the calculation process is correct and if the final result matches the answer in the original question-and-answer pair.

5. The formula-based numerical reasoning question-answering method according to claim 1, characterized in that: The training of the formula generation model and the parameter recognition model is as follows: the formula generation model and the parameter recognition model are large language models based on a decoder architecture. For the formula generation model, prompt words are constructed to describe the formula generation task that the model needs to perform. The question is used as the input of the task in the prompt words, and the output of the task is the formula information. The formula information in the formula dataset is used as the supervision label of the model output, and supervised learning is performed. For the parameter recognition model, prompt words are constructed to describe the parameter recognition task that the model needs to perform. The question and parameter name are used as inputs to the task in the prompt words, and the output of the task is parameter information. The parameter information of the parameter dataset is used as the supervision label of the model output, and supervised learning is performed.

6. The formula-based numerical reasoning question-answering method according to claim 1, characterized in that: Use Numbat as a formula calculator.

7. An electronic device comprising a processor and a memory, characterized in that: The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the formula-based numerical reasoning question-answering method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the formula-based numerical reasoning question-answering method according to any one of claims 1-6.

Images