A mathematical application question automatic answering method based on large model knowledge distillation

Abstract

The application relates to a mathematical application question automatic answering method based on large model knowledge distillation, which comprises the following steps: inputting a to-be-answered mathematical application question into a large model-based knowledge distillation model to generate a mathematical expression as a solution program sequence, wherein the large model-based knowledge distillation model comprises a teacher model and a student model, the teacher model is constructed based on a large language model (LLM), and the student model is obtained by training data generated by the teacher model. The application uses a large language model as a teacher model, first designs to generate various mathematical application questions to practically enhance the student model, and second dynamically evaluates the learning state of the student model to generate various mathematical application question variants, improves the understanding ability of the student model to mathematical application question semantics and scenes, solves the problem of poor solving ability of a model with few parameters, and enables the student model to obtain mathematical application question solving ability close to a large language model with few parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for automatically solving mathematical word problems based on large-model knowledge distillation. Background Art

[0002] Solving math word problems (MWPs) lies at the intersection of multiple natural language processing techniques, involving currently popular intelligent technologies such as machine reading comprehension, knowledge question answering, and human-computer dialogue. It is not only a core task in mathematical reasoning but also a key benchmark for machine intelligence, and has long attracted widespread attention from researchers. With the emergence of PLMs and LLMs, the knowledge distillation paradigm has become popular in natural language processing for better application in downstream tasks. The fundamental idea behind this approach is to extract task-specific models with smaller parameters from large language models. Due to the relatively small size of the MWP dataset, large general-purpose networks are fine-tuned for downstream processing of word problem solving. Knowledge distillation enables the model to concentrate the knowledge of a general model into a smaller, more focused model. When performing knowledge distillation based on LLMs, most previous studies have emphasized the use of CoT methods. CoT uses a large model as a teacher model to generate distilled knowledge for the reasoning steps, and fine-tunes a smaller student model. While these methods have successfully improved the performance of the student model, they also exhibit certain limitations.

[0003] At present, the research on MWP solving based on LLMs still has certain defects: First, LLMs have hundreds of billions of parameters and cannot be deployed on a large scale. Some require API availability to achieve reproducibility. Targeted fine-tuning of downstream tasks also requires a lot of time and computing resources. These further limit the promotion and use of LLMs in professional fields. Secondly, in knowledge distillation based on LLMs, small student models find it difficult to understand the reasoning steps of CoT, which affects their learning efficiency. Research evidence shows that only extremely large language models have the ability to execute CoT during the reasoning process. Therefore, many student models trained on CoT reasoning steps do not achieve satisfactory accuracy in MWP solving tasks. Finally, existing knowledge distillation methods based on LLMs lack feedback from student models to LLMs teacher models, ignore the evaluation of the status of student knowledge learning, and further affect the performance of student models. In order to solve the above problems, the present invention proposes a method for automatically solving mathematical word problems based on large-model knowledge distillation. Summary of the Invention

[0004] The purpose of this invention is to provide a method for automatically solving mathematical word problems based on large-scale model knowledge distillation, so that the student model can obtain mathematical word problem-solving capabilities close to those of a large language model with very few parameters.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for automatically solving mathematical word problems based on large-scale model knowledge distillation, including:

[0007] Get math word problems to be solved;

[0008] The mathematical word problems to be solved are input into the knowledge distillation model based on the large model to generate mathematical expressions as a sequence of equation solutions, wherein the knowledge distillation model based on the large model includes a teacher model and a student model, the teacher model is constructed based on the large language model LLMs, and the student model is obtained by training with data generated by the teacher model.

[0009] Optionally, the teacher model includes a diversity sub-question generator, a diversity analogy question generator, and a data filter;

[0010] The diversity sub-question generator is used to input the initial source data set to generate multiple questions with the same semantic description and sub-equations and corresponding answers;

[0011] The diversity analogy problem generator is used to generate mathematical word problems with the same problem but different semantic descriptions from the initial source data set input;

[0012] The data filter is used to screen and filter the questions generated by the diversity sub-question generator and the diversity analogy question generator to obtain a training data set.

[0013] Optionally, screening and filtering the questions generated by the diversity sub-question generator and the diversity analogy question generator include: filtering questions with incorrect formats, completing incomplete questions based on the zero-sample prompt technology of LLMs, eliminating questions that cannot be answered and have incorrect formats, and obtaining the training data set.

[0014] Optionally, training the student model using data generated by the teacher model includes:

[0015] S1. After training the student model a preset number of times using the training dataset, evaluating the learning status of the student model after the preset number of training times using an evaluation dataset to obtain unresolved problems, wherein the evaluation dataset is selected from data generated by the teacher model;

[0016] S2. Calling the diversity sub-problem generator and the diversity analogy problem generator in the teacher model to generate targeted data for the unsolvable problem to obtain an enhanced data set;

[0017] S3. Merge the enhanced dataset into the training dataset, and return to S1 until a preset stopping condition is reached.

[0018] Optionally, the student model includes a model based on an encoder-decoder architecture and a model based on a decoder architecture.

[0019] Optionally, the model based on the encoder-decoder architecture includes: using GTS as a decoder, the backbone network LSTM, RoBERTa-base and RoBERTa-large as encoders, wherein the encoder is used to model the input sequence and return the hidden state of the input mathematical problem, and the decoder is used to generate a distribution of solution equations based on the output features of the encoder.

[0020] Optionally, the model based on the decoder architecture is a large language model with a decoder architecture, wherein the large language model with the decoder architecture is fine-tuned using a re-parameterization method during training.

[0021] The beneficial effects of the present invention are as follows: (1) Targeted Knowledge Distillation (TKD) uses LLMs as the role of teacher models and designs a generation module for generating diversified MWP variants. It mainly generates two types of variants from the source MWP: sub-problems with unchanged text descriptions but diversified problems, and similar problems with unchanged problem descriptions but diversified text descriptions and scenarios. At the same time, a problem filter is designed to clean and filter the generated problems to obtain high-quality MWPs. TKD does not require the CoT capability of the small model, so that the small model can learn more effectively. (2) TKD designs an evaluation module for evaluating the learning status of the model, uses the generated comprehensive and diverse problems to evaluate the student model, finds the weak points of the solution-solving ability based on the learning status of the student model, and feeds back to LLMs for targeted dynamic generation. TKD dynamically checks and fills in the gaps in the student model's weaknesses during learning, further improving the learning efficiency and solution-solving ability of the student model. (3) Compared with existing work, TKD is based on the GPT-4 API application. The student model design is divided into two categories. One category uses GTS as the decoder and three different backbone networks LSTM, RoBERTa-base and RoERTa-large as the encoder. The other category uses large models with few parameters, Llama2-7b and Qwen2.5-7b. Experimental results show that TKD not only significantly outperforms the fine-tuning baseline in accuracy while reducing the actual application cost, but also achieves solving capabilities comparable to LLMs with a student model using only a very small number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a framework diagram of a large-model-based knowledge distillation model according to an embodiment of the present invention;

[0024] Figure 2 A comparison chart of ablation experiments on the MAWPS dataset according to an embodiment of the present invention;

[0025] Figure 3 A comparison chart of ablation experiments on the SVAMP dataset according to an embodiment of the present invention;

[0026] Figure 4 This is a comparison chart of the ablation experiment on the MathQA dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] This embodiment provides a method for automatically solving mathematical word problems based on large-scale model knowledge distillation, including:

[0030] The mathematical word problems to be solved are input into the knowledge distillation model based on the large model to generate mathematical expressions as a sequence of equation solutions. The knowledge distillation model based on the large model includes a teacher model and a student model. The teacher model is built based on the large language model LLMs, and the student model is trained using data generated by the teacher model.

[0031] Furthermore, the teacher model includes a diversity sub-question generator, a diversity analogy question generator, and a data filter;

[0032] A diverse sub-question generator, which is used to generate multiple questions and corresponding answers with the same semantic description and sub-equations from the initial source dataset input;

[0033] A diverse analogy problem generator, which uses the initial source dataset as input to generate mathematical word problems with the same problem but different semantic descriptions;

[0034] The data filter is used to screen and filter the questions generated by the diversity sub-question generator and the diversity analogy question generator to obtain the training data set.

[0035] Specifically, this embodiment combines the characteristics of LLM, designs two question generators for data generation and one data filter for data filtering, and uses MWP source Represents the initial source data set as the initial input. Before starting training, source Generate diverse data.

[0036] In the diverse sub-problem generator G1, MWPs with the same semantic description and different questions will be generated. To verify the model's situational understanding ability, when the model understands the context of the mathematical problem and its underlying logic, it should be able to solve the perceptual question generator in that context and generate corresponding sub-problems. This study is based on BERT's decomposition of semantics and equations to generate diverse sub-equations and sub-problems. Now, due to the powerful functionality of LLMs, this embodiment can generate multiple questions and corresponding answers with the same semantic description and sub-equations through a small number of examples and prompts.

[0037] In the diversity analogy question generator G2, MWPs with the same questions and different semantic descriptions will be generated. In order to verify the model's understanding of the problem, MathCheck proposed a rewriting variant: converting the original question into a new question, using different wording or different sentence structure, but without changing the mathematical logic of the original version. It focuses on semantic robustness and aims to test whether the model can correctly reason when dealing with different descriptions of the same mathematical logic. Naturally, this type of rewriting variant is used for model training, and it can also enhance the model's further learning and understanding of semantics. Similarly, this embodiment uses LLMs to generate multiple analogy MWPs with the same questions and different semantic descriptions as the source problem through simple prompt design and zero-sample implementation.

[0038] This embodiment designs a data filter to screen the problems generated by the above two generators. Filtering out harmful enhanced data can improve data quality and the performance of downstream solvers.

[0039] Furthermore, the questions generated by the diversity sub-question generator and the diversity analogy question generator are screened and filtered, including: filtering out incorrectly formatted questions, completing incomplete questions based on the zero-sample prompt technology of LLMs, and eliminating unanswerable and incorrectly formatted questions to obtain a training dataset.

[0040] Furthermore, training the student model using data generated by the teacher model includes:

[0041] S1. After training the student model a preset number of times using the training dataset, the student model is evaluated using an evaluation dataset to identify unresolved problems, wherein the evaluation dataset is selected from data generated by the teacher model;

[0042] S2. Call the diverse sub-problem generator and diverse analogy problem generator in the teacher model to generate targeted data for unsolvable problems and obtain an enhanced dataset;

[0043] S3. Merge the enhanced dataset into the training dataset and return to S1 until the preset stopping condition is reached.

[0044] Specifically, this embodiment uses MWP train Represents the portion of the dataset used to train the student model, using MWP val Represents the portion of the dataset used to evaluate the learning status of the student model. A certain proportion of data is randomly extracted from the data generated by the LLMs-based data generation module as the MWP val Follow-up evaluation is conducted and the remaining part is used as MWP train At the beginning of training, this embodiment uses S to represent the student model solver in the framework, and makes the student model S in MWP train After learning a certain amount of knowledge, that is, training a certain number of times, the learning status of S is evaluated, and the evaluation data set MWP generated by S is used to evaluate the learning status of S. val The performance of the above problem pairs (X, Y) that cannot be solved by S reflects the insufficiency of S in learning knowledge during training. The data generation module G based on LLMs is called to generate targeted data. The generated (X, Y) constitutes an enhanced data set. In this embodiment, MWP is used. aug After each round of targeted data enhancement, MWP aug Merged into MWP train In the training process, every time S learns enough knowledge, it will be evaluated and enhanced with targeted data, and then added to the subsequent training.

[0045] Furthermore, the student model includes a model based on an encoder-decoder architecture and a model based on a decoder architecture.

[0046] Furthermore, the model based on the encoder-decoder architecture includes: using GTS (Goal-driven TreeStructured) as the decoder, the backbone network LSTM, RoBERTa-base and RoBERTa-large as encoders, where the encoder is used to model the input sequence and return the hidden state of the input mathematical problem, and the decoder is used to generate the distribution of the solution equation based on the output features of the encoder.

[0047] Furthermore, the model based on the decoder architecture is a large language model with a decoder architecture, wherein the large language model with the decoder architecture is fine-tuned using a re-parameterization method during training.

[0048] Specifically, the models based on the decoder architecture in this embodiment are Llama2-7b and Qwen2.5-7b which only have a decoder architecture.

[0049] The method of this embodiment is further described below:

[0050] This embodiment assumes that the numbers in problem X and expression Y use {N0, N1, ..., N j} means, where N i Refers to the i-th number, j is the maximum value of the number in X and Y. The numbers in math problems are theoretically impossible to be exhaustive, so the general quantifier V num The mapping function constructed by arithmetic expression preprocessing stores all the numbers that appear in X and maps them to the digital token list X according to the order of the numbers in X. q ={N0, N1, ..., N j}. When the system generates expression Y, if the specified number N is generated i , the corresponding number will be automatically taken out from the mapping function. The advantages of this definition are: (1) It is applicable to the data preprocessing technology widely used in MWP solution, which can unify the representation of quantity and reduce the vocabulary. (2) In the data generation process, since the goal of the generator is to generate MWP variants, the number mapping can prevent the generation of variants that only change the value of the quantity.

[0051] The MWP solving task is evaluated based on the accuracy of the model's predictions, divided into two criteria: answer accuracy and expression accuracy. Answer accuracy indicates that the generated expression is considered correct if the predicted calculated value is equal to the answer. Expression accuracy measures whether the generated expression tree matches the target expression tree. There is a fundamental difference between the two: with expression accuracy, the answer is considered correct only if the generated expression is correct, and any difference from the target expression is considered incorrect. For example, generating "N0+N1-N2" and "(N0+N1)-N2" may yield the same answer, but the expression is different. In this case, the answer accuracy is true, while the expression accuracy is false.

[0052] This embodiment proposes a large-model-based targeted knowledge distillation framework TKD. The entire model can be divided into three modules: a diversity data generation module based on LLMs, a student model learning status evaluation and targeted generation module, and a teacher model and student model selection module based on LLMs. Among them, the diversity data generation module based on LLMs generates a variety of mathematical word problem variants based on the input original mathematical word problems, forming a data set for data enhancement and evaluation, and further improving the student model's problem-solving ability; the student model learning status evaluation and targeted generation module mainly uses the evaluation data set generated by the diversity data generation module based on LLMs to comprehensively evaluate the student model that has learned certain knowledge, and feedback the student model's learning status to achieve further targeted data generation; the teacher model and student model module based on LLMs uses LLMs as a teacher role to train suitable student models. Figure 1 This is the TKD model architecture. Each module will be described in detail below.

[0053] 1. Data generation module based on LLMs:

[0054] In recent research, a new paradigm for mathematical reasoning evaluation, MathCheck, was proposed. It requires models to understand semantics, resist interference, and understand scenarios. It uses four question forms (including the original question and its three rewritten variants) to check the model's reasoning robustness. The question variants it proposes are comprehensive and diverse. It is not only a new evaluation paradigm for MWP, but can also be used for model training. Inspired by the above work, this embodiment combines the characteristics of LLM to design two question generators for data generation and a data filter for data filtering.

[0055] This example uses MWP source Represents the initial source data set as the initial input. Before starting training, source Generate diverse data.

[0056] In the diversity sub-problem generator G1, this embodiment will generate MWPs with the same semantic description and different questions. In order to verify the model's situational understanding ability, when the model understands the scenario of a mathematical problem and its underlying logic, they should be able to solve other problems in that scenario, and can generate rewritten variants that change their problems. In previous work, for example, the diversified equation generator represents the source MWP as (text description, problem, equation), and the original equation is decomposed into multiple sub-equations through the diversity equation generator. Each generated sub-equation and original text description are input into the equation-aware problem generator to generate the corresponding sub-problem. This study decomposes semantics and equations based on BERT to generate diverse sub-equations and sub-problems. Now, due to the powerful functionality of LLMs, this embodiment is based on the few-sample prompt technology of LLMs, and is designed to use variant problems derived from three source problems and examples of corresponding answers as samples, combined with "generate variants for mathematical word problems, the answers to the variants should be correct and different from the source problems, but have exactly the same format as the source problems, containing not only operands but also operators, but not the modulo operator, and using N0, N1,... to represent numbers, without other numerical representations." As a prompt, a variety of variant problems and corresponding answers with the same semantic description but different sub-equations can be generated.

[0057] In the diversity analogy problem generator G2, this embodiment will generate MWPs with the same problem and different semantic descriptions. In order to verify the model's understanding of the problem, MathCheck proposes a rewriting variant: converting the original problem into a new problem, using different wording or different sentence structure, but without changing the mathematical logic of its original version. It focuses on semantic robustness and aims to test whether the model can correctly reason when dealing with different descriptions of the same mathematical logic. Naturally, this type of rewriting variant is used for model training, and can also enhance the model's further learning and understanding of semantics. Similarly, this embodiment is based on the zero-sample prompt technology of LLMs, and is designed to "generate a mathematical word problem and its answer, the format of which is exactly the same as the previous one, without explanation, using N0, N1,... to represent numbers, instead of other numerical representations." As a prompt, multiple analogous mathematical word problems and corresponding answers with the same problems and equations as the source problem but different semantic descriptions can be generated without giving samples.

[0058] This embodiment designs a data filter to screen the questions generated by the above two generators. Filtering out harmful augmented data can improve data quality and the performance of downstream solvers. Due to the existence of large model hallucinations, the data generated by LLMs contains a variety of problems. For example, the generated questions are incomplete, the second half is missing, or the generated questions are expressed in an inaccurate format, etc., which are difficulties that need to be solved. This embodiment first performs a format check and uses regular expressions to filter out questions with incorrect formats. Secondly, it proofreads the answers based on the zero-sample prompt technology of LLMs and designs "Generate variant questions and correct answers without explanation. Use N0, N1,... to represent numbers. No other numbers are required. Follow the same format as the following format" as a prompt. For the questions generated by the above two generators, based on LLMs, the incomplete questions are completed and the questions that cannot be answered and have incorrect formats are eliminated, and finally a high-quality training data set is obtained.

[0059] Compared with the previous use of manual human annotation or ordinary models, the diverse sub-problem generator and diverse analogy problem generator designed in this embodiment realize the automation and enrichment of problem variants and ensure high quality, thereby improving the upper limit of the solver.

[0060] 2. Status assessment and targeted generation module:

[0061] Evaluation requires diverse data to reflect the solver's learning state. A solver may memorize the MWPs in the training set rather than fully understand them, so the training set cannot be used directly to validate the student solver. Therefore, this embodiment randomly selects a certain percentage of data from the high-quality dataset generated by the data generation module to form an evaluation dataset, which is used to evaluate the student model's solving ability.

[0062] During the student model training, this embodiment is designed to use the evaluation data set to evaluate the learning status of the student model after the student model has learned certain knowledge. The problems that the student model cannot solve reflect that there is still a gap between the student model and the ideal state, and the data generation module based on LLMs is called to generate targeted enhanced data for the weak points of the student model, and the enhanced data is supplemented to the next round of training. During the student model training, multiple evaluations and multiple targeted data enhancements can be performed. The algorithm pseudo code is shown in Algorithm 1. This embodiment uses MWP train Represents the portion of the dataset used to train the student model, using MWP val Represents the portion of the dataset used to evaluate the learning status of the student model. A certain proportion of data is randomly extracted from the data generated by the LLMs-based data generation module as the MWP val Follow-up evaluation is conducted and the remaining part is used as MWP trainAt the beginning of training, this embodiment uses S to represent the student model solver in the framework, and makes the student model S in MWP train After learning a certain amount of knowledge, that is, training a certain number of times, the learning status of S is evaluated, and the evaluation data set MWP generated by S is used to evaluate the learning status of S. val The performance of the above problem pairs (X, Y) that cannot be solved by S reflects the insufficiency of S in learning knowledge during training. The data generation module G based on LLMs is called to generate targeted data. The generated (X, Y) constitutes an enhanced data set. In this embodiment, MWP is used. aug After each round of targeted data enhancement, MWP aug Merged into MWP train This helps S in the next round of training. During training, each time S acquires a sufficient amount of knowledge, it is evaluated and targeted with data augmentation, and then incorporated into subsequent training. This module can be iterated multiple times during S training. Therefore, TKD is a dynamic evaluation and generation process, characterized by a loop and continuous iteration until data generation is accurate and the specified number of times has been achieved. The evaluation and targeted generation algorithm is shown in Table 1.

[0063] Table 1

[0064]

[0065]

[0066] The state evaluation and targeted generation modules proposed in this embodiment gradually increase the size of the training set, cover more and broader knowledge, and specifically make up for the shortcomings of the student model, thereby producing a more powerful solver.

[0067] 3. Student Model Based on LLMs:

[0068] The TKD framework proposed in this example, based on the concept of knowledge distillation, requires the selection and design of an appropriate student model for training. Based on GPT-4 API calls, this example selects solvers with varying parameter sizes as student models, ranging from 20M to 7B, fully demonstrating the generalizability of the TKD framework. The design of student models falls into two categories, which are described in detail below.

[0069] A small model with 3.1 million parameters:

[0070] A small model with few parameters was used, following an encoder-decoder architecture. GTS was selected as the decoder, and three different backbone networks, LSTM, RoBERTa-base, and RoBERTa-large, were used for the encoder. The original encoder-decoder model was pre-trained on the MWP dataset. Given a math problem X, the student model first uses the encoder to model the input sequence and returns the hidden state of the input math problem, represented as:

[0071] h=Encoder(X) (1)

[0072] Where h is the output hidden state of the encoder, Encoder∈{LSTM, RoBERTa-base, RoERTa-large} is one of the existing text encoder models, and the output features of the encoder are input into the tree-based decoder to generate the distribution of the solution equation, which is expressed as:

[0073] P(Y|X)=TreeDecoder(h) (2)

[0074] Large language model with 3.2 billion parameters:

[0075] LLMs with fewer parameters are used, including Llama2-7b and Qwen2.5-7b, which only have decoder architectures. When training Llama2-7b on the MWP dataset, due to the large number of model parameters, this example uses the parameter-efficient fine-tuning (PEFT) method for fine-tuning. This example uses the reparameterization-based LoRA method.

[0076] Reparameterization-based methods aim to transform network weights using low-rank techniques. This approach effectively reduces the number of trainable parameters while retaining the ability to handle high-dimensional matrices. LoRA introduces a simple method to update the parameters of the weight matrix by decomposing it into the product of two low-rank matrices. It can be expressed as:

[0077] H0=H i W0+H i ΔW=H i W0+H i BA (3)

[0078] Among them H i ∈R t×d and H0∈R t×d are the input and output of a specific layer, such as the attention layer or the fully connected layer. d×d Can be any pre-trained weight matrix, including weights in MLP or Attention layers. B∈R r×d and A∈R r×dIt is a low - rank matrix used to cover the pre - trained parameter ΔW. r << d is an important hyperparameter of LoRA. During training, the original dense weight matrix of the pre - trained model is frozen, and additional trainable low - rank matrices can be introduced in each layer of the transformer architecture to represent the part of the weight change. This method retains the general language features captured by the pre - trained model and, due to the use of low - rank matrices, greatly reduces the storage and computational requirements, thus enabling lightweight fine - tuning of Llama2 - 7b and Qwen2.5 - 7b models.

[0079] Experiments:

[0080] 1. Datasets:

[0081] In this embodiment, the performance of TKD is evaluated on three English MWP datasets: MAWPS, SVAMP, and MathQA. The dataset statistics are shown in Table 2.

[0082] Table 2

[0083]

[0084] (1) MAWPS: MAWPS is a commonly used dataset with 2,373 arithmetic application problems. In this embodiment, the sub - dataset MAWPS - S provided by the paper is used. This dataset converts the equation - form answers into arithmetic expressions and removes invalid equation groups, resulting in a total of 1,987 problems. In this embodiment, a 5 - fold cross - validation experiment is conducted on this dataset.

[0085] (2) SVAMP: SVAMP is a dataset sampled from existing datasets and manually modified. Some types of problems in the original data are carefully fine - tuned to test the robustness of the model and its sensitivity to questions. Since this dataset has no test split, in this embodiment, the experimental settings of the original paper are followed, with a total of 4,138 problems.

[0086] (3) MathQA: MathQA is a comprehensive dataset that can be regarded as both a dataset of math application problems and a multi - step reasoning dataset. In the original MathQA dataset, a certain number of instances with annotated equations cannot obtain correct numerical answers. In this embodiment, the experimental settings of Tan et al. are followed and processed to remove the problems with incorrect answers, resulting in a total of 20,207 problems.

[0087] 2. Baseline models:

[0088] To evaluate the effectiveness of the TKD method, this embodiment selects three types of advanced models with different numbers of parameters as baseline models for comparison. The baseline models considered are as follows:

[0089] The first category is large language models with hundreds of billions of parameters. For this type of model, the best LLMs are selected for comparison with the best open-source LLMs. This example compares the closed-source GPT4 and the open-source PaLM, which perform best in multiple tests, and their performance on the CoT method. The PaLM has 540 bytes of parameters.

[0090] The second type is a large language model with billions of parameters. This type of model selects LLMs with fewer parameters among LLMs with large parameters. In this embodiment, Llama2-7b and Qwen-7b are selected, with a parameter count of 7B.

[0091] The third category is small models with millions of parameters. These models are based on deep learning and PLMs, approaches that predate the emergence of LLMs. They improve task accuracy through the complex structure of neural networks and enhanced representations using PLMs. This example uses GTS as the decoder and LSTM, RoBERTa-base, and RoBERTa-large as the encoders, with parameters of 20M, 144M, and 377M, respectively.

[0092] 3. Experimental setup:

[0093] In this work, all experiments were performed on an NVIDIA RTX A6000 graphics card and implemented in Python using the PyTorch framework. For small models with millions of parameters, this embodiment uses the AdamW optimizer with 50 epochs, a batch size of 16, an initial learning rate of 8e-6, and a halving every 20 epochs. The dropout during training is 0.01. Evaluation and targeted generation methods will be performed at the 10th and 30th epochs. For large language models with billions of parameters, this embodiment uses the LoRA parameter r of 16, lora_alpha of 32, dropout of 0.1, an initial learning rate of 1e-4, and epochs of 3. Evaluation and targeted generation methods will be performed at the second epoch. The LLMs data generator and filter call the GPT-4 API. The evaluation metric used in this embodiment is answer accuracy.

[0094] Experimental analysis:

[0095] Table 2 shows the experimental results of the TKD method proposed in this example on the MAWPS, SVAMP, and MathQA datasets. Llama2-7b and Qwen2.5-7b are the results obtained using open-source code with the same fine-tuning parameter settings. GPT-4 on the MathQA dataset is the result obtained using the API test in this example. The results of other methods are reported in the original paper. The following analysis is performed based on the results in the table:

[0096] (1) Compared with small models with millions of parameters, the TKD method proposed in this embodiment achieved better results on all datasets. Compared with "LSTM+GTS", the "LSTM+GTS+TKD" proposed in this embodiment improved by 8.1% (90.7% vs. 82.6%), 11.6% (42.4% vs. 30.8%), and 7.6% (78.9% vs. 71.3%) on the three datasets respectively; compared with "RoBERTa-base+GTS+TKD" proposed in this embodiment, the "RoBERTa-base+GTS+TKD" proposed in this embodiment improved by 4.1% (92.6% vs. 88.5%), 20.5% (61.5% vs. 41.0%), and 6.0% (80.1% vs. 74.1%) on the three datasets respectively;

[0097] Compared with "RoBERTa-large+GTS", the "RoBERTa-large+GTS+TKD" proposed in this example improves by 2.9% (93.3% vs. 90.4%), 21.3% (70.8% vs. 49.5%), and 4.8% (81.7% vs. 76.9%) on the three datasets, respectively. This demonstrates the effectiveness of the method proposed in this example.

[0098] (2) Compared with the large language models Llama2-7b and Qwen2.5-7b with billions of parameters, the TKD method proposed in this embodiment achieved better results on all datasets. Compared with "Llama2-7b", the "Llama2-7b+TKD" proposed in this embodiment improved by 11.3% (70.1% vs. 58.8%), 4.7% (51.1% vs. 46.4%) and 5.3% (27.9% vs. 22.6%) on the three datasets; compared with "Qwen2.5-7b", the "Qwen2.5-7b+TKD" proposed in this embodiment improved by 2.2% (95.4% vs. 93.2%), 2.6% (76.6% vs. 74.0%) and 3.9% (61.6% vs. 57.7%) on the three datasets. This further proves the effectiveness of the method proposed in this embodiment.

[0099] (3) Compared with the large language models GPT-4 and PaLM with hundreds of billions of parameters, the TKD method proposed in this embodiment achieves superior or similar results on all data sets with fewer parameters. On the MAWPS dataset, the "Qwen2.5-7b+TKD" proposed in this embodiment uses 7B parameters, which is 2.1% higher than the PaLM+CoT with 540B parameters (95.4% vs. 93.2%), and is close to the result of GPT-4; on the SVAMP dataset, the "RoBERTa-large+GTS+TKD" and "Qwen2.5-7b+TKD" proposed in this embodiment use 377M and 7B parameters, which are 1.4% higher (70.8% vs. 69.4%) and 7.2% higher (76.6% vs. 69.4%) respectively compared with the PaLM with 540B parameters; on the MathQA dataset, the "RoBERTa-large+GTS+TKD" proposed in this embodiment uses 377M parameters, which is 0.8% higher (81.7% vs. 80.9%) than GPT-4. The effectiveness of the method proposed in this embodiment is further illustrated.

[0100] The experimental results on the MAWPS, SVAMP, and MathQA datasets are shown in Table 3. The results in Table 3 and the above analysis indicate that the TKD method generates diverse data based on LLMs, and this diverse data improves the solving performance of the student model. Furthermore, the evaluation and targeted generation modules of the TKD method not only provide feedback on the learning status of the solver but also generate further targeted data, resulting in a more powerful student model.

[0101] Table 3

[0102]

[0103] Ablation experiment:

[0104] The model structure of this embodiment mainly includes two modules: a module for generating diverse data based on LLMs and a module for evaluating the learning status of the student model and generating targeted data. To verify the effectiveness of each module, this embodiment designed two types of ablation experiments:

[0105] (1) To study the effectiveness of the data generation module: A represents the removal of the LLMs-based diversity data generation module.

[0106] (2) To study the effectiveness of the evaluation and targeted generation modules: B represents the removal of the evaluation and targeted generation methods.

[0107] The experimental results of the ablation experiment are as follows Figure 2 、 Figure 3 、 Figure 4 As shown in the figure, the following inferences can be drawn from the results:

[0108] (1) TKD outperforms A, which indicates that LLMs-based diverse data generation can effectively generate comprehensive and diverse data, improving the solving ability of the student model;

[0109] (2) TKD is better than B. It can be inferred that evaluating the student model in training can provide feedback on the learning status of the student model, thereby further generating targeted data and improving the solving ability of the student model.

[0110] Case Study:

[0111] In addition, this embodiment further selects several examples to illustrate the superiority of the TKD method. The problem generation examples are shown in Table 4. For the selected source problems, the LLMs-based diversity data generation module in the TKD method generates data and obtains two types of data.

[0112] (1) Compared with the source questions, the generated diverse sub-question types show that LLMs successfully generate questions and answers with the same semantic description but different questions;

[0113] (2) Compared with the source questions, the generated diverse analogy question types show that LLMs successfully generate questions and answers with the same problem but different descriptions.

[0114] Table 4

[0115]

[0116]

[0117] In order to obtain high-quality data, this embodiment further filters the generated questions and answers.

[0118] The data filtering example is shown in Table 5. The answer format obtained by the generation module is incomplete and wrong, while the answer format after passing the data filter is complete and correct.

[0119] Table 5

[0120]

[0121] This embodiment aims at solving mathematical word problems. The previous knowledge distillation framework is limited to the model structure, cannot effectively improve the performance of the student model, and lacks learning feedback for the student model. Therefore, a method for solving mathematical word problems based on targeted knowledge distillation of a large language model is proposed. The TKD method proposed in this embodiment performs data enhancement in the diversity data generation module based on LLMs. The generated diversity data is effectively filtered, which can further improve the solving performance of the student model. At the same time, in order to better feedback the learning status of the student model, a variety of evaluation data sets are introduced in the status evaluation and targeted generation module to evaluate the learning status of the student model, and then generate data in a targeted manner to supplement the deficiencies in the student model learning. The experimental results on the MAWPS, SVAMP and MathQA datasets show the effectiveness of the method proposed in this embodiment.

[0122] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for automatically solving mathematical word problems based on large-scale model knowledge distillation, characterized in that: include: Get math word problems to be solved; Inputting the mathematical word problem to be solved into a knowledge distillation model based on a large model to generate a mathematical expression as a sequence of equations to be solved, wherein the knowledge distillation model based on the large model includes a teacher model and a student model, the teacher model is constructed based on a large language model (LLMs), and the student model is trained using data generated by the teacher model; The teacher model includes a diversity sub-question generator, a diversity analogy question generator, and a data filter; The diversity sub-question generator is used to input the initial source data set to generate multiple questions with the same semantic description and sub-equations and corresponding answers; The diversity analogy problem generator is used to input the initial source data set to generate mathematical word problems with the same problem but different semantic descriptions; The data filter is used to screen and filter the questions generated by the diversity sub-question generator and the diversity analogy question generator to obtain a training data set.

2. The method for automatically solving mathematical word problems based on large-scale knowledge distillation according to claim 1 is characterized in that: Screening and filtering the questions generated by the diversity sub-question generator and the diversity analogy question generator include: filtering questions with incorrect formats, and using LLMs-based zero-sample prompt technology to complete incomplete questions, eliminating questions that cannot be answered and have incorrect formats, and obtaining the training data set.

3. The method for automatically solving mathematical word problems based on large-scale knowledge distillation according to claim 1 is characterized in that: Training the student model using data generated by the teacher model includes: S1. After training the student model a preset number of times using the training dataset, evaluating the learning status of the student model after the preset number of training times using an evaluation dataset to obtain unresolved problems, wherein the evaluation dataset is selected from data generated by the teacher model; S2. Calling the diversity sub-problem generator and the diversity analogy problem generator in the teacher model to generate targeted data for the unsolvable problem to obtain an enhanced data set; S3. Merge the enhanced dataset into the training dataset, and return to S1 until a preset stopping condition is reached.

4. The method for automatically solving mathematical word problems based on large-scale knowledge distillation according to claim 1 is characterized in that: The student model includes a model based on an encoder-decoder architecture and a model based on a decoder architecture.

5. The method for automatically solving mathematical word problems based on large-scale knowledge distillation according to claim 4 is characterized in that: The model based on the encoder-decoder architecture includes: a decoder and an encoder, using GTS as the decoder, and the backbone network LSTM, RoBERTa-base and RoBERTa-large as the encoder, wherein the encoder is used to model the input sequence and return the hidden state of the input mathematical problem, and the decoder is used to generate the distribution of the solution equation according to the output features of the encoder.

6. The method for automatically solving mathematical word problems based on large-scale knowledge distillation according to claim 4 is characterized in that: The decoder architecture-based model is a large language model with a decoder architecture, wherein the large language model with a decoder architecture is fine-tuned using a reparameterization method during training.

Images