A mathematical application question solving system and method based on BERT encoding

Abstract

The application provides a BERT encoding-based mathematical application question solving system and method, and relates to the technical field of application solving. : As the encoder of Chinese questions, BERT is the encoder of English questions; the encoding process first acquires a text sequence Q, and maps the text sequence Q into a vector representation matrix composed of word vectors one by one; the overall representation vector of the text sequence is obtained by averaging each word vector in the vector representation matrix; a tree decoder is used to generate an expression tree; a classifier is used to capture question representations that lead to the generation of incorrect expressions; if the output result of the classifier is 1, it indicates that the expression generated by the decoder corresponds to the question; if the output result is 0, it indicates that the question representation vector generated by the encoder leads to the generation of an incorrect expression and needs to be further optimized; the system is trained, thereby realizing the solving of mathematical application questions.

Description

Technical Field

[0001] This invention relates to the field of application problem-solving technology, and in particular to a mathematical application problem-solving system and method based on BERT encoding. Background Technology

[0002] In recent years, the automatic solution of MathWordPro (MWP) has attracted increasing attention from scholars. This task applies artificial intelligence to mathematics education, representing a practical application of natural language understanding technology in smart education. It is also a complex process of knowledge extraction, representation, reasoning, and computation. MathWordPro integrates descriptive vocabulary with inherent numerical logic. The core of solving it is to extract effective information and numbers from the problem and form a correct knowledge representation to deduce the answer. Example 1 in Table 1 shows an example of MWP. The solution model extracts quantities such as "1250" from the problem, understands the quantitative relationships, infers the operators between the quantities, maps the problem to the expression "x = 1250 - 32 - 30", and then calculates the final answer "1188".

[0003] Table 1: MWP Examples

[0004]

[0005] Various end-to-end neural network problem-solving models have been proposed, capable of solving problems without human intervention. Among them, "Modeling Intra-Relation in Math Word Problems with DifferentFunctional Multi-Head Attentions" considers the unique linguistic features of word problem texts, introducing a multi-attention mechanism in the model's encoding stage to mine different types of text features, and using LSTM as a decoder to generate specific mathematical expressions. "Xie, Z., Sun, S.: A goal-driven tree-structured neural model for mathword problems. In: IJCAI.pp.5299–5305 (2019)" utilizes a bidirectional GRU to encode the problem and proposes a tree-structured decoder, transforming the expression generation method from autoregression to the construction of a mathematical expression tree through a goal-driven approach.

[0006] The approach "Modeling Intra-Relation in Math Word Problems with DifferentFunctional Multi-Head Attentions" divides the problem into a quantity region and a question region using commas or periods. The quantity region contains one or more quantities and their neighboring words, while the question region consists of the sentence containing the question, typically the last sentence of the problem. It utilizes multi-head attention to mine the relationships between the quantities and their relationships with their respective neighboring words and the words in the question region to optimize the problem encoding representation. While the problem representation obtained using this method is richer and more accurate than that obtained using only an RNN-based encoder, the constraint that the model output should be a mathematically compliant expression cannot be guaranteed by using an LSTM as the decoder and generating the mathematical expression through autoregression.

[0007] To address the issue of RNN-based neural networks generating mathematically incorrect expressions as decoders, the paper "Xie, Z., Sun, S.: A goal-driven tree-structured neural model for math word problems. In: IJCAI.pp.5299–5305 (2019)" proposes a top-down, bottom-up approach to generate mathematically compliant expressions. Specifically, it first focuses on the objective quantity to be solved, then captures relevant information within the problem that helps achieve the objective. If the objective can be directly achieved using this information, the solution is complete; otherwise, the objective is decomposed based on the relevant information, and this process is repeated for each sub-objective until all objectives are achieved. While this approach ensures that the generated mathematical expressions strictly conform to mathematical rules, it only focuses on expression generation, neglecting the context of the problem and failing to guarantee the quality of the generated expressions. In fact, to cultivate students' language comprehension and mathematical logic, math word problems are often not intuitively presented yet contain mathematical logic. Taking Example 2 in Table 1 as an example, compared to Example 1, the wording of the problem has changed slightly, but the logic has changed significantly. In this case, the GTS model simply generates an expression directly from the problem statement, lacking an accurate understanding of the numerical logic. It fails to deduce the multiplicative relationship between the quantities "32" and "30," thus generating an incorrect expression. This illustrates that the current model's understanding of the problem is insufficient; it can only grasp the superficial meaning of the problem and cannot deeply explore the mathematical relationships it contains. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a mathematical word problem solving system and method based on BERT encoding.

[0009] A mathematical word problem solving system based on BERT encoding specifically includes: an encoding module, a decoding module, a supervision module, and a fine-tuning module.

[0010] The encoding module is responsible for encoding mathematical word problems, that is, converting mathematical word problems in natural language form into encoded results in vector form;

[0011] The decoding module receives the encoding result output by the encoding module and infers the mathematical expression corresponding to solving each mathematical application problem;

[0012] The supervision module is a regularization term used to verify the correctness of mathematical expressions;

[0013] The fine-tuning module uses the encoding results to predict the number of mathematical symbols in the mathematical expression corresponding to the problem, allowing the encoder to perceive the mathematical logic corresponding to the problem before the solution begins.

[0014] A method for solving mathematical word problems with a supervised module based on BERT encoding, implemented based on the aforementioned system for solving mathematical word problems with a supervised module based on BERT encoding, includes the following steps:

[0015] Step 1: Input a text sequence Q = {q1, q2, ..., q...} of length n into the math word problem solving system. n}, where q i For natural language words or numbers, n is the total number of characters contained in the text sequence Q;

[0016] Step 2: Perform the coding phase using MacBERT : BERT acts as an encoder for Chinese questions and an encoder for English questions. The encoding process first obtains the text sequence Q and then maps the text sequence Q into a vector representation matrix Z∈R composed of individual character vectors. n*h ;

[0017] Z = Encoder(Q) (1)

[0018] Where n is the total number of characters in the problem, and h is the dimension of the implicit vector; R n*h Let be the vector representation matrix of the problem, and Encoder represent the encoding of the mathematical word problem in natural language form using a pre-trained language model as an encoder;

[0019] Step 3: Represent each word vector z in the vector representation matrix Z s The average is used to obtain the overall representation vector of the text sequence Q.

[0020]

[0021] Step 4: Perform the fine-tuning phase, and... The input is fed into a feedforward neural network (FFN) consisting of two fully connected layers (FC) and one ReLU activation layer, and the number of mathematical symbols O in the corresponding mathematical expression is obtained. pre FFN() represents a feedforward neural network:

[0022]

[0023] Step 5: Use a tree decoder to generate an expression tree;

[0024] Step 5.1: During the decoding process, utilize the overall representation vector. The feature vector q of the root node of the expression tree root :

[0025]

[0026] Step 5.2: Using the representation vector q of the current node and the problem representation vector matrix Z, calculate the relationship between the current node and each character z in the problem. s Attention weight a s Then use a s Calculate the context vector of the current node. Then use q and The vector representation e(y) of each mathematical symbol is used to calculate the probability of each mathematical symbol corresponding to the current node, and the maximum value of these probabilities is taken to achieve the representation of the mathematical symbol. Prediction:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] In the formula, [] represents the concatenation operation, and tanh() is the tanh activation function. W a , W s V represents the parameters that the model needs to train. op M is a set of mathematical operators. op M is the word embedding matrix corresponding to the operator. op (y) represents the word embedding vector of the operator y, V con M is a set of constants that may be used to solve mathematical word problems. con M is the word embedding matrix corresponding to a constant. con (y) represents the word embedding vector of the constant y, n p Let loc(y) be the set of all numbers appearing in each problem, where loc(y) represents the position of the number y in the problem, and z... loc(y) This indicates that the vector at position loc(y) is selected from the overall vector matrix representing the problem.

[0035] If the predicted label If it is a number or a constant, then the subtree representation vector t of the subtree rooted at that node is directly obtained from... accomplish;

[0036]

[0037] If the predicted label If it is an operator, then we need to continue predicting the left and right subtrees of that node. The representation vector q of the left child node of that node... left It is based on the predicted label of the current node. Context vector And the vector q is calculated using the following formula, after obtaining q left Then, using the previous prediction formula, the mathematical symbol corresponding to the left child node of this node is predicted.

[0038]

[0039]

[0040]

[0041] g l =σ(W gl ·h l )

[0042] Q le =tanh(W le ·h l )

[0043]

[0044] In the formula, [] denotes concatenation operation, and · denotes matrix multiplication. σ is the sigmoid activation function, tanh() is the tanh activation function, and W ol W cl W gl W le These are the training parameters.

[0045] Step 5.3: Continue constructing the expression tree rooted at the left child node according to the preorder traversal order until the mathematical symbol corresponding to the left child node of a certain node q is a number or a constant. Then, construct the right subtree of node q. First, use the following formula to obtain the target vector q of the right child of node q. right Then use q right Predict the mathematical symbol corresponding to the right child node.

[0046]

[0047]

[0048]

[0049] g r =σ(W gr ·[h r ,t left ])

[0050] Q re =tanh(W re ·[h r ,t left ])

[0051]

[0052] If the prediction result If the operator is q, then the target decomposition of node q needs to continue until the prediction results of the left and right child nodes of a certain node u are both numbers or constants. At this point, the left subtree of node u can be used to represent the vector u. left The right subtree of node u represents vector u. right Mathematical symbols corresponding to node u Begin constructing the subtree representation vector t of the subtree rooted at u. u And build upwards layer by layer until the root node of the entire tree is reached.

[0053]

[0054]

[0055]

[0056]

[0057] In the formula, [] denotes the concatenation operation, σ() is the sigmoid activation function, tanh() is the tanh activation function, and W gu and W cu These are the training parameters.

[0058] Step 6: Use a classifier to capture the question representation that leads to the generation of incorrect expressions. If the classifier output is 1, it means that the expression generated by the decoder corresponds to the question. If the output is 0, it means that the question representation vector generated by the encoder led to the generation of incorrect expressions and needs to be further optimized.

[0059] Step 6.1: Utilize a multi-head attention mechanism to capture the relationships between mathematical symbols in the mathematical expression and obtain the vector representation matrix hA∈R of the mathematical expression. m*h .

[0060] hA=MultiheadAttention(A,A,A)(15)

[0061] Where MultiheadAttention represents the multihead attention mechanism, A is the vector representation matrix of the mathematical expression predicted by the model, m is the length of the mathematical expression predicted by the model, and h is the dimension of the hidden vector.

[0062] Step 6.2: Average hA and then compare it with the overall representation vector of the problem. After concatenation, the data is fed into a classifier (FC). The classification result u (0 or 1) determines whether the mathematical expression predicted by the model is consistent with the problem.

[0063]

[0064]

[0065] [] denotes a concatenation operation. FC is a classifier, which is a feedforward neural network consisting of two fully connected layers and a tanh activation function.

[0066] Step 7: Use the negative sampling algorithm to simultaneously provide the classifier with the true mathematical expression A corresponding to each question. positive and according to A posutive The generated negative example expression A negative ;

[0067] Step 8: Training in solving mathematical word problems;

[0068] Step 8.1: Predict the number of each operator in the mathematical expression corresponding to the math problem: Set the learning objective to minimize the root mean square error loss.

[0069] L Fine-Tuning =MSE(O pre O truth (16)

[0070] Where MSE represents the root mean square error, O pre O represents the number of mathematical symbols in the expression corresponding to the problem predicted by the model. truth It is the actual number of mathematical symbols in the expression;

[0071] Step 8.2: Let the classifier determine whether the question representation and the expression representation match, and set the learning objective to minimize the binary cross-entropy loss:

[0072]

[0073] Where θ represents the network parameters to be optimized in the supervision module;

[0074] Step 8.3: To maximize the probability of generating the corresponding mathematical expression given the problem, the encoder should maximize the consistency between representation Z and answer A. Therefore, the loss function to be minimized is:

[0075]

[0076] Where h represents the network parameters in the encoder and decoder modules, and parameter α adjusts the loss weights during training;

[0077] Step 9: Output a mathematical expression of length m, A = {a1, a2, ..., a...} m}, a i It is one of the following three parts, the first part including the number V that appears in the question text. num The second part is the external auxiliary constant, denoted as V. con The third part is a list of mathematical symbols, denoted as V. op To solve mathematical word problems;

[0078] The beneficial effects of adopting the above technical solution are as follows:

[0079] This invention provides a system and method for solving mathematical word problems based on BERT encoding, which has the following beneficial effects:

[0080] (1) This invention proposes a mathematical word problem solving system with a supervised module based on BERT encoding. On the one hand, in order to confirm the correctness of the expressions generated by the model, we designed a supervised module based on multi-head attention to check whether the generated solution expressions are correct in a binary classification manner: First, we learn the consistency relationship between each problem and its corresponding mathematical expression by concatenating the problem representation vector and the expression representation vector and feeding them into the classifier. Then, in the solution stage, we use the consistency relationship between the problem representation and the equation representation learned by the classifier model to check whether the vector representation of the currently generated solution expression matches the problem vector representation. When a mismatch is found, the model performance is improved by optimizing the problem encoding representation that leads to the generation of incorrect solution expressions.

[0081] (2) This invention leverages the advantages of the pre-trained language model BERT in natural language understanding to learn the relationships between different parts of a question and obtain a contextual representation of the question. Simultaneously, a fine-tuning task is designed to predict the number of operators in the expression corresponding to each question, enabling the encoder to learn the mathematical relationships corresponding to each question, making the question representations with the same operators and the number of each operator more similar. This makes the matching relationship between the question representation vector and the expression representation vector more obvious, and the concatenated feature vector is more representative, thereby reducing the classification difficulty of the supervision module in identifying erroneous expressions generated by the decoding module. Attached Figure Description

[0082] Figure 1 This is a model architecture diagram in an embodiment of the present invention;

[0083] Figure 2 This is a scatter plot representing the problem in an embodiment of the present invention;

[0084] Figure (a) shows the encoding of the problem using the original MacBERT, and Figure (b) shows the encoding of the problem using a fine-tuned MacBERT.

[0085] Figure 3 This is a bar chart showing the accuracy of the model on problems with different expression lengths in the embodiments of the present invention. Detailed Implementation

[0086] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0087] A mathematical word problem solving system based on BERT encoding specifically includes: an encoding module, a decoding module, a supervision module, and a fine-tuning module.

[0088] The encoding module is responsible for encoding mathematical word problems, that is, converting mathematical word problems in natural language form into encoded results in vector form;

[0089] The decoding module receives the encoding result output by the encoding module and infers the mathematical expression corresponding to solving each mathematical application problem;

[0090] The supervision module is a regularization term used to verify the correctness of mathematical expressions;

[0091] The fine-tuning module uses the encoding results to predict the number of mathematical symbols in the mathematical expression corresponding to the problem, allowing the encoder to perceive the mathematical logic corresponding to the problem before the solution begins.

[0092] A method for solving mathematical word problems with a supervised module based on BERT encoding, implemented based on the aforementioned system for solving mathematical word problems with a supervised module based on BERT encoding, includes the following steps:

[0093] Step 1: Input a text sequence Q = {q1, q2, ..., q...} of length n into the math word problem solving system. n}, where q i For natural language words or numbers, n is the total number of characters contained in the text sequence Q;

[0094] Step 2: Perform the coding phase using MacBERT : BERT acts as an encoder for Chinese questions and an encoder for English questions. The encoding process first obtains the text sequence Q and then maps the text sequence Q into a vector representation matrix Z∈R composed of individual character vectors. n*h ;

[0095] Z = Encoder(Q) (1)

[0096] Where n is the total number of characters in the problem, and h is the dimension of the implicit vector; R n*h Let be the vector representation matrix of the problem, and Encoder represent the encoding of the mathematical word problem in natural language form using a pre-trained language model as an encoder;

[0097] Step 3: Represent each word vector z in the vector representation matrix Z s The average is used to obtain the overall representation vector of the text sequence Q. The fine-tuning phase involves predicting the number of mathematical symbols in the expression.

[0098]

[0099] Step 4: Perform the fine-tuning phase, and... The input is fed into a feedforward neural network (FFN) consisting of two fully connected layers (FC) and one ReLU activation layer, and the number of mathematical symbols O in the corresponding mathematical expression is obtained. pre FFN() represents a feedforward neural network:

[0100]

[0101] Introducing an operator prediction task allows the encoder to perceive the mathematical relationships inherent in the problem. This provides a more accurate representation of the problem for both the decoding and supervision modules, thus aiding in the generation of correct expressions and the identification of incorrect ones. To demonstrate our idea, we selected 15 Chinese math problems each with equation templates of “N / N”, “N*N / N”, and “N*N+N”, respectively. Figure 2 The problem involves circles, triangles, and squares. We then encode these problems using the original MacBERT and our fine-tuned MacBERT, and use the T-SNE method to reduce the dimensionality of the problem representation. The specific results are as follows: Figure 2 As shown in (a) and (b) in the figure, after fine-tuning, the encoder can better separate problem representations with different mathematical expression templates, and make problems with similar mathematical expression templates have more similar vector representations, while the representations of unrelated problems are further separated. This allows the decoder to generate solution expressions better during the decoding process, and the supervision module can easily check for mismatched problem representation-equality representation pairs.

[0102] Step 5: Use a tree decoder to generate an expression tree. An example of an expression tree is shown below. Figure 1 As shown in "DecoerModule".

[0103] Step 5.1: During the decoding process, utilize the overall representation vector. The feature vector q of the root node of the expression tree root :

[0104]

[0105] Step 5.2: Using the representation vector q of the current node and the problem representation vector matrix Z, calculate the relationship between the current node and each character z in the problem. s Attention weight a s Then use a s Calculate the context vector of the current node. Then use q and The vector representation e(y) of each mathematical symbol is used to calculate the probability of each mathematical symbol corresponding to the current node, and the maximum value of these probabilities is taken to achieve the representation of the mathematical symbol. Prediction:

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] In the formula, [] represents the concatenation operation, and tanh() is the tanh activation function. W a , W s V represents the parameters that the model needs to train. op M is a set of mathematical operators. op M is the word embedding matrix corresponding to the operator. op (y) represents the word embedding vector of the operator y, where y may be '+' or '-', etc. V con M is a set of constants that may be used to solve mathematical word problems. con M is the word embedding matrix corresponding to a constant. con (y) represents the word embedding vector of the constant y, where y could be "1" or "3.14", etc. p Let loc(y) be the set of all numbers appearing in each problem, where loc(y) represents the position of the number y in the problem, and z... loc(y) This indicates that the vector at position loc(y) is selected from the overall vector matrix representing the problem.

[0114] If the predicted label If it is a number or a constant, then the subtree representation vector t of the subtree rooted at that node is directly obtained from... accomplish.

[0115]

[0116] If the predicted label If it is an operator, then we need to continue predicting the left and right subtrees of that node. The representation vector q of the left child node of that node... left It is based on the predicted label of the current node. Context vector And the vector q is calculated using the following formula, after obtaining q leftThen, using the previous prediction formula, the mathematical symbol corresponding to the left child node of this node is predicted.

[0117]

[0118]

[0119]

[0120] g l =σ(W gl ·h l )

[0121] Q le =tanh(W le ·h l )

[0122]

[0123] In the formula, [] denotes concatenation operation, and · denotes matrix multiplication. σ is the sigmoid activation function, tanh() is the tanh activation function, and W ol W cl W gl W le These are the parameters that the model needs to be trained with.

[0124] Step 5.3: Continue constructing the expression tree rooted at the left child node according to the preorder traversal order until the mathematical symbol corresponding to the left child node of a certain node q is a number or a constant. Then, construct the right subtree of node q. First, use the following formula to obtain the target vector q of the right child of node q. right Then use q right Predict the mathematical symbol corresponding to the right child node. This process also requires utilizing the left subtree representation vector t of node q. left :

[0125]

[0126]

[0127]

[0128] g r =σ(W gr ·[h r ,t left ])

[0129] Q re =tanh(W re ·[h r ,leeft ])

[0130]

[0131] If the prediction result If the operator is q, then the target decomposition of node q needs to continue until the prediction results of the left and right child nodes of a certain node u are both numbers or constants. At this point, the left subtree of node u can be used to represent the vector u. left The right subtree of node u represents vector u. right Mathematical symbols corresponding to node u Begin constructing the subtree representation vector t of the subtree rooted at u. u And build upwards layer by layer until the root node of the entire tree is reached.

[0132]

[0133]

[0134]

[0135]

[0136] In the formula, [] denotes the concatenation operation, σ() is the sigmoid activation function, tanh() is the tanh activation function, and W gu and W cu These are the parameters that the model needs to be trained with.

[0137] Step 6: The supervision module in this invention is as follows Figure 1 As shown in the SupervisionModule, the classifier captures the question representation that leads to the generation of incorrect expressions. If the classifier output is 1, it means that the expression generated by the decoder corresponds to the question. If the output is 0, it means that the question representation vector generated by the encoder leads to the generation of incorrect expressions and needs to be further optimized.

[0138] Step 6.1: Utilize a multi-head attention mechanism to capture the relationships between mathematical symbols in the mathematical expression and obtain the vector representation matrix hA∈R of the mathematical expression. m*h .

[0139] hA=MultiheadAttention(A,A,A)(15)

[0140] Where MultiheadAttention represents the multihead attention mechanism, A is the vector representation matrix of the mathematical expression predicted by the model, m is the length of the mathematical expression predicted by the model, and h is the dimension of the hidden vector.

[0141] Step 6.2: Average hA and then compare it with the overall representation vector of the problem. After concatenation, the data is fed into a classifier (FC). The classification result u (0 or 1) determines whether the mathematical expression predicted by the model is consistent with the problem.

[0142]

[0143]

[0144] [] denotes a concatenation operation. FC is a classifier, which is a feedforward neural network consisting of two fully connected layers and a tanh activation function.

[0145] Step 7: To improve the classification performance of the classifier and avoid errors caused by the limitations of the classifier, a negative sampling algorithm is used to simultaneously provide the classifier with the true mathematical expression A corresponding to each question. positive and according to A positive The generated negative example expression A negative As shown in Algorithm 1, the parameter λ is a probability threshold between 0 and 1, used to determine whether to change A. positive A mathematical symbol in the text.

[0146] In this embodiment, λ is set to 0.1.

[0147] Algorithm 1: Negative Sampling Algorithm

[0148]

[0149] Step 8: Training in solving mathematical word problems;

[0150] Step 8.1: Predict the number of each operator in the mathematical expression corresponding to the math problem: Set the learning objective to minimize the root mean square error loss.

[0151] L Fine-Tuning =MSE(O pre O truth (16)

[0152] Where MSE represents the root mean square error, O pre O represents the number of mathematical symbols in the expression corresponding to the problem predicted by the model. truth It is the actual number of mathematical symbols in the expression;

[0153] Step 8.2: Let the classifier determine whether the question representation and the expression representation match, and set the learning objective to minimize the binary cross-entropy loss:

[0154]

[0155] Where θ represents the network parameters to be optimized in the supervision module, including the parameters in the multi-head attention module and the classifier.

[0156] Step 8.3: To maximize the probability of generating the corresponding mathematical expression given the problem, the encoder should maximize the consistency between representation Z and answer A. Therefore, the loss function to be minimized is:

[0157]

[0158] Where h represents the network parameters in the encoder and decoder modules, and parameter α adjusts the loss weights during training, which is set to 0.05 in this embodiment.

[0159] Step 9: Output a mathematical expression of length m, A = {a1, a2, ..., a...} m}, a i It is one of the following three parts, the first part including the number V that appears in the question text. num The second part is the external auxiliary constant, denoted as V. con The third part is a list of mathematical symbols, denoted as V. op To solve mathematical word problems;

[0160] In this embodiment, the training process is divided into three stages. In the fine-tuning stage, we set the number of iterations to 10, where we train our encoder separately and minimize Equation 16. In the solution stage, we set the number of iterations to 80. In the first 30 iterations, we train the supervision module and the Encoder-Decoder module separately and minimize Equation 17 and the first part of Equation 18. After the first 30 iterations, the supervision module can accurately supervise the question representation and expression representation. Therefore, in the last 50 iterations, we minimize Equations 17 and 18 to allow the supervision module to guide the encoder to generate a more accurate question representation. In all three stages, we set the batch size to 32, use the Adam optimizer to ensure training stability, and set the initial learning rate to 3e-5 for the encoder and 3e-4 for the other modules. The learning rate is halved every 30 iterations. We set dropout to 0.5 to prevent overfitting. In the testing stage, we set the beam search size to 5. Our fine-tuning phase was conducted on four NVIDIA TESLA V100 graphics cards, while the solution phase was conducted on one NVIDIA RTX A4000 graphics card.

[0161] In this embodiment, we use three publicly available datasets in this field: Math23K, Ape210K, and MathQA, as experimental data. Math23K is a Chinese dataset containing 23,162 questions; Ape210K is a larger and more complex Chinese MWP dataset containing 210,488 questions with 56,532 templates. Due to the presence of noisy data in Ape210K, the Ape-clean dataset, containing 81,225 questions, was used after data filtering for experimental purposes. To verify the effectiveness of this invention on other languages, the English dataset MathQA, containing 37,259 math questions, was used. To avoid training difficulties, questions with more than 100 words or expressions longer than 20 words were removed. The statistics of the processed dataset are shown in Table 2.

[0162] Table 2: Statistical Results of the Dataset

[0163]

[0164] Accuracy was used as the evaluation metric. For the Math23k dataset, two validation methods were employed: the first used the standard partition provided by the dataset, denoted as Math23K; the second used a five-fold cross-validation method, denoted as Math23K*. For the Ape-clean dataset, the given partition was used, containing 79,388 training problems and 1,837 test problems. The training set from Ape-clean and the remaining 129,263 problems from Ape210K were combined for fine-tuning MacBERT, and problems from MathQA were used to fine-tune BERT.

[0165] Table 3: Comparison of the accuracy of solutions obtained by BEM-SM and benchmark methods

[0166]

[0167] Table 3 shows the comparison results of the proposed BEM-SM model with other models. First, it can be seen that the overall results of the BEM-SM model on the Math23K dataset are better than all other methods, and it achieves a solution accuracy of 80% on the Ape-clean dataset and 81.7% on the MathQA dataset. This proves that the BEM-SM model and its components are effective for the MWP problem. The system learns the potential relationship between the problem and the expression by predicting the number of expression operators, and further improves the accuracy of the generated expression through a supervision module.

[0168] Ablation experiments were conducted on the innovations of each part of the system. First, we conducted a fine-tuning task ablation experiments, and the results are shown in Tables 4 and 5. Among them, BEM-SM(-FT) uses only the BERT without fine-tuning in the encoding part. We can see that when the fine-tuning task is not considered, the model performance decreases. The introduction of the fine-tuning task helps the model learn the potential relationship between each question and the corresponding equation, thereby achieving higher solution performance.

[0169] We also conducted ablation experiments on the supervised module, and the results are shown in Tables 3 and 4, where BEM-SM (-SM) indicates that the model does not use the supervised module. It can be observed that without the supervised module, the model performance decreases in all four validation methods. This directly proves that the supervised module can check for erroneous expression representations, optimize the model's encoder, and thus improve the accuracy of the model's generated expressions.

[0170] Table 4: Comparison of Accuracy Rates of Solutions to Ablation Experiments

[0171]

[0172] Table 5: Comparison of accuracy rates in solving equations from ablation experiments

[0173]

[0174] We compared different negative sampling methods, and the results are shown in Table 6, where Ex-Acc and An-Acc represent the expression accuracy and answer accuracy, respectively. It can be seen that changing both operators and numbers simultaneously exhibits better performance than changing only one. Furthermore, when only the operators in the equation are changed, the overall performance of the model is better than changing only the numbers in the equation. This is because the encoder of this invention learns the number of operators in each problem and its corresponding expression during the fine-tuning phase. Therefore, when the operators in the correct expression are changed, the mathematical meaning contained in the correct expression and the counterexample expression differs. However, the operator information contained in the problem representation has the same meaning as the operators in the correct expression. Therefore, the classifier in the supervision module of this invention can effectively classify problem-expression pairs. When the number of operators in the incorrect expression generated by the model during the solution process is inconsistent with that of the correct expression, the supervision module can more easily obtain the problem representation that led to the generation of the incorrect expression and further optimize the encoder. However, when only the numbers in the correct expression are changed, our encoder cannot correctly identify the difference in operators between the correct and counterexample expressions, and cannot achieve the above effect. This result further shows that our designed fine-tuning task can achieve better performance in a system with a supervision module.

[0175] In addition, we conducted experiments on the α parameter, and the results are shown in Table 7. It can be seen that an α parameter that is too large (0.1) or too small (0.01) is not conducive to improving the system accuracy, while maintaining it at 0.05 shows a better effect.

[0176]

[0177] We also conducted an experiment on the number of negative examples generated by negative sampling, and the results are shown in Table 8. The results show that generating 5 negative examples for each positive example yields the best results.

[0178] Furthermore, the results of parameter experiments on the number of heads in multi-head attention are shown in Table 9. The model performs best overall when the number of heads is set to 8.

[0179]

[0180] We also tested the system's performance on questions with different expression lengths, such as... Figure 3 As shown, it can be observed that BEM-SM exhibits the best solution performance in all cases, and the gap between our system and other systems is widening when the expression is longer (11+), which to some extent indicates that our model has an advantage in more complex problems.

[0181] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A mathematical application problem solving method based on BERT encoding with a supervision module, characterized by, The method comprises the following steps: Step 1: input a text sequence of length n into a mathematical application problem solving system wherein is a natural language word or number, n is the total number of characters contained in the text sequence Q; Step 2: Perform the encoding stage, using MacBERT as the encoder for Chinese questions and BERT as the encoder for English questions; the encoding process first obtains the text sequence Q, and maps the text sequence Q into a vector representation matrix Z∈ , where n is the total number of characters contained in the question, and h is the dimension of the hidden vector; is the vector representation matrix of the question; Step 3: Representing each word vector in the matrix Z as a vector Taking the average to get the overall representation vector for the text sequence ;​ Step 4: Perform the fine-tuning phase, and... The input is fed into a feedforward neural network (FFN) consisting of two fully connected layers (FC) and one ReLU activation layer, and the number of mathematical symbols O in the corresponding mathematical expression is obtained. pre FFN() represents a feedforward neural network: ； Step 5: generating an expression tree using a tree decoder; Step 6: capturing the question representation that leads to the generation of an incorrect expression using a classifier, if the classifier output is 1, it means that the expression generated by the decoder corresponds to the question, if the output is 0, it means that the question representation vector generated by the encoder leads to the generation of an incorrect expression and needs to be further optimized; Step 7: The negative sampling algorithm is used to provide the classifier with the real mathematical expression corresponding to each question at the same time and according to generated negative example expressions ; Step 8: training of the mathematical application question solving system; Step 9: Output the mathematical expression of length m , is one of the following three parts, the first part includes the numbers appearing in the title text , the second part is the external auxiliary constant, denoted as , the third part is the list of mathematical symbols denoted as , and the solution of the mathematical application problem is realized.

2. The mathematical application problem solving method based on BERT encoding with a supervision module according to claim 1, characterized in that, The vector representation matrix Z in step 2 is as follows: ； Wherein, the Encoder represents encoding the mathematical application question in natural language form using a pre-trained language model as an encoder.

3. The mathematical application problem solving method with a supervised module based on BERT encoding according to claim 1, characterized in that, The overall representation vector described in step 3 As shown in the following equation: 。 4. The mathematical application problem solving method with a supervised module based on BERT encoding according to claim 1, characterized in that, The step 5 specifically comprises the following steps: Step 5.1: Utilize the overall representation vector in the decoding process Feature vector as root node in expression tree : ； Step 5.2: Compute the attention weights of the current node to each token in the question using the current node's representation vector q and the question representation vector matrix Z Then compute the context vector of the current node using Then compute the probability of each mathematical symbol for the current node using q and and the vector representation of each mathematical symbol and take the maximum value to achieve the prediction of the mathematical symbol​​​ ； ； ； ； ； ； ； where [ ] denotes concatenation operation, is the tanh activation function, , , , are the parameters to be trained for the model, is a set of mathematical operators, is the word embedding matrix corresponding to the operator, denotes the word embedding vector of the operator y, is a set of constants used in solving mathematical word problems, is the word embedding matrix corresponding to the constant, denotes the word embedding vector of the constant y, n p is a set of numbers appearing in each problem, loc(y) denotes the position of the number y appearing in the problem, denotes the vector at the loc(y) position from the overall problem representation vector matrix; If the predicted tag is a number or a constant, then the subtree representation vector t of the subtree rooted at this node is directly implemented by t = {t1, t2,..., tn} ； If the prediction flag is an operator, it is necessary to continue predicting the left and right child trees of the node; the representation vector of the left child node of the node is calculated according to the prediction flag of the current node, the context vector and the representation vector q by the following formula, and the mathematical symbol corresponding to the left child node of the node is predicted by using the previous prediction formula after is obtained ; ； ； ； ； ； ； where [ ] denotes concatenation operation, • denotes matrix multiplication; σ is sigmod activation function, is tanh activation function, , , , is training parameter; Step 5.3: Continue to construct the expression tree rooted at the left child node in the pre-order traversal order until the mathematical symbol corresponding to the left child node of a node q is a number or a constant, then construct the right subtree of the node q, first get the target vector of the right child node of q using the following formula , then predict the mathematical symbol corresponding to the right child node of q using :​ ； ； ； ； ； ； If the predicted result is an operator, then the target decomposition of the q node should be continued until the predicted results of the left and right child nodes of a certain node u are both numbers or constants, and then the left child tree of u is used to represent the vector , the right child tree of u is used to represent the vector , and the mathematical symbol corresponding to u is used to represent the operator The sub-tree representation vector of the sub-tree rooted at u is constructed , and the construction is performed layer by layer upwards until the root node of the whole tree is reached. ； ； ； ； where [ ] denotes concatenation operation, σ() is sigmod activation function, is tanh activation function, and is training parameter.

5. The mathematical application problem solving method based on BERT encoding with a supervision module according to claim 4, characterized in that, The step 6 specifically comprises the following steps: Step 6.1: Utilize multi-head attention mechanism to capture the relationship between each mathematical symbol in the mathematical expression and get the vector representation matrix hA∈ ; ； wherein, denotes a multi-head attention mechanism, A is a matrix of vector representations of the mathematical expressions predicted by the model, m is the length of the mathematical expressions predicted by the model, and h is the dimension of the hidden vectors. Step 6.2: hA is averaged and combined with the overall expression vector of the question After splicing, it is sent to the classifier FC, and the classification result u is 0 or 1 to determine whether the mathematical expression predicted by the model is consistent with the question: ； ； Wherein, [] represents the splicing operation; FC is a classifier, which is a feedforward neural network composed of two fully connected layers and a tanh activation function.

6. The mathematical application problem solving method with a supervised module based on BERT encoding according to claim 1, characterized in that, The step 8 specifically comprises the following steps: Step 8.1: predicting the number of operators in the mathematical expression corresponding to the mathematical question: setting the learning goal to minimize the root mean square error loss: ； where MSE represents the mean square error, represents the number of each mathematical symbol in the expression corresponding to the question predicted by the model, is the actual number of each mathematical symbol in the expression. Step 8.2: let the classifier judge whether the question representation and the expression representation match, set the learning goal to minimize the binary cross-entropy loss: ； Wherein, θ represents the network parameters to be optimized in the supervision module; Step 8.3: maximize the probability of generating the corresponding mathematical expression of the given question, the encoder should maximize the consistency of the representation Z and the answer A, so the loss function to be minimized is: ； Wherein, η represents the network parameters in the encoder and decoder modules, and the parameter α adjusts the loss weight in the training process.

7. A BERT encoding-based mathematical application question solving system for implementing the BERT encoding-based mathematical application question solving method of claim 1, characterized in that, Specifically comprising: The encoding module, the decoding module, the supervision module and the fine-tuning module; The encoding module is responsible for encoding the mathematical application question, i.e. converting the mathematical application question in natural language form into a vector form encoding result; The decoding module receives the encoding result output by the encoding module to infer the mathematical expression corresponding to solving each mathematical application question; The supervision module is a regularization term used to verify the correctness of the mathematical expression; The fine-tuning module uses the encoding result to predict the number of mathematical symbols in the mathematical expression corresponding to solving the question, so that the encoder can perceive the mathematical logic corresponding to the question before solving.

Images