A mathematical application question solving system and method based on contrast learning

Abstract

The application provides a mathematical application question solving system and method based on contrast learning, and relates to the technical field of application solving.The system comprises an encoding module, a decoding module, a supervision module and a contrast learning module; Chinese application questions are encoded by using an encoder MacBert, and English application questions are encoded by using an encoder Bert; a tree-shaped decoder SAU is used to generate an expression tree, that is, a mathematical expression corresponding to each question is generated; a classifier is used to capture question representations that lead to the generation of incorrect expressions; if the output result of the classifier is 1, it indicates that the expression generated by the decoder corresponds to the question; if the output result is 0, it indicates that the question representation vector generated by the encoder leads to the generation of an incorrect expression and needs to be further optimized; then the system is trained, and finally the solving of mathematical application questions is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of application solving, and in particular to a mathematical application question solving system and method based on contrast learning. BACKGROUND

[0002] The implementation of a mathematical application question (MWP) solver is a challenging problem in the field of natural language processing. Its task is to automatically solve mathematical problems described in natural language. It needs to encode the expression of a mathematical application question from natural language into a logical form that machines can understand, then map it into the corresponding mathematical expression through a decoding module, and finally calculate the answer. In each MWP, the number of equations is always equal to the number of unknowns, so people divide MWP into arithmetic word problems (AWP) with a single unknown and a single equation, and equation set problems (ESP) with multiple unknowns and multiple equations according to the number of equations and unknowns corresponding to each question. Problem 1 and problem 2 in Table 1 are two MWP examples taken from the Math23K dataset. Problem 1 is an ESP problem, in which unknowns x and y represent the number of large bowls and small bowls respectively. Problem 2 is an AWP problem, in which x represents the price of The Romance of the Three Kingdoms. Compared with AWP, ESP often corresponds to multiple unknowns, and a set of equations need to be established to obtain the final solution.

[0003] Table 1: MWP examples

[0004]

[0005] With the wide application of neural network-based natural language processing technology in various industries, people have developed various end-to-end deep learning models to solve MWP problems. In order to solve the problem that the output of the model for solving MWP problem is a mathematical expression, GTS proposes a tree decoder (Qin J, Lin L, Liang X, et al. Semantically-aligned universal tree-structured solver for math word problems[J]. arXiv preprint arXiv:2010.06823, 2020.), which first constructs a target mathematical expression into an expression tree and recursively generates a solution expression tree in the decoding stage of the model in the preorder sequence. Thus, each non-leaf node of the expression tree calculated is an operator, and the leaf node is a number. Therefore, the output expression of the model is strictly in accordance with the mathematical meaning, thereby avoiding the problem of generating an unsuitable mathematical solution expression using an RNN-based neural network as a decoder, and improving the model effect. On this basis, the SAU model proposes a universal expression tree method, which converts each equation in the equation set corresponding to each ESP problem into a corresponding expression tree, and takes them as the sub-trees of the root node of the universal expression tree, thereby uniformly representing the ESP expression tree and the AWP expression tree. And by aligning the nodes in the tree being generated with the problem representation in the context in the decoding stage, the semantic consistency of the generated expression tree is further strengthened, thereby further improving the solving accuracy.

[0006] Although the SAU model proposes a universal expression tree method to uniformly solve AWP problems and ESP problems, it is found through experiments that in a complex language environment and mathematical relationship, the model often has difficulty in distinguishing whether a relatively complex question is an AWP problem or an ESP problem, and such problems often lead to low efficiency of the solving model. Specifically, the SAU model sometimes incorrectly regards an AWP problem as an ESP problem and generates two solutions, and sometimes incorrectly regards an ESP problem as an AWP problem and generates only one solution. As shown in Table 1: the SAU model has difficulty in distinguishing between the two types of MWP, and incorrectly predicts problem 1 (ESP problem) as AWP and problem 2 (AWP) as ESP. The problem representation vectors of some problems are seriously confused after encoding, which directly leads to the generation of incorrect solutions due to serious deviation in understanding the problem. SUMMARY

[0007] In view of the defects of the prior art, the application provides a mathematical application question solving system and method based on contrast learning.

[0008] A mathematical application question solving system based on contrastive learning, comprising an encoding module, a decoding module, a supervision module, and a contrastive learning module;

[0009] The encoding module is responsible for encoding mathematical application questions, i.e., converting mathematical application questions in natural language form into encoded results in vector form;

[0010] The decoding module receives the encoded results output by the encoding module to infer the mathematical expressions corresponding to each mathematical application question;

[0011] The supervision module is a regularization term used to verify the correctness of the mathematical expressions;

[0012] The contrastive learning module is responsible for separating the vector representations of different types of mathematical application questions as much as possible using the encoded results of the questions, so that the vector representations of the same type of mathematical application questions are as close as possible;

[0013] On the other hand, a mathematical application question solving method based on contrastive learning is implemented based on the aforementioned mathematical application question solving system based on contrastive learning, and specifically comprises the following steps:

[0014] Step 1: input a sequence of natural language description of length n into the mathematical application question solving system, i.e., the question title Q, Q = {q1, q2,..., qn}, where q n is a natural language word or a number; i

[0015] Step 2: encode the Chinese application question using the encoder MacBert and the English application question using the encoder Bert;

[0016] Step 2.1: encode the positive example question corresponding to the question title Q and the negative example question corresponding to the question title Q to obtain the vector representation of the three questions;

[0017] First, input the question title Q into the encoder MacBERT or BERT, where MacBERT is used to encode the Chinese application question and BERT is used to encode the English application question, to obtain the vector representation matrix Z of the question title Q, Z ∈ R n*h :

[0018] Z = Encoder (Q) (1)

[0019] where n is the total number of characters in the question, h is the dimension of the hidden vector, and R n*h is the vector representation matrix of the question, and Encoder represents using a pre-trained language model as an encoder to encode the mathematical application question in natural language form;

[0020] ​Step 2.2: Represent each word vector z in the vector representation matrix Z s The average is taken to obtain the overall representation vector of the target problem Q of length n.

[0021]

[0022] Calculate the global representation vector separately The problem representation of the positive example problem corresponding to the target problem Q. Cosine similarity between and the overall representation vector The problem representation of the counterexample problem corresponding to the target problem Q. Cosine similarity between The training objective is to increase Simultaneously reduce

[0023] Step 3: Use the tree decoder SAU to generate an expression tree, that is, to generate the mathematical expression corresponding to each question;

[0024] Step 3.1: During the decoding process, utilize the overall representation vector. The feature vector q of the root node of the expression tree root :

[0025]

[0026] Step 3.2: Using the representation vector q of the current node and the problem representation vector matrix Z, calculate the relationship between the current node and each character z in the problem. s Attention weight a s Then use a s Calculate the context vector of the current node. Then use q and The vector representation e(y) of each mathematical symbol is used to calculate the probability of each mathematical symbol corresponding to the current node, and the maximum value of these probabilities is taken to achieve the representation of the mathematical symbol. Prediction:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] where [ ] denotes concatenation operation, tanh() is the tanh activation function, Wa, W s is the training parameter, V op is the set of mathematical operators, M op is the word embedding matrix corresponding to the operator, M op (y) denotes the word embedding vector of operator y, V con is the set of constants, M con is the word embedding matrix corresponding to the constant, M con (y) denotes the word embedding vector of constant y, n p is the set of numbers appearing in each question, loc(y) denotes the position of number y appearing in the question, z loc(y) denotes the vector at loc(y) position from the overall question vector matrix;

[0035] If the predicted token is a number or a constant, the subtree representation vector t of the subtree rooted at this node is directly implemented as :

[0036]

[0037] If the predicted token is an operator, the left and right subtrees of this node need to be predicted; the representation vector q left of the left child node of this node is calculated according to the predicted token of the current node, the context vector and the representation vector q by the following formula, after obtaining q left , the mathematical symbol corresponding to the left child node of this node is predicted by using the previous prediction formula

[0038]

[0039]

[0040] h l = o l ⊙C l

[0041] g l = σ(W gl ·h l )

[0042] Q le= tanh(W le · h l )

[0043] q left = g l ⊙ Q le

[0044] where [] denotes concatenation operation, · denotes matrix multiplication. σ is sigmod activation function, tanh() is tanh activation function, W ol , W cl , W gl , W le are training parameters.

[0045] Step 3.3: Continue to construct the expression tree rooted at the left child node in the pre-order traversal order until the mathematical symbol corresponding to the left child node of a node q is a number or a constant, then construct the right child tree of q node. First, get the target vector q right of the right child of q node using the following formula right , and then predict the mathematical symbol corresponding to the right child node

[0046]

[0047]

[0048] h r = o r ⊙ C r

[0049] g r = σ(W gr · [h r , t left ])

[0050] Q re = tanh(W re · [h r , t left ])

[0051] q right = g r ⊙ Q re

[0052] If the prediction result is an operator, the target decomposition of q node needs to be continued until the prediction results of the left and right child nodes of a node u are both numbers or constants, then start to use the left child tree of u node to represent the vector u left , the right child tree of u node to represent the vector u right , and the mathematical symbol corresponding to u node Start to build the subtree representation vector t of the subtree with u as the root node u And build layer by layer upwards until the root node of the whole tree is built.

[0053]

[0054]

[0055]

[0056]

[0057] In the formula, [] represents the splicing operation, σ() is the sigmod activation function, tanh() is the tanh activation function, W gu and W cu are the parameters to be trained by the model.

[0058] Step 4: In the decoding process, each generated computable subtree can be regarded as a computable unit. The regularization of subtree-level semantic alignment is proposed, and the subtree representation t of each subtree and z i (1≤i≤n) are calculated by formula (12), and then a two-layer feedforward neural network with tanh activation is used to convert t and a to the same semantic space respectively, and d sa and e sa are used to represent them respectively:

[0059]

[0060] e sa = W e2 tanh(W e1 a) (13)

[0061] d sa = W d2 tanh(W d1 t) (14)

[0062] Where tanh() is the tanh activation function, W a , V a , W e1 , W e2 , W d1 , W d2 are the parameter matrices to be trained by the model.

[0063] Step 5: Capture the question representation that leads to the generation of the incorrect expression using the classifier. If the classifier output is 1, it means that the expression generated by the decoder corresponds to the question. If the output is 0, it means that the question representation vector generated by the encoder leads to the generation of an incorrect expression and needs to be further optimized.

[0064] Step 5.1: Use the multi-head attention mechanism to capture the relationship between each mathematical symbol in the mathematical expression and obtain the vector representation matrix hA∈R m*h .

[0065] hA=MultiheadAttention(A,A,A) (15)

[0066] where MultiheadAttention represents the multi-head attention mechanism, A is the vector representation matrix of the mathematical expression predicted by the model, m is the length of the mathematical expression predicted by the model, and h is the dimension of the hidden vector.

[0067] Step 5.2: Take the average of hA and concatenate it with the overall question representation vector to input into the classifier FC. The classification result u is 0 or 1 to determine whether the mathematical expression predicted by the model is consistent with the question:

[0068]

[0069]

[0070] where [] represents the concatenation operation, FC is the classifier, and is a feedforward neural network composed of two fully connected layers and a tanh activation function.

[0071] The negative sampling algorithm is used to provide the classifier with the true mathematical expression A positive for each question and the negative example expression A positive generated according to A negative .

[0072] Step 6: Train the mathematical application question solving system, increase and decrease . The loss function is set as:

[0073]

[0074] where is the question representation of the positive example question, is the question representation of the negative example question, and MarginRankingLoss() is the MarginRankingLoss loss function.

[0075] The learning objective is set to minimize the binary cross-entropy loss:

[0076]

[0077] Where θ represents the network parameters to be optimized in the supervision module, u is the result of whether the expression predicted by the classifier matches the question, and takes the value of 0 or 1, and P(X) is the probability of event X occurring.

[0078] The encoder maximizes the consistency between representation Z and answer A, and maximizes the semantic consistency between each node in the expression tree and the corresponding part of the question. Therefore, the loss function minimized is:

[0079]

[0080] in η7 The parameters represent the network parameters in the encoder and decoder modules. Parameter α adjusts the weight of the subtree semantic alignment mechanism loss during training, parameter β adjusts the weight of the teacher module loss during training, m is the number of computable subtrees for each solution tree, and d is the number of subtrees. sa and e sa These are the subtree representation vector and subtree semantic alignment vector for each computable subexpression tree, respectively.

[0081] Step 7: Output a mathematical expression of length m, A = {a1, a2, ..., a...} m}, a i It is one of the following three parts, the first part including the number V that appears in the question text. num The second part is the external auxiliary constant, denoted as V. con The third part is a list of mathematical symbols, denoted as V. op This enables the solution of mathematical word problems.

[0082] The beneficial effects of adopting the above technical solution are as follows:

[0083] This invention provides a mathematical word problem solving system and method based on contrastive learning, which has the following beneficial effects:

[0084] (1) A contrastive learning method was designed to enable the pre-trained language model to better understand the mathematical logic contained in the problem. This method achieves the effect of clustering problem representations with the same mathematical expression template and separating problem representations with different mathematical expression templates. At the same time, the supervision module we designed further improves the solution accuracy.

[0085] (2) Introduce the Chinese MWP dataset HMWP as experimental data, and we use the small cow translation tool to translate it into English and merge it with two small English datasets that contain both AWP problems and ESP problems into a new English MWP dataset Merge. Experiments are carried out on the two datasets, and the experimental results show that the performance of our model is better than the current optimal method, and the model has scientificity and effectiveness. BRIEF DESCRIPTION OF DRAWINGS

[0086] Figure 1 MWPSM-CL model architecture diagram in the embodiment of the application;

[0087] Wherein figure (a) is SAU-Solver, figure (b) is MacBERT-CL;

[0088] Figure 2 MWPSM-CL model architecture diagram in the embodiment of the application;

[0089] Figure 3 AWP problem experimental result graph in the embodiment of the application;

[0090] Figure 4 ESP problem experimental result graph in the embodiment of the application. DETAILED DESCRIPTION

[0091] The specific embodiments of the application will be further described in detail below in combination with the drawings and examples. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

[0092] A mathematical application question solving system based on contrast learning, comprising: an encoding module, a decoding module, a supervision module and a contrast learning module;

[0093] The encoding module is responsible for encoding mathematical application questions, i.e. converting mathematical application questions in natural language form into encoded results in vector form;

[0094] The decoding module receives the encoded results output by the encoding module to infer the mathematical expressions corresponding to each mathematical application question;

[0095] The supervision module is a regularization term for checking the correctness of the mathematical expressions;

[0096] The contrast learning module is responsible for separating the vector representations of different types of mathematical application questions as much as possible using the encoded results of the questions, so that the vector representations of the same type of mathematical application questions are as close as possible;

[0097] On the other hand, a mathematical application question solving method based on contrast learning is realized based on the aforementioned mathematical application question solving system based on contrast learning, and specifically comprises the following steps:

[0098] Step 1: Input a sequence of length n described in natural language into the math word problem solving system, i.e., the target problem Q, where Q = {q1, q2, ..., q...} n}, where q i Natural language words or numbers;

[0099] Step 2: The proposed MWPSM-CL model architecture is as follows Figure 2 As shown, the encoder MacBert is used to encode Chinese word problems, and the encoder Bert is used to encode English word problems. A contrastive learning task is designed to enable the encoder to better understand the mathematical information contained in the problem, so as to avoid the situation where the encoder cannot understand the confusing encoded representation generated by the problem, thus leading the decoder to generate incorrect solutions.

[0100] Step 2.1: Encode the target question Q, the positive example question corresponding to the target question Q, and the negative example question corresponding to the target question Q respectively, and obtain the vector representation of the three questions;

[0101] First, the target problem Q is fed into the encoder MacBERT or BERT. MacBERT is used to encode Chinese word problems, while BERT is used to encode English word problems, resulting in the vector representation matrix Z∈R of the target problem Q. n*h :

[0102] Z = Encoder(Q) (1)

[0103] Where n is the total number of characters in the problem, and h is the dimension of the implicit vector; R n*h Let be the vector representation matrix of the problem, and Encoder represent the encoding of the mathematical word problem in natural language form using a pre-trained language model as an encoder;

[0104] Step 2.2: Represent each word vector z in the vector representation matrix Z s The average is taken to obtain the overall representation vector of the target problem Q of length n. Used in the subsequent contrastive learning process, decoding module, and supervision module:

[0105]

[0106] During the comparative learning phase, the overall representation vector is calculated separately. The problem representation of the positive example problem corresponding to the target problem W. Cosine similarity between and the overall representation vector The problem representation of the counterexample problem corresponding to the target problem Q. Cosine similarity between where the training target is to increase So that the problem representation of the corresponding equation template consistent problem is more similar, let the encoder understand the potential relationship between the problems with the same mathematical meaning but different problem text representations, so as to better learn the same mathematical logic implied by mathematical problems in different expressions. At the same time, reduce So that the problem representation of different types of MWP problems (such as AWP and ESP), the corresponding mathematical relationship is not completely the same, so that the problem representation of the problem is as far away as possible in the vector space, so as to avoid the confusion caused by the encoder's insufficient understanding of the problem and then lead to the generation of wrong solutions.

[0107] In order to verify our idea, we use the MacBert trained by the contrast learning method proposed in the invention to encode the 30 questions of each of the equation templates mentioned in the third section N*(1+x) 2 =N and x+y=N; N*x+N*y=N, and the vector representation of them is also reduced by the T-SNE method, and the specific results are as follows Figure 1 (b) The triangular and circular shapes shown in: After our training, the problem representation corresponding to the same equation template is more Figure 1 (a) closely distributed in the vector space, while the problem representation corresponding to different equation templates is more Figure 1 (a) separated far apart. Under such conditions, the decoder can use more accurate problem representation to generate correct solution expressions.

[0108] Step 3: Use the tree decoder SAU to generate the expression tree, that is, generate the mathematical expression corresponding to each question; In addition, we also introduce the supervision module we designed before to check the error mathematical expressions generated by the decoder.

[0109] Use the tree decoder SAU to generate the expression tree, an example of an expression tree is shown in Figure 2 “DecoerModule” in.

[0110] Step 3.1: In the decoding process, use the overall representation vector as the feature vector qroot of the root node in the expression tree:

[0111]

[0112] Step 3.2: Use the current node's representation vector q and the question representation vector matrix Z to calculate the attention weight a s of each word z s in the question for the current node, and then use a s to calculate the context vector of the current node Then the probabilities of each mathematical symbol corresponding to the current node are calculated using q and and the vector representation e(y) of each mathematical symbol, and the maximum of these probabilities is taken to realize the prediction of the mathematical symbol :

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] where [] represents the concatenation operation, and tanh() is the tanh activation function, W a , W s is the training parameter, V op is the set of mathematical operators, M op is the word embedding matrix corresponding to the operator, and e op (y) represents the word embedding vector of the operator y, where y can be '+' or '-' and the like. V con is the set of constants, M con is the word embedding matrix corresponding to the constant, and e con (y) represents the word embedding vector of the constant y, where y can be "1" or "3.14" and the like. n p is the set of each number appearing in each question, loc(y) represents the position of the number y appearing in the question, and z loc(y) represents the vector at the loc(y) position taken from the overall question representation vector matrix.

[0121] If the predicted label is a number or a constant, the subtree representation vector t of the subtree rooted at this node is directly realized by :

[0122]

[0123] If the predicted label is an operator, it is necessary to continue to predict the left and right subtrees of this node; the representation vector q leftis the predicted label of the current node Context vector and the representation vector q is computed by the following equation left The mathematical symbol corresponding to the left child node of the node q is predicted by the following prediction equation

[0124]

[0125]

[0126] h l = o l ⊙ C l

[0127] g l = σ(W gl · h l )

[0128] Q le = tanh(W le · h l )

[0129] q left = g l ⊙ Q le

[0130] where [ ] denotes the concatenation operation, · denotes the matrix multiplication. σ is the sigmod activation function, tanh() is the tanh activation function, W ol , W cl , W gl , W le are the training parameters.

[0131] Step 3.3: Continue to construct the expression tree rooted at the left child node in the pre-order traversal order until the mathematical symbol corresponding to the left child node of a node q is a number or a constant, then construct the right child tree of q. First, get the target vector q right of the right child of q by the following equation right Then predict the mathematical symbol corresponding to the right child node of q This process also needs to use the left child tree representation vector t left of q node:

[0132]

[0133]

[0134] h r = o r ⊙ C r

[0135] g r = σ(W gr · [h r , t left ])

[0136] Q re = tanh(W re · [h r , t left ])

[0137] q right = g r ⊙ Q re

[0138] If the predicted result is an operator, then the target decomposition of the q node needs to be continued until the predicted results of the left and right child nodes of a certain node u are both numbers or constants, and then the left subtree of u node represents the vector u left , the right subtree of u node represents the vector u right , and the mathematical symbol corresponding to u node starts to build the subtree representation vector t u of the subtree with u as the root node, and builds layer by layer upwards until the root node of the whole tree is built.

[0139]

[0140]

[0141]

[0142]

[0143] In the formula, [] represents the splicing operation, σ() is the sigmod activation function, tanh() is the tanh activation function, W gu and W cu are the parameters to be trained by the model.

[0144] Step 4: In the decoding process, each generated computable subtree can be regarded as a computable unit, and each computing unit is respectively inferred from different parts of the target objective Q and should maintain semantic consistency with these parts. Therefore, the SAU model proposes regularization of subtree-level semantic alignment, calculates the subtree semantic alignment vector a through formula (12) using the subtree representation t of each subtree and z i (1≤i≤n), and then uses a two-layer feedforward neural network with tanh activation to convert t and a to the same semantic space, respectively, using d sa and e sadenotes:

[0145]

[0146] e sa = W e2 tanh(W e1 a) (13)

[0147] d sa = W d2 tanh(W d1 t) (14)

[0148] where tanh() is a tanh activation function, W a , V a , W e1 , W e2 , W d1 , W d2 are parameter matrices to be trained by the model.

[0149] Step 5: The supervision module in the present application is as shown in Figure 2 SupervisionModule, which uses the classifier to capture the question representation that leads to the generation of the incorrect expression. If the output of the classifier is 1, it indicates that the expression generated by the decoder corresponds to the question. If the output is 0, it indicates that the question representation vector generated by the encoder leads to the generation of an incorrect expression and needs to be further optimized.

[0150] Step 5.1: Use the multi-head attention mechanism to capture the relationship between each mathematical symbol in the mathematical expression and obtain the vector representation matrix hA∈R m*h .

[0151] hA= MultiheadAttention(A, A, A) (15)

[0152] where MultiheadAttention represents the multi-head attention mechanism, A is the vector representation matrix of the mathematical expression predicted by the model, m is the length of the mathematical expression predicted by the model, and h is the dimension of the hidden vector.

[0153] Step 5.2: Take the average of hA and concatenate it with the overall question representation vector to send it to the classifier FC. The classification result u is 0 or 1 to determine whether the mathematical expression predicted by the model is consistent with the question:

[0154]

[0155]

[0156] Where [] represents the concatenation operation, FC is the classifier, which is a feedforward neural network consisting of two fully connected layers and a tanh activation function.

[0157] To improve the classification performance of the classifier and avoid errors caused by its limitations, a negative sampling algorithm is used to simultaneously provide the classifier with the true mathematical expression A corresponding to each question. positive and according to A positive The generated negative example expression A negative As shown in Table 1, the parameter λ is a probability threshold between 0 and 1, used to determine whether to change A. positive A mathematical symbol in the language. In this embodiment, λ is set to 0.1.

[0158] Table 1: Negative Sampling Algorithm

[0159]

[0160]

[0161] Step 6: Train the mathematical word problem solving system to increase... and reduce Therefore, the loss function is set as follows:

[0162]

[0163] in, The problem is represented as a positive example problem. The problem is represented as a counterexample.

[0164] MarginRankingLoss() is the MarginRankingLoss loss function, and its hyperparameter is set to 0.2 in this embodiment.

[0165] The goal of the supervised module is to enable the classifier to determine whether the question representation and the expression representation match. The learning objective is to minimize the binary cross-entropy loss.

[0166]

[0167] Where θ represents the network parameters to be optimized in the supervision module, including the parameters in the multi-head attention module and the classifier. u is the result of whether the expression predicted by the classifier matches the question, taking a value of 0 or 1, and P(X) is the probability of event X occurring.

[0168] The goal of the solver module is to maximize the probability of generating the corresponding mathematical expression given a problem. Furthermore, under the supervision of the teacher module, the encoder maximizes the consistency between representation Z and answer A, and maximizes the semantic consistency between each node in the expression tree and its corresponding part in the problem. Therefore, the minimized loss function is:

[0169]

[0170] Where η represents the network parameters in the encoder and decoder modules, parameter α adjusts the weight of the subtree semantic alignment mechanism loss during training, and is set to 0.01 in this embodiment; parameter β adjusts the weight of the teacher module loss during training, and is set to 0.05 in this embodiment; m is the number of subtrees that can be computed for each solution tree; and ds... a and es a These are the subtree representation vector and subtree semantic alignment vector for each computable sub-expression tree, respectively.

[0171] Our training process is divided into three phases. In the contrastive learning phase, we set the number of iterations to 10. Here, we train our encoder separately and minimize Equation 18. In the solution phase, we set the number of iterations to 80. In the first 30 iterations, we train the supervisory module and the Encoder-Decoder module separately and minimize Equation 19 and the first two parts of Equation 20. After the first 30 iterations, the supervisory module can accurately supervise the question representation and expression representation. Therefore, in the last 50 iterations, we minimize Equations 19 and 20 to allow the supervisory module to guide the encoder to generate a more accurate question representation. Throughout all three phases, we set the batch size to 32, use the Adam optimizer to ensure training stability, and set the initial learning rate to 3e-5 for the encoder and 3e-4 for the other modules. The learning rate is halved every 30 iterations. We set dropout to 0.5 to prevent overfitting. In the testing phase, we set the beam search size to 5. Our comparative learning phase was conducted on four NVIDIA TESLA V100 graphics cards, while the solution phase was conducted on one NVIDIA RTX A4000 graphics card.

[0172] Step 7: Output a mathematical expression of length m, A = {a1, a2, ..., a...} m}, a i It is one of the following three parts, the first part including the number V that appears in the question text. num The second part consists of some external auxiliary constants, such as (1, π), needed in solving mathematical word problems, denoted as V. conThe third part is a list of mathematical symbols consisting of ‘+’, ‘-’, ‘*’, ‘ / ’ and ‘^’ denoted as V op , and solving the mathematical application questions.

[0173] In this embodiment, the public data set HMWP in the field is used as Chinese experimental data. In addition, since the research on ESP problems is not deep enough, there are only two small data sets of English ESP problem data sets at present. In order to solve the problem that the deep learning model needs a large amount of data for model training, we use the small cow translation tool to translate the problems in HWMP into English, and add ALG514 and Draw to summarize a new English MWP data set Merge. See Table 2 for analysis of each data set.

[0174] Table 2: Data set statistics table

[0175]

[0176] During the experiment, because too long questions and equations will seriously affect the training of the model, we removed all the questions with more than 100 words and the equations with more than 20 characters. Then we carried out sufficient experiments on HMWP and Merge in the way of five-fold cross-validation, and reproduced the code according to the SAU-Solver public code. The experimental results are shown in Tables 3-6: First, the equation accuracy and answer accuracy of our model on the two data sets are much higher than those of SAU-Slover, especially on the HMWP data set, our model reaches a solving accuracy of 60.4%, which is 10% higher than that of SAU-Solver, which proves the scientificity and effectiveness of our invention. Then we carried out ablation experiments on the contrast learning method we proposed. MWPSM-CL(-CL) represents the model without contrast learning. The results show that without using contrast learning, the solving accuracy of the model decreases to different degrees, which further proves that the contrast learning method we proposed is indeed helpful to improve the solving accuracy of the model.

[0177] Table 3: CLM-SM and baseline method answer accuracy comparison

[0178]

[0179] Table 4: CLM-SM and baseline method equation accuracy comparison

[0180]

[0181] Table 5: CLM-SM ablation experiment answer accuracy comparison

[0182]

[0183] Table 6: Comparison of Accuracy of CLM-SM Ablation Experiment Equations

[0184]

[0185] To observe the experimental results in more detail, we also took the test results of the first fold of the five-fold cross-validation on the HMWP dataset and analyzed them first by question type. The statistical results of the first fold test data are shown in Table 7, and the specific experimental results analyzed by question type are shown in Tables 8 and 9.

[0186] Table 7: HMWP First Fold Test Data Statistics

[0187]

[0188]

[0189] Table 8: Comparison of Answer Accuracy Rates in HMWP First Fold

[0190]

[0191] Table 9: Comparison of Accuracy Rates of HMWP First-Fold Equations

[0192]

[0193] The results in the table show that our model outperforms SAU-Solver in solving both the ASP and ESP problems. We further compared the accuracy of the two models for equations of different lengths, with the experimental results for the AWP problem shown below. Figure 3 As shown, the experimental results for the ESP problem are as follows: Figure 4 As shown in the figure, the results show that our proposed model outperforms the SAU-Solver model in accuracy across all equation lengths, whether on the AWP or ESP problem. This effect is more pronounced as the equation length increases, indicating that our model performs better when solving more complex problems.

[0194] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A mathematical word problem-solving system based on contrastive learning, characterized in that, It includes an encoding module, a decoding module, a supervision module, and a contrastive learning module; The encoding module is responsible for encoding mathematical word problems, that is, converting mathematical word problems in natural language form into encoded results in vector form; The decoding module receives the encoding result output by the encoding module and infers the mathematical expression corresponding to solving each mathematical application problem; The supervision module is a regularization term used to verify the correctness of mathematical expressions; The contrastive learning module is responsible for using the coding results of the questions to separate the vector representations of different types of mathematical word problems, so that the vector representations of the same type of mathematical word problems are clustered. The aforementioned system for solving mathematical word problems based on contrastive learning is used to implement a method for solving mathematical word problems based on contrastive learning, specifically including the following steps: Step 1: Input a sequence of length n, described in natural language, into the math word problem solving system; this is the target problem Q. , where q i Natural language words or numbers; Step 2: Encode the Chinese word problems using the MacBert encoder, and encode the English word problems using the Bert encoder; Step 3: Use the tree decoder SAU to generate an expression tree, that is, to generate the mathematical expression corresponding to each question; Step 4: During the decoding process, each time a computable subtree is generated, this subtree can be regarded as a computable unit. A subtree-level semantic alignment regularization is proposed, utilizing the subtree representation t of each subtree and... Calculate the semantic alignment vector 'a' of the subtree, and then use the... The activated two-layer feedforward neural network transforms t and a into the same semantic space, respectively, using... and express; The and As shown in the following formula: (12)； (13)； (14)； in The tanh activation function is used. For parameter matrices; Step 5: Use a classifier to capture the question representation that leads to the generation of incorrect expressions. If the classifier output is 1, it means that the expression generated by the decoder corresponds to the question. If the output is 0, it means that the question representation vector generated by the encoder led to the generation of incorrect expressions and needs to be further optimized. Step 6: Train the mathematical word problem solving system; Step 7: Output a mathematical expression of length m , It is one of the following three parts, the first part including the numbers that appear in the question text. The second part is the external auxiliary constant, denoted as The third part is a list of mathematical symbols, denoted as... This enables the solution of mathematical word problems.

2. The mathematical word problem solving system based on contrastive learning according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Encode the target question Q, the positive example question corresponding to the target question Q, and the negative example question corresponding to the target question Q respectively, and obtain the vector representation of the three questions; First, the target problem Q is fed into the encoder MacBERT or BERT. MacBERT is used to encode Chinese word problems, while BERT is used to encode English word problems, resulting in the vector representation matrix Z∈ of the target problem Q. : (1)； Where n is the total number of characters in the question, and h is the dimension of the implicit vector; Let be the vector representation matrix of the problem, and Encoder represent the encoding of the mathematical word problem in natural language form using a pre-trained language model as an encoder; Step 2.2: Represent each word vector in the vector representation matrix Z The average is taken to obtain the overall representation vector of the target problem Q of length n. : (2)； Calculate the global representation vector separately The problem representation of the positive example problem corresponding to the target problem Q. cosine similarity between them (sim( ), and the overall representation vector The problem representation of the counterexample problem corresponding to the target problem Q. cosine similarity between them (sim( The training objective is to increase sim( ), while reducing sim( ).

3. The mathematical word problem solving system based on contrastive learning according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: During the decoding process, utilize the overall representation vector. The feature vector of the root node in the expression tree : (3)； Step 3.2: Using the representation vector q of the current node and the problem representation vector matrix Z, calculate the relationship between the current node and each word in the problem. Attention weights Then use Calculate the context vector of the current node. Then use q and and vector representations of various mathematical symbols Calculate the probability of each mathematical symbol corresponding to the current node, and take the maximum value of these probabilities to achieve the desired mathematical symbol. Prediction: ； ； ； ； ； ； ； In the formula, [] represents the splicing operation. The tanh activation function is used. , , , For training parameters, A set of mathematical operators. The word embedding matrix corresponding to the operator. The word embedding vector representing the operator y. For the set of constants, The word embedding matrix corresponding to the constant. The word embedding vector representing the constant y. Let loc(y) be the set of all numbers that appear in each problem, and let loc(y) represent the position of number y in the problem. This indicates that the vector at position loc(y) is selected from the overall vector matrix representing the problem. If the predicted label If it is a number or a constant, then the subtree representation vector t of the subtree rooted at that node is directly obtained from... accomplish: If the predicted label If it is an operator, then it is necessary to continue predicting the left and right subtrees of that node; the representation vector of the left child node of that node. It is based on the predicted label of the current node. Context vector and representing vectors The result is obtained by calculating using the following formula. Then, using the previous prediction formula, the mathematical symbol corresponding to the left child node of this node is predicted. ; ； ； ； ； ； ； In the formula, [] represents the concatenation operation, · represents matrix multiplication, and σ is the sigmoid activation function. The tanh activation function is used. , , , For training parameters; Step 3.3: Continue constructing the expression tree rooted at the left child node according to the preorder traversal order until the mathematical symbol corresponding to the left child node of a certain node q is a number or a constant. Then, construct the right subtree of node q. First, use the following formula to obtain the target vector of the right child of node q. Then use Predict the mathematical symbol corresponding to the right child node. : ； ； ； ； ； ； If the prediction result If the operator is q, then the target decomposition of node q needs to continue until the prediction results of the left and right child nodes of a certain node u are both numbers or constants. At this point, the left subtree of node u can be used to represent the vector. The right subtree of node u represents a vector. Mathematical symbols corresponding to node u Begin constructing the subtree representation vector of the subtree rooted at u. And build upwards layer by layer until the root node of the entire tree is reached; ； ； ； ； In the formula, [] denotes the concatenation operation, and σ() is the sigmoid activation function. The tanh activation function is used. and These are the parameters that the model needs to be trained with.

4. The mathematical word problem solving system based on contrastive learning according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Utilize a multi-head attention mechanism to capture the relationships between mathematical symbols in the mathematical expression and obtain the vector representation matrix hA∈ of the mathematical expression. ; (15)； in, This represents the multi-head attention mechanism, where A is the vector representation matrix of the mathematical expression predicted by the model, m is the length of the mathematical expression predicted by the model, and h is the dimension of the hidden vector. Step 5.2: [The sentence is incomplete and requires more context to be translated accurately.] After averaging, the vector represents the entire problem. After concatenation, the data is fed into a classifier (FC). The classification result u (0 or 1) determines whether the mathematical expression predicted by the model is consistent with the problem. (16)； (17)； Where [] represents the concatenation operation, FC is a classifier, which is a feedforward neural network consisting of two fully connected layers and a tanh activation function; The negative sampling algorithm is used to simultaneously provide the classifier with the true mathematical expression corresponding to each question. and according to Generated negative example expression .

5. A mathematical word problem-solving system based on contrastive learning according to claim 1, characterized in that, The training described in step 6 specifically includes: Increase sim( and reduce sim( Therefore, the loss function is set as follows: (18)； in, The problem is represented as a positive example problem. For the counterexample problem, MarginRankingLoss() is the MarginRankingLoss loss function; The learning objective is set to minimize the binary cross-entropy loss: (19)； Where θ represents the network parameters to be optimized in the supervision module, u is the result of whether the expression predicted by the classifier matches the question, and takes the value of 0 or 1, and P(X) is the probability of event X occurring. The encoder maximizes the consistency between representation Z and answer A, and maximizes the semantic consistency between each node in the expression tree and the corresponding part of the question. Therefore, the loss function minimized is: (20)； Where η represents the network parameters in the encoder and decoder modules, parameter α adjusts the weight of the subtree semantic alignment mechanism loss during training, parameter β adjusts the weight of the teacher module loss during training, and m is the number of subtrees that can be computed for each solution tree. and These are the subtree representation vector and subtree semantic alignment vector for each computable subexpression tree, respectively.

Images