A question structure reasoning method and system

Abstract

The present invention discloses a question structure reasoning method and system. The method comprises: receiving input text and constructing relationships between content to obtain semantic relationships; acquiring prior knowledge and integrating it with the input text to obtain implicit knowledge; combining the implicit knowledge with the semantic relationships to generate a tree-structured target; and parsing the tree-structured target according to a predetermined traversal order to obtain an equation expression. The system comprises a hierarchical reasoning encoder, a knowledge encoder, a tree-structured encoder, and a tree-structured decoder. By using the present invention, mathematical expressions can be constructed based on the question description. As a question structure reasoning method and system, the present invention can be widely applied in the field of natural language processing.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a question structure reasoning method and system. Background Art

[0002] Math problem-solving tasks involve reasoning about mathematical queries based on a description of the problem. They are an interdisciplinary research topic connecting mathematics and natural language processing. A problem is described through a brief narrative, followed by a question about an unknown number. In recent years, multi-task processing research based on deep learning methods has received increasing attention. Typically, to answer this type of mathematical reasoning task, users not only need to parse the question and understand the context, but also apply external knowledge. However, previous methods have only learned text descriptions from short and limited narratives, without using any background knowledge not present in the description. This has limited the model's ability to infer mathematical problems from a global perspective. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a question structure reasoning method and system to construct a mathematical expression based on the problem description.

[0004] The first technical solution adopted by the present invention is: a question structure reasoning method, comprising the following steps:

[0005] Receive input text and build relationships between contents to obtain semantic relationships;

[0006] Acquire prior knowledge and fuse it with input text to obtain implicit knowledge;

[0007] Combine implicit knowledge and semantic relationships and generate tree-structured targets;

[0008] The tree structure target is parsed according to the predetermined traversal order to obtain the equation expression.

[0009] Furthermore, the mutual relationship between the contents includes a word-level reasoning relationship and a sentence-level reasoning relationship. The step of receiving the input text and constructing the mutual relationship between the contents to obtain the semantic relationship specifically includes:

[0010] Receive input text;

[0011] Construct word-level reasoning relationships based on the word-level reasoning layer;

[0012] Construct sentence-level reasoning relations based on the sentence-level reasoning layer;

[0013] The word-level reasoning layer and the sentence-level reasoning layer both adopt GRU-based sequence encoding.

[0014] Furthermore, the step of constructing a word-level reasoning relationship based on the word-level reasoning layer specifically includes:

[0015] Encode words based on bidirectional GRU;

[0016] Incorporate contextual information into word-level representations to generate word representations;

[0017] The attention mechanism is introduced to extract important words, and the word-level context vector is used to measure the importance of words to obtain word-level reasoning relationships;

[0018] Aggregate important word representations into sentence vectors.

[0019] Furthermore, the step of constructing a sentence-level reasoning relationship based on the sentence-level reasoning layer specifically includes:

[0020] Encode sentences based on bidirectional GRU;

[0021] Merge the information of adjacent sentences into the sentence-level representation to generate sentence representation;

[0022] The attention mechanism is introduced and the sentence-level context vector is used to measure the importance of the sentence, thus obtaining the sentence-level reasoning relationship.

[0023] Furthermore, the step of acquiring prior knowledge and fusing the prior knowledge with the input text to obtain implicit knowledge specifically includes:

[0024] Acquire prior knowledge based on a Chinese pre-trained model on a large corpus;

[0025] The prior knowledge is integrated with the input text to extract the implicit knowledge of the input text.

[0026] Furthermore, the step of combining implicit knowledge and semantic relationships to generate a tree structure target specifically includes:

[0027] Adaptively merge implicit knowledge and semantic relationships based on dot products and generate enhanced representations through linear mapping functions;

[0028] Divide the nodes of the tree structure into mathematical operators, common sense values ​​and numbers;

[0029] Initialize the root node vector according to the enhanced representation;

[0030] Combined with the target vocabulary with candidate words, the root node vector with trainable vectors is iteratively used to predict the probability of belonging to the node;

[0031] Generate tree nodes according to the predicted probability and obtain the tree structure target.

[0032] Furthermore, the expression of the enhanced representation is as follows:

[0033] Y=F([w p Y p ,w h Y h ])

[0034] In the above formula, Y represents the enhancement representation, Y p Indicates semantic relationship, Y h represents implicit knowledge, w p and w h represents the importance of the correspondence, [·] represents the linking operation, and F represents the linear mapping function.

[0035] Furthermore, the step of parsing the tree structure target according to the predetermined traversal order to obtain the equation expression specifically includes:

[0036] Parse the tree structure target;

[0037] Generate an intermediate operator based on the top node of the tree structure target;

[0038] Recursively parse all nodes from the left child node to the right child node to obtain the equation expression.

[0039] The second technical solution adopted by the present invention is: a question structure reasoning system, comprising:

[0040] Hierarchical reasoning encoder, which receives input text and constructs the relationship between the content to obtain semantic relationships;

[0041] Knowledge encoder, which is used to acquire prior knowledge and fuse it with input text to obtain implicit knowledge;

[0042] Tree structure encoder, used to combine implicit knowledge and semantic relations and generate tree structure targets;

[0043] A tree structure decoder is used to parse the tree structure target according to a predetermined traversal order to obtain an equation expression.

[0044] The beneficial effects of the method and system of the present invention are as follows: the present invention effectively integrates implicit knowledge into the model based on the knowledge encoder, which can help the model correctly parse the semantics of words from complex texts, builds the relationship between words and sentences based on the hierarchical reasoning encoder, realizes the connection between the entity domain and the context domain, and thus realizes the construction of mathematical expressions based on the problem description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the steps of a question structure reasoning method of the present invention;

[0046] Figure 2It is a structural block diagram of a question structure reasoning system of the present invention;

[0047] Figure 3 It is a flow chart of a specific embodiment of the present invention.

[0048] Figure 4 It is a schematic diagram of hierarchical reasoning according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0050] A math problem (MWP) can be represented as (P,E), where P is the problem text and E is the solution expression. Assume that the description of a MWP has L sentences s i , each sentence contains T i Words, w it (t∈[1,T]) represents the word in the i-th sentence. Our proposed encoder projects the original problem description into a vector representation, based on which a tree-structured decoder is constructed to predict mathematical expressions.

[0051] Reference Figure 1 and Figure 3 The present invention provides a question structure reasoning method, which includes the following steps:

[0052] S1, receiving input text and constructing the relationship between the contents to obtain semantic relationships;

[0053] S2, obtain prior knowledge and fuse it with the input text to obtain implicit knowledge;

[0054] S3, combining implicit knowledge and semantic relations to generate a tree structure target;

[0055] S4. Parse the tree structure target according to the predetermined traversal order to obtain the equation expression.

[0056] As a further preferred embodiment of the present method, the mutual relationship between the contents includes a word-level reasoning relationship and a sentence-level reasoning relationship. The step of receiving the input text and constructing the mutual relationship between the contents to obtain the semantic relationship specifically includes:

[0057] S11, receiving input text;

[0058] S12, constructing word-level reasoning relationships based on the word-level reasoning layer;

[0059] S13, constructing sentence-level reasoning relations based on the sentence-level reasoning layer;

[0060] The word-level reasoning layer and the sentence-level reasoning layer both adopt GRU-based sequence encoding.

[0061] Specifically, this step is implemented by a hierarchical reasoning encoder, as shown in the schematic diagram. Figure 4 , taking into account that different parts of a mathematical description do not have similar relevant information. Furthermore, determining relevant parts involves modeling the interactions between words, not just their isolated presence in the text. Therefore, taking this aspect into account, the model includes two levels of reasoning mechanisms. One is word-level reasoning, and the other is sentence-level reasoning, which allows the model to focus more or less on individual words and sentences when building the entire description representation. The hierarchical reasoning encoder consists of two layers. The first layer is the word-level reasoning layer, and the second layer is the sentence-level reasoning layer. GRU-based sequence encoding is used in both layers of reasoning.

[0062] GRU-based sequence encoding uses a gating mechanism to track the state of the sequence without using separate memory units. There are two types of gates: reset gate r t and update gate z t . They jointly control how information is updated to the state. At time t, the GRU calculates the new state as:

[0063]

[0064] This is the previous state h t-1 and the current new state calculated using the new sequence information Linear interpolation of . Update gate z t Decide how much past information to keep and how much new information to add. t Updated to:

[0065] z t =σ(W z x t +U z h t-1 +b z )

[0066] where x t Is the sequence vector at time t. Candidate state Calculated as:

[0067]

[0068] where r t To reset the gate, control the contribution of the previous state to the candidate state. t If it is 0, it means forgetting the past state. t Updated to:

[0069] r t =σ(W r x t +U r h t-1 +b r )

[0070] The above W and U are learnable matrix weights, and b is a learnable bias vector.

[0071] As a further preferred embodiment of the present method, the step of constructing a word-level reasoning relationship based on the word-level reasoning layer specifically includes:

[0072] S121, encode words based on bidirectional GRU;

[0073] S122, merging context information into word-level representation to generate word representation;

[0074] S123. Introduce the attention mechanism to extract important words, and use the word-level context vector to measure the importance of the words, and obtain the word-level reasoning relationship;

[0075] S124. Aggregate important word representations into sentence vectors.

[0076] Specifically, word-level reasoning. In this layer, the model uses a bidirectional GRU to generate word representations by integrating information from both directions. Therefore, it incorporates contextual information into word-level representations. Given a word w it , t∈[1,T] and an embedding matrix W e , a bidirectional GRU consists of a forward For sentence s i From the word w i1 Read the word w iT , and a reverse From the word w iT Read the word w i1 :

[0077] x it =W e w it ,t∈[1,T],

[0078]

[0079]

[0080] Given a word w it The word-level representation of is obtained by concatenating the forward hidden state and the backward hidden state, i.e. Can be combined with w itThe information of the entire sentence centered on . Note that not all words have the same effect on expressing the meaning of the sentence. Therefore, we introduce an attention mechanism to extract the words that are important to the sentence and aggregate the representations of these informative words into a sentence vector. Specifically,

[0081] u it =tanh(W w h it +b w ),

[0082]

[0083] s i =∑α it h it .

[0084] We first input word-level features h through a single-layer MLP it , to obtain u it As h it The hidden representation of . Then use the word-level context vector u w Measure the importance of words and get the normalized importance weight α through the softmax function it We then compute the weighted sum of the word representations according to the learnable weights as the sentence vector s i During the training process, the word context vector u w Perform random initialization and joint learning

[0085] As a further preferred embodiment of the present method, the step of constructing a sentence-level reasoning relationship based on the sentence-level reasoning layer specifically includes:

[0086] S131, encode the sentence based on bidirectional GRU;

[0087] S132, merging information of adjacent sentences into sentence-level representation to generate sentence representation;

[0088] S133. Introduce the attention mechanism and use the sentence-level context vector to measure the importance of the sentence and obtain the sentence-level reasoning relationship.

[0089] Specifically, sentence-level reasoning. Given a sentence vector s i , we use a similar method to get the question description vector. We use a bidirectional GRU to encode the sentence:

[0090]

[0091]

[0092] in, and Denote forward GRU and reverse GRU respectively. We will and Connect them to get the target representation of sentence i h i The information of the neighboring sentences of sentence i is integrated, but the focus is still on sentence i. In order to reward sentences related to the correct parsing of the problem description, we use the attention mechanism again and introduce a sentence-level context vector u s To measure the importance of a sentence, the formula is:

[0093] u it =tanh(W s h i +b s ),

[0094]

[0095] v=∑α i h i ,

[0096] Where v is the global context vector, which summarizes all the information of a sentence in a description. Similarly, during training, the sentence-level context vector u s Also randomly initialized and learned jointly.

[0097] As a further preferred embodiment of the method, the step of acquiring prior knowledge and fusing the prior knowledge with the input text to obtain implicit knowledge specifically includes:

[0098] S21. Acquire prior knowledge based on a Chinese pre-trained model on a large corpus;

[0099] S22. Fuse the prior knowledge with the input text to extract the implicit knowledge of the input text.

[0100] Specifically, this step is implemented by pre-training the knowledge encoder, using a transformer-based language model and building an encoder that is pre-trained with the Roberta model that has been pre-trained on large corpora BooksCorpus and Wikipedia to capture implicit knowledge. Like in BERT, we use WordPiece to tag a description get tokens, embed them into the pre-trained Roberta embedding, and modify Roberta's positional encoding to obtain a series of d-dimensional token representations We feed this information into a pre-trained knowledge encoder based on transformers to fine-tune the representation during training. We average the outputs of all transformer steps to obtain the combined implicit knowledge representation Y p .

[0101] As a further preferred embodiment of the method, the step of combining implicit knowledge and semantic relationships to generate a tree structure target specifically includes:

[0102] S31, adaptively merge implicit knowledge and semantic relations based on dot product, and generate enhanced representation through linear mapping function;

[0103] Specifically, the result Y is obtained by pre-training the knowledge encoder and the hierarchical reasoning encoder respectively. p and Y h Then, we use Figure 1 The parser at the end of the two encoders shown adaptively merges Y p and Y h , and obtain an enhanced representation Y for final decoding. The enhanced representation Y can be expressed as:

[0104] Y=F([w p Y p ,w h Y h ]),

[0105] where w p and w h By Y p and Y h Derived from , it is used to calculate the importance of the task. [ ] represents the link operation. We use a simple dot product to merge Y p and Y h These two representations are then transformed into an enhanced representation Y using a linear mapping function F, such as a fully connected layer, for final decoding. p and w h It can be calculated as:

[0106]

[0107]

[0108] Among them, W p and W h are all trainable weight matrices, and Indicates different MLPs.

[0109] S32, adaptively merge implicit knowledge and semantic relations based on dot product, and generate enhanced representation through linear mapping function;

[0110] S33, dividing the nodes of the tree structure into mathematical operators, common sense numerical values ​​and numbers;

[0111] S34, initializing the root node vector according to the enhanced representation;

[0112] S35, combining a pre-prepared target vocabulary with candidate words, iteratively using the root node vector with the trainable vector to predict the probability of belonging to the node;

[0113] S36. Generate tree nodes according to the predicted probability to obtain a tree structure target.

[0114] Specifically, our model initializes the root node vector according to the global context representation Y from the two encoders. The expression tree in the decoder contains three types of nodes: mathematical operators V op , a commonsense value V encountered in the target expression but not in the question text con (e.g., a rabbit has 4 legs) and the number n encountered in problem P P . For the target expression V tar For each token y of , its token embedding e(y|P) is defined as:

[0115]

[0116] Among them, M op and M con are two problem-independent trainable word embedding matrices. P In the value, we take the corresponding hidden state from the encoder as its token embedding, where loc(y,P) is the index position of the value y in P. Mathematical operator V op Occupies a non-leaf position. n P The representation of y depends on some MWP description. Because y has to obtain the corresponding hidden state from the encoder output V op and V con The representation of is represented by two embedding matrices M op and M con Get it independently.

[0117] The tree-structured decoder consists of a tree-structured encoder and a tree-structured decoder. Similar to the tree-structured encoder, we prepare candidate words for operators and numbers in the target vocabulary, and then iteratively use the root vector with the trainable vector to predict the probability of the node token y in the target vocabulary. Then, according to the rule in formula (19), the specific y with the highest probability is replaced with the tree node.

[0118] As a further preferred embodiment of the method, the expression of the enhanced representation is as follows:

[0119] Y=F([w p Y p ,w h Y h ])

[0120] In the above formula, Y represents the enhancement representation, Y p Indicates semantic relationship, Y h represents implicit knowledge, w p and w h represents the importance of the correspondence, [·] represents the linking operation, and F represents the linear mapping function.

[0121] As a further preferred embodiment of the method, the step of parsing the tree structure target according to a predetermined traversal order to obtain an equation expression specifically includes:

[0122] S41, parsing the tree structure target;

[0123] S42, generating an intermediate operator according to the topmost node of the tree structure target;

[0124] S43. Recursively parse all nodes from the left child node to the right child node to obtain the equation expression.

[0125] Specifically, mathematical equations typically consist of operators and variables. First, variables are defined as leaf nodes, with each operator node having two child nodes. The tree-structured decoder then parses the equation expression according to a predetermined traversal order. It first generates the center operator, followed by the left child node. This generation process continues recursively until the last leaf node is reached. Next, the right child node is generated in the same manner.

[0126] As a further preferred embodiment of this method, since the MWP task can be expressed as (P, E), we define its loss function as It can be formulated as predicting the token y of node t t Formally, the objective function for training the optimizer is:

[0127]

[0128] Among them, m is the size of E, q t and Y t are the target vector and context vector of the tth node. p is calculated using the distribution calculation function in the target-driven tree structure.

[0129] like Figure 2 As shown, a question structure reasoning system includes:

[0130] Hierarchical reasoning encoder, which receives input text and constructs the relationship between the content to obtain semantic relationships;

[0131] Knowledge encoder, which is used to acquire prior knowledge and fuse it with input text to obtain implicit knowledge;

[0132] Tree structure encoder, used to combine implicit knowledge and semantic relations and generate tree structure targets;

[0133] A tree structure decoder is used to parse the tree structure target according to a predetermined traversal order to obtain an equation expression.

[0134] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A question structure reasoning method, characterized in that: The following steps are involved: Receive input text and build relationships between contents to obtain semantic relationships; Acquire prior knowledge and fuse it with input text to obtain implicit knowledge; Combine implicit knowledge and semantic relationships and generate tree-structured targets; Parse the tree structure target according to the predetermined traversal order to obtain the equation expression; The mutual relationship between the contents includes a word-level reasoning relationship and a sentence-level reasoning relationship. The step of receiving the input text and constructing the mutual relationship between the contents to obtain the semantic relationship specifically includes: Receive input text; Construct word-level reasoning relationships based on the word-level reasoning layer; Construct sentence-level reasoning relations based on the sentence-level reasoning layer; Combining word-level reasoning relations and sentence-level reasoning relations to obtain semantic relations; The word-level reasoning layer and the sentence-level reasoning layer both use GRU-based sequence encoding; The step of acquiring prior knowledge and integrating the prior knowledge with the input text to obtain implicit knowledge specifically includes: Acquire prior knowledge based on a Chinese pre-trained model on a large corpus; The prior knowledge is integrated with the input text. In the language model, the outputs of all transformer steps are averaged to extract the implicit knowledge of the input text.

2. A question structure reasoning method according to claim 1, characterized in that: The step of constructing a word-level reasoning relationship based on the word-level reasoning layer specifically includes: Encode words based on bidirectional GRU; Incorporate contextual information into word-level representations to generate word representations; The attention mechanism is introduced to extract important words, and the word-level context vector is used to measure the importance of words to obtain word-level reasoning relationships; Aggregate important word representations into sentence vectors.

3. A question structure reasoning method according to claim 2, characterized in that: The step of constructing a sentence-level reasoning relationship based on the sentence-level reasoning layer specifically includes: Encode sentences based on bidirectional GRU; Merge the information of adjacent sentences into the sentence-level representation to generate sentence representation; The attention mechanism is introduced and the sentence-level context vector is used to measure the importance of the sentence, thus obtaining the sentence-level reasoning relationship.

4. A question structure reasoning method according to claim 2, characterized in that: The step of combining implicit knowledge and semantic relationships to generate a tree structure target specifically includes: Adaptively merge implicit knowledge and semantic relationships based on dot products and generate enhanced representations through linear mapping functions; Divide the nodes of the tree structure into mathematical operators, common sense values ​​and numbers; Initialize the root node vector according to the enhanced representation; Combined with the target vocabulary with candidate words, the root node vector with trainable vectors is iteratively used to predict the probability of belonging to the node; Generate tree nodes according to the predicted probability and obtain the tree structure target.

5. A question structure reasoning method according to claim 4, characterized in that: The expression of the enhanced representation is as follows: Y=F([w p Y p ,w h Y h ]) In the above formula, Y represents the enhancement representation, Y p Indicates semantic relationship, Y h represents implicit knowledge, w p and w h represents the importance of the correspondence, [·] represents the linking operation, and F represents the linear mapping function.

6. A question structure reasoning method according to claim 5, characterized in that: The step of parsing the tree structure target according to the predetermined traversal order to obtain the equation expression specifically includes: Parse the tree structure target; Generate an intermediate operator based on the top node of the tree structure target; Recursively parse all nodes from the left child node to the right child node to obtain the equation expression.

7. A question structure reasoning system, characterized in that: The method for executing a question structure reasoning method as claimed in claim 1 comprises: Hierarchical reasoning encoder, which receives input text and constructs the relationship between the content to obtain semantic relationships; Knowledge encoder, which is used to acquire prior knowledge and fuse it with input text to obtain implicit knowledge; Tree structure encoder, used to combine implicit knowledge and semantic relations and generate tree structure targets; A tree structure decoder is used to parse the tree structure target according to a predetermined traversal order to obtain an equation expression.

Images