Mathematical application question intelligent answering method and system fused with large model knowledge
By inspiring, verifying and injecting knowledge into the intelligent answers of mathematical application problems, and using large language models to generate symbols, the logical errors and insufficient interpretability in the inference process in the existing technology are solved, and more accurate and explanatory Answers to mathematical application questions have enhanced the reasoning ability of artificial intelligence.
Patent Information
- Application Number
- CN202411077460.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-07
AI Technical Summary
The existing automatic solution methods for mathematical application problems are prone to logical errors and factual errors in the reasoning process, and cannot effectively model human gradual application of knowledge, lack interpretability, and are far from the goal of general artificial intelligence.
Using an intelligent solution method that integrates big model knowledge, by inspiring, verifying and injecting necessary knowledge in each inference step, using the large language model to generate symbols and combine them into solution expressions, and updating inferences with knowledge encoding and text representation vectors Target vector, realizes interpretability and accuracy of the inference process.
Improve the accuracy and interpretability of intelligent solutions of mathematical application problems, enhance the rationality of the reasoning process, can be extended to a variety of machine learning tasks and large language models, and provides feasible solutions with strong reasoning capabilities, with high educational and practical value. .
Smart Images

Figure CN119005333A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for intelligently solving mathematical application problems integrating large model knowledge. Background Art
[0002] Automatically solving math word problems (MWP) is an important task in exploring general artificial intelligence. Its goal is to enable machines to read a math problem described in natural language and infer the corresponding numerical answer. This process requires machines to understand mathematical logic, learn and use mathematical knowledge for reasoning like humans. Therefore, the ability to automatically solve math word problems is regarded as an important indicator of the level of machine intelligence and has always been a basic but very important research direction in the field of artificial intelligence.
[0003] At present, there are mainly the following methods for automatically solving mathematical word problems:
[0004] (1) Rule-based automatic solution method.
[0005] Rule-based automatic solving methods rely on manually constructed patterns or formal languages. By matching the question text with predefined question templates, the question is mapped into a specific propositional form, and then the reasoning results are gradually generated using manually defined rules.
[0006] (2) Automatic solution method based on statistics.
[0007] The statistical-based automatic solution method uses traditional machine learning models such as support vector machines and log-linear models to identify entities, quantities, and operators from the question text, and then substitutes them into predefined expression templates to obtain the answer through simple symbolic reasoning.
[0008] (3) Automatic solution method based on semantic analysis.
[0009] The automatic solution method based on semantic parsing maps the problem text into a structured semantic representation (such as a semantic parse tree) and converts the semantic information into mathematical expressions through certain logical rules.
[0010] (4) Automatic solution method based on deep learning.
[0011] The automatic solution method based on deep learning mainly adopts the encoder-decoder structure, which automatically mines and understands the characteristics of the question text through the encoder, and then uses the decoder to reason and solve the expression. Based on this structure, the existing work mainly focuses on improving the semantic understanding ability of the encoder and the reasoning ability of the decoder. The method focusing on semantic understanding ability enriches the mining of question information by introducing the semantic structure, numerical size, numerical relationship, etc. of the question text, so as to achieve a more accurate understanding of the problem and obtain more reasonable reasoning results. The method focusing on reasoning ability designs different neural network architectures based on different reasoning modes, such as using recurrent neural networks to directly generate symbol sequences in the sequence-to-sequence (Seq2Seq) architecture, designing a goal-oriented tree structure decoder to generate expression trees from top to bottom based on the human problem decomposition mechanism in the sequence-to-tree (Seq2Tree) architecture, and using a relational extraction decoder in the deductive reasoning architecture to gradually combine numerical values and operators to form sub-expressions and iteratively generate expression trees from bottom to top.
[0012] (5) Automatic solution method based on large language model.
[0013] Large Language Model (LLM) is a general deep neural network model with massive parameters trained on a large-scale corpus, which has strong context understanding ability. Existing research has enabled it to acquire strong thinking planning ability through technologies such as Chain-of-Thought and Tree-of-Thought, and has gradually achieved excellent results in tasks such as solving mathematical word problems.
[0014] However, the above-mentioned automatic solution methods are still some distance away from human-level artificial intelligence, and they are prone to logical errors, factual errors and other problems in reasoning.
[0015] In view of this, the present invention is proposed. Summary of the invention
[0016] The purpose of the present invention is to provide a method and system for intelligently solving mathematical word problems that integrates large model knowledge, which overcomes the shortcomings of the prior art that cannot model the gradual application of knowledge in the reasoning process, lacks explainability, and is far from the goal of general artificial intelligence, and can more accurately achieve intelligent solutions to mathematical word problems.
[0017] The objective of the present invention is achieved through the following technical solutions:
[0018] An intelligent solution method for mathematical word problems integrating large model knowledge, comprising:
[0019] Step 1: Enter the question text;
[0020] Step 2: In the current reasoning step, based on the question text and the generated symbols, the knowledge required for the current reasoning step is stimulated based on the large language model;
[0021] Step 3: Encode the knowledge stimulated by step 2, verify and filter the knowledge using the obtained knowledge encoding and the inference target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the inference target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; then update the inference target vector of the current reasoning step in combination with the calculated two attention weights; use the updated inference target vector of the current reasoning step to calculate the symbol of the current reasoning step, and generate the inference target vector of the next reasoning step; where, if the current reasoning step is the first step, the inference target vector of the current reasoning step is the initial inference target vector obtained using the question text; generate all symbols by repeatedly executing steps 2 and 3, and combine them in sequence to form a solution expression;
[0022] Step 4: Use the expression to solve the answer.
[0023] An intelligent mathematical problem-solving system integrating large model knowledge, including:
[0024] An input unit, used to input question text;
[0025] The knowledge system is used to stimulate the knowledge required for the current reasoning step based on the question text and the generated symbols based on the large language model;
[0026] The reasoning system is used to encode the stimulated knowledge, verify and filter the knowledge using the obtained knowledge encoding and the reasoning target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; update the reasoning target vector of the current reasoning step in combination with the calculated two attention weights; calculate the symbol of the current reasoning step in combination with the updated reasoning target vector of the current reasoning step, and generate the reasoning target vector of the next reasoning step; wherein, if the current reasoning step is the first step, the reasoning target vector of the current reasoning step is the initial reasoning target vector obtained using the question text; generate all symbols by repeatedly executing steps 2 and 3, and combine them in sequence into a solution expression;
[0027] The solving unit is used to solve the answer by solving the expression.
[0028] A processing device, comprising: one or more processors; a memory for storing one or more programs;
[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0030] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.
[0031] It can be seen from the technical solution provided by the present invention that the necessary knowledge is stimulated, verified, and injected in each reasoning step, so as to focus on how to ensure the rationality of the stimulated knowledge in each reasoning step and how to apply the knowledge to the symbol generation of the current step, and finally achieve the improvement of the interpretability and accuracy of the reasoning process. The solution provided by the present invention can not only be expanded to include a variety of machine learning tasks and use a variety of large language models, but also provide a feasible solution for realizing artificial intelligence with powerful reasoning ability; it can also provide highly interpretable problem-solving logic ideas for personalized online guidance, which has high educational and practical value. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0033] Figure 1 A flowchart of a method for intelligently solving mathematical word problems integrating large model knowledge provided by an embodiment of the present invention;
[0034] Figure 2 A schematic diagram of an intelligent problem-solving framework provided by an embodiment of the present invention;
[0035] Figure 3 A schematic diagram of the principles of knowledge stimulation, knowledge verification, and knowledge injection provided by an embodiment of the present invention;
[0036] Figure 4 A schematic diagram of an intelligent mathematical word problem solving system integrating large model knowledge provided by an embodiment of the present invention;
[0037] Figure 5 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0039] First, the terms that may be used in this article are explained as follows:
[0040] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.
[0041] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.
[0042] The following is a detailed description of a method and system for intelligently solving mathematical word problems that integrates large model knowledge provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals and technicians in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturer is not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.
[0043] Embodiment 1
[0044] The embodiment of the present invention provides a method for intelligently solving mathematical word problems by integrating large model knowledge, such as Figure 1 As shown, it mainly includes the following steps:
[0045] Step 1. Enter the question text.
[0046] Step 2: Knowledge stimulation process.
[0047] In the embodiment of the present invention, a knowledge system is constructed to stimulate the knowledge required for the current reasoning step based on the large language model according to the question text and the generated symbols. The knowledge system mainly includes: a large language model and a knowledge invoker; the large language model is used as a storage knowledge base for common sense knowledge, mathematical knowledge, etc., and the knowledge invoker inspires knowledge from the large language model at the reasoning step level.
[0048] Step 3: Reasoning process.
[0049] In the embodiment of the present invention, knowledge verification and injection are performed based on the stimulated knowledge to guide logical thinking and ensure the rationality of reasoning.
[0050] The reasoning process is as follows: encode the knowledge stimulated by step 2, verify and filter the knowledge using the obtained knowledge encoding and the reasoning target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; then update the reasoning target vector of the current reasoning step in combination with the calculated two attention weights; use the updated reasoning target vector of the current reasoning step to calculate the symbol of the current reasoning step and generate the reasoning target vector of the next reasoning step; where, if the current reasoning step is the first step, the reasoning target vector of the current reasoning step is the initial reasoning target vector obtained using the question text; generate all symbols by repeatedly executing steps 2 and 3, and combine them in sequence to form a solution expression.
[0051] Step 4: Use the expression to solve the answer.
[0052] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.
[0053] 1. General overview of the plan.
[0054] The existing automatic solution methods are still a long way from human-level artificial intelligence, and they are prone to logical errors, factual errors and other problems in mathematical reasoning. Even for large models, they will produce results that violate basic common sense and mathematical laws in the reasoning chain, which is called hallucination. For this reason, compared with the mathematical reasoning process of humans, cognitive psychology theory points out that the reasoning process is carried out step by step, and each step requires the stimulation, verification, and injection of necessary knowledge to guide logical thinking and ensure the rationality of reasoning. Therefore, how to make the model gradually use knowledge in the reasoning process is a key research to improve its intelligence level, and it is also a core goal of achieving general artificial intelligence.
[0055] In order to overcome the shortcomings of the prior art that humans cannot model the gradual application of knowledge in the reasoning process, lack of interpretability, and are far from the goal of general artificial intelligence, an embodiment of the present invention provides an intelligent solution method for mathematical word problems that integrates large model knowledge. Construct a "knowledge system-reasoning system" as an intelligent problem-solving framework. The knowledge system (KnowledgeSystem) uses a large language model as a storage knowledge base for common sense knowledge, mathematical knowledge, etc., and includes a knowledge stimulator (Knowledge Invoker) to stimulate knowledge from the large language model at the reasoning step level. The inference system (InferenceSystem) adopts an encoder-decoder structure to gradually apply the knowledge in the knowledge system to reason mathematical expressions. In particular, the decoder includes a knowledge verifier (Knowledge Verifier) and a knowledge injector (Knowledge Injector), which respectively focus on how to ensure the rationality of the stimulated knowledge in the current reasoning step, and how to apply the knowledge to the symbol generation of the current step, and finally achieve the improvement of the interpretability and accuracy of the reasoning process. The present invention can also be extended to include a variety of machine learning tasks and use a variety of large language models, providing a feasible solution for realizing artificial intelligence with powerful reasoning capabilities.
[0056] 2. Detailed introduction of the plan.
[0057] 1. Problem definition and formalization.
[0058] The goal of the automatic mathematical word problem solving task is to generate the answer to the mathematical expression calculation problem by performing logical reasoning based on the word problem text. P Defined as a sequence X consisting of n words P =[x1,x2,...,x n ], each x represents a word, and the subscript is the number of the word, X P The set of values that appear in is defined as N P (e.g. {400,6}). The solution expression Y for problem P P Defined as a sequence Y consisting of m operators, constants, or numerical values in the problem (collectively referred to as symbols) P =[y1,y2,…,y m ], where each symbol y is the set V of decoding symbols for the math word problem P P An element of , the subscript is the serial number of the symbol. Decoding symbol set V P By the operator set V O (e.g. {+,-,×,÷}), constant set V C (e.g. {1,π}) and a set of values NP Composition, namely V P =V O ∪V C ∪N P Since N for each problem P Different, each question's V P The answer to question P is also different. P is the value of the unknown variable to be found (e.g. weight is "550"), and is expressed by the mathematical expression Y P The goal of the automatic mathematical word problem solving task is to learn a model that reads the problem text X P , generating the expression Y P , and based on Y P Calculation problem variable a P .
[0059] 2. Data collection and preprocessing.
[0060] (1) Data collection.
[0061] The invention uses math word problems in a broad sense as input data sets, and the problems must include question text, mathematical expressions, and numerical answers. Examples of such data are the open source math word problem dataset (Math23K) released by Tencent and the open source dataset (MAWPS) released by Microsoft. In addition, the input data set can also be obtained by crawling the web or collecting math problem sets for primary and secondary school students' homework or exams offline.
[0062] (2) Data preprocessing
[0063] Before building a model, the collected data needs to be preprocessed to ensure the effectiveness of the model. Preprocessing mainly includes the following:
[0064] (2.1) Data filtering.
[0065] The embodiments of the present invention are mainly aimed at mathematical word problems that contain only one question, the answer is a numerical value and can be directly calculated through a mathematical expression. Therefore, it is necessary to select questions in the data set that contain only one mathematical expression and a numerical answer, and filter out questions that lack an expression or have two or more expressions and answers.
[0066] (2.2) Sampling.
[0067] Random sampling is performed in each type of data set, and a subset of the original data set is selected to train the model.
[0068] 3. Intelligent problem-solving framework.
[0069] In the embodiment of the present invention, a "knowledge system-reasoning system" is constructed as an intelligent problem-solving framework, such as Figure 2As shown, the intelligent problem-solving framework includes a knowledge system and a reasoning system. The knowledge system is mainly responsible for implementing the aforementioned step 2, using a large language model as a knowledge repository, and using a knowledge stimulator to stimulate the knowledge required for each reasoning step. The reasoning system is mainly responsible for implementing the aforementioned step 3, which infers the solution expression of a given mathematical problem in an encoder-decoder manner. The present invention designs a knowledge verifier and a knowledge injector in the decoder, which are respectively used to filter irrelevant knowledge stimulated in the knowledge system and apply the knowledge to guide the symbolic reasoning of the current step.
[0070] like Figure 3 As shown, the main principles of knowledge stimulation, knowledge verification and knowledge injection are presented.
[0071] (1) Knowledge system.
[0072] The knowledge system is responsible for the storage and stimulation of knowledge. Since the large language model has implicitly stored knowledge in the parameters through training on massive data, the knowledge system directly uses the current large language model (such as GPT4, ChatGPT) as the knowledge base K.
[0073] In the embodiment of the present invention, the current reasoning step is defined as the tth step (reasoning step t), given the knowledge base K, the knowledge stimulator is based on the question text X P and the expression generated before step t The knowledge K required to stimulate the reasoning at step t t '. This process uses prompt engineering technology to splice X P , For a query Q t Input the large language model and use the text output by the large language model as K t '(For example Figure 3 The text in "This question requires the application of the formula 'Total weight = quantity multiplied by unit weight', multiplying the unit weight of bananas by the total number of boxes."), recorded as r is the length of knowledge.
[0074] (2) Reasoning system.
[0075] The reasoning system aims to use the knowledge stimulated by the knowledge system to reason about the expression of a given mathematical word problem. Figure 2 As shown, it adopts the encoder-decoder approach to first encode the question text and then decode it to generate an expression tree.
[0076] (2.1) Encoder-Decoder.
[0077] For a given math word problem P, the encoder Enc reads the problem text X P =[x1,x2,...,xn ], generate a text representation vector sequence H = [h1,h2,...,h n ] and the initial inference target vector s1:
[0078] (H,s1)=Enc([x1,x2,...,x n ])
[0079] Enc can be implemented using RNN, BERT, the encoder of the existing MWP model, and a large language model.
[0080] Based on H and s1, the decoder gradually generates the solution expression Y P =[y1,y2,…,y m Specifically, the t-th step (t = 1, 2, ..., m) consists of two stages: 1) Symbol prediction stage: using the inference target vector s of the t-th step t Calculate symbol y t The distribution of p(y t |y1,y2,...,y t-1 );2) Target generation stage: based on the selected symbol y t , generate the inference target vector s for the next step t+1 These two stages are performed alternately, and all the symbols generated are combined in sequence to solve the expression Y P .
[0081] The present invention mainly focuses on the decoding process, gradually using the knowledge stimulated by the knowledge system to guide the symbol prediction stage in the decoder. In the decoder, the present invention designs two modules: a knowledge verifier module and a knowledge injector module. In the tth step, the knowledge verifier verifies the knowledge K obtained by the knowledge stimulator. t ′ The rationality of y is verified and filtered, and the knowledge injector injects the filtered knowledge into the symbol prediction of the tth step, which is used to assist in generating y t .
[0082] (2.2) Knowledge Verifier.
[0083] Knowledge K inspired by large language models t ′ There may be redundancy, e.g. Figure 2 The correct knowledge that "bananas and oranges are both fruits" may be stimulated at the same time, but it is irrelevant to reasoning. This may bring noise to the knowledge application of the model. Therefore, the knowledge obtained by stimulation is encoded by the knowledge verifier, and the knowledge is verified and filtered using the obtained knowledge encoding and the reasoning target vector of the current reasoning step to obtain the filtered knowledge.
[0084] First, encode it in the following way:
[0085]
[0086] in, Representation knowledge K t The first word and the rth word in ′; for The corresponding encoding vector, f θ For text encoders (such as BERT, T5, etc.); To define the symbol, K t For knowledge K t ′’s encoding vector, k t Encode knowledge.
[0087] Then, combined with the inference target vector s t and knowledge encoding k t , computing knowledge K t ′ The adaptation score and adaptation probability p of the current reasoning step t :
[0088] score = tanh(W v1 ·[s t ,k t ]+b v1 )
[0089] p t =σ(W v2 score+b v2 )
[0090] Among them, σ is the sigmoid function, W v1 ,W v2 is the weight matrix, b v1 ,b v2 is the bias term, and tanh is the hyperbolic activation function.
[0091] Based on the above adaptation probability, the present invention adopts the threshold ε>0 to t ′ Filter if p t >ε, then the filtered knowledge is the encoding vector K of the knowledge t , otherwise the filtered knowledge is a zero vector
[0092] In order to train the knowledge verifier, we need to minimize the following training objective L v , where δ t ∈{0,1} is whether the knowledge K should be filtered in the tth step t ′ The true label of ist =1, otherwise 0.
[0093]
[0094] Among them, L v is the loss of the knowledge verifier and m is the number of reasoning steps.
[0095] (2.3) Knowledge Injector.
[0096] The knowledge injector injects the filtered knowledge into the symbol prediction, which is mainly used to calculate the attention weight of the filtered knowledge in combination with the inference target vector of the current reasoning step, and to calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; and then the two calculated attention weights are combined to update the inference target vector of the current reasoning step.
[0097] Inspired by the dual mechanism of human cognition, on the one hand, the filtered knowledge is expressed in text form, in which the knowledge is often sparse, and in the reasoning process, people tend to focus on the most important part (for example, the formula "total weight = quantity multiplied by unit weight"). On the other hand, based on knowledge, humans can in turn find the most relevant information in the question text (for example, find the "unit weight of bananas" and the "total number of boxes"), which can further reduce the difficulty of reasoning.
[0098] Based on this idea, the present invention designs the following dual attention reasoning mechanism to calculate the attention weight of the filtered knowledge and the attention weight of the question text:
[0099]
[0100] in, is the attention weight of the i-th word in the filtered knowledge, The attention weight of the jth word in the question text, is the encoding vector of the i-th word in the filtered knowledge, W k1 ,W k2 ,W h1 ,W h2 are all weight matrices; softmax is a normalized exponential function, tanh is a hyperbolic activation function, t means the current inference step is the tth step, s t is the inference target vector, k t Encode knowledge; j is the text representation vector of the jth word in the question text.
[0101] The present invention combines the above two attention weights to update the inference target vector s t , the update method is:
[0102]
[0103] in, is the attention weight of the i-th word in the filtered knowledge, The attention weight of the jth word in the question text, is the encoding vector of the i-th word in the filtered knowledge, h j is the text representation vector of the jth word in the question text, β1,β2>0 is the hyperparameter to balance the two attention weights. If β1 is large, it means that more attention should be paid to the stimulated knowledge during reasoning, and if β2 is large, more attention should be paid to the existing information in the question; s ′ t is the updated inference target vector of the current inference step t.
[0104] (2.4)Decoding module.
[0105] The decoding module is also an important component of the decoder, which is used to use the updated inference target vector of the current inference step to calculate the sign of the current inference step and generate the inference target vector of the next inference step.
[0106] In the embodiment of the present invention, combined with the updated inference target vector s ′ t , calculate the symbol of the current reasoning step t, expressed as:
[0107] p(y t |y1,y2,...,y t-1 )=softmax(Dec1(y1,y2,...,y t-1 ,s′ t ))
[0108] Among them, y1,y2,...,y t-1 represents the generated symbols, the subscripts correspond to the order of the reasoning steps, p(y t |y1,y2,...,y t-1 ) means generating the symbol y of the current reasoning step t under the condition of the symbols that have been generated t ; softmax is a normalized exponential function, and Dec1 is a decoding module.
[0109] The inference target vector s for the next inference step is generated using the following formula: t+1 :
[0110] s t+1 = Dec2(s t ′,e(y t ))
[0111] Among them, Dec2 is another decoding module; e(yt ) represents the symbol y t The embedding vector of .
[0112] Exemplarily, Dec1 and Dec2 can be implemented using the output gate and forget gate in the LSTM model.
[0113] 4. Model training.
[0114] In the embodiment of the present invention, the intelligent problem-solving framework needs to be trained in advance, and the training loss function L is expressed as:
[0115] L=∑ P ∑ t -logp(y t |y1,y2,...,y t-1 )+β·L v
[0116] Among them, P represents the mathematical application problem to which the question text belongs; t represents that the current reasoning step is the tth step, y1, y2, ..., y t-1 represents the generated symbols, the subscripts correspond to the order of the reasoning steps, p(y t |y1,y2,...,y t-1 ) means generating the symbol y of the current reasoning step t under the condition of the symbols that have been generated t The probability of L v is the loss function of the knowledge verifier, and β>0 is a hyperparameter that balances the symbol prediction loss function (i.e., the first term in the above formula) and the knowledge filter loss function.
[0117] The parameters of the model are optimized based on the training loss function until the stopping condition is met (for example, the training reaches the set maximum number of times or the loss function converges). After that, the solution task can be completed based on the above steps 1 to 4. The parameter optimization process involved in this part can be implemented with reference to conventional technologies, so it will not be described in detail.
[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0119] Embodiment 2
[0120] The present invention also provides a system for intelligently solving mathematical word problems integrating large model knowledge, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes:
[0121] Input unit (performs the above step 1), used to input question text;
[0122] The knowledge system (performs the above step 2) is used to stimulate the knowledge required for the current reasoning step based on the large language model according to the question text and the generated symbols at the current reasoning step;
[0123] The reasoning system (executes the aforementioned step 3) is used to encode the stimulated knowledge, verify and filter the knowledge using the obtained knowledge encoding and the reasoning target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; update the reasoning target vector of the current reasoning step in combination with the calculated two attention weights; calculate the symbol of the current reasoning step in combination with the updated reasoning target vector of the current reasoning step, and generate the reasoning target vector of the next reasoning step; wherein, if the current reasoning step is the first step, the reasoning target vector of the current reasoning step is the initial reasoning target vector obtained using the question text; generate all symbols by repeated execution, and combine them in sequence into a solution expression;
[0124] The solving unit (performs the aforementioned step 4) is used to solve the answer using the solving expression.
[0125] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.
[0126] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.
[0127] Embodiment 3
[0128] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.
[0129] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0130] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:
[0131] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;
[0132] The output device may be a display terminal;
[0133] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.
[0134] Embodiment 4
[0135] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.
[0136] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.
[0137] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. An intelligent method for solving mathematical word problems integrating large model knowledge, characterized in that: include: Step 1: Enter the question text; Step 2: In the current reasoning step, based on the question text and the generated symbols, the knowledge required for the current reasoning step is stimulated based on the large language model; Step 3: Encode the knowledge stimulated in step 2, verify and filter the knowledge using the obtained knowledge encoding and the reasoning target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; Then, the inference target vector of the current reasoning step is updated by combining the two calculated attention weights; the symbol of the current reasoning step is calculated using the updated inference target vector of the current reasoning step, and the inference target vector of the next reasoning step is generated; wherein, if the current reasoning step is the first step, the inference target vector of the current reasoning step is the initial inference target vector obtained using the question text; all symbols are generated by repeatedly executing steps 2 and 3, and are combined in sequence into a solution expression; Step 4: Use the expression to solve the answer.
2. According to claim 1, a method for intelligently solving mathematical word problems integrating large model knowledge is characterized in that: The encoding of the knowledge stimulated in step 2 includes: The current reasoning step is recorded as step t, and the knowledge stimulated by step 2 is recorded as K t ′, is encoded in the following way: in, Representation knowledge K t The first word and the rth word in ′; for The corresponding encoding vector, f θ is a text encoder; To define the symbol, K t For knowledge K t ′’s encoding vector, k t Encode knowledge.
3. The method for intelligently solving mathematical word problems integrating large model knowledge according to claim 2 is characterized in that: The knowledge is verified and filtered using the acquired knowledge encoding and the inference target vector of the current inference step. The filtered knowledge includes: The current reasoning step is recorded as step t, and the reasoning target vector is recorded as s t , the knowledge code is denoted as k t ; Calculate the knowledge K′ t The adaptation score and adaptation probability p of the current reasoning step t t : score=tanh(W v1 ·[s t ,k t ]+b v1 ) p t =σ(W v2 ·score+b v2 ) Among them, σ is the sigmoid function, W v1 ,W v2 is the weight matrix, b v1 ,b v2 is the bias term, tanh is the hyperbolic activation function; If p t >ε, then the filtered knowledge is the encoding vector K of the knowledge t , otherwise the filtered knowledge is a zero vector.
4. The method for intelligently solving mathematical word problems integrating large model knowledge according to claim 1 is characterized in that: The step of calculating the attention weight of the filtered knowledge in combination with the inference target vector of the current inference step, and calculating the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text includes: The following dual attention reasoning mechanism is designed to calculate the attention weight of the filtered knowledge and the attention weight of the question text: in, is the attention weight of the i-th word in the filtered knowledge, The attention weight of the jth word in the question text, is the encoding vector of the i-th word in the filtered knowledge, W k1 ,W k2 ,W h1 ,W h2 are all weight matrices; softmax is a normalized exponential function, tanh is a hyperbolic activation function, t means the current inference step is the tth step, s t is the inference target vector, k t Encode knowledge; j is the text representation vector of the jth word in the question text.
5. The method for intelligently solving mathematical word problems integrating large model knowledge according to claim 1 is characterized in that: The inference target vector of the current inference step is updated by combining the two calculated attention weights, including: The current reasoning step is recorded as step t, and the reasoning target vector is recorded as s t , the update method is: in, is the attention weight of the i-th word in the filtered knowledge, The attention weight of the jth word in the question text, is the encoding vector of the i-th word in the filtered knowledge, h j is the text representation vector of the jth word in the question text, β1, β2 are hyperparameters for balancing the two attention weights; s′ t is the updated inference target vector of the current inference step t.
6. The method for intelligently solving mathematical word problems integrating large model knowledge according to claim 1 is characterized in that: The combining the updated inference target vector of the current inference step, calculating the sign of the current inference step, and generating the inference target vector of the next inference step includes: The current reasoning step is recorded as step t, and the updated reasoning target vector of the current reasoning step t is recorded as s′ t , the symbolic representation for calculating the current reasoning step t is: p(and t |y1,y2,...,y t-1 )=softmax(Dec1(y1,y2,...,y t-1 ,s′ t )) Among them, y1,y2,...,y t-1 represents the generated symbols, the subscripts correspond to the order of the reasoning steps, p(y t |y1,y2,...,y t-1 ) means generating the symbol y of the current reasoning step t under the condition of the symbols that have been generated t The probability of; softmax is a normalized exponential function, Dec1 is a decoding module; The inference target vector s for the next inference step is generated using the following formula: t+1 : s t+1 =Dec2(s′ t ,e(y t )) Among them, Dec2 is another decoding module; e(y t ) represents the symbol y t The embedding vector of .
7. The method for intelligently solving mathematical word problems integrating large model knowledge according to claim 1 is characterized in that: The step 2 is implemented by a knowledge system, and the step 3 is implemented by a reasoning system. The reasoning system adopts an encoder-decoder structure. The encoder is responsible for encoding the question text to obtain a text representation vector of the question text and an initial reasoning target vector; the decoder includes a knowledge verifier, a knowledge injector and a decoding module. The knowledge verifier is used to encode the knowledge stimulated by step 2, and use the obtained knowledge encoding and the reasoning target vector of the current reasoning step to verify and filter the knowledge to obtain the filtered knowledge; the knowledge injector is used to calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; Then, the inference target vector of the current inference step is updated by combining the two calculated attention weights; the decoding module is used to calculate the sign of the current inference step using the updated inference target vector of the current inference step, and generate the inference target vector of the next inference step; The knowledge system and the reasoning system form an intelligent problem-solving framework; the intelligent problem-solving framework is trained in advance, and the training loss function L is expressed as: L=∑ p ∑ t -log p(y t |y1,y2,...,y t-1 )+β·L v Among them, P represents the mathematical word problem to which the question text belongs; t represents that the current reasoning step is the tth step, y1, y2, ..., y t-1 represents the generated symbols, the subscripts correspond to the order of the reasoning steps, p(y t |y1,y2,...,y t-1 ) means generating the symbol y of the current reasoning step t under the condition of the symbols that have been generated t The probability of L v is the loss function of the knowledge verifier, and β>0 is a hyperparameter; When the knowledge verifier verifies and filters the knowledge, it generates the adaptation score p of the knowledge stimulated by step 2 to the current reasoning step t t , and the loss function calculated from this is expressed as: Among them, L v is the loss of the knowledge verifier, m is the number of reasoning steps, δ t ∈{0,1} is the true label at the current inference step t.
8. An intelligent mathematical problem-solving system integrating large model knowledge, characterized in that: include: An input unit, used to input question text; The knowledge system is used to stimulate the knowledge required for the current reasoning step based on the question text and the generated symbols based on the large language model; The reasoning system is used to encode the knowledge stimulated by step 2, verify and filter the knowledge using the obtained knowledge encoding and the reasoning target vector of the current reasoning step, obtain the filtered knowledge, and calculate the attention weight of the filtered knowledge in combination with the reasoning target vector of the current reasoning step, and calculate the attention weight of the question text in combination with the knowledge encoding and the text representation vector of the question text; Combine the two calculated attention weights to update the inference target vector of the current inference step; Combine the updated inference target vector of the current inference step to calculate the symbol of the current inference step and generate the inference target vector of the next inference step; If the current inference step is the first step, the inference target vector of the current inference step is the initial inference target vector obtained using the question text; Generate all symbols by repeated execution and combine them in order to form a solution expression; The solving unit is used to solve the answer by solving the expression.
9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Image reasoning question and answer method based on priori knowledge inspired large language model
CN116595151A
Answer model training method
CN117350383A
Method for intelligent dialogue based on machine reading comprehension, device, and terminal
WO2019242297A1
