A machine answering method based on deep learning

Abstract

The application belongs to the technical field of machine answering, and particularly relates to a machine answering method based on deep learning. The application applies deep learning to automatic solving of machine answering, improves an implicit feature pyramid reasoning network model, and uses the improved implicit feature pyramid reasoning network model for automatic solving. The automatic solving needs to be based on direct statement relations extracted from question data, implicit relations discovered and features of question data extracted, fully uses the question data, realizes automatic and fast solving of answers to questions, can be applied to solving practical problems such as machine answering of mathematical questions and physical circuit questions, and improves the machine answering capability.

Description

Technical Field

[0001] This invention belongs to the field of machine problem-solving technology, specifically relating to a machine problem-solving method based on deep learning. Background Technology

[0002] Machine-based problem-solving is a highly challenging research problem in the field of intelligent education, its challenge lying in the mechanization of intellectual work. The goal of machine-based problem-solving is to build an intelligent algorithm that reads in the problem in the most convenient way, automatically understands and solves it, and outputs the correct solution process and result. Machine-based problem-solving is a key technology for improving the intelligence level of education systems. It can effectively enhance the intelligence level of functions such as learning guidance, automatic grading, and automatic diagnosis, and its research results have significant application prospects in scientific research and education.

[0003] Early machine learning research primarily focused on the automatic proof of geometry problems, with problem understanding and automatic solution remaining semi-automated. In recent years, with the rapid development of artificial intelligence technology, automated, multimodal problem understanding and automatic solution have become new research priorities. Problem understanding is the most crucial aspect of machine learning research. It involves extracting information for solving the problem from all provided materials, including text, graphics, and images, and inputting this information into the automatic solution model. A good problem understanding method can significantly improve the efficiency of machine learning. Automatic solution is an important component of machine learning. Building upon problem understanding, it further reasons and calculates based on the acquired information, uncovering implicit information and deducing the internal logic of the solution process, thereby obtaining the solution process and result.

[0004] The machine-based problem-solving process can be mainly divided into two aspects: problem understanding and automatic solution. Automatic solution involves automating the reasoning and solving processes based on the problem understanding to arrive at the solution. This is not only a crucial step in machine-based problem-solving methods but also determines whether machine-based solutions can better guide students and serve teaching. Currently, automatic solution methods in the field of machine-based problem-solving mainly include the following three categories: framing methods, search-based reasoning methods, and algebraic methods.

[0005] 1) Frame Method. Kintsch first proposed the two-frame reasoning calculus solution process, which involves first obtaining a knowledge frame and a solution frame from the problem, and then performing reasoning calculus on the two frames to obtain the solution process. This research opened the prelude to using the frame method to achieve automatic problem solving, but strictly speaking, this solution process is only a set of guiding rules, not an algorithm in the strict sense. Ma et al. extended Kintsch's frame problem-solving theory and proposed a knowledge representation method based on a cognitive model, realizing the automatic solution of elementary arithmetic word problems from one step to multiple steps. This method first extends the semantic information representation model set by Kintsch for one-step addition and subtraction arithmetic word problems, and proposes a representation model that can represent multi-step word problems, while establishing a frame model for automatic solution; secondly, it proposes a natural language understanding method based on semantic sentence patterns for problem text understanding, which achieves problem text understanding by matching the problem text with semantic sentence patterns and representing them with a semantic information representation model; finally, it uses a frame-based automatic calculation method to perform reasoning calculus on the known conditions and knowledge base, thereby achieving automatic problem solving. This method initially achieved problem understanding and automatic solution for multi-step elementary arithmetic word problems. However, due to the existence of various complex natural language expressions, the construction of semantic sentence models will be a massive system engineering project. Subsequently, Kushman et al. developed a system for solving simple arithmetic word problems through verb classification. This system can automatically solve arithmetic word problems without pre-set templates, representing another implementation of the King's Framework problem-solving theory. The results of solving standard elementary arithmetic word problems show that the system achieved good results, but it can only automatically solve simple arithmetic word problems containing addition and subtraction operations.

[0006] 2) Search-based reasoning. Search-based reasoning primarily obtains a complete reasoning path to the problem-solving goal by continuously searching and matching rules with problem-solving information. Academician Zhang Jingzhong et al. first proposed a forward-reasoning-based search method for proving geometric theorems. This method searches the reasoning steps in the proof process until a forward indeterminate point is encountered, thus forming a complete reasoning path. Building on this, Pan et al. proposed a parallel forward-reasoning method for machine proof of geometric theorems, Jiang et al. proposed a heuristic search algorithm for readable proofs of polynomial equation-type geometric theorems, and Zhong et al. proposed a search-based reasoning method combining ontology and rules for proving plane geometric theorems. The advantages of search-based reasoning are: firstly, it can produce readable proofs; secondly, regardless of whether the final conclusion is correctly deduced, it can generate a large amount of reasoning information during the reasoning process, which can be used for heuristic learning and teaching. Its disadvantage is that the addition of a large amount of new knowledge during the reasoning process reduces the efficiency of the reasoning. Building upon search-based reasoning methods, Academician Zhang Jingzhong proposed a point-elimination method based on higher-order geometric invariant reasoning. This method eliminates construction points in the conclusions of geometric propositions by using search-based reasoning with geometric invariants, thereby proving geometric theorems. The point-elimination method is more concise and clear, enabling readable proofs, and each step in the proof process has a precise geometric meaning.

[0007] 3) Algebraic Methods. Early on, Academician Wu Wenjun and others proposed an algebraic method for proving elementary geometric theorems of equal form. This method is widely recognized as a landmark breakthrough in the field of automated proofs, leading to unprecedented activity in research on automated proofs of geometric theorems based on algebraic methods. Several successful new methods subsequently emerged, such as... Basic methods, numerical methods, and illustrative methods are among the methods used. Currently, most automatic solution processes based on algebraic methods boil down to automatically solving systems of equations. Koncel et al. explored the automatic solution of algebraic textual problems and proposed a method to resolve algebraic textual problems into systems of equations. Yu et al. proposed an automatic solution method that solves the linear part of a relational set step by step, and generated a readable solution process through semantic recovery of the solution process.

[0008] In summary, research on automated solution methods is still relatively weak. However, studying automated solution methods is an important aspect of improving machine problem-solving capabilities and promoting the application of machine solutions. Summary of the Invention

[0009] The purpose of this invention is to provide a machine learning-based solution method to achieve automatic and fast automatic solution in machine learning.

[0010] Therefore, this invention provides a machine learning-based solution method, comprising the following steps:

[0011] 1) Obtain the question data and extract the features B1,…,B1 of the directive relationships obtained directly from the question data. n To uncover the implicit relationships P1,…,P that are necessary for solving the problem data but are not directly expressed in the problem data. n And extract the feature Q of the question data included in the question data;

[0012] 2) Extract the features B1,…,B of the indicative relations. n Features of the implicit relationships discovered: P1, ..., P n The extracted problem data features Q are then input into the constructed improved implicit feature pyramid reasoning network model to generate the solution results for the problem data.

[0013] The improved implicit feature pyramid inference network model is used to first integrate the features B1,…,B of the directive relationship. n Features P1,…,P corresponding to the implicit relations n The results are added together and then subjected to an implicit function of nonlinear transformation to obtain the equilibrium feature P1. * ,…,P n * :P i * =G θ (P i +B i ), i = 1, 2, ..., n, G θ This represents a multilayer converter for nonlinear transformation; further, it incorporates the equalization feature P1. * ,…,P n * Characteristics of direct relation B1,…,B n The features Q of the problem are input into the relational reasoning network for reasoning and solving, generating the solution RN(A): RN(A) = F φ (∑ i,j P i * B j ,Q), j=1,2,…,n, where RN() represents the composite function of relational reasoning, A represents the feature of the solution, F φ This represents a multi-layered converter in a relational reasoning network.

[0014] Its beneficial effects are as follows: This invention applies deep learning to the automatic solution of machine problems, improves the implicit feature pyramid inference network model, and uses the improved implicit feature pyramid inference network model for automatic solution. During automatic solution, it needs to rely on the direct relationships extracted from the problem data, the implicit relationships discovered, and the features of the extracted problem data, making full use of the problem data and realizing the automatic and fast extraction of the answer to the problem. It can be applied to solving practical problems such as machine solution of mathematical problems and physics circuit problems, and improves the machine solution capability.

[0015] Furthermore, the question data is either unimodal or multimodal data. The unimodal data is either text data or image data, and the multimodal data includes both text data and image data.

[0016] Its beneficial effects are: it can process both single-modal and multi-modal data, and has strong universality.

[0017] Furthermore, in step 1), the constructed question understanding model is used to extract features of direct relationships, discover features of implicit relationships, and extract features of question data. The question understanding model is trained using a question dataset labeled with question labels, direct relationship feature labels, and implicit relationship feature labels.

[0018] Its beneficial effects are: as long as the question comprehension model is trained and constructed, it can quickly extract the features of direct relationships, discover the features of implicit relationships, and extract the features of question data. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall scheme of the problem-solving model of the present invention;

[0020] Figure 2 This is a feature learning framework diagram for multimodal data in this invention;

[0021] Figure 3 This is a diagram of the comparative learning framework for multimodal data in this invention;

[0022] Figure 4 This is a framework diagram for extracting declarative relationships according to the present invention;

[0023] Figure 5-1 This is a diagram of the intrinsic association learning framework based on declarative relations, which is adopted in mode 1 of the present invention;

[0024] Figure 5-2 This is a diagram of the intrinsic association learning framework based on declarative relations, which is adopted in mode 2 of the present invention.

[0025] Figure 6 This is a framework diagram for discovering implicit relationships in this invention;

[0026] Figure 7 This is a structural diagram of the implicit feature pyramid inference network model of the present invention. Detailed Implementation

[0027] The focus of this invention is on automatic problem solving in machine learning. After the problem understanding is completed, an improved implicit feature pyramid reasoning network model is designed to achieve automated reasoning and problem solving, so as to obtain the final solution result.

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0029] Examples of machine learning-based solution methods:

[0030] An embodiment of a machine learning-based solution method according to the present invention, the entire process of which is as follows:

[0031] Step 1: Obtain the question data. The question data can be unimodal data (image or text data) or multimodal data (image and text data). Extract the features B1,…,B1, which are direct declarative relationships obtained directly from the question data. n To uncover the implicit relationships P1,…,P that are necessary for solving the problem data but are not directly expressed in the problem data. n Then, the feature Q of the problem data included in the problem data is extracted. This step is actually the problem understanding process.

[0032] Adopting such Figure 1 The model shown implements question understanding. The entire question understanding model includes Text and / or Image embedding, Encoder, Multi-Modal Contrastive Learning, and Decoder.

[0033] like Figure 2 As shown, the text and / or image embedding is a feature embedding layer, including a text marker embedding layer and an image region embedding layer. The text marker embedding layer transforms the text into a set of text marker feature embedding sequences through position embedding, marker embedding, and task-learned embedding. Where E (Text Embedding) represents the text embedding marker, t represents the learning task embedding marker, [CLS] (Classification, CLS) represents the classification marker at the beginning of the sequence, and [SEP] (Separator, SEP) represents the separator marker between or at the end of the sequence. The image region embedding layer obtains the image region feature embedding sequence by fusing visual feature embedding, positional feature embedding, and learning task embedding. Where E (Image Embedding) represents the region feature embedding label of the image, i represents the learning task embedding label, and [IMG](Image,IMG) represents the classification label at the beginning of the sequence. If the question data is unimodal, the corresponding output is the text label feature embedding sequence. Or image region feature embedding sequence If the data in the question is multimodal, then the text tag feature embedding sequence and the image region feature embedding sequence are concatenated into a text-image pair feature embedding sequence.

[0034] Encoder and Decoder are the encoder and decoder for the multimodal, multi-task model UniT, such as Figure 2 As shown, the Encoder includes a text encoder and an image encoder. The text encoder learns contextualized representations of text features. The image encoder then outputs the learning of the contextual region representation of the image. And output it.

[0035] like Figure 3 As shown, Multi-Modal Contrastive Learning is a multi-modal contrastive learning model. This model first transforms the feature representations of text and / or images into the same embedding space through a fully connected layer, and then calculates their cosine similarity to measure the distance d(V,W). Specifically, it first randomly selects examples, such as randomly selecting examples corresponding to images. Examples corresponding to the text Two related examples are generated for each example by using two independent augmentation functions; then, a basic neural network encoder f(·) is used to extract representation vectors from the augmented data. The extracted representation vector is The extracted representation vector is Again, a neural network projection head g(·) is used to map the extracted vectors onto the contrastive loss space, thereby obtaining the result. corresponding and with corresponding Finally, a contrastive loss function is trained for the contrastive prediction task, which attracts relevant examples to each other and repels irrelevant examples to each other, in order to achieve semantic alignment of multimodal data.

[0036] like Figure 4 and Figure 6As shown, the Decoder uses a shared decoder to decode the input according to each task to obtain implicit features, and obtains the output features of each task based on the specific output header of each task. The tasks include a direct relation extraction task and a latent relation discovery task. For the direct relation extraction task, direct relation features are directly extracted from the question data. For the latent relation discovery task, the latent relation features of the question data need to be inferred from the question data using a latent relation library.

[0037] Having obtained the indicative relations, it is necessary to construct an intrinsic association learning model to associate the high-level semantics of the indicative relations. This involves semantically associating the implicit features of each indicative relation with the implicit features related to that indicative relation at a high level. Since the same modality contains different indicative relations, and the same indicative relation may be expressed in different modalities, two association methods can be used in the intrinsic association model: the first is to first associate the features of the same indicative relation across different modalities, and then associate the implicit features of different indicative relations at a high level, such as... Figure 5-1 As shown; the second method involves first associating the implicit features of different indicative relations for each modality, and then associating the implicit features of indicative relations between different modalities at a higher level, such as... Figure 5-2 As shown, this can solve the problem of the lack of inherent connection between declarative relationships.

[0038] Of course, the multimodal multitasking UniT model is used in this step. As another implementation, other multimodal multitasking models in the prior art, such as the MURAL model, can also be used.

[0039] Step 2, extract the features B1,…,B of the declarative relations. n Features of the implicit relationships discovered: P1, ..., P n The extracted problem data features Q are then input into the constructed improved implicit feature pyramid inference network model to generate the solution results for the problem data.

[0040] This step involves obtaining the explicit and implicit relations in the multimodal data. In order to correctly answer the question, we need to reason about the implicit relations. That is, starting from the implicit relations discovered in the question, we combine the obtained explicit and implicit relations to reason layer by layer and calculate the numerical solutions in each layer, thereby reasoning about and solving the question.

[0041] like Figure 7The diagram shows the constructed improved Implicit Feature Pyramid Network (IFPN) inference model, which is used for inference and solution. Specifically, firstly, the features P1,…,P1 of the implicit relationships discovered in step one are... n The extracted indicative relations are characterized by B1, ..., B1. n The features Q of the extracted problem data are input into IFPN. When inputting into IFPN, the features B1,…,B of the indicative relations are first input into IFPN. n Features P1,…,P corresponding to the implicit relations n The addition process is performed; then, the implicit function of the nonlinear transformation in IFPN is constructed using the nonlinear components and cross-scale connections introduced by the deep equalization model, and the result of the addition process is passed through the implicit function of the nonlinear transformation to obtain the equalization feature P1. * ,…,P n * :P i * =G θ (P i +B i ), where G θ P represents a multilayer converter for nonlinear transformation, indicating the process of using a multilayer converter to handle nonlinear transformations. i As a feature of implicit relationships, B i The characteristics of the directive relation are then determined; next, the equilibrium characteristic P1 of the implicit function is solved using the equilibrium characteristic solver. * ,…,P n * Finally, the equilibrium feature P1 * ,…,P n * Characteristics of direct relation B1,…,B n The features Q of the problem are input into the relational reasoning network for reasoning and solving, thereby generating the solution to the problem, i.e., RN(A) = F. φ (∑ i,j P i * B j ,Q), where RN represents the composite function of relational reasoning, A is the feature of the solution, and F φ This is a multi-level converter for relational feature reasoning, representing the process of using a multi-level converter to handle summation. Q represents the feature of the problem, and P... * For balanced features, B represents the features of indicative relationships. It should be noted that the feature Q of the question data is processed together with the question text in the Encoder, except that it is given an additional question label, and there is generally only one question.

[0042] The machine learning-based solution method of this invention extracts the features B1,…,B1 of the declarative relations. n Discover the features of implicit relationships P1,…,P n Based on the extracted features Q of the problem data, and using an improved implicit feature pyramid inference network model, the answer to the problem can be inferred regardless of whether the problem data is unimodal image data or text data, or multimodal image data and text data. This is crucial for expanding the scope of problem-solving and improving the quality of problem-solving, and has important theoretical research significance and academic research value.

[0043] The methods described above can be implemented using computer program code. For example, a machine learning-based solution device can be designed, comprising a memory, a processor, and an internal bus. The processor and memory communicate and interact with each other via the internal bus. The processor executes computer execution instructions stored in the memory to implement the deep learning-based machine learning solution method of this invention. In other words, it should be understood that the method described in this embodiment can be implemented using computer program instructions to follow the flow of a deep learning-based machine learning solution method.

[0044] Specifically, the processor can be a microprocessor (MCU), a programmable logic device (FPGA), or other processing device, and can run various operating systems, such as Windows or Linux. The memory can be any type of memory that stores information using electrical energy, such as RAM or ROM; it can also be any type of memory that stores information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; it can also be any type of memory that stores information using optical methods, such as CDs and DVDs; and of course, it can also be other types of memory, such as quantum memories and graphene memories.

Claims

1. A machine learning-based solution method, characterized in that, include: 1) Obtain unimodal or multimodal question data, and use a question comprehension model to extract features of direct relationships obtained directly from the question data. To uncover the implicit relationships in the data that are necessary to solve the problem but are not directly expressed in the data. And extract the features of the question data included in the question data. ; Unimodal data consists of text or image data, while multimodal data includes both text and image data. The question comprehension model includes a feature embedding layer and a multimodal contrastive learning model. If the data is unimodal, the output is a text-labeled feature embedding sequence. or image region feature embedding sequence If it is multimodal data, then... and These are concatenated into a sequence of image-text pair feature embeddings; a multimodal contrastive learning model randomly extracts examples corresponding to images and examples corresponding to text, and trains a contrastive loss function through a contrastive prediction task, attracting related examples to each other and repelling irrelevant examples to achieve semantic alignment of the multimodal data; in obtaining Based on this, an intrinsic association learning model is constructed to associate high-level semantics of indicative relations. The association method of the intrinsic association model is any one of the following: the first method is to first associate different modal features of the same indicative relation, and then associate the implicit features of different indicative relations at a high level; the second method is to first associate the implicit features of different indicative relations of each modality, and then associate the implicit features of indicative relations of different modalities at a high level. 2) Extracted Excavated And the extracted The input is fed into the constructed improved implicit feature pyramid reasoning network model to generate the solution results for the question data; Among them, the improved implicit feature pyramid inference network model is used to first... With the corresponding The sum is then processed by addition, and the result is subjected to an implicit function of nonlinear transformation to obtain the equilibrium characteristics. : , , A multilayer converter representing a nonlinear transformation; and further... , and The input is fed into a relational reasoning network for inference and solution generation to produce the solution results. : , , The composite function representing relational reasoning. This indicates the characteristics of the solution. This represents a multi-layered converter in a relational reasoning network.

2. The machine learning-based solution method according to claim 1, characterized in that, The encoder and decoder in the problem comprehension model are the multimodal, multi-task model UniT.

3. The machine learning-based solution method according to claim 1, characterized in that, The question understanding model used in step 1) is trained using a question dataset with question labels, explicit relation feature labels, and implicit relation feature labels.

Images