A question understanding method for machine answering

Abstract

The present application belongs to the technical field of machine answering, and particularly relates to a question understanding method for machine answering. The present application firstly realizes feature extraction and unified semantic expression of single-mode / multi-mode data through a feature embedding module, an encoder and a contrast learning module, then defines direct statement relations and implied relations, extracts direct statement relations directly shown in the question in the question understanding process to realize fusion understanding of data, and reasons out implied relations required for answering the question, thereby laying a foundation for automatic solution of the question. The present application establishes a basic theory of machine answering, breaks through key technologies of machine answering, is very crucial for expanding a problem solving range and improving problem solving quality, has important theoretical research significance, academic research value and broad application prospect, and meets application services of machine answering.

Description

Technical Field

[0001] This invention belongs to the field of machine problem-solving technology, specifically relating to a problem understanding method for machine problem-solving. Background Technology

[0002] In recent years, driven by the combined forces of market demand in educational technology and the rapid development of artificial intelligence technologies such as natural language understanding, image understanding, and automated reasoning, machine problem-solving has become a new research hotspot. Currently, research on machine problem-solving methods for multimodal questions is still a relatively new field, and its research is in the dual exploration stage of models and applications. The main difficulties and challenges can be summarized in the following three aspects:

[0003] 1) Machine-solved multimodal problems exhibit heterogeneity, and the data within each modality lacks semantic coherence. To clarify the issues in the problems more clearly, vividly, and realistically, problems across disciplines are often presented in multimodal formats such as text, graphics, and images. This leads to heterogeneity among the different modalities and poses a challenge to the unified representation of multimodal problems. Multimodal problems also contain a large amount of multimodal data, including text, symbols, formulas, and graphics. The differences in statistical characteristics among these modalities result in a lack of semantic coherence. Different modal expressions of the same entity contain not only shared information but also unique information. Failure to semantically correlate this information will affect the accuracy of multimodal data understanding, thereby hindering problem comprehension and automatic solution. Therefore, how to construct a feature representation mechanism for multimodal data, simultaneously addressing the heterogeneity between multimodal problems and the semantic coherence between different modalities, and achieving feature extraction and unified semantic representation of multimodal data, is the first challenge that machine-solved multimodal problems need to overcome.

[0004] 2) The machine-aided multimodal understanding process suffers from a semantic gap. Understanding multimodal questions is not based on the understanding of underlying text and image features, but rather on the semantic understanding of the objects or events described by the text and images. Multimodal questions are described in the form of text, graphics, and images. Feature representation of these multimodal questions is the foundation of multimodal understanding, while semantic understanding requires extracting the directly stated information. Furthermore, the directly stated information extracted from each modality describes the same object or event from different perspectives, and they lack inherent connections. Therefore, how to construct an understanding model for multimodal data that can effectively extract directly stated information from multimodal data and learn their inherent connections, thereby bridging the semantic gap and improving the quality and effectiveness of multimodal understanding, is the second challenge that machine-aided multimodal question solving needs to address.

[0005] 3) The multimodal problem-solving process in machine learning lacks the discovery and reasoning of implicit information. To correctly solve a problem, it's necessary not only to extract the directly stated information but also to uncover implicit information by combining theorems, laws, and reasoning from various disciplines. This implicit information, though not directly stated in the problem, is essential for automatic solution. Discovering and adding implicit information from problems is a generative problem; the required implicit information should be provided by an implicit information database. Therefore, establishing an implicit information database containing theorems, laws, reasoning, and common sense is a fundamental task. Since the knowledge scope involved in machine-solving problems across different disciplines is defined, the implicit information related to those disciplines constitutes a definite set. The multimodal problem-solving process needs to integrate the directly stated and implicit information from the problem into a unified multimodal data reasoning and solution model, enabling them to work together to correctly solve the problem. Therefore, how to construct a reasoning and problem-solving model for multimodal data that can effectively uncover the implicit information in multimodal data and perform reasoning and problem-solving to achieve automatic solution of multimodal problems is the third challenge that machine solutions for multimodal problems need to address.

[0006] The machine problem-solving process can be mainly divided into two aspects: problem understanding and automatic solution. Problem understanding is the foundation of automatic solution and a key step in exploring machine problem-solving methods. For computers, problem understanding involves extracting all the solution information contained in the problem and applying it to the reasoning and problem-solving model to obtain the solution process and result. Currently, a good problem understanding method can not only expand the scope of problem understanding but also improve the accuracy of reasoning and problem-solving. At present, problem understanding methods in the field of machine problem-solving can be mainly divided into three categories: text-based problem understanding methods, graph-based problem understanding methods, and multimodal data-based problem understanding methods.

[0007] 1) Text-based Question Understanding Methods. Question text is also a form of natural language, and it can be understood through lexical, syntactic, semantic, and dependency relation analyses within the field of natural language processing. Text-based question understanding methods can be broadly categorized into three types: semantic role-based methods, methods combining part-of-speech tagging to extract mathematical relationships, and methods using machine learning classification to obtain equation systems. ① Semantic Role-Based Methods: These methods understand the meaning of the question text through semantic role annotation and keywords, thereby deriving the knowledge and relationships within the question. Methods include sentence model double-frames, sentence model mathematical expressions, parse tree mathematical formulas, language analysis trees, and paradigmatic language mathematical formulas. The commonality of these methods is that they use tools such as defining semantic sentence models, parse trees, language analysis trees, and paradigmatic languages ​​to understand the semantic roles of the question text. The advantage of semantic role-based methods is that the process is easy to understand; however, they generally lack generalization ability. The main reason is that these methods all encounter the curse of semantic variability, i.e., the difficulty in generalization caused by the diverse expressions of the same mathematical meaning in natural language, resulting in a large number of semantic role templates but a small range of comprehensible questions. Furthermore, none of these methods emphasize the ability to uncover implicit information. ② A method combining part-of-speech variation to extract mathematical relations: This method uses a set of mathematical relations as the target form for question comprehension and employs a heuristic algorithm using a syntactic-semantic model to extract directly stated mathematical relations (referred to as: declarative relations). Simultaneously, it combines bag-of-words and support vector machines to construct and discover implicit mathematical relations (referred to as: implicit relations). This combination forms a new method that can extract both declarative and implicit relations. However, this method also has limitations: the syntactic-semantic model is limited by natural language word classification methods, lacks a process for optimizing declarative relations, and has limited ability to discover implicit relations. ③ A method using machine learning classification to extract equation sets: Existing machine learning-based question comprehension methods generally involve taking the question text as input, classifying and assigning variables through machine learning, and finally outputting a set of equations. This type of machine learning method has two limitations: first, the expressive power of existing equation set templates is limited, and this method still cannot obtain satisfactory question comprehension results for various types of questions; second, these deep networks do not embed domain knowledge to improve question comprehension capabilities.

[0008] 2) Graphical Data-Based Problem Understanding Methods. In education, graphics are a common form of problem representation. They can intuitively describe the objects stated in the problem and the relationships between them. This visual expression can effectively convey the question setter's thoughts, intentions, and the questions being tested. Currently, graphic understanding problems mainly focus on geometry and circuit problems. Graphical understanding methods can be broadly divided into graphic drawing-based understanding methods and graphic recognition-based understanding methods. ① Graphic drawing-based understanding methods: This involves using drawing tools to draw the graphics in the problem. During the drawing process, the drawing tools acquire the positional information and relationships of symbols such as points, lines, and components. Once the graphics are drawn, the understanding of the graphics is achieved. These methods lack understanding of complex sentence structures, and the natural language understanding models are pre-designed and lack self-learning capabilities, requiring manual addition of new vocabulary and sentence structures. ② Graphic recognition-based understanding methods: This involves using image processing technology to recognize graphics, and then analyzing the entities, structures, and relationships within the graphics to extract useful problem-solving information. The difficulty of this method lies in how to design a graphic understanding scheme to analyze and mine the recognized graphics to obtain effective problem-solving information. Image recognition-based methods typically use Hough transform techniques to identify basic entities in an image, then analyze the relationships between these entities to obtain problem-solving information. Traditional methods include using gradient Hough transform to identify circles, using improved asymptotic probability Hough transform to identify lines, and combining Hough transform with random detection algorithms to improve circle recognition accuracy. These methods can effectively recognize standard images generated by drawing tools; however, their ability to recognize hand-drawn images is limited because they struggle to identify imprecise instances of lines and circles. The advantage of using Hough transform for image recognition and analysis is that it allows the use of mature image recognition techniques for image understanding. However, these methods have two limitations: first, using single image data for image understanding is not ideal, and there is a lack of new methods to integrate image and text data to improve image understanding; second, they are limited to processing two-dimensional planar images and cannot achieve satisfactory image understanding results for three-dimensional images such as solid geometry.

[0009] 3) Problem Understanding Methods Based on Multimodal Data. Machine-solved multimodal problems not only contain natural language descriptions of the problem text but also corresponding graphics, symbols, and formulas, such as circuit diagrams, geometric figures, and chemical formulas. For some complex problems, text sometimes struggles to describe structural, relational, and ambiguous information, while graphics can provide a more intuitive expression and analysis. Examples include geometric analysis based on auxiliary lines and node analysis of circuit diagrams. Text and graphics complement each other in problem expression; therefore, many problems are typically expressed using both text and graphics modalities. For these text-graphic problems, both text and graphics contain a portion of the problem's meaning. The problem text uses natural language to describe phenomena in a real-world context and poses the problem to be solved, while graphics express objects and their relationships in a more intuitive and clear way, effectively eliminating ambiguity in the text. Therefore, combining text and graphics in problem understanding is an important task.

[0010] In summary, each of the current problem-solving methods has its advantages and disadvantages, but none of them can achieve the best problem-solving effect. Furthermore, due to the large number of tasks that machines need to solve, current machine learning methods cannot achieve the integration of multimodal problem-solving and automatic problem-solving. Moreover, the problem-solving process in machine solutions cannot generate a readable and understandable process, which still falls short of the requirements of intelligent education applications. Summary of the Invention

[0011] The purpose of this invention is to provide a problem understanding method for machine solving, in order to solve the problem of poor problem understanding effect caused by existing problem understanding methods.

[0012] To address the aforementioned technical problems, this invention provides a method for understanding questions in machine-aided problem-solving, comprising the following steps:

[0013] 1) Obtain question data, wherein the question data is multimodal data or unimodal data, wherein the unimodal data is text data or image data, and the multimodal data includes text data and image data;

[0014] 2) Input the question data into the constructed question comprehension model to extract the features of direct statements and infer the features of implicit statements in order to complete the question comprehension; the direct statements are the relationships directly obtained from the question data, and the implicit statements are the relationships required to solve the question data but not directly stated in the question.

[0015] The question understanding model includes a feature embedding module, a multi-layer self-attention converter, and a contrastive learning module. The multi-layer self-attention converter includes an encoder and a decoder.

[0016] The feature embedding module is used to perform feature embedding processing on the input question data: if the input question data is unimodal data, then a text tag feature embedding sequence or an image region feature embedding sequence is obtained accordingly; if the input question data is multimodal data, then a text tag feature embedding sequence and an image region feature embedding sequence are obtained accordingly, and the two sequences are connected into a text-image pair feature embedding sequence.

[0017] The encoder is used to encode the question data after feature embedding using the included self-attention masking mechanism, thereby obtaining the context label representation of text features and / or the context region representation of image regions.

[0018] The contrastive learning module is used to semantically align the context label representation and / or context region representation and map them to a unified semantic space, thereby obtaining the corresponding text feature sequence and / or image feature sequence.

[0019] The decoder is used to use a shared decoder to decode the input according to each task to obtain implicit features, and to obtain the features output by each task based on the specific output header of each task; wherein, each task includes a declarative relation extraction task and an implicit relation reasoning task. The declarative relation extraction task is used to extract declarative relation features from the question data, and the implicit relation reasoning task is used to infer the implicit relation features of the question data from the question data in combination with an implicit relation library.

[0020] Its beneficial effects are as follows: This invention first achieves feature extraction and unified semantic representation of single-modal / multimodal data through a feature embedding module, encoder, and contrastive learning module. Then, it defines explicit and implicit relations. During the problem understanding process, it not only extracts the explicit relations intuitively presented in the problem to achieve data fusion understanding, but also infers the implicit relations required to solve the problem, laying the foundation for automatic problem solving. This invention establishes the fundamental theory of machine problem solving, breaks through key technologies in machine problem solving, and is crucial for expanding the scope of problem solving and improving the quality of problem solving. It has significant theoretical research significance, academic research value, and broad application prospects, meeting the application service needs of machine problem solving.

[0021] Furthermore, the multi-layer self-attention converter adopts a multimodal multitasking UniT model.

[0022] Its beneficial effects are as follows: the UniT model adopted can jointly learn multiple tasks across different domains through a unified transformer, and can process multiple tasks on multiple datasets at the same time. It achieves powerful performance for each task through compact model parameters.

[0023] Furthermore, the encoder includes a text encoder that encodes text data after feature embedding and / or an image encoder that encodes image data after feature embedding. The text encoder is trained alternately using two methods: bidirectional prediction and sequence-to-sequence generation. The image encoder is trained using feature regression or region classification.

[0024] Its beneficial effects are as follows: For text encoders, the alternating training method of bidirectional prediction and sequence-to-sequence generation can learn better contextual representations of text; for image encoders, the training method of feature regression or region classification can learn better contextual representations of images.

[0025] Furthermore, the contrastive learning module includes an enhancement function, a neural network encoder, and a neural network projector head; the enhancement function is used to generate two correlated enhancement data for each context label representation / context region representation using two independent enhancement functions; the neural network encoder is used to extract representation vectors from the enhancement data; and the neural network projector head is used to map the representation vectors to a contrastive loss space to obtain a text feature sequence / image feature sequence.

[0026] Its beneficial effects are as follows: In order to model the feature extraction of multimodal questions and have the ability to process various types of data, the question understanding model establishes a feature learning module for multimodal data, performs self-attention masking on text features and image features in multimodal data respectively, and realizes the interaction between multimodal data through multi-layer transformers, thereby generating better feature representation of multimodal data and supporting downstream multimodal tasks.

[0027] Furthermore, when training the contrastive learning module, it is necessary to make relevant context label representations and / or context regions attractive to each other, and irrelevant context label representations and / or context regions repulsive to each other.

[0028] Its beneficial effect is that during training, related text features and / or image features attract each other in the representation space, while unrelated features repel each other in the representation space, thereby achieving semantic alignment of multimodal data.

[0029] Furthermore, when training the contrastive learning module, if it is contrastive learning of the same modality data, the text rewriting method is used to semantically rewrite the single modality data to generate more than a certain number of negative samples, and the positive and negative samples are used for training; if it is contrastive learning of cross-modality data, the text rewriting method is used to semantically rewrite the multimodality data to generate more than a certain number of positive and negative samples.

[0030] Its beneficial effects are as follows: generating positive and negative samples by using semantic rewriting can help the model learn more detailed semantic alignment from different levels of images or text, learn more accurate semantic alignment of text or image features, and thus achieve unified semantic feature representation of multimodal data.

[0031] Furthermore, when training the decoder, the high-level semantics of indicative relations need to be associated, and either of the following two association methods should be used: Association Method 1: First associate the features of different modalities of the same indicative relation, and then associate the implicit features of different indicative relations at a high level; Association Method 2: First associate the implicit features of different indicative relations of each modality, and then associate the implicit features of indicative relations of different modalities at a high level.

[0032] Its beneficial effects are as follows: This invention constructs an intrinsic association learning model to associate the high-level semantics of indicative relations, that is, to semantically associate the implicit features of each indicative relation with the implicit features related to that indicative relation at a high level, which can solve the problem of the lack of intrinsic association between indicative relations.

[0033] Furthermore, if the input question data includes text data, then when the feature embedding module performs feature embedding processing, it needs to perform position embedding, label embedding, and learning task embedding on the text data, so that the resulting text label feature embedding sequence W is represented as: Where E represents the text embedding marker, t represents the learning task embedding marker, [CLS] represents the classification marker at the beginning of the sequence, and [SEP] represents the separator marker between or at the end of the sequence; w1,…,w n This represents each word segmentation marker in the text data; if the input question data includes image data, then when the feature embedding module performs feature embedding processing, it needs to fuse visual features, location features, and learning task embedding for the image data, thereby obtaining the image region feature embedding sequence V, represented as follows: Where E represents the image embedding label, i represents the learning task embedding label, [IMG] represents the classification label at the beginning of the sequence, and v1,…,v n This indicates the labels for different regions in the image data.

[0034] Furthermore, if the input question data includes image data, the feature embedding module needs to encode the positional features of each region of the image before performing feature embedding processing, using a 5-dimensional vector. The encoding is performed, where (x1,y1) and (x2,y2) represent the coordinates of the lower left and upper right corners of the region, respectively, and W and H represent the width and height of the input image, respectively. Attached Figure Description

[0035] Figure 1 This is a general framework diagram of the problem understanding method for machine-based problem solving according to the present invention;

[0036] Figure 2 This is a schematic diagram of the overall scheme of the problem understanding method for machine solving according to the present invention;

[0037] Figure 3 This is a feature learning framework diagram for multimodal data in this invention;

[0038] Figure 4 This is a feature learning framework diagram for single-mode text data according to the present invention;

[0039] Figure 5 This is a feature learning framework diagram for single-mode image data according to the present invention;

[0040] Figure 6 This is a diagram of the comparative learning framework for multimodal data in this invention;

[0041] Figure 7 This is a diagram of the indicative relation extraction framework based on multimodal learning of the present invention;

[0042] Figure 8(a) is a diagram of the intrinsic association learning framework based on declarative relations using the first method of the present invention;

[0043] Figure 8(b) is a diagram of the intrinsic association learning framework based on declarative relations using the second approach of the present invention;

[0044] Figure 9 This is a diagram of the implicit relationship discovery framework based on multimodal learning of the present invention. Detailed Implementation

[0045] This invention utilizes a deep learning network to construct a question understanding model. After training, the acquired question data is directly input into the constructed question understanding model to obtain the explicit and implicit relationships embodied in the question data, thereby achieving question understanding. This question understanding model is used to realize three levels of content, including feature representation of multimodal data, fusion understanding of multimodal data, and reasoning and solving of multimodal data. Feature representation of multimodal data is the foundation of this invention. It primarily addresses the heterogeneity between multimodal questions and the semantic correlation between different modalities by establishing a feature representation mechanism. This enables feature extraction and unified semantic representation of multimodal data, providing a foundation for further research. The fusion and understanding of multimodal data is a key issue. It utilizes a multimodal learning model to fuse and understand the features of the aforementioned multimodal data, extracting direct relationships. Addressing the lack of inherent correlation between these direct relationships, a correlation learning model is established to achieve multimodal data understanding. The research results provide theoretical and algorithmic reserves for understanding multimodal questions in machine learning and offer technical support for further research. Reasoning and solving multimodal data is the focus of this invention. Due to the characteristics of machine learning applications, simply extracting direct relationships from multimodal data is insufficient for accurate problem solving; it is necessary to uncover implicit relationships within the multimodal data. This layer utilizes a multimodal learning model combined with an implicit relationship database to discover implicit relationships within multimodal data.

[0046] It should be noted that the direct relationship here refers to the content directly stated in the question, while the implicit relationship refers to the content not directly stated in the question but necessary to solve the problem. For example, if the question stem states that the length, width, and height of a cube are 5 cm, 3 cm, and 8 cm respectively, and the question asks for the volume of the cube, the direct relationship for this question is the specific values ​​of the length, width, and height, while the implicit relationship is the formula for the volume of the cube needed to solve the problem, i.e., volume = length × width × height.

[0047] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] Example of a problem comprehension method for machine-based solutions:

[0049] like Figure 1 The diagram shows the overall framework of the problem-solving model of this invention. It includes the following parts: 1) Feature representation of multimodal data, including feature learning and comparative learning of multimodal data. 2) Fusion understanding of multimodal data, including indicative relation extraction based on multimodal learning and intrinsic association learning based on indicative relations. 3) Reasoning and solving of multimodal data, including implicit relation discovery based on multimodal learning. Correspondingly, each part is explained through... Figure 2The network structure shown is implemented below. The details of each part are explained in conjunction with the accompanying diagram.

[0050] I. Feature representation of multimodal data.

[0051] 1. Feature learning of multimodal data.

[0052] To model feature extraction for multimodal questions and enable the processing of various modalities of data, this invention establishes a feature learning model for multimodal data. Figure 2 The Feature Representation method performs self-attention masking on text and image features in multimodal data, and realizes the interaction between multimodal data through the decoder of a multi-layer self-attention converter, thereby generating better feature representation of multimodal data and supporting downstream multimodal tasks.

[0053] like Figure 3 As shown, for text and image inputs of multimodal question data, firstly, the feature embedding module of single-modal data is used. Figure 2 The feature embedding module (text and / or image embedding) includes a text marker embedding layer and an image region embedding layer, which are used to obtain the text marker feature embedding sequence, respectively. and image region feature embedding sequence Secondly, depending on whether the learning task is unimodal (i.e., text-based or image-based questions) or multimodal (image-text pair questions), it is determined whether to concatenate the two sequences into a single image-text pair feature embedding sequence. Specifically, in multimodal tasks, two sequences need to be concatenated into a single image-text pair feature embedding sequence; finally, the feature embedding sequence is input into the encoder of a multilayer self-attention converter. Figure 2 The encoder in the model (including text encoders and image encoders, which will be described in detail below) learns multimodal feature representations of image regions and text tags. The unimodal representation is: the contextual label representation of text features. Or the context region representation of the image region Multimodal representation as The feature representation is then output. Specifically, considering that different modalities of learning tasks may require extracting different types of features, the learning task is also embedded into a vector w. task This is added to the encoder of the multi-layer self-attention converter to allow it to extract task-specific information at output. It should be noted that the parameter definitions in this paragraph will be explained in detail below.

[0054] It should be noted that the multilayer self-attention converter of the present invention (including Figure 2The Encoder and Decoder in this model use the existing multimodal multitasking UniT model. This multi-layer self-attention converter, based on image region embedding layers and text tag embedding layers, utilizes a multi-layer converter (…). Figure 3 The N Transformer Blocks in the model implement a single data stream interaction, with each transformer layer using a self-attention masking mechanism. Figure 3 Multi-head Self Attention and point-based feedforward neural networks (in the context of multi-head self-attention and point-based feedforward neural networks) Figure 3 The model consists of a Feed Forward algorithm and selects GeLU as the activation function. Furthermore, this model is structurally identical to the BERT model, so its weights can be initialized using pre-trained BERT weights, thus improving the usability of the original pre-trained model. Specifically, this section will use a multi-layer self-attention transformer for both text and image encoding to encode features of the question text and image, respectively, thereby generating better contextual feature representations.

[0055] 1) Text encoder.

[0056] like Figure 4 As shown, similar to the pre-trained natural language processing model BERT and its improved models, let the text input in a multimodal question be a sequence of words w = {w1, ..., w...} n First, the text tag embedding layer transforms it into a set of text tag feature embedding sequences through position embedding, tag embedding, and learning task embedding. Where E (Text Embedding) represents the text embedding token, t represents the learning task embedding token, [CLS] (Classification, CLS) represents the classification token at the beginning of the sequence, and [SEP] (Separator, SEP) represents the separator token between or at the end of the sequence. Then, a text encoder with a self-attention masking mechanism is used to train the text's tokenized features to learn the contextual tokenized representation of the text features. The contextual label is then output. Specifically, the learning task embedding vector is used. As part of the BERT input, it is added to the text tag feature embedding sequence, and in downstream tasks, it can be extracted from the implicit features of the output text. Its purpose is to label various downstream training tasks and share implicit features, such as: declarative relation extraction, implicit relation discovery, and reasoning solution.

[0057] To better learn contextualized representations of text features, this invention trains an encoding model for a question text using two types of language modeling tasks: Bidirectional Prediction (BP) and Sequence-to-Sequence (Seq2Seq). This model utilizes a specific self-attention masking mechanism to control the contextual content of the prediction conditions. Simultaneously, to improve the language learning process, this invention first uses a syntactic-semantic model to detect semantically complete phrases in the question text, such as name entities, and then treats them as a whole in the subsequent masking strategy. During the training of BP and Sequence-to-Sequence (Seq2Seq), a sequence of complete words or phrases is always sampled, rather than word tags. Throughout the training process, this invention will uniformly alternate between the BP and Sequence-to-Sequence (Seq2Seq) objectives to ultimately obtain the contextualized representations of text features.

[0058] ① Bidirectional prediction: For text tag feature embedding sequences This invention performs iterative span sampling from the sequence until 12% of the tokens are selected. The word span is randomly selected from a geometric distribution of l to Geo(p), where l... max =10, p=0.2, all markers in the selected region will be replaced by a masked marker, a random marker, or the original marker with probabilities of 80%, 10%, and 10%, respectively. The goal is to adjust the value based on the surrounding context w. \m To predict the mask marker w m That is, using the minimization of the negative log-likelihood function for prediction: L BP (θ)=-E w∈D log P θ (w m |w \m ), where θ is a trainable parameter, and each word or phrase w is sampled from the training set D.

[0059] ② Sequence-to-sequence generation: For text tag feature embedding sequences This invention performs iterative fragment sampling from the sequence until 25% of the budget is spent. For each iteration, a fragment length is first selected from a range uniformly distributed in the range l to U(4,32), and then fragment sampling is performed using this length. Each selected fragment... The start and end markers [CLS] and [SEP] will be added to form the sequence. All selected segments are removed from the text and concatenated to form the target sequence T, while the remaining parts of the text are concatenated to form the source sequence S. Finally, the following model is trained on the source sequences to generate the target sequence under autoregressive conditions: LSeq2Seq (θ)=-E (S,T)∈D log P θ (T|S), where θ is a trainable parameter, and both the target sequence and the source sequence (S,T) in the text are sampled from the training set D.

[0060] 2) Image encoder.

[0061] like Figure 5 As shown, this invention uses the visual regions of an image as input. First, it employs a circuit pattern recognition method based on Faster-RCNN to detect the Regions of Interest (ROIs) of the image and extract their visual features. Since the self-attention mechanism in the multi-layer converter of the next module is unordered, a 5-dimensional vector will be used. The positional features of each region are encoded, where (x1, y1) and (x2, y2) represent the coordinates of the bottom-left and top-right corners of each region, respectively, and W and H represent the width and height of the input image, respectively. A region feature embedding sequence is obtained by fusing visual feature embedding, positional feature embedding, and learning task embedding. Where E (Image Embedding) represents the image region feature embedding label, i represents the learning task embedding label, and [IMG](Image,IMG) represents the classification label at the beginning of the sequence; then, an image encoder with a self-attention masking mechanism is used to train the image region features to learn a more realistic image context region representation. The context region representation is then output. Specifically, the task embedding vector is learned. When added to the region feature embedding sequence, it can be extracted from the implicit features of the output image in downstream tasks, i.e.: Its purpose is to label various downstream training tasks and share implicit features, such as learning tasks like object detection, visual entailment, and relational reasoning.

[0062] Similar to the self-attention masking modeling for text features described above, the self-attention masking model for image features samples image regions and masks their visual features with a 15% probability. The visual features of the masked regions are replaced with 0. Image regions are often highly overlapping. To avoid information leakage, all regions with a high proportion of overlap are selected for masking. Meanwhile, regions are randomly selected as masking anchors, and regions with an anchor borrow greater than 0.3 are masked. The goal is to mask from other regions v. \m Reconstruction of the covered area v m :L V (θ)=E v∈D f θ (v m|v \m ), where θ is a trainable parameter, and each image region v is sampled from the training set D. Unlike text tags represented by discrete labels, visual features are high-dimensional and continuous, and cannot be supervised learning using likelihood functions. Therefore, this invention will use two types of image modeling tasks, Feature Regression (FR) and Region Classification (RC), to train an encoding model for a title image, thereby learning a better representation of the image's contextual regions.

[0063] ① Feature Regression: First, use the function L(θ) = E v∈D f θ (v m |v \m The masked regions of the image are reconstructed, where θ are trainable parameters, and each image feature v is sampled from the training set D. Feature regression learns the transformed output for each masked region. To its visual features The regression, that is: Where r represents a fully connected layer used to process visual features Transform into a with Vectors of the same dimension.

[0064] ②Region Classification: Region classification identifies the target semantic classification for each masked region. First, the masked regions output by the transformation model are... The input is fed into a fully connected layer to compute scores for K target classes, and then transformed into a normalized distribution function by a softmax function. Since no target category is provided, the target category with the highest detected confidence is used as the label for the mask region from the Faster R-CNN target detection output, and then transformed into a single vector. The ultimate goal is to minimize the cross-entropy (CE) loss, i.e.:

[0065] 2. Comparative learning of multimodal data.

[0066] Based on multimodal data feature extraction, this invention addresses the lack of semantic correlation between different modalities (i.e., between data within the same modality and between data across modalities) by proposing a multimodal contrastive learning model. Figure 2Multi-Modal Contrastive Learning (also known as the Multi-Modal Contrastive Learning Module) in this model semantically aligns text and / or image representations at different levels and maps them to a unified semantic space. The idea behind Multi-Modal Contrastive Learning is to attract related text features and / or image features in the representation space while repelling unrelated features, thereby achieving semantic alignment of multimodal data.

[0067] like Figure 6 As shown, the multimodal contrastive learning model first transforms the feature representations of text and / or images into the same embedding space through a fully connected layer, and then calculates the cosine similarity between them to measure the distance d(V,W). That is, it first randomly selects text and image examples, for example... and Two related examples are generated for each example by using two independent augmentation functions; then, a basic neural network encoder f(·) is used to extract representation vectors from the augmented data, i.e. and Again, a neural network projection head g(·) is used to map the representation vector to the contrastive loss space, thus obtaining... and Finally, a contrastive loss function is trained for the contrastive prediction task, which attracts relevant examples to each other and repels irrelevant examples to each other.

[0068] 1) Comparative learning of data within the same modality. Data in single-modal text or images also lacks semantic correlation. For example, the resistance value, current flowing through the resistor, and voltage across the resistor lack semantic correlation in feature representation. Similarly, the series and parallel structures of resistors in a circuit diagram lack semantic correlation in feature representation. To enhance multi-granularity semantic alignment between data, this section utilizes text rewriting techniques to semantically rewrite single-modal text or images. Text rewriting generates a large number of negative samples, rather than randomly sampling them as in previous methods. This helps the model learn more detailed semantic alignment from different levels of the image or text, i.e., by training the contrastive loss function:

[0069]

[0070] Among them, W + and W - d(Z,W) represents the positive and negative examples of text or image Z, respectively. +) represents the distance between positive examples of text or image Z, d(Z,W') represents the distance between positive and negative examples of text or image Z, and τ represents the temperature parameter. By learning more accurate semantic alignment of text or image features, a unified semantic feature representation of multimodal data can be achieved.

[0071] 2) Contrastive Learning Across Modal Data: For semantic alignment between title text and image cross-modal data, it is necessary not only to connect the scene displayed in the image with the text in the title, but also to align the entities in the image and their positional relationships with the descriptions in the text. Many existing multimodal pre-training methods attempt to align visual and text representations by performing simple image-text matching using a limited corpus of image-text pairs. They randomly sample negative images or text for each image-text pair from the same training batch and use a classifier to determine whether the image and text match. Since the randomly sampled negative images or text are usually very different from the original images or text, only a very coarse alignment between text and visual representations can be learned. This section also uses a text rewriting method to create a large number of positive and negative examples from the original image-text pair data, instead of randomly sampling negative samples, and uses these positive and negative examples to train the contrastive loss function:

[0072]

[0073] Among them, W + and W - Let V represent positive and negative examples of image V, and τ represent the temperature parameter. This allows for more accurate semantic alignment between image and text representations, enabling unified semantic feature representation of multimodal data.

[0074] II. Understanding the fusion of multimodal data.

[0075] 1. Indicative relation extraction based on multimodal learning.

[0076] Direct statements are those directly stated in the problem statement, such as the radius of a circle being 3 centimeters, and the length and width of a cube being 5 centimeters and 8 centimeters, respectively. This invention, based on the established multimodal feature learning and contrastive learning models, applies a multilayer self-attention converter decoder (…). Figure 2 The Decoder in the algorithm outputs the implicit features of the multimodal data and extracts the indicative relations from the text data, image data, and image-text pairs data according to the indicative relation extraction learning task. The extraction results are then analyzed to generate expressions of the indicative relations, thereby enabling the understanding of multimodal questions.

[0077] A framework for extracting indicative relations based on multimodal learning, such as... Figure 7As shown. For the learning task of extracting indicative relations, we add a class head to generate classification output, a box head to generate boundary outputs for each location of the implicit features in the decoder, and an attribute head to generate attribute outputs for the implicit features, i.e.: c l =class_head(h dec,l );b l =box_head(h dec,l );a l =attr_head(h dec,l c l ), where c l b l and a l These are the classification, location boundary, and attribute output sequence, respectively, h dec,l The decoded output sequence is the implicit feature sequence, which is composed of the encoded input sequence h. enc and task sequence vector q task Composition, i.e., {h dec,l}=D(h enc ,q task Each position of the implicit feature in the decoder generates an object or event class, which contains the three types of outputs mentioned above. By associating and analyzing the output of each object or event class, the indicative relationships in the multimodal data can be extracted.

[0078] 1) Indicative Relation Extraction from Text Data. This invention achieves text data understanding by constructing an indicative relation extraction model based on multimodal learning. Specifically, it predicts masked labels of text features by minimizing the negative log-likelihood function to generate classification probabilities and attribute information, while simultaneously using a sequence-to-sequence generation task to produce a target sequence under autoregressive conditions. The goal of training this model is to maximize the prediction probability of masked labels from given text data, thereby obtaining the mathematical relationship between source and target sequence features, and converting it into an indicative relation for output.

[0079] 2) Indicative Relation Extraction from Image Data. This invention achieves image data understanding by constructing an indicative relation extraction model based on multimodal learning. First, Faster-RCNN is used to pre-train the image data in the problem, selecting salient regions and extracting their features. Second, a self-attention masking model is used to predict the masks of these salient regions. Third, a softmax function is used to calculate feature regression and region classification for the image data. Finally, a cross-entropy loss function is used to predict the masked regions. The goal of training this model is to maximize the probability of predicting mask labels from given image data, thereby obtaining the mathematical relationships between image region feature sequences and converting them into indicative relations for output.

[0080] 3) Extraction of Indicative Relationships Between Text and Image Pairs. This invention achieves text-image pair data understanding by constructing an indicative relationship extraction model based on multimodal learning. First, a text masking model and an image region masking model are used to predict the masking of randomly masked text sequences and image region sequences, respectively. Second, positive and negative examples obtained using text rewriting techniques are used to train the contrastive loss function in the multimodal learning model, semantically aligning text and image representations at different levels. Finally, the distance d(V,W) between them is measured by calculating their cosine similarity. The goal of this model training is to obtain the semantic correlation between text-image pair features by calculating the distance between text-image pairs and performing maximum correlation analysis on image and text features, and then converting this correlation into indicative relationships for output.

[0081] 2. Learning based on intrinsic associations of declarative relations.

[0082] Based on the acquisition of multimodal indicative relations, this section addresses the lack of intrinsic correlation between these relations by constructing an intrinsic correlation learning model to associate the high-level semantics of indicative relations. This involves semantically associating the implicit features of each indicative relation with the implicit features related to that indicative relation at a high level. Since different indicative relations exist within the same modality, and the same indicative relation may be expressed in different modalities, this section employs two association methods in the intrinsic correlation model: one is to first associate the features of the same indicative relation across different modalities, and then associate the implicit features of different indicative relations at a high level, as shown in Figure 8(a); the other is to first associate the implicit features of different indicative relations within each modality, and then associate the implicit features of indicative relations across different modalities at a high level, as shown in Figure 8(b). In specific modeling, the extracted indicative relations can be mapped to a semantic space first, and then the association method can be selected based on the classification, location, and attribute features of the indicative relations, followed by association learning to obtain a more accurate semantic understanding of objects or event classes.

[0083] III. Reasoning and solving of multimodal data.

[0084] Implicit relations refer to relationships needed in problem-solving but not explicitly stated in the problem statement, including theorems and formulas required to solve the problem. Students often add implicit relations based on their existing knowledge during problem-solving, which raises the research question of discovering and mining implicit relations from problems in machine learning. Currently, most machine learning algorithms lack a systematic method for discovering implicit relations, instead using only a small number of common-sense relations during knowledge extraction. Implicit relations needed for the solution process can be discovered from the problem information, but adding implicit relations requires a corresponding implicit relation library to provide such relations. Since the scope of knowledge involved in machine learning for each subject is fixed, the implicit relations related to each subject constitute a definite set. Therefore, an implicit relation library can be used to manage the implicit relations involved in each subject.

[0085] Different subjects contain various implicit relationships. Discovering these implicit relationships is a generative process, while solving them is a reasoning process. Reasoning with multimodal data plays a decisive role in discovering implicit relationships and directly affects the reasoning results. This invention, based on a multimodal learning model, applies the implicit features of multimodal data output by the decoder of a multilayer self-attention converter. Then, according to the learning task of discovering implicit relationships, it combines a implicit relationship database to discover the implicit relationships in the multimodal data, thereby obtaining the feature representation of the implicit relationships and generating implicit relationship expressions.

[0086] like Figure 9 As shown, in order to discover the implicit relationships needed for the solution process from the problem, it is necessary to establish a mapping relationship between multimodal data representation and each implicit relationship. Based on the learning task of implicit relationship discovery, we also add a class head (class_head) to generate classification output, a box head (box_head) to generate boundary outputs for each position of the implicit features in the decoder, a depot head (depot_head) to generate the discovered implicit relationship output, and an attribute head (attr_head) to generate the attribute outputs of the implicit features, i.e.: c l =class_head(h dec,l b l =box_head(h dec,l ),d l =depot(h dec,l ), a l =attr_head(h dec,l c l d l ), where c l b ld l and a l These are classification, location boundary, implicit relation, and attribute output sequence, respectively. dec,l The decoded output sequence is the implicit feature sequence, which is composed of the encoded input sequence h. enc and task sequence vector q task Composition, i.e., {h dec,l}=D(h enc ,q task Each position of the implicit feature in the decoder generates an object or event class, which contains the four types of outputs mentioned above. By associating and analyzing the output of each object or event class, the implicit relationships in the multimodal data can be extracted.

[0087] This concludes the introduction to the structure of the question comprehension model and the corresponding training method. After constructing and training the question comprehension model, it can be applied to actual question comprehension to extract the explicit relationships embodied in the question and infer the implicit relationships. The overall process of a question comprehension method for machine solving according to this invention, implemented using the trained question comprehension model, is as follows:

[0088] Step 1: Obtain the question data. This question data is multimodal data, which includes text data and image data.

[0089] Step two involves inputting both text and image data into the question comprehension model to extract the features of the direct relationships reflected in the question data and infer the features of the implicit relationships reflected in the question data, thereby completing the question comprehension.

[0090] In summary, this invention, starting from the application requirements of machine learning, conducts research on feature representation, fusion understanding, and reasoning methods for multimodal data. It proposes a learning method for feature representation of multimodal data in machine learning, which effectively achieves feature extraction and unified semantic representation of multimodal data while simultaneously considering the heterogeneity between multimodal problems and the lack of semantic correlation between different modalities. Furthermore, it proposes a multi-task cross-modal deep learning framework capable of simultaneously performing multi-task learning such as extracting indicative relations and discovering implicit relations from data in multimodal problems. This framework shares the same model parameters for all tasks, rather than fine-tuning specific tasks individually. This provides new ideas and methods for understanding and solving multimodal problems. The proposed problem understanding method can be applied to machine learning of specific subject problems, solving practical problems in machine learning of elementary mathematics and physics circuit problems, effectively enhancing the application value of multimodal processing technology and demonstrating significant cutting-edge advancement.

Claims

1. A problem comprehension method for machine-based solutions, characterized in that, Includes the following steps: 1) Obtain question data, wherein the question data is multimodal data or unimodal data, wherein the unimodal data is text data or image data, and the multimodal data includes text data and image data; 2) Input the question data into the constructed question comprehension model to extract the features of direct statements and infer the features of implicit statements in order to complete the question comprehension; the direct statements are the relationships directly obtained from the question data, and the implicit statements are the relationships required to solve the question data but not directly stated in the question. The question understanding model includes a feature embedding module, a multi-layer self-attention converter, and a contrastive learning module. The multi-layer self-attention converter includes an encoder and a decoder. The feature embedding module is used to perform feature embedding processing on the input question data: if the input question data is unimodal data, then a text tag feature embedding sequence or an image region feature embedding sequence is obtained accordingly; if the input question data is multimodal data, then a text tag feature embedding sequence and an image region feature embedding sequence are obtained accordingly, and the two sequences are connected into a text-image pair feature embedding sequence. The encoder is used to encode the question data after feature embedding using the included self-attention masking mechanism, thereby obtaining the context label representation of text features and / or the context region representation of image regions. The contrastive learning module is used to semantically align the context label representation and / or context region representation and map them to a unified semantic space, thereby obtaining the corresponding text feature sequence and / or image feature sequence. The decoder is used to use a shared decoder to decode the input according to each task to obtain implicit features, and to obtain the output features of each task based on the specific output header of each task; wherein, each task includes a declarative relation extraction task and an implicit relation reasoning task. The declarative relation extraction task is used to extract declarative relation features from the question data, and the implicit relation reasoning task is used to infer the implicit relation features of the question data from the question data by combining the implicit relation library.

2. The problem understanding method for machine-based solutions according to claim 1, characterized in that, The multi-layer self-attention converter adopts a multimodal multi-task UniT model.

3. The problem comprehension method for machine-based solutions according to claim 2, characterized in that, The encoder includes a text encoder that encodes text data after feature embedding and / or an image encoder that encodes image data after feature embedding. The text encoder is trained alternately using two methods: bidirectional prediction and sequence-to-sequence generation. The image encoder is trained using feature regression or region classification.

4. The problem understanding method for machine solving according to claim 1, characterized in that, The contrastive learning module includes an enhancement function, a neural network encoder, and a neural network projector head. The enhancement function is used to generate two correlated enhancement data for each context label representation / context region representation using two independent enhancement functions. The neural network encoder is used to extract representation vectors from the enhancement data. The neural network projector head is used to map the representation vectors to the contrastive loss space to obtain a text feature sequence / image feature sequence.

5. The problem comprehension method for machine-based solutions according to claim 4, characterized in that, When training the contrastive learning module, it is necessary to make relevant context label representations and / or context regions attractive to each other, and irrelevant context label representations and / or context regions repulsive to each other.

6. The problem comprehension method for machine-based solutions according to claim 4, characterized in that, When training the contrastive learning module, if it is contrastive learning of the same modality data, the text rewriting method is used to semantically rewrite the single modality data to generate more than a certain number of negative samples, and the positive and negative samples are used for training. For comparative learning of cross-modal data, text rewriting methods are used to semantically rewrite the multimodal data to generate more than a certain number of positive and negative samples.

7. The problem comprehension method for machine-based solutions according to claim 1, characterized in that, When training the decoder, the high-level semantics of declarative relations need to be associated, and either of the following two association methods should be used: Association Method 1: First, associate different modal features with the same indicative relationship, and then associate the implicit features with different indicative relationships at a higher level; Association Method 2: First, associate the implicit features of different indicative relations for each modality, and then associate the implicit features of indicative relations for different modalities at a higher level.

8. The problem comprehension method for machine-based solutions according to claim 1, characterized in that, If the input question data includes text data, then when the feature embedding module performs feature embedding processing, it needs to perform position embedding, label embedding, and learning task embedding on the text data, so that the resulting text label feature embedding sequence W is represented as: Where E represents the text embedding marker, t represents the learning task embedding marker, [CLS] represents the classification marker at the beginning of the sequence, and [SEP] represents the separator marker between or at the end of the sequence; w1,…,w n This represents each word segmentation marker in the text data; If the input question data includes image data, then when the feature embedding module performs feature embedding processing, it needs to fuse visual features, location features, and learning task embedding for the image data, resulting in an image region feature embedding sequence V represented as follows: Where E represents the image embedding label, i represents the learning task embedding label, [IMG] represents the classification label at the beginning of the sequence, and v1,…,v n This indicates the labels for different regions in the image data.

9. The problem comprehension method for machine-based solutions according to claim 8, characterized in that, If the input question data includes image data, the feature embedding module needs to encode the positional features of each region of the image before performing feature embedding processing. This encoding uses a 5-dimensional vector. The encoding is performed, where (x1,y1) and (x2,y2) represent the coordinates of the lower left and upper right corners of the region, respectively, and W and H represent the width and height of the input image, respectively.

Images