A multimodal exercise representation method based on knowledge graph

Abstract

The present invention discloses a multimodal exercise representation method based on knowledge graph, which relates to the technical field of multimodal representation, including: obtaining mathematical exercise information and related knowledge points of multimodal information, extracting all elements involved in the multimodal exercise; extracting image and text features of the multimodal exercise; based on surface understanding rules, clarifying the spatial relationship between elements in the exercise text and the picture, and constructing a knowledge graph representing the surface information; based on the rules of hidden understanding, converting related theorems and formulas involved in the exercise into formalized language, and constructing a knowledge graph representing the hidden information; fusing the knowledge graphs representing the surface information and the knowledge graphs representing the hidden information into a knowledge graph representing the complete exercise information; based on the knowledge graph nodes and edges representing different exercise information, calculating the similarity and difficulty between the exercises and the matching degree between the text knowledge graph and the image knowledge graph; based on an automatic problem-solving model, obtaining a deep representation of the exercise.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal representation, and in particular to a multimodal exercise representation method based on knowledge graph. Background Art

[0002] Online learning platforms such as Coursera and Udemy have attracted widespread attention. These platforms have accumulated a large amount of exercise data and use these exercises to provide tailored educational applications. Among such problems, the representation of mathematical problems is quite challenging because they are inherently abstract and lack intuitive understanding. In particular, geometry problems in mathematics are more difficult to understand because they involve a wider range of symbols and complex relationships. However, geometry problems often exist in multimodal forms, with diagrams and text parts complementing each other. Therefore, the solution of multimodal geometry problems greatly tests the logical reasoning ability of the model, and its representation has become a research hotspot in the field of intellectual education. In recent studies, various methods have been explored to achieve the representation of multimodal exercises. These methods include pre-training methods and structured knowledge representation (such as knowledge graph) technology. Pre-training methods help to self-supervise the learning of multimodal data features based on task or problem attributes. However, this method requires comprehensive information acquisition from a large number of problem data sets, and it is difficult to fully represent the relationship between mathematical symbols. Structured knowledge representation technology, such as knowledge graphs, has shown significant effectiveness in representing multimodal data. However, some popular methods mainly focus on partial content information of unimodal text questions, which is insufficient for understanding geometry exercises containing multiple modalities. Therefore, it is a significant challenge to accurately represent multimodal information such as geometric images of exercises and implicit information such as basic theorems and mathematical symbols without relying on large datasets, and effectively integrate them to improve subsequent tasks. At present, it is difficult to effectively capture comprehensive information through a large amount of exercise data in the existing technology, which hinders the improvement of downstream task performance. Therefore, it is urgent to propose a multimodal exercise representation method based on knowledge graph. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a multimodal exercise representation method based on knowledge graph to improve the performance of downstream tasks related to multimodal exercises.

[0004] To achieve the above-mentioned object, the present invention provides a multimodal exercise representation method based on knowledge graph, comprising: obtaining multimodal exercise information and related knowledge points, and preprocessing the multimodal exercise information to obtain preprocessed multimodal exercise information;

[0005] Extracting image element information and text key information from the preprocessed multimodal exercise information, converting the image element information and the text key information into formalized language, and obtaining image formalized language and text formalized language;

[0006] Based on the principle of surface understanding, a formal language set of the image formal language and the text formal language is constructed to clarify the spatial relationship between the elements in the exercise text and the image, and to construct a knowledge graph representing the surface information;

[0007] Based on the hidden layer understanding rules, the relevant theorems and formulas involved in the exercises are converted into formal language, a knowledge graph representing the hidden layer information is constructed, and the knowledge graph representing the surface layer information and the knowledge graph representing the hidden layer information are merged to obtain a knowledge graph representing the complete exercise information;

[0008] Converting the image formal language and the text formal language into an image knowledge graph and a text knowledge graph respectively, and calculating the similarity and difficulty between exercises and the matching degree between the image knowledge graph and the text knowledge graph based on the nodes and edges in the knowledge graph representing the complete exercise information;

[0009] The image formalization language, the text formalization language and the knowledge graph representing complete exercise information are input into an automatic exercise solving model to obtain a deep representation of the exercise.

[0010] Optionally, preprocessing the multimodal exercise information to obtain preprocessed multimodal exercise information includes:

[0011] Performing image-text separation on the multimodal exercise information to obtain exercise images and exercise texts;

[0012] Using a target detection algorithm to extract key edges, corners and vector representations of values ​​of the image from the exercise image;

[0013] A regular matching method is adopted for the exercise text, and a custom matching rule is used to filter out useless information in the exercise.

[0014] Optionally, the method for constructing a knowledge graph representing surface information is:

[0015] Rule 1: If or but The definition is as follows:

[0016]

[0017] Rule 2: If p∈Gr, is defined as follows:

[0018]

[0019] in, A knowledge graph that represents surface information; Representing knowledge graph Entities in Representing knowledge graph the relationship between Represents a set of geometric shapes; represents a set of arithmetic functions; Indicates a combination relationship; It indicates a predicate; Representation predicate part of; Indicates parameters; Represents a set of geometric relationships; and Indicates different parameters; Indicates the calculation dimension; Represents the knowledge graph in the test results Entities in Represents the knowledge graph in the test results The relationship in.

[0020] Optionally, the method for constructing a knowledge graph representing hidden layer information is:

[0021]

[0022]

[0023] Among them, {f} represents the relevant theorem involved in the exercise; The formal language corresponding to the exercises; A knowledge graph representing latent information; Representing knowledge graph Entities in Represents the entities in the knowledge graph in a round of test results; Representing knowledge graph the relationship between Represents the relationship in the knowledge graph in a round of test results.

[0024] Optionally, the method for calculating the similarity between exercises is:

[0025]

[0026] in, Indicates the similarity between two exercises; and Represent two different exercises respectively; and Represent two training parameters respectively; Represents the entity of the knowledge graph corresponding to exercise a; Represents the entity of the knowledge graph corresponding to exercise b; Represents the relationship of the knowledge graph corresponding to exercise a; Represents the relationship of the knowledge graph corresponding to exercise b; ( ) represents the Jaccard function, which is a measure of the similarity between two sets.

[0027] Optionally, the method for calculating the difficulty between exercises is:

[0028]

[0029] in, Indicates the difficulty of the exercises; and Represent two training parameters respectively; is degree centrality; for connectivity; Represents the knowledge graph corresponding to the exercise.

[0030] Optionally, the method for calculating the matching degree between the image knowledge graph and the text knowledge graph is:

[0031]

[0032] in, Indicates the matching degree between the image knowledge graph and the text knowledge graph; Represents a text knowledge graph; Represents the image knowledge graph; and Represent two training parameters respectively; ( ) represents the Jaccard function, which is used to measure the similarity between two sets; Represent entities in the image knowledge graph; Represent entities in the text knowledge graph; Representing relationships in the image knowledge graph; Represents relations in a text knowledge graph.

[0033] Optionally, obtaining a deep representation of the problem involves:

[0034] The image formalization language and the text formalization language are processed by using a bidirectional gated recurrent unit to capture the contextual relationship of the exercise semantics and obtain the context representation of the exercise;

[0035] HGAN is used to capture the graph structure information of the knowledge graph representing the complete exercise information, and obtain the information representation of each element of the knowledge graph;

[0036] Input the context representation of the exercise and the information representation of each element of the knowledge graph into the Transformer-based encoder to capture the long-distance dependency relationship between the text context information and the knowledge graph structure information, and obtain the deep representation of the exercise;

[0037] GTS is used to generate corresponding output results based on the deep representation of the input, convert the deep representation of the exercise into the solution formula of the question, realize the decoding of the result vector, and use the automatic problem-solving task as an indirect downstream task to verify the quality of the exercise representation.

[0038] Technical effect of the present invention: The present invention discloses a multimodal exercise representation method based on knowledge graph, which realizes the effective integration of problem-related theorems and contents. Traditional methods can generally extract symbolic data from multimodal problems, but ignore the clear logical connections between these symbols; the method proposed by the present invention constructs a problem knowledge graph to understand the key content and logical relationships in multimodal geometry problems; the present invention proposes a two-layer strategy for representing multimodal exercises, which captures the key information of geometry exercises at different levels by constructing a knowledge graph and clearly extracts their relationships, which helps to comprehensively represent multimodal exercises; these different graphs are merged to construct a comprehensive problem knowledge graph; the present invention is applied to multiple downstream tasks and experiments are carried out on two data sets containing multimodal geometry exercise information; the traditional large model method mainly relies on large-scale data pre-training models, the present invention establishes three direct downstream tasks and one indirect downstream task, uses the knowledge graph to adjust the problem representation, and improves the performance of downstream tasks related to multimodal exercises. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A schematic diagram of a flow chart of a multimodal exercise representation method based on a knowledge graph according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the process of constructing a knowledge graph according to an embodiment of the present invention;

[0042] Figure 3 The figure is a flow chart of an embodiment of the present invention for realizing indirect downstream tasks of automatically solving problems by using images and text information of geometry exercises. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] like Figure 1 As shown, this embodiment provides a multimodal exercise representation method based on knowledge graph, including: S1, preprocessing the multimodal mathematical exercises of the public data set GeoQA and Geometry3K to separate images and texts. Then, the object detection algorithm is used for the exercise image to extract the vector representation of the key edges, corners and numerical elements of the image; the regular matching method is used for the exercise text, and the matching rules are customized to filter out the useless information in the exercise;

[0046] S2: Using the data preprocessed in step S1, convert the image element information and text key information into a formal language consisting of nested predicate (parameter) pairs. The image and text formal languages ​​of the exercise transformed in step S2 are combined and the union of the two parts is taken to form the formal language set of the exercise itself.

[0047] S3. Convert the formal language set into a knowledge graph for the exercise as the surface understanding information of the exercise.

[0048] Further, the steps in S2 and S3 include:

[0049] When converting geometry exercises into heterogeneous graphs, since they contain many abstract symbols and lines, the content of the problem cannot be easily constructed into a heterogeneous graph as with traditional retrieval methods. It is necessary to design rules to create a surface-level understanding of the heterogeneous graph. Used to represent specific geometric problems. The construction of surface understanding heterogeneous graphs includes the following steps.

[0050]

[0051] The main goal in surface understanding involves Convert to The construction rules of . Where V represents the entities in the graph and E represents the relationships in the graph. Each problem can be solved using a formal language The language consists of predicate-argument pairs By encapsulating important geometric elements, such as geometric shape sets ( ), geometric relationship set ( ) and a set of arithmetic functions ( ), these predicates are used in constructing Therefore, it is necessary to formulate rules for constructing a knowledge graph based on predicates:

[0052] Rule 1: If or but The definition is as follows:

[0053]

[0054] Indicates a composition relationship. Specifically, v is a part of p.

[0055] Rule 2: If p∈Gr, is defined as follows:

[0056]

[0057] p means and The geometric relationship between them represents different parameters respectively;

[0058] S4: According to the knowledge points and reasoning formulas related to the exercises, such as the Pythagorean theorem, they are converted into knowledge graphs as the implicit understanding information of the exercises in the manner of S2 and S3.

[0059] Furthermore, the construction process described in S4 includes the following steps:

[0060] Convert the theorem {f} involved in the problem into a knowledge graph of latent information Then, the implicit heterogeneous graph The explicit heterogeneous graph obtained from step S3 The information is integrated into a knowledge graph as a complete information representation of the exercise. Similar to surface understanding, deep understanding can also be simplified into the following process.

[0061]

[0062] Different from step S3, the theorem needs to be converted into a formal language based on logical sets; secondly, new rules need to be designed to construct implicit heterogeneous graphs. .Conversion The process consists of four parts: logical operators ( ), constituent elements ( ), logical relationship ( ), and constants ( ). Specifically It consists of multiple nested logical operation unit layers Composition, of which , o∈ Initially or ; In the conversion process, e can also represent the logical operation unit of the previous layer. Then, a rule that can express abstract symbols and relationships is established and converted into an implicit heterogeneous graph Gim. The rule is as follows.

[0063] Rule 3: The definition is as follows

[0064]

[0065] If a problem involves only one theorem and ; If multiple theorems are involved, then is the entity set consisting of other theorems in the current problem Following the same principles in its composition;

[0066] S5. For the knowledge graph obtained in step S4, convert the edges and entities in the graph into a triple format of (entity, edge, entity) as the serialized information representation of the exercise; using the obtained exercise knowledge graph and serialized data, design four independent downstream tasks: similarity prediction, difficulty assessment, image text retrieval, and geometric question answering to check the representation quality of the knowledge graph.

[0067] Furthermore, the construction process for each task described in S5 includes the following steps:

[0068] Utilize the entity and relation similarity in the associated knowledge between questions to calculate the and Similarity between For the problem and Knowledge Graph = and = :

[0069]

[0070] ( ) represents the Jaccard function, which is a measure of the similarity between two sets.

[0071] The difficulty of question q is directly related to the complexity of the knowledge graph. Therefore, two indicators derived from the question knowledge graph are used: degree centrality and connectivity To calculate the complexity of the knowledge graph corresponding to the question to reflect the difficulty of the exercise:

[0072]

[0073] Image knowledge graph for geometry exercises = and text knowledge graph = The similarity evaluation between entities or relations of two knowledge graphs is performed using the Jaccard function. This method calculates the similarity between the graph and the text. To evaluate the matching degree of image and text, the formula is as follows:

[0074]

[0075] All three tasks use cross entropy as the objective function;

[0076] S6. Use the exercise formal language and knowledge graph obtained in S2 and S4 as input data for the indirect downstream task of evaluating the quality of knowledge graph representation - the exercise automatic problem-solving model.

[0077] Further, the process in S6 includes the following steps:

[0078] Bidirectional Gated Recurrent Units (BiGRU) are used to fully capture the subtle semantic information associated with each token. This choice is intended to enhance the semantic details in the explicit question information. Subsequently, the hidden state representation is derived, denoted as H = { ··· , where N represents the vocabulary and the number of values ​​contained in the formal language. Meanwhile, M represents the dimension of the hidden vector. It should be noted that in the implicit information, the representation of the logical operators and the constants should be kept independent. Therefore, two different embedding matrices are needed to obtain their respective representations.

[0079] The HGAN-based graph transformer encoder is used, taking the above hidden state representation H as input. Due to the scale-free nature of the heterogeneous graph of the problem, the variance of the graph data is quite large. To solve this problem, the HGAN process needs to be iterated K times to obtain In the HGAN learning process, a transformer-based procedure is used to enhance the learned representation of the generated graph G Finally, a global pooling operation is applied to all learned node representations to learn a graph representation that includes global context. information.

[0080] Using the GTS-based tree decoder, the root node of the initialized tree is = Then the expression tree The method nI for constructing each node representation is constructed using the pre-order traversal method. The loss function L is defined as the log-likelihood of the probability of the sum of negative numbers, and the formula is as follows:

[0081]

[0082] represents the i-th token traversed in the target expression, I is the number of tokens in the target expression, and prob is the distribution calculation function obtained in GTS. The training goal is to minimize the loss function.

[0083] The embodiment of the present invention consists of two key components: a problem understanding module and an evaluation task module. The former extracts explicit and implicit heterogeneous graph information of multimodal exercises through surface understanding and deep understanding, and then fuses these two heterogeneous graphs into a comprehensive exercise knowledge graph. The latter demonstrates the advantages of motion knowledge graphs in understanding motion through direct and indirect tasks at different levels. On the one hand, direct tasks, such as similarity and difficulty assessment, demonstrate the accuracy and interpretability of understanding motion knowledge graphs. On the other hand, indirect tasks, such as automatic problem-solving tasks, illustrate the superiority of motion knowledge graphs in problem solving.

[0084] like Figure 2 As shown in Figure 1, the process of constructing question information into a knowledge graph includes:

[0085] The surface understanding of the exercise includes converting the geometry problem itself into a formal language and extracting explicit information from the formal language into an explicit heterogeneous graph G. Since the process of converting the image information of the problem into a formal language can be achieved through data annotation tools and target detection algorithms, the focus of surface information extraction is to realize the process of converting the formal language into a knowledge graph based on custom rules.

[0086] The explicit information of the geometry exercises is converted into 10 formal languages ​​using data annotation tools. In order to extract more explicit relational information from the formal language, the parameter information needs to be enriched. For example, here is Equals(LengthOf(Line(Q, R), num), 2)", where "num" is a specific numeric relationship between "Lengthof" and "Line", indicating the length of the line. Rules that identify the relationship between entities from the inside out are applied to extract key information in the formal language.

[0087] A deep understanding of the problem involves extracting the implicit information contained in the exercise itself into an implicit heterogeneous graph G. First, all the core theorems are decomposed into four parts: logical operators ( ), constituent elements ( ), logical relationship ( ), and constants ( ). Then, each theorem is reconstructed in a formal language. Finally, an implicit heterogeneous graph is constructed based on a rule-based approach.

[0088] Then, the explicit heterogeneous graph needs to be integrated with the implicit heterogeneous graph. The fusion of two heterogeneous graphs essentially involves establishing the relationship between similar entities from the two graphs. Therefore, the similarity between the two entities needs to be evaluated. The respective entity sets are extracted from the two graphs and these entities are represented by vectors. The relationship between two entities can be determined based on the similarity of their corresponding vectors. The formula is as follows:

[0089]

[0090]

[0091] Among them, yuim,uex represents the similarity dot product vector between different entities of two knowledge graphs. If If the value of is relatively high, a new relationship is established between the two entities, thus completing the establishment of the relationship between the two knowledge graphs.

[0092] like Figure 3 As shown in the figure, the process performed by the automatic problem-solving model uses BiGRU to initialize the representation of each token (including words and numbers). Subsequently, the graph transformer HGAN is introduced to learn the features of entities and relations in the graph. Finally, a tree decoder based on the goal-driven tree structure (GTS) is used to construct the expression of the solution and promote the understanding of the overall information in the exercise. The specific process includes:

[0093] The hidden state representation H = { ··· }∈R T×d , where T represents the vocabulary and number of values ​​in the formal language, and d represents the dimension of the hidden vector. The representation learned by BiLSTM for each token is used as the input of the graph transformer encoder. In the implicit information of this geometric exercise, the representations of logical operators and constants are independent, that is, their representations are obtained by two independent embedding matrices.

[0094] Furthermore, the HGAN graph transformer encoder is used to learn the representation of the knowledge graph. Since the knowledge graph of the geometry exercise is a heterogeneous graph, the basic idea of ​​learning node features in a heterogeneous graph is to learn a set of weight parameter matrices corresponding to each relationship type. Therefore, it is necessary to construct a meta-path Φ based on the relationship type and search for adjacent nodes.

[0095]

[0096] in, represent the initial representations of node x and node y respectively, It is a deep neural network that learns the attention between two nodes on the current path.

[0097] Then, the importance between node pairs is obtained based on the meta-path, and they are normalized to obtain the weight coefficients:

[0098]

[0099] in, (·) is the activation function, ⊕ represents the connection operation, z represents the adjacent node z in the meta-path Φ, and h is the vector representation of node z. Then, the meta-path embedding of node x can be aggregated using the shadow features of its adjacent meta-paths,

[0100]

[0101] in, Learned embedding for node x representing the meta-path Φ.

[0102] Then, the aggregation algorithm and deep neural network are used to integrate the representations of the same node from different paths to obtain the final embedding features of the node. , and finally aggregate the representations of all nodes in the graph to form the representation of the entire knowledge graph.

[0103] Furthermore, a goal-driven tree structure (GTS) is used as a decoder to construct the solution expression. The numerical nodes are set as leaf nodes, and each operator node must have two child nodes. Then, the expression tree is generated in a pre-order traversal manner. After that, the negative log-likelihood function is used as the loss function, and the training goal is to minimize the loss function.

[0104] The present invention discloses a method for representing multimodal exercises based on a knowledge graph, which realizes the effective integration of problem-related theorems and contents. Traditional methods can generally extract symbolic data from multimodal problems, but ignore the clear logical connections between these symbols. The method proposed in the present invention constructs a problem knowledge graph to understand the key content and logical relationships in multimodal geometry problems. The present invention proposes a two-layer strategy for representing multimodal exercises. By constructing a knowledge graph, the key information of geometry exercises at different levels is captured and their relationships are clearly extracted, which helps to comprehensively represent multimodal exercises. These different graphs are merged to construct a comprehensive problem knowledge graph. The present invention is applied to multiple downstream tasks and experiments are carried out on two data sets containing multimodal geometry exercise information. Traditional large model methods mainly rely on large-scale data pre-training models. The present invention establishes three direct downstream tasks and one indirect downstream task, uses knowledge graphs to adjust problem representation, and improves the performance of downstream tasks related to multimodal exercises.

[0105] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal exercise representation method based on knowledge graph, characterized in that: include: Acquire multimodal exercise information and related knowledge points, and preprocess the multimodal exercise information to obtain preprocessed multimodal exercise information; Extracting image element information and text key information from the preprocessed multimodal exercise information, converting the image element information and the text key information into formalized language, and obtaining image formalized language and text formalized language; Based on the principle of surface understanding, a formal language set of the image formal language and the text formal language is constructed to clarify the spatial relationship between the elements in the exercise text and the image, and to construct a knowledge graph representing the surface information; The method of constructing a knowledge graph representing surface information is: Rule 1: If or but The definition is as follows: Rule 2: If p∈Gr, is defined as follows: in, A knowledge graph that represents surface information; Representing knowledge graph Entities in Representing knowledge graph the relationship between Represents a set of geometric shapes; represents a set of arithmetic functions; Indicates a combination relationship; It indicates a predicate; Representation predicate part of; Represents parameters; Represents a set of geometric relationships; and Indicates different parameters; Indicates the calculation dimension; Represents the knowledge graph in the test results Entities in Represents the knowledge graph in the test results the relationship between Based on the hidden layer understanding rules, the relevant theorems and formulas involved in the exercises are converted into formal language, a knowledge graph representing the hidden layer information is constructed, and the knowledge graph representing the surface layer information and the knowledge graph representing the hidden layer information are merged to obtain a knowledge graph representing the complete exercise information; Converting the image formal language and the text formal language into an image knowledge graph and a text knowledge graph respectively, and calculating the similarity and difficulty between exercises and the matching degree between the image knowledge graph and the text knowledge graph based on the nodes and edges in the knowledge graph representing the complete exercise information; The image formalization language, the text formalization language and the knowledge graph representing complete exercise information are input into an automatic exercise solving model to obtain a deep representation of the exercise.

2. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: Preprocessing the multimodal exercise information to obtain preprocessed multimodal exercise information includes: Performing image-text separation on the multimodal exercise information to obtain exercise images and exercise texts; Using a target detection algorithm to extract key edges, corners and vector representations of values ​​of the image from the exercise image; A regular matching method is adopted for the exercise text, and a custom matching rule is used to filter out useless information in the exercise.

3. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: The method to construct a knowledge graph representing hidden information is: Among them, {f} represents the relevant theorem involved in the exercise; The formal language corresponding to the exercises; A knowledge graph representing latent information; Representing knowledge graph Entities in Represents the entities in the knowledge graph in a round of test results; Representing knowledge graph the relationship between Represents the relationship in the knowledge graph in a round of test results.

4. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: The method for calculating the similarity between exercises is: in, Indicates the similarity between two exercises; and Represent two different exercises respectively; and Represent two training parameters respectively; Represents the entity of the knowledge graph corresponding to exercise a; Represents the entity of the knowledge graph corresponding to exercise b; Represents the relationship of the knowledge graph corresponding to exercise a; Represents the relationship of the knowledge graph corresponding to exercise b; Represents the Jaccard function, a metric used to measure the similarity between two sets.

5. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: The method for calculating the difficulty between exercises is: in, Indicates the difficulty of the exercises; and Represent two training parameters respectively; is degree centrality; for connectivity; Represents the knowledge graph corresponding to the exercise.

6. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: The method for calculating the matching degree between the image knowledge graph and the text knowledge graph is: in, Indicates the matching degree between the image knowledge graph and the text knowledge graph; Represents a text knowledge graph; Represents the image knowledge graph; and Represent two training parameters respectively; represents the Jaccard function, a measure of the similarity between two sets; Represent entities in the image knowledge graph; Represent entities in the text knowledge graph; Representing relationships in the image knowledge graph; Represents relations in a text knowledge graph.

7. The multimodal exercise representation method based on knowledge graph according to claim 1, characterized in that: Obtaining a deep representation of the problem involves: The image formalization language and the text formalization language are processed by using a bidirectional gated recurrent unit to capture the contextual relationship of the exercise semantics and obtain the context representation of the exercise; HGAN is used to capture the graph structure information of the knowledge graph representing the complete exercise information, and obtain the information representation of each element of the knowledge graph; Input the context representation of the exercise and the information representation of each element of the knowledge graph into the Transformer-based encoder to capture the long-distance dependency relationship between the text context information and the knowledge graph structure information, and obtain the deep representation of the exercise; GTS is used to generate corresponding output results based on the deep representation of the input, convert the deep representation of the exercise into the solution formula of the question, realize the decoding of the result vector, and use the automatic problem-solving task as an indirect downstream task to verify the quality of the exercise representation.

Images