A method for establishing a multimodal knowledge graph and its application

Abstract

This application discloses a method for establishing a multimodal knowledge graph, comprising: collecting open source text knowledge graphs and target text knowledge graphs from multiple disciplines and extracting structural features, visual features, and text features; fusing the open source text knowledge graph with the target text knowledge graph and processing inconsistent information to obtain a multimodal knowledge graph; collecting learners' interactive information and extracting entities and relationships to obtain head entity embedding representations, tail entity embedding representations, and relationship embedding representations of the interactive information, thereby converting the interactive information into triples; constructing a pre-trained language model of the target knowledge to learn the multimodal knowledge graph, and providing feedback information on the input triple-form interactive information. This method can solve the problem that the traditional method of constructing subject knowledge graphs based on single subject knowledge has poor interdisciplinary learning effects, resulting in low efficiency in the learning process of using subject knowledge graphs for knowledge information retrieval and intelligent question answering.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence education technology, and more specifically, to a method for establishing a multimodal knowledge graph, a device for establishing a multimodal knowledge graph, an electronic device, and a computer-readable storage medium. Background Art

[0002] Interdisciplinary learning builds a comprehensive knowledge system for students, enabling them to flexibly apply their knowledge to solve new problems. Compared to traditional independent learning, interdisciplinary learning not only integrates knowledge within disciplines but also connects knowledge across disciplines, strengthening the collaborative educational function of the curriculum.

[0003] In the current intelligent education scenario, establishing subject knowledge graphs is an important teaching method. However, existing subject knowledge graphs are based on single subject knowledge and do not establish connections between different subject knowledge. Therefore, existing subject knowledge graphs cannot achieve good interdisciplinary learning effects, making students' learning processes such as knowledge information retrieval and intelligent question-answering in subject knowledge graphs inefficient, and it is difficult to strengthen students' comprehensive knowledge system. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention proposes a method and application for establishing a multimodal knowledge graph, aiming to solve the problem that the traditional method of constructing subject knowledge graphs based on a single subject knowledge has poor interdisciplinary learning effect, resulting in low efficiency in the learning process of using subject knowledge graphs for knowledge information retrieval and intelligent question answering, and difficulty in strengthening students' comprehensive knowledge system.

[0005] On the one hand, the present invention proposes a method for establishing a multimodal knowledge graph, including: collecting open source text knowledge graphs and target text knowledge graphs of multiple disciplines and extracting structural features, visual features and text features; fusing the open source text knowledge graph and the target text knowledge graph according to the structural features, the visual features and the text features and processing inconsistent information to obtain a multimodal knowledge graph; collecting learners' interactive information and extracting entities and relationships to obtain head entity embedding representation, tail entity embedding representation and relationship embedding representation of the interactive information to convert the interactive information into a triple form; constructing a pre-trained language model of target knowledge, using the pre-trained language model to learn the multimodal knowledge graph, and using the multimodal knowledge graph to provide feedback information on the interactive information in the form of triples input.

[0006] In one embodiment of the present invention, the extraction of structural features, visual features and text features includes: using a graph convolutional network encoder with a ConvE model loss to extract the structural features by using the neighbor aggregation representation as a triple energy function; the visual encoder uses a residual neural network pre-trained on the ImageNet dataset to obtain the image embedding of the corresponding learning resource entity, and uses the output of the last hidden layer of the Softmax function operation as the visual feature; and using the BRET model as a learning resource text encoder to extract the text features.

[0007] In one embodiment of the present invention, the open source text knowledge graph and the target text knowledge graph are merged and inconsistent information is processed to obtain a multimodal knowledge graph, including: forcibly aligning entity pairs representing the same meaning in the open source text knowledge graph and the target text knowledge graph so that the entity pairs are represented by the same embedding vector; removing duplicate aligned entities in a unified subject relationship vector set, and maintaining the unified subject relationship vector set.

[0008] In one embodiment of the present invention, the fusion of the open source text knowledge graph and the target text knowledge graph and processing of inconsistent information to obtain a multimodal knowledge graph includes: using a bilinear model Tucker decomposition to decompose the tensor into a core tensor M c And 4 mode factors; continue to decompose the core tensor M c All modal representations are merged into the same space with element-wise product to achieve fusion; constraints are added to limit the correlation between different modal representations of the same learning resource entity, and contrastive learning of different entities and modalities is used as an additional learning objective for regularization to enhance multimodal fusion.

[0009] In one embodiment of the present invention, the open source text knowledge graph is merged with the target text knowledge graph and inconsistent information is processed to obtain a multimodal knowledge graph, including: using a single-graph attention mechanism to select adjacent nodes in one discipline, and using a cross-graph attention mechanism to select adjacent nodes in multiple disciplines; the single-graph attention mechanism and the cross-graph attention mechanism respectively calculate the attention score function according to the importance scoring function of the target node relative to the adjacent nodes, and aggregate neighbor messages to obtain node representations in different states; according to the node representation, the outputs of the single-graph attention mechanism and the cross-graph attention mechanism in different states are integrated to obtain the final representation of the target node; according to the final representation, the candidate entity is linked to the target text knowledge graph to obtain the complete multimodal knowledge graph.

[0010] In one embodiment of the present invention, the method of collecting learners' interactive information and extracting entities and relationships to obtain head entity embedding representation, tail entity embedding representation and relationship embedding representation of the interactive information to convert the interactive information into triple form includes: collecting learners' interactive information and dividing it into Tokens through byte pair encoding, inputting it into the ELMo model, and outputting context-aware Token embedding; treating entity and relationship extraction as a set prediction problem, and modeling it using a transformer-based non-autoregressive decoder; using bipartite matching loss to score the predicted triples according to the basic facts, selecting the triple with the best ranking, and obtaining the head entity embedding representation, tail entity embedding representation and relationship embedding representation in the problem, thereby converting the problem into triple form.

[0011] In one embodiment of the present invention, the multimodal knowledge graph is used to provide feedback information on the interactive information in the form of triples input, including: given a question que, a subject entity Q and an answer entity A, all shortest paths between Q and A are extracted from the multimodal knowledge graph, and the shortest paths are regarded as related to the question que, and pre-trained using contrastive learning loss; the retrieval module of the multimodal knowledge graph is adjusted to retrieve a subgraph of the question que to provide an inference scope for the inference module; and the inference module of the multimodal knowledge graph is adjusted to obtain a matching score and select the best-ranked entity as the answer to the question.

[0012] In one embodiment of the present invention, the retrieval module for adjusting the multimodal knowledge graph includes: obtaining the question que and triples (e according to the multimodal knowledge graph) ai ,p i ,e bi ) The vector after t iterations captures the semantic matching features Expressed as: in, is the parameter of the projection layer after t iterations, m and d are the hidden dimensions of the pre-trained language model and feature vector respectively, σ is the sigmoid activation function, and ⊙ is the Hadamard product; the semantic matching features generated by the matching information propagation module are aggregated to update the entity representation and obtain the entity matching score. In the tth step, use The matching score of the head entity calculated in the first step is used as a weight to aggregate the matching features from adjacent triples to obtain the representation of the tail entity and update the entity matching score; the matching scores of the nodes in the multimodal knowledge graph subgraph are generated by adopting the matching information propagation module, and the basic truth vector represented by the matching score is obtained by using the marked answer to minimize the KL loss between the basic fact matching score vectors.

[0013] In one embodiment of the present invention, the adjustment of the reasoning module of the multimodal knowledge graph includes: initializing the parameters of the reasoning module using the parameters of the retrieval module, and using a matching information propagation module similar to the retrieval stage to generate matching scores of nodes in the multimodal knowledge graph subgraph; adjusting the KL loss by fitting the learned matching score with the basic truth vector according to the KL loss, and using the learned reasoning model to select the best-ranked entity as the answer to the question according to the matching score.

[0014] On the other hand, an embodiment of the present invention further proposes an electronic device, comprising: a memory and one or more processors connected to the memory, the memory storing a computer program, and the processor being used to execute the computer program to implement the steps of the method described in any one of the aforementioned embodiments.

[0015] On the other hand, an embodiment of the present invention further proposes a computer-readable storage medium, which stores a computer program that can be executed by an access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the method described in any one of the aforementioned embodiments.

[0016] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0017] 1) The multimodal knowledge graph construction method proposed in this invention uses open source text knowledge graphs from multiple disciplines and target text knowledge graphs to extract structural, visual, and textual features, and fuses them to obtain a multimodal knowledge graph. The multimodal knowledge graph is then used to train an intelligent interaction model, thereby effectively improving students' learning efficiency in knowledge information retrieval and intelligent question-answering within the subject knowledge graph, thereby strengthening students' comprehensive knowledge system.

[0018] 2) Use bilinear model Tucker decomposition to decompose the tensor into core tensor M c And 4 mode factors, continue to decompose the core tensor M c The representations of all modalities are merged into the same space with element-by-element product to achieve fusion, thus realizing multimodal information fusion between the open source text knowledge graph and the target text knowledge graph;

[0019] 3) It uses both single-graph attention and cross-graph attention mechanisms to handle information fusion issues within the same discipline and between different disciplines, effectively solving the information inconsistency problem between knowledge graphs of different disciplines.

[0020] 4) The matching information propagation module enables the language model to share the relevance information of the retrieval stage during the reasoning stage, effectively solving the problem of information loss such as context information and entity relevance information during the reasoning stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a method for establishing a multimodal knowledge graph provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the network structure of a method for establishing a multimodal knowledge graph provided by an embodiment of the present invention;

[0024] Figure 3 An overall principle block diagram of a method for establishing a multimodal knowledge graph provided by an embodiment of the present invention;

[0025] Figure 4 Schematic diagram of application scenarios of the multimodal knowledge graph provided by an embodiment of the present invention;

[0026] Figure 5 A schematic structural diagram of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0028] like Figure 1As shown, the first embodiment of the present invention provides a method for establishing a multimodal knowledge graph, for example, including: step S1, collecting open source text knowledge graphs and target text knowledge graphs of multiple disciplines and extracting structural features, visual features and text features; step S2, fusing the open source text knowledge graph and the target text knowledge graph according to the structural features, the visual features and the text features and processing inconsistent information to obtain a multimodal knowledge graph; step S3, collecting learners' interaction information and extracting entities and relationships to obtain the head entity embedding representation, tail entity embedding representation and relationship embedding representation of the interaction information to convert the interaction information into a triple form; step S4, constructing a pre-trained language model of the target knowledge, using the pre-trained language model to learn the multimodal knowledge graph, and using the multimodal knowledge graph to provide feedback information on the interaction information in the form of triples input.

[0029] In step S1, for example, the existing text knowledge graphs of physics, chemistry, and biology are collected as open source knowledge graphs, and the learning resources of middle school physics, chemistry, and biology are collected as target knowledge graphs. The extracted features include structural features, visual features, and text features.

[0030] Step S1 specifically includes the following sub-steps:

[0031] Step 1.2, the knowledge graph is defined as Among them, ε and Represent the collection of entities and relations respectively, Representing the relation triples of the knowledge graph, we use the modality extractor to extract the structural, visual, and textual features of the learning resources. The specific process is as follows:

[0032] In step 1.2.1, a graph convolutional network (GCN) encoder with ConvE loss is used to extract structural features by taking the neighbor aggregation representation as a triplet energy function, i.e. Among them, f represents a nonlinear function, vec represents a vectorized operation, represents the embedded representation after two-dimensional reshaping, represents the relationship parameter, w represents the convolution filter, W e Represents the transformation matrix, * represents the convolution operation;

[0033] In step 1.2.2, the visual encoder uses ResNet pre-trained on ImageNet to obtain the image embedding of the corresponding learning resource entity, and uses the output of the last hidden layer of the Softmax operation as the visual feature.

[0034] In step 1.2.3, the entity description contains richer and more complex knowledge than the pure knowledge graph. In order to fully extract the complex knowledge, BERT is used as the learning resource text encoder.

[0035] Step 1.3: Graph the knowledge of physics, chemistry, and biology: By forcing the alignment of entity pairs that represent the same meaning in the three disciplines a and e b , so that e a and e b are represented by the same embedding vector.

[0036] Step 1.4: Remove duplicate aligned entities from the unified subject entity vector set ε and maintain a unified subject relationship vector set.

[0037] In step S2, the extracted structural, visual and textual features are integrated into the science text knowledge graph, and the inconsistent information between the three different modal knowledge graphs of the subjects is processed to form a complete multimodal middle school science knowledge graph.

[0038] Combine Figure 2 and Figure 3 As shown, step S2 specifically includes the following sub-steps:

[0039] Step 2.1: In a multimodal knowledge graph, each entity in the knowledge graph is represented by multiple features from different modalities. Define the set of modalities T = {l, r, h, v}, where l, r, h, v represent the structural modality, visual modality, textual modality, and multimodal modality, respectively. The link prediction model is expected to learn a scoring function for the relation triple to estimate the likelihood of the triple. This function is formulated as: The fusion stage uses the bilinear model Tucker decomposition to decompose the tensor into the core tensor M c And 4 mode factors:

[0040] M=(((M c ×H l )×H r )×H h )×H v , (1)

[0041] in, represents the transformation matrix, Represents the core tensor. The entity embedding is first projected into a low-dimensional space and then combined with the core tensor M c Fusion, by decomposing the core tensor M c The computational complexity is further reduced by merging the representations of all modalities into the same space with element-wise products:

[0042]

[0043] in, Represents potential representation, is the original embedding representation, is the decomposition transformation matrix of each modality. Constraints are added to limit the correlation between different modal representations of the same learning resource entity, and contrastive learning of different entities and modalities is used as an additional learning objective for regularization. In contrastive learning, paired representations of the same entity in different modalities are used as positive samples, and paired representations of different entities are used as negative samples. The training goal is to make the distance between negative samples greater than the distance between positive samples to enhance multimodal fusion, that is:

[0044] L(W(x),W(x + ))<<L(W(x),W(x - )), (3)

[0045] Where L(·,·) represents the distance metric, W(·) represents the embedding function, and the superscripts + and - represent positive and negative samples, respectively. We randomly sample g entities from the learning resource entity set as a mini-batch and define the contrastive learning loss on it. Positive sample pairs are naturally obtained from the same entity, and negative sample pairs are shared by all other entities. We use the latent representation And using cosine similarity As a distance metric, each learning resource entity has the following contrastive loss function:

[0046]

[0047] Where S = {(l,r), (r,h), (l,h)} represents a set of multimodal sample pairs.

[0048] In step 2.2, a multi-graph attention mechanism is used to capture the diverse information from the three disciplines. The graph neural network encoder with multi-graph attention partitions the unified multimodal science knowledge graph into three subgraphs, obtaining three hidden representations of the target node. These three subgraphs fall into two categories: single-graph attention, where the target node and its adjacent nodes are of the same discipline; and cross-graph attention, where the target node and its adjacent nodes are of different disciplines.

[0049] The single-graph attention mechanism uses N to represent the total number of subjects, and the representation of the obtained node is:

[0050]

[0051] in, Represents entity e iIn the hidden representation of layer l, σ(·) is a nonlinear activation function, Represents entity e i The neighbor node set of the i-th subject knowledge graph, Represents entity e i In the relation triple (e i ,r,e j ), Represents the attention score of each message from the neighbor node. Neighbor Message The calculation method is:

[0052]

[0053] in It is nth i The transformation matrix of the knowledge graph, d is the dimension of entity and relationship embedding, and Concat(·) is the vector connection function. The attention score is defined as:

[0054]

[0055] in is the importance scoring function of the target node's neighbor nodes relative to the target node. The calculation formula is:

[0056]

[0057] Among them, γ is a learnable variable used to measure the importance of relationship r. It is nth i Two transformation matrices of the subject knowledge graph.

[0058] The cross-graph attention mechanism, the target node e i From the nth i The subject knowledge graph, whose neighbor nodes come from the nth j In a subject knowledge graph (j≠i), the embedding representation of the target node in the cross graph can be updated as follows:

[0059]

[0060] in Represents entity e i In the nth j The hidden representation of the lth layer in the subject knowledge graph, Indicates e i nth j The neighbor node set of a subject knowledge graph. The calculation method of neighbor information is:

[0061]

[0062] in It is nth j The transformation matrix of the subject knowledge graph, d is the dimension of entity and relationship embedding, and Concat(·) is the vector connection function. The attention score is defined as:

[0063]

[0064] in is the importance scoring function of the target node's neighbor nodes relative to the target node. The calculation formula is:

[0065]

[0066] Among them, γ is a learnable variable used to measure the importance of relationship r. They are nth i and nth j Two transformation matrices of the subject knowledge graph.

[0067] In step 2.4, based on the embedding representation of the given entity and relation, the score of the candidate triple is calculated through the knowledge graph decoder ConvE:

[0068]

[0069] Among them, f represents a nonlinear function, vec represents a vectorized operation, represents the embedded representation after two-dimensional reshaping, represents the relationship parameter, w represents the convolution filter, W e Indicates the transformation matrix, * indicates the convolution operation. At the same time, in order to increase the correct triple (e a ,r,e b ) score, reducing the error triples (e a ,r,e b ), minimizing the margin-based ranking loss function:

[0070]

[0071] in, is the set of triples consisting of all triples of the given three-subject knowledge graph, λ>0 is the margin hyperparameter, [x] + =max(x,0). Based on the model prediction results, the candidate entities are linked to the science text knowledge graph to form a complete multimodal middle school science knowledge graph.

[0072] In step S3, for example, questions raised by learners are collected, entities and relations are extracted from them, and the embedding representation of the head entity in the question is obtained. ai , tail entity embedding representation e bi and relation embedding representation p i , thus converting the problem into (e ai ,p i ,e bi ) in triple form.

[0073] Step S3 specifically includes the following sub-steps:

[0074] Step 3.1: Collect the questions raised by the learner and use the ELMo model as the question encoder. The questions input by the learner are segmented into tokens through byte pair encoding and then input into the ELMo model. The output of the ELMo model is context-aware token embedding, which is expressed as Where l is the length of the question (including the two special start and end tokens [CLS] and [SEP]), and d is the number of hidden units in the ELMo model.

[0075] In step 3.2, entity and relation extraction is formulated as a set prediction problem and modeled using a transformer-based non-autoregressive decoder. Before decoding begins, the decoder needs to know the size of the target triplet set. This is accomplished by requiring the non-autoregressive decoder to generate a fixed-size h prediction set for each question, where h is set to be larger than the number of triplets in the question. The decoder input is initialized with h learnable embeddings, called triplet queries, which are identical for all sentences.

[0076] The non-autoregressive decoder consists of a stack of X identical transformer layers. In each transformer layer, there is a multi-head self-attention mechanism to model the relationship between triplets, and a multi-head mutual attention mechanism to fuse the information of a given sentence. The non-autoregressive decoder does not have the constraint of autoregressive factorization of the output and does not need to prevent earlier decoding steps from accessing information from later steps. Therefore, an unmasked self-attention mechanism is used in the multi-head self-attention mechanism. The non-autoregressive decoder transforms h triplet queries into h output embeddings, represented as The output is then embedded into M through a feedforward network d Decode into relation types and entities independently, and obtain h final predicted triplets. Specifically, given d Embedded The output of , the predicted relationship type is obtained as follows:

[0077] p=softmax(W r g d ). (15)

[0078] The predicted head and tail entities are decoded by using four x-class classifiers to predict the start and end indices respectively:

[0079]

[0080] in, and is a learnable parameter and t is the total number of relation types.

[0081] use represents the set of ground truth triples, represents a set of h predicted triples, where h is greater than n. Each element i of the ground truth set can be viewed as Among them, P i is the target relationship type, Is the head entity e a or tail entity e b The starting or ending index of . Each element i of the predicted triple set is represented by

[0082]

[0083] Step 3.3, using the bipartite matching loss to score the predicted triples according to the ground truth, is divided into two steps: finding the best match and calculating the loss function. In order to find the ground truth triple set π * and predicted triple sets The optimal match between * element:

[0084]

[0085] Here, Π(h) is the space of all permutations of length h. ) is the ground truth value Y i and the pairwise matching cost between the predicted triples with index π(i). By considering the prediction of relation type and the prediction of entity span, Defined as:

[0086]

[0087] Optimal allocation π * The Hungarian algorithm is used in polynomial time (O(h 3 )). Consider the basic fact set S as a group of people in the assignment problem and the prediction triple set Consider it as a set of work. i (Person i) and The cost of the assignment (job j) is defined as The optimal matching with the minimum total cost is easily computed using the classic Hungarian algorithm.

[0088] Compute the loss function for all pairs matched in the previous step:

[0089]

[0090] Get the head entity embedding representation e in the question ai , tail entity embedding representation e bi and relation embedding representation p i , thus converting the problem into (e ai ,p i ,e bi ) in triple form.

[0091] In step S4, combined Figure 4 As shown, for example, a multimodal middle school science learning resource pre-training language model is designed to enable the model to learn the multimodal middle school science knowledge graph constructed above, and learn the question matching task on this basis, and put all candidate triples (e ai ,p i ,e bi ) Input retrieval and reasoning modules, the model gives corresponding answers based on the questions raised by the learners.

[0092] Step S4 specifically includes the following sub-steps:

[0093] Step 4.1, adopt the contrastive learning objective to align the representations of relevant learning question-relation pairs, while pushing away other question-relation pairs. To collect relevant question-relation pairs, given an example consisting of a question que, a subject entity Q, and an answer entity A, all the shortest paths between Q and A are extracted from the entire multimodal middle school science knowledge graph, and all the relations within these paths are considered to be related to que, denoted as p + , pre-trained using contrastive learning loss:

[0094]

[0095] where τ is the temperature hyperparameter, is the randomly sampled negative relation, and sim(que,p) is the cosine similarity.

[0096] In step 4.2, we fine-tune the multimodal science knowledge graph retrieval module to retrieve the subgraph of the given question que, thereby providing the reasoning scope for the reasoning module. The specific process is as follows:

[0097] Step 4.2.1: Use the semantic matching module and matching information propagation module to aggregate semantic matching features to update entity representation, obtain entity matching scores, and fine-tune the multimodal middle school science knowledge graph retrieval module. The semantic matching module is used to generate question que and triples (e ai ,p i ,e bi ). In this process, the tail entities in the triples with the same prefix are merged to generate the corresponding abstract node e b ′ is used to represent the set of tail entities. The same operation can be performed on the head entity. The text of question que and relation p is encoded using the pre-trained language model, and the output representation of [CLS] token is used as their representation. By adopting the corresponding projection layer, the question que and triple (e ai ,p i ,e bi ) The vector after t iterations captures the semantic matching features

[0098]

[0099] in, are the parameters of the projection layer after t iterations, m and d are the hidden dimensions of the pre-trained language model and feature vector, respectively, σ is the sigmoid activation function, and ⊙ is the Hadamard product.

[0100] Step 4.2.2, use the matching information propagation module to aggregate the generated semantic matching features to update the entity representation and obtain the entity matching score. In order to initialize the matching score, for a question que and a multimodal middle school science knowledge graph For each entity Replace que and e i The matching score between is set to: i is the subject entity (the entity mentioned in the knowledge graph about the question), then otherwise 0. In step t, use The matching score of the head entity calculated in the first step is used as the weight and the matching features from adjacent triples are aggregated to obtain the representation of the tail entity:

[0101]

[0102] in is the representation of entity e in step t, is a learnable matrix. In the first step, the representation of its single-hop relationship is aggregated into entity representation: in Is a learnable matrix. Based on all entities The representation of , uses the Softmax function to update the entity matching score as:

[0103]

[0104] in Is a learnable vector. After T steps of iteration, the final entity matching score z is obtained (T) .

[0105] Step 4.2.3, generate the matching score z of the nodes in the interdisciplinary multimodal subgraph according to the matching information propagation module A , use the marked answer to get the A The ground truth vector if The abstract node in contains the answer entity, which is set to 1, and the KL divergence between the learned and ground truth matching score vectors is minimized as:

[0106]

[0107] After fine-tuning the loss, a retrieval model can be effectively learned. It is used to retrieve the subgraph for a given question q, selecting the top K nodes based on the matching score, i.e., nodes within a reasonable distance from the subject entity.

[0108] In step 4.3, fine-tune the multimodal science knowledge graph reasoning module and select the best-ranked entity as the answer to the question based on the model's matching score. The specific process is as follows:

[0109] Step 4.3.1, use the matching information propagation module to fine-tune the subgraph obtained in the retrieval stage in the inference stage. As the retrieval model is fine-tuned, a smaller subgraph is obtained for each question que Initialize the parameters of the inference model with the parameters of the retrieval model, and use the same matching information propagation module as the retrieval stage, but do not need to merge the tail entities in the triples with the same prefix, and generate the matching score z of the node in the subgraph of the multimodal middle school science knowledge graph. R ;

[0110] Step 4.3.2, the learned matching score z is calculated based on the KL loss R and the ground truth vector Fitting:

[0111]

[0112] Where the subscript R indicates that the node comes from the retrieved subgraph. After fine-tuning the loss, the learned inference model can be used to select the best-ranked entity as the answer to the question based on the matching score.

[0113] In summary, the multimodal knowledge graph establishment method proposed in the first embodiment of the present invention utilizes the open source text knowledge graphs and target text knowledge graphs of multiple disciplines to extract structural, visual and text features, and performs fusion processing to obtain a multimodal knowledge graph. The multimodal knowledge graph is used to train the intelligent interaction model, thereby effectively improving the learning efficiency of students in knowledge information retrieval and intelligent question answering in the subject knowledge graph, and strengthening the students' comprehensive knowledge system; the bilinear model Tucker decomposition is used to decompose the tensor into the core tensor M c And 4 mode factors, continue to decompose the core tensor M c The representations of all modalities are merged into the same space with element-by-element multiplication to achieve fusion, realizing multimodal information fusion between the open source text knowledge graph and the target text knowledge graph; the single-graph attention mechanism and the cross-graph attention mechanism are used to handle the information fusion problems between the same disciplines and between different disciplines respectively, effectively solving the information inconsistency problem between knowledge graphs of different disciplines; the matching information propagation module is used to enable the language model to share the relevance information of the retrieval stage in the reasoning stage, effectively solving the problem of information loss such as context information and entity relevance information in the reasoning stage.

[0114] like Figure 5 As shown, the second embodiment of the present invention also provides a computer-readable storage medium 40 on which a computer program is stored. When the program is executed by the processor, the steps of the multimodal knowledge graph establishment method described in the first embodiment are implemented, and the computer-readable storage medium provided in this embodiment has the same beneficial effects as the multimodal knowledge graph establishment method provided in the first embodiment.

[0115] Among them, computer-readable storage media may include, but are not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0116] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0117] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0119] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0120] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for establishing a multimodal knowledge graph, characterized in that: include: Collect open source text knowledge graphs and target text knowledge graphs from multiple disciplines and extract structural features, visual features, and text features; fusing the open source text knowledge graph with the target text knowledge graph according to the structural features, the visual features, and the text features, and processing inconsistent information to obtain a multimodal knowledge graph; Collecting learners' interactive information and extracting entities and relationships to obtain head entity embedding representation, tail entity embedding representation, and relationship embedding representation of the interactive information, so as to convert the interactive information into triple form; Constructing a pre-trained language model of target knowledge, using the pre-trained language model to learn the multimodal knowledge graph, and using the multimodal knowledge graph to provide feedback information on the input triple-form interactive information; The step of fusing the open source text knowledge graph with the target text knowledge graph and processing inconsistent information to obtain a multimodal knowledge graph includes: Forcefully aligning entity pairs representing the same meaning in the open source text knowledge graph and the target text knowledge graph so that the entity pairs are represented by the same embedding vector; removing duplicate aligned entities in a unified subject relationship vector set, and maintaining the unified subject relationship vector set; The step of fusing the open source text knowledge graph with the target text knowledge graph and processing inconsistent information to obtain a multimodal knowledge graph includes: Use bilinear model Tucker decomposition to decompose the tensor into core tensors And 4 mode factors; continue to decompose the core tensor All modal representations are merged into the same space with element-wise product to achieve fusion; constraints are added to limit the correlation between different modal representations of the same learning resource entity, and contrastive learning of different entities and modalities is used as an additional learning objective for regularization to enhance multimodal fusion.

2. The method for establishing a multimodal knowledge graph according to claim 1, wherein: The extraction of structural features, visual features and text features includes: A graph convolutional network encoder with ConvE model loss is used to extract the structural features by representing the neighborhood aggregation as a triplet energy function; The visual encoder uses a residual neural network pre-trained on the ImageNet dataset to obtain the image embedding of the corresponding learning resource entity, and uses the output of the last hidden layer of the Softmax function as the visual feature; The BRET model is used as a learning resource text encoder to extract the text features.

3. The method for establishing a multimodal knowledge graph according to claim 1, wherein: The step of fusing the open source text knowledge graph with the target text knowledge graph and processing inconsistent information to obtain a multimodal knowledge graph includes: A single-graph attention mechanism is used to select adjacent nodes in one subject, and a cross-graph attention mechanism is used to select adjacent nodes in multiple subjects; The single-graph attention mechanism and the cross-graph attention mechanism respectively calculate an attention score function based on the importance score function of the target node relative to the adjacent nodes, and aggregate neighbor messages to obtain node representations in different states; Integrating the outputs of the single-graph attention mechanism and the cross-graph attention mechanism in different states according to the node representation to obtain a final representation of the target node; The candidate entities are linked to the target text knowledge graph according to the final representation to obtain the complete multimodal knowledge graph.

4. The method for establishing a multimodal knowledge graph according to claim 1, wherein: The step of collecting learners' interactive information and extracting entities and relationships to obtain a head entity embedding representation, a tail entity embedding representation, and a relationship embedding representation of the interactive information, so as to convert the interactive information into a triple form, includes: Collect learners’ interaction information and segment it into tokens through byte pair encoding, then input it into the ELMo model to output context-aware token embedding; Entity and relation extraction is treated as a set prediction problem and modeled using a transformer-based non-autoregressive decoder; The predicted triplets are scored according to the ground truth using the bipartite matching loss, and the best-ranked triples are selected to obtain the head entity embedding representation, tail entity embedding representation, and relation embedding representation in the question, thereby converting the question into a triple form.

5. The method for establishing a multimodal knowledge graph according to claim 1, wherein: Providing feedback information on the input interactive information in triple form by using the multimodal knowledge graph includes: Given a problem , main entity and answer entities , extracted from the multimodal knowledge graph and All shortest paths between and consider the shortest paths as the same as the problem Related,pretraining using contrastive learning loss; Adjust the retrieval module of the multimodal knowledge graph to retrieve questions The subgraph provides the reasoning scope for the reasoning module; Adjust the reasoning module of the multimodal knowledge graph to obtain matching scores and select the best-ranked entity as the answer to the question.

6. The method for establishing a multimodal knowledge graph according to claim 5, wherein: The retrieval module for adjusting the multimodal knowledge graph includes: Obtain questions based on the multimodal knowledge graph and triples The vector after t iterations captures semantic matching features , expressed as: ； in, , are the parameters of the projection layer after t iterations, 、 are the hidden dimensions of the pre-trained language model and feature vector, is the sigmoid activation function, is the Hadamard product; The semantic matching features generated by the matching information propagation module are aggregated to update the entity representation and obtain the entity matching score. In the tth step, the entity matching score is obtained by using The matching score of the head entity calculated in the first step is used as a weight to aggregate the matching features from adjacent triples to obtain the representation of the tail entity and update the entity matching score; The matching scores of the nodes in the multimodal knowledge graph subgraph are generated according to the matching information propagation module, and the basic truth vector represented by the matching score is obtained by using the marked answers to match the basic truth score vectors. Minimize losses.

7. The method for establishing a multimodal knowledge graph according to claim 6, wherein: The reasoning module for adjusting the multimodal knowledge graph includes: Initialize the parameters of the inference module using the parameters of the retrieval module, and use a matching information propagation module similar to the retrieval stage to generate matching scores for nodes in the multimodal knowledge graph subgraph; according to The loss fits the learned matching scores to the ground truth vectors The loss is adjusted and the learned inference model is used to select the best-ranked entity as the answer to the question based on the matching score.

8. A computer-readable storage medium, characterized in that It stores a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the method according to any one of claims 1 to 7.

Images