A few-sample knowledge graph completion method based on multimodal fusion in the field of education

Abstract

The present invention provides a method for completing a few-sample knowledge graph based on multimodal fusion in the field of education, which relates to the field of completing a few-sample knowledge graph, including: S1: collecting multi-source information such as images, texts and attributes of knowledge related to the field of education; S2: extracting features from the collected multi-source information, extracting attribute features by using a sub-attribute mapping matrix to construct a hierarchical type mapping matrix method, and generating feature representations of the multi-source information; S3: using a feature fusion method based on a recurrent neural network to represent the above features, and performing fusion representation after entity alignment; S4: extracting semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node; S5: combining the knowledge representation results obtained by the structure-based knowledge representation and the adjacent feature aggregation layer, and using a knowledge reasoning method based on a neural network to predict the tail entity. The present invention has the advantages of high link prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of few-sample knowledge graph completion, and relates to a few-sample knowledge graph completion method based on multimodal fusion in the field of education. Background Art

[0002] Although traditional knowledge graph completion models have achieved good results on various benchmark datasets, they all require a large number of relational facts to train the models. Compared with traditional knowledge graph completion technologies, few-shot knowledge graph completion technology can better adapt to actual needs. It can not only effectively reduce the processing of artificial samples, but also enable the model to quickly adapt to the endless stream of new samples. Therefore, the problem of few-shot knowledge graph completion has attracted the attention of a large number of researchers and has become an important research direction in the field of knowledge graphs.

[0003] In the knowledge graph of education, it is often very expensive to obtain information in the field of education. Moreover, in the education knowledge graph, since some knowledge entities have a small number of relationships connected to them, it is difficult to learn their semantic features, which to a certain extent restricts the completion ability of the knowledge graph. The reasoning problem of few-sample knowledge is extremely important and is of great significance to improving the ability of knowledge reasoning and graph completion. Therefore, how to combine the semantic information in the few samples to complete the knowledge graph is the first challenge to be solved.

[0004] Classical knowledge representation learning models often focus on the triple structured information of the knowledge graph itself, while ignoring the rich knowledge contained in multi-source information. These multi-source information include multi-modal information such as text information about entity descriptions, attribute information of entity types, and visual information of entity images, which can effectively supplement the triple structured information of the knowledge graph and help build more accurate knowledge representation results. Therefore, how to combine multi-source information such as knowledge itself, text descriptions, attribute categories, and visual images to establish cross-modal fusion of knowledge in a unified semantic vector space is the second challenge that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a method for completing a knowledge graph with a few samples based on multimodal fusion in the field of education in response to the above-mentioned problems existing in the existing technology. The technical problem to be solved by the present invention is how to complete the knowledge graph by combining the semantic information in the few samples.

[0006] The purpose of the present invention can be achieved through the following technical solutions: A method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, characterized in that it includes:

[0007] S1: Collect multi-source information such as images, texts, and attributes of relevant knowledge in the field of education;

[0008] S2: Extract features from the collected multi-source information, use the sub-attribute mapping matrix to construct a hierarchical type mapping matrix method to extract attribute features, and generate feature representations of multi-source information;

[0009] S3: The above features are represented by a feature fusion method based on a recurrent neural network, and then fused after entity alignment;

[0010] S4: Extract the semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node;

[0011] S5: Combining the knowledge representation results obtained by structure-based knowledge representation and the adjacent feature aggregation layer, a neural network-based knowledge reasoning method is used to complete the knowledge graph.

[0012] Preferably, the extracted image features, text features, and attribute features are extracted by a method comprising:

[0013] Image feature extraction: Use convolutional neural network (CNN) to obtain the feature representation of each image in the knowledge space, and use AlexNet convolutional neural network model to extract image features from entity images.

[0014] Text feature extraction: The Word2vec model is used to train on a large-scale corpus to obtain the word vector of the text, and used as the input of the convolutional neural network model.

[0015] Attribute feature extraction: A recursive hierarchical encoder based on a convolutional neural network is used to construct a hierarchical type mapping matrix using a sub-attribute mapping matrix. As the recursive hierarchical encoder is constructed from top to bottom, the hierarchical type mapping matrix is ​​constructed by recursively multiplying the sub-type mapping matrix to obtain the final attribute-based feature representation.

[0016] Preferably, the step of extracting image features from the entity image includes:

[0017] First, for the knowledge triple (h, r, t), all entity images of the head entity and the tail entity are taken as input, and the feature representation of each image in the knowledge space is obtained through the convolutional neural network CNN.

[0018] In the triple (h, r, t), h represents the head entity of the triple, r represents the relationship of the triple, and t represents the tail entity of the triple.

[0019] The attention mechanism is used to integrate the representations of each image and obtain the image-based feature representation of the entity.

[0020] Finally, the structure-based representation h s / t s and image-based representation h i / ti They are learned uniformly under the framework of the TransE translation distance model, and the knowledge semantic vector and the parameters of the convolutional neural network are updated through back propagation.

[0021] where h s / t s and h i / t i They are structure-based and image-based head and tail entity feature representations respectively.

[0022] Preferably, the step of extracting text features from the entity includes:

[0023] Two representation vectors are constructed for each entity: the first is a structure-based representation using h s and t s Represent the structure-based representation of the head entity and the tail entity respectively. This representation vector is completely determined by the structural information of the knowledge graph, which is similar to the distributed vector learned by the previous translation model;

[0024] The second is a text-based representation using h t and t t They represent the text-based representation of the head entity and the tail entity respectively. By fusing the text-based feature representation with the structured information based on the knowledge graph triples, the alignment and joint training of the text space and the knowledge space are achieved.

[0025] The maximum margin loss function is used for training, and finally the text-based feature representation h is obtained. t and t t .

[0026] Preferably, the step of extracting attribute-based feature representation from the entity includes: using a recursive hierarchical encoder constructed based on a convolutional neural network, and using a sub-attribute mapping matrix to construct a hierarchical type mapping matrix.

[0027] First, the input data is processed by the recursive hierarchical encoder of the convolutional neural network to obtain a preliminary feature representation;

[0028] Use the sub-attribute mapping matrix to map the preliminary features to the corresponding sub-attribute space to obtain a more fine-grained feature representation;

[0029] According to the sub-attribute representation, the sub-attribute is mapped to a higher-level hierarchical type space through the hierarchical type mapping matrix to obtain a more hierarchical feature representation;

[0030] With the help of the mapping matrix and the knowledge inference model based on the translation distance, training is performed according to the given scoring function. Finally, the attribute-based feature representation h is obtained through training. a and t a .

[0031] Preferably, after learning the feature representations based on images, texts and attributes respectively, a recurrent neural network model is used to fuse all the features after entity alignment, and finally a knowledge representation result integrating multi-source information is obtained. The specific steps include:

[0032] The feature representations based on images, texts and attributes are concatenated as the model input, and a bidirectional long short-term memory network is used as the feature fusion method to encode the semantic features of images, texts and attributes in sequence to generate a combined representation. A multimodal fusion method is used to model the interaction between different modalities. Subsequently, combined with the cross-modal attention mechanism, the single modal features are used to learn the association between modalities in parallel with the low-rank fusion modality, and further generate a comprehensive entity feature representation. By comparing each single modal feature with the overall feature representation of the entity, the entity similarity is learned, and the overall feature representation of the entity is continuously updated. The similarity is calculated based on the updated overall feature representation of the entity, and finally the most similar entity pair is determined, and the entity is aligned. Finally, the output is sent to a fully connected MLP network for decoding to obtain the knowledge representation result of multi-source information fusion.

[0033] Preferably, the specific steps of extracting the semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node include:

[0034] First, the adjacent node features are aggregated step by step through the entity and relationship levels, so that different types of semantic feature information can be aggregated based on the relationship path;

[0035] Secondly, in the relation-level feature aggregation part, different types of semantic information will be aggregated through the relation paths connected to entities;

[0036] Finally, the attention mechanism is used to obtain the importance of different relationship paths, so as to selectively aggregate important adjacent node features.

[0037] The present invention has the following beneficial effects:

[0038] By collecting multi-source information such as images, texts and attributes of relevant knowledge and performing feature extraction, a feature representation based on multi-source information is generated, and then the feature fusion method based on recurrent neural network is used to fuse the features, and the semantic features of adjacent entities are extracted to form an adjacent feature aggregation layer to supplement the semantic features of the central node. Finally, the knowledge representation results obtained by combining the structure-based knowledge representation and the adjacent feature aggregation layer are combined, and the knowledge reasoning method based on neural network is used to predict the tail entity. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0040] Figure 2 Diagram of the knowledge representation learning method for multi-source information fusion;

[0041] Figure 3 Figure 2 is a diagram of the entity and relationship feature aggregation method. DETAILED DESCRIPTION

[0042] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0043] Reference Figure 1 The present invention provides a method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, comprising:

[0044] S1: Collect multi-source information such as images, texts, and attributes of relevant knowledge in the field of education;

[0045] S2: Extract features from the collected multi-source information, use the sub-attribute mapping matrix to construct a hierarchical type mapping matrix method to extract attribute features, and generate feature representations of multi-source information;

[0046] S3: The above features are represented by a feature fusion method based on a recurrent neural network, and then fused after entity alignment;

[0047] S4: Extract the semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node;

[0048] S5: Combining the knowledge representation results obtained by structure-based knowledge representation and the adjacent feature aggregation layer, a neural network-based knowledge reasoning method is used to complete the knowledge graph.

[0049] Furthermore, the steps of extracting the knowledge graph in the field of education in the present invention are as follows:

[0050] S11: Collect various data related to the field of education, including text and non-text data from various sources such as textbooks, course outlines, teacher training materials, academic papers, student reports, etc.

[0051] S12: Clean and preprocess the collected data, including removing noise and redundant information, and performing natural language processing operations such as word segmentation, part-of-speech tagging, and syntactic analysis to prepare for subsequent knowledge extraction work.

[0052] S13: Use named entity recognition technology to identify entities related to the education field in the text, and extract the correlation, dependency, etc. between entities from the text.

[0053] S14: Based on the extraction results of entities and relationships, a knowledge graph model is established. Graph database methods are used to represent entities and relationships, confirm the attributes of entities and relationships, and associate them with corresponding data.

[0054] S15: Verify the constructed knowledge graph to ensure the accuracy and completeness of entities and relationships.

[0055] Furthermore, the specific steps of extracting image features from the entity image in step S2 are:

[0056] S21.1: For the knowledge triple (h, r, t), all entity images of the head entity and the tail entity are taken as input, and the feature representation of each image in the knowledge space is obtained through the convolutional neural network CNN.

[0057] S21.2: Use the attention mechanism to integrate the image representations and obtain the image-based feature representation of the entity.

[0058] S21.3: Finally, the structure-based representation h s / t s and the image-based representation h i / t i They are learned uniformly under the framework of the TransE translation distance model, and the knowledge semantic vector and the parameters of the convolutional neural network are updated through back propagation.

[0059] In the triple (h, r, t), h represents the head entity of the triple, r represents the relationship of the triple, and t represents the tail entity of the triple. s / t s and h i / t i They are structure-based and image-based head and tail entity feature representations respectively.

[0060] Specifically, in order to extract the feature representation of image information, the classic AlexNet convolutional neural network model is used to extract image features from entity images. In addition, in order to transfer the image feature vector from the image space to the knowledge space, an image mapping module is constructed, and the mapping matrix is ​​used to establish the representation of the image in the knowledge space. The process is as follows:

[0061] p i =M·f(img i ),#

[0062] in is the mapping matrix, d i represents the dimension of the image feature vector, and d s Represents the dimension of the knowledge space vector. Entity image img iThe image feature vector f(igm i ), and then the entity image is represented in the knowledge space through the image mapping module. i .

[0063] In addition, in order to enable the model to automatically select high-quality entity images to form a more accurate entity-based image representation, the present invention combines the image representation in the knowledge space with the entity structure-based knowledge representation to construct an instance-level attention model. The calculation method is as follows:

[0064]

[0065] in represents the structure-based knowledge representation of the kth entity, and att(x) represents the attention mechanism. According to the attention obtained by each entity image, each image instance is weighted when jointly constructing the entity-based image representation. The more similar the vector of the entity image in the knowledge space is to the structure-based representation of its corresponding entity, the higher the attention result, and the greater the influence of this entity image in the final construction of the entity-based image representation. Based on the model assumption of the translation distance, the following scoring function is determined:

[0066]

[0067] Among them, α1, α2, α3, and α4 are hyperparameters that control the scores of each item. s / h i and t s / t i They are the structure-based and image-based head and tail entity feature representations respectively. After model training, the image-based feature representation is finally obtained. i (i.e. h i and t i ).

[0068] Furthermore, the specific steps of extracting text features in step S2 are:

[0069] Two representation vectors are constructed for each entity: the first is a structure-based representation using h s and t s Represent the structure-based representation of the head entity and the tail entity respectively. This representation vector is completely determined by the structural information of the knowledge graph, which is similar to the distributed vector learned by the previous translation model;

[0070] The second is a text-based representation using h t and t tThey represent the text-based representation of the head entity and the tail entity respectively. By fusing the text-based feature representation with the structured information based on the knowledge graph triples, the alignment and joint training of the text space and the knowledge space are achieved.

[0071] Specifically, in order to extract important text information from entity descriptions as completely as possible and reduce the noise that may exist in the text information, the Word2vec model is used to train the text word vectors on a large-scale corpus and use them as the input of the convolutional neural network model. Formally, the model input uses M = {m1, ..., m n}, where n represents the length of the entity text after preprocessing. First, the convolutional layer will perform a window operation, the process is as follows:

[0072] e i =f(vec(f([M])*Ω)W),#

[0073] Where f(x) represents the nonlinear function ReLU, the symbol [x] represents the concatenation operation, * represents the convolution operation, Ω represents the convolution kernel, W represents the mapping matrix, and vec(x) represents the full connection operation. Similarly, based on the model assumption of the translation distance, the following scoring function is determined:

[0074]

[0075] where h s / h t and t s / t t are the structure-based and description-based head and tail entity feature vectors, respectively. The former is used to model the objective structural information in the knowledge graph, and the latter is used to integrate the text features of the entity description. The maximum margin loss function is also used for training, and finally the text-based feature representation e is obtained. t (i.e. h t and t t ).

[0076] Furthermore, the specific steps of extracting attribute features in step S2 are:

[0077] A recursive hierarchical encoder based on convolutional neural network is adopted, and the hierarchical type mapping matrix is ​​constructed using the sub-attribute mapping matrix.

[0078] First, the input data is processed by the recursive hierarchical encoder of the convolutional neural network to obtain a preliminary feature representation;

[0079] Use the sub-attribute mapping matrix to map the preliminary features to the corresponding sub-attribute space to obtain a more fine-grained feature representation;

[0080] According to the sub-attribute representation, the sub-attributes are mapped to a higher-level hierarchical type space through the hierarchical type mapping matrix to obtain a more hierarchical feature representation. For a type c with m sub-layers, the recursive hierarchical encoder constructs a type mapping matrix.

[0081] With the help of the mapping matrix and the knowledge inference model based on the translation distance, training is performed according to the given scoring function. Finally, the attribute-based feature representation h is obtained through training. a and t a Specifically, for a type c with m layers of subtypes, the recursive hierarchical encoder constructs its type mapping matrix as follows:

[0082]

[0083] where c (m) Represents the coarsest-grained sub-attribute in the hierarchy of c, c (1) Represents the finest-grained sub-attribute of c, Indicates subtype c (i) The corresponding mapping matrix.

[0084] With the help of the mapping matrix and based on the knowledge reasoning model assumption of the translation distance, the following scoring function is proposed to be defined:

[0085]

[0086] Among them, M r，h，1 and M r，t，2 Represents the type-specific mapping matrix based on the head entity and the tail entity, for example, M r，h，1 It represents the type highlighted by h as the head entity in relation r. During the training process, the maximum margin loss function is used for model training:

[0087]

[0088] Among them, γ represents the interval distance, and (h′, r, t′) represents the wrong triple, that is, the head entity or the tail entity in the correct triple is replaced by random selection to generate an incorrect triple that does not exist in the knowledge graph. Through model training, the attribute-based feature representation e is finally obtained. a (i.e. h a and t a ).

[0089] Furthermore, in step S3, after learning the feature representations based on images, texts and attributes respectively, all features are fused using a recurrent neural network model after entity alignment. The specific process is as follows:

[0090] Represent features based on images, texts and attributes i, e t , e a The concatenation is used as the model input, and the bidirectional long short-term memory network is used as the feature fusion method to encode the semantic features of the image, text and attributes in sequence. The calculation process is as follows:

[0091]

[0092] Among them, i, f, o and g represent the input, forgetting, output and gating coefficients in the recurrent neural network, c and x represent hidden layer features, U and W represent shared matrices, and σ represents the activation function. The combined representation h is generated by BiLSTM network encoding and input into a fully connected MLP network with one layer for decoding, and finally the knowledge representation result e of multi-source information fusion is obtained.

[0093] Among them, in low-rank modal learning, learning modal associations in parallel allows the features of each modality to be learned through independent paths, and learning to interact between modalities through shared parameters. The specific formula is as follows:

[0094]

[0095] Where Z represents the matrix of correlations between modes, M represents the number of modes, and W is a shared parameter matrix.

[0096] Among them, in the similarity comparison learning, the cosine similarity indicator is used to measure the similarity between entities. The specific formula is as follows:

[0097]

[0098] Where A and B represent the vectors of entity A and entity B.

[0099] Furthermore, in step S4, the present invention extracts semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node, wherein the adjacent feature aggregation layer includes entity level aggregation and relationship level aggregation.

[0100] For entity-level aggregation, due to the heterogeneity of knowledge graphs in the education field, we first use the relationship path-based entity-level feature aggregation. and Represents randomly initialized entity and relationship semantic features. The specific formula for entity-level feature aggregation is as follows:

[0101]

[0102] in Represents the adjacent entity based on the relationship path r, symbol and are the i-th entity and the aggregated features based on the relationship path r. The entity-level feature aggregation function Φ agg Following the GCN model. Therefore, the formula is:

[0103]

[0104] Where |x| represents the length of the set. For a given relationship path set Then there is The groups have different semantic aggregate features, each of which is generated based on a single relationship path and can represent one type of semantic information.

[0105] For relational level aggregation, specifically, in order to aggregate more comprehensive semantic information, a relation-based attention mechanism is established by aggregating features from different relational paths, and then used to aggregate various types of semantic information. The specific process of relational level feature aggregation is as follows:

[0106] against To aggregate semantic features at the entity level, we first calculate each relationship path The attention weight is calculated as follows:

[0107]

[0108] The relationship path weight expression is as follows:

[0109]

[0110] Symbol Φ att represents the attention function designed by a deep neural network, which can learn the path importance of different relations and then be used to selectively aggregate useful informative features.

[0111] In order to learn the weight value of each relationship path, we first analyze the semantic features of the relationship Perform nonlinear transformation (such as a single-layer perceptron model). Then, the importance of the relationship path is calculated through the relationship attention vector q. Finally, the weight value of the relationship path It can be obtained through the activation function. The specific calculation process is as follows:

[0112]

[0113] Where W represents the shared matrix in the nonlinear mapping process, b is the bias vector, q is the relational attention vector, and σ is the Sigmoid activation function used to map values ​​from 0 to 1. It should be noted that the above parameters are independent of entity or relational semantic features. Weight value The higher it is, the more important the relationship path r is.

[0114] After obtaining the importance of each relationship path, we can The weighting of the entity-level aggregated features is done as follows:

[0115]

[0116] Among them, CONCAT represents the concatenation operation, and Φ agg is the aggregation function. Finally, the next layer of entity and relationship feature aggregation can be expressed as:

[0117]

[0118] in and are the shared matrices related to entities and relationships, respectively, and f(x) represents the nonlinear activation function ReLU. After the feature aggregation process of layer l, the entity and relationship vectors with enhanced semantic features can be obtained. and

[0119] Furthermore, in step S5, the knowledge representation result obtained by combining the structure-based knowledge representation and the adjacent feature aggregation layer is used to predict the tail entity using a knowledge reasoning method based on a neural network. The specific steps are as follows:

[0120] First, the input matrix M of the neural network can be obtained by the following formula:

[0121] M=CONCAT{h,r}

[0122] CONCAT represents the concatenation operation. In this case, the convolution operation can extract the semantic interaction between h and r to output the feature map.

[0123] In addition, in order to model the complex relationships in the knowledge graph of the education field, a convolution kernel associated with each relationship is generated. It is used to extract the semantic features of entities under specific relationships. Therefore, the convolution kernel can extract relationship-specific features from the semantic features of the head entity, which is more effective in distinguishing the differences between entities. The specific process is as follows:

[0124]

[0125] Where * represents the convolution operation, v n represents the feature map generated after the convolution operation. Therefore, the scoring function can be defined as:

[0126] φ(h, r, t)=f((vec(f(V))W))t, #

[0127] Where vec(x) represents a full connection operation. By semantically matching with the feature vector t of the tail entity, the score of the knowledge triple (h, r, t) is obtained, and finally the confidence of the knowledge triple is obtained through the Sigmoid function.

[0128] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A few-sample knowledge graph completion method based on multimodal fusion in the field of education, characterized in that: include: S1: Collect multi-source information of images, texts and attributes related to education; S2: Extract features from the collected multi-source information, use the sub-attribute mapping matrix to construct a hierarchical type mapping matrix method to extract attribute features, and generate feature representations of multi-source information; S3: The above features are represented by a feature fusion method based on a recurrent neural network, and then fused after entity alignment; S4: Extract the semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node; S5: Combining the knowledge representation results obtained by structure-based knowledge representation and the adjacent feature aggregation layer, a neural network-based knowledge reasoning method is used to complete the knowledge graph; The step of extracting attribute features by using the sub-attribute mapping matrix to construct a hierarchical type mapping matrix method in step S2 includes: A recursive hierarchical encoder based on convolutional neural network is used to construct a hierarchical type mapping matrix using the sub-attribute mapping matrix; First, the input data is processed by the recursive hierarchical encoder of the convolutional neural network to obtain a preliminary feature representation; Use the sub-attribute mapping matrix to map the preliminary features to the corresponding sub-attribute space to obtain a more fine-grained feature representation; According to the sub-attribute representation, the sub-attribute is mapped to a higher-level hierarchical type space through the hierarchical type mapping matrix to obtain a more hierarchical feature representation; With the help of the mapping matrix and the knowledge reasoning model based on the translation distance, training is performed according to the given scoring function, and finally the attribute-based feature representation h is obtained. a and t a .

2. According to claim 1, a few-sample knowledge graph completion method based on multimodal fusion in the field of education is characterized in that: The method for extracting the image features, text features, and attribute features includes: Use convolutional neural network (CNN) to obtain the feature representation of each image in the knowledge space, and use AlexNet convolutional neural network model to extract image features from entity images; The Word2vec model is used to train large-scale corpora to obtain word vectors of texts, and used as input for the convolutional neural network model; A recursive hierarchical encoder based on a convolutional neural network is used to construct a hierarchical type mapping matrix using a sub-attribute mapping matrix. As the recursive hierarchical encoder moves from top to bottom, the hierarchical type mapping matrix is ​​constructed by recursively multiplying the sub-type mapping matrix to obtain the final attribute-based feature representation.

3. According to claim 2, a method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, characterized in that: The steps of extracting image features from entity images include: First, for the knowledge triple (h, r, t), all entity images of the head entity and the tail entity are taken as input, and the feature representation of each image in the knowledge space is obtained through the convolutional neural network CNN; In the triple (h, r, t), h represents the head entity of the triple, r represents the relationship of the triple, and t represents the tail entity of the triple; Then, the attention mechanism is used to integrate the image representations to obtain the image-based feature representation of the entity; Finally, the structure-based representation h s / t s and image-based representation h i / t i They are learned uniformly under the framework of the TransE translation distance model, and the knowledge semantic vector and the parameters of the convolutional neural network are updated through back propagation; where h s / t s and h i / t i They are structure-based and image-based head and tail entity feature representations respectively.

4. According to claim 2, a method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, characterized in that: The steps to extract text features from entities include: First, two representation vectors are constructed for each entity: the first is a structure-based representation using h s and t s Represent the structure-based representation of the head entity and the tail entity respectively; this representation vector is completely determined by the structural information of the knowledge graph, which is similar to the distributed vector learned by the previous translation model; The second is a text-based representation using h t and t t Represent the text-based representation of the head entity and the tail entity respectively; by fusing the text-based feature representation with the structured information based on the knowledge graph triple, the alignment and joint training of the text space and the knowledge space are achieved; The maximum margin loss function is used for training, and finally the text-based feature representation h is obtained. t and t t .

5. According to claim 2, a method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, characterized in that: After learning the feature representations based on images, texts, and attributes respectively, all features are fused after entity alignment, and finally the knowledge representation result of fusion of multi-source information is obtained. The specific steps include: The feature representations based on images, texts and attributes are concatenated as the model input, and a bidirectional long short-term memory network is used as the feature fusion method to encode the semantic features of images, texts and attributes in sequence to generate a combined representation. A multimodal fusion method is used to model the interaction between different modalities. Subsequently, combined with the cross-modal attention mechanism, the single modal features are used to learn the association between modalities in parallel with the low-rank fusion modality, and further generate a comprehensive entity feature representation. By comparing each single modal feature with the overall feature representation of the entity, the entity similarity is learned, and the overall feature representation of the entity is continuously updated. The similarity is calculated based on the updated overall feature representation of the entity, and finally the most similar entity pair is determined, and the entity is aligned. Finally, the output is sent to a fully connected MLP network for decoding to obtain the knowledge representation result of multi-source information fusion.

6. According to claim 1, a method for completing a knowledge graph based on a few samples in the field of education based on multimodal fusion, characterized in that: The specific steps of extracting the semantic features of adjacent entities to form an adjacent feature aggregation layer to supplement the semantic features of the central node include: First, the adjacent node features are aggregated step by step through the entity and relationship levels, so that different types of semantic feature information can be aggregated based on the relationship path; Secondly, in the relation-level feature aggregation part, different types of semantic information will be aggregated through the relation paths connected to entities; Finally, the attention mechanism is used to obtain the importance of different relationship paths, so as to selectively aggregate important adjacent node features.

7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a method for completing a few-sample knowledge graph based on multimodal fusion in the field of education as described in any one of claims 1 to 6.

8. A device for completing a few-sample knowledge graph based on multimodal fusion in the field of education, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a method for completing a few-sample knowledge graph based on multimodal fusion in the field of education as described in any one of claims 1 to 6.

Images