A Text-Image Enhanced Multimodal Knowledge Graph Embedding Method

Abstract

This invention discloses a text-image enhanced multimodal knowledge graph embedding method, comprising: improving the quality of entity graph image sets through image simulation algorithms; realizing high-quality image and text features using ResNet and BERT models respectively; and finally achieving effective fusion of multimodal features through a two-dimensional neural network model, thereby obtaining a more comprehensive and accurate representation of entities and relationships. This invention not only fully utilizes multimodal information to enhance entity representation but also improves the model's expressive power through its underlying network architecture, providing better support for knowledge graph applications.

Description

Technical Field

[0001] This invention relates to the technical field of knowledge graph embedding, and more particularly to a text-image enhanced multimodal knowledge graph embedding method. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, knowledge graphs have received widespread attention as a way to constitute knowledge representation. Knowledge Representation Learning (KRL) technology transforms triples in knowledge graphs into empowered forms, embedding entities into a low-dimensional operating space, providing important support for the application of knowledge graphs.

[0003] However, existing knowledge relation representation learning methods have several limitations: First, while traditional translation models such as TransE and TransH can handle even-one relations, their performance drops significantly when dealing with complex relations such as even-many and many-to-one relationships, severely impacting the representational capabilities of knowledge graphs. Second, current mainstream knowledge graph embedding models primarily employ single-mode feature learning, such as using only textual information for entity representation. This approach overlooks the rich semantic information contained in entity modalities (such as images and audio), resulting in incomplete and inaccurate entity representations. Third, while existing shallow neural network models have efficiency advantages, their limited expressive power makes it difficult to capture the complex mathematical connections between entities and relations, especially when dealing with hierarchical and progressive knowledge. Furthermore, in practical applications, knowledge graphs often face the problem of data sparsity, meaning that a large number of entities lack direct relational connections, making it difficult for traditional embedding methods relying solely on structural information to learn accurate entity representations.

[0004] For existing multimodal knowledge graphs, the image datasets of entities generally suffer from problems such as low correlation between images and entities and inaccurate extraction, which directly affects the learning effect of knowledge representation based on multimodal information. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by this invention is the lack of single-modal representation, limited expressive power of shallow models, and poor quality of multimodal data in existing knowledge graph embedding methods.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: using an image filtering algorithm to preprocess all entity image sets to obtain a high-quality image set corresponding to each entity;

[0009] Based on the high-quality image set, image features are extracted using a deep residual network ResNet, and entity image feature representations are obtained through global average pooling.

[0010] The text descriptions of all entities are input into the BERT model to obtain text feature vectors, and the entity text feature representations are obtained through fully connected layers.

[0011] The entity structure representation, text feature representation, and image feature representation of all entities are input into a 2D modal neural network model. The 2D modal neural network model adopts the ConvE model structure for multimodal fusion and knowledge graph embedding to obtain entity / relation representations of fused text and images.

[0012] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the image preprocessing includes:

[0013] Perform unification and color space conversion operations on all entity images;

[0014] Calculate and normalize the histogram data of the image;

[0015] The similarity score between images is calculated using the histogram correlation comparison method;

[0016] A high-quality image set related to the entity is obtained by filtering based on the set similarity threshold.

[0017] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the calculation formula for the histogram relevance comparison method is as follows:

[0018]

[0019] Where N is the total number of bins in the histogram. H is the histogram mean of the k-th image. k (J) represents the value of the Jth bin of the histogram of the kth image, d(H1,H2) represents the similarity score of the histograms of the two images, and H1 and H2 are the two histograms to be compared.

[0020] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the deep residual network ResNet includes an input layer, an original surface layer, multiple residual block groups, a global average pooling layer, and a fully connected layer.

[0021] The multiple residual block groups achieve nonlinear mapping of features through residual connections, and the global average pooling layer is used to compress the spatial dimension of the feature map.

[0022] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the scoring function for entity structure representation in the 2D morphological neural network model is:

[0023]

[0024] Among them, R r ∈R k， It is a relation parameter that depends on the relationship R between the head and tail entities. H represents s and R r The two-dimensional structure, correspondingly, if H s R r ∈R k ,So At this time, k = k w k h * indicates a convolution operation, w is the filter of the 2D convolutional layer, W is the fully connected projection matrix, and H s T represents the structural representation of the head entity of a triple in a knowledge graph. s The structural representation of the tail entity of a triple in a knowledge graph.

[0025] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the scoring function for entity text feature representation in the 2D morphological neural network model is:

[0026]

[0027] in, Both the head and tail entities in the text are entity text features, while The head entity in the diagram represents the entity's structural features, while the tail entity represents the entity's textual features. In this context, the head entity represents the entity's textual features, while the tail entity represents the entity's structural features. t The text representation of the head entity of a triple, T t The text representation of the tail entity of a triple.

[0028] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the scoring function for entity image feature representation in the 2D morphological neural network model is defined as:

[0029]

[0030] in, The head and tail entities in the preceding part are both entity image features, while In this context, the head entity represents the entity's structural features, while the tail entity represents the entity's image features. In the diagram, the head entity represents entity image features, while the tail entity represents entity structural features. i The image representation of the head entity of a triple, T i A visual representation of the tail entity of a triple.

[0031] As a preferred embodiment of the text-image enhanced multimodal knowledge graph embedding method described in this invention, the 2D shape neural network model needs to be trained, including:

[0032] A scoring function combining entity structure representation, entity text feature representation, and entity image feature representation is used to score the entity;

[0033] Constructing a binary cross-entropy loss function:

[0034]

[0035] Where N is the sample size, t i For feature labels, s i To predict the score, for a 1-1 rating, t is a dimension R 1×1 The label vector is denoted by R, and for scores of 1-N, the dimension is represented as R. 1×N ;

[0036] The loss function is then optimized and trained using the Adam optimizer.

[0037] The beneficial effects of this invention are:

[0038] 1. By using image filtering algorithms, the quality of entity image datasets can be improved, images with low relevance to entities can be removed, and higher quality knowledge representations can be obtained;

[0039] 2. By training the entity relationship representation using a 2D convolutional neural network model, the feature interaction between entities and relationships can be maximized;

[0040] 3. Use a 2D convolutional neural network model to fuse features from various modalities, thereby more efficiently integrating entity text features and entity image features into entity structural features. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0042] Figure 1 This is a flowchart illustrating the text-image enhanced multimodal knowledge graph embedding method of the present invention;

[0043] Figure 2 This is a schematic diagram of the process for obtaining entity text representations according to the present invention;

[0044] Figure 3 This is a schematic diagram of the process for extracting image features according to the present invention;

[0045] Figure 4 This is a schematic diagram of the residual block group structure shown in this invention;

[0046] Figure 5 This is a schematic diagram illustrating the process of representing an entity image according to the present invention;

[0047] Figure 6 This is a schematic diagram of the ConVE model structure shown in this invention. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0049] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a text-image enhanced multimodal knowledge graph embedding method, which specifically includes:

[0052] S1. Use image filtering algorithms to preprocess all entity image sets to obtain a high-quality image set corresponding to each entity.

[0053] S2. Based on a high-quality image set, image feature extraction is performed using a deep residual network ResNet, and entity image feature representation is obtained through global average pooling.

[0054] S3. Input the text descriptions of all entities into the BERT model to obtain text feature vectors, and obtain entity text feature representations through fully connected layers;

[0055] S4. Input the entity structure representation, text feature representation and image feature representation of all entities into the 2D modal neural network model. The 2D modal neural network model adopts the ConvE model structure for multimodal fusion and knowledge graph embedding to obtain entity / relation representations of fused text and images.

[0056] The following is combined Figures 2-6 The flowchart shown, along with some preferred or optional examples of the present invention, more specifically describes the implementation process and / or effects of certain embodiments of the present invention.

[0057] Image preprocessing

[0058] It should be noted that the image filtering algorithm used in this embodiment is used to filter out images that are not related to the entity. Specifically, by using an image similarity algorithm (histogram comparison method), the set of images associated with the entity is found, that is, the histogram data of the two images to be compared is obtained. Then, the histogram data is normalized to finally obtain a similarity index. By setting the boundary of the similarity index, it is possible to determine whether the images are the same entity.

[0059] In an optional implementation, a histogram comparison method is used to find the set of images associated with the entity, including:

[0060] All images are preprocessed, including resizing to a uniform size, converting to the same color space (such as HSV), and denoising, to ensure consistency in the comparison results;

[0061] Calculate the histogram of the image, select appropriate color channels (such as H and S channels in HSV), divide the pixel values ​​into a certain number of bins, count the frequency of pixels in each bin, and normalize the histogram to the [0,1] interval to eliminate the influence of image size on the comparison.

[0062] The similarity between the reference image and the candidate image is calculated using a histogram comparison method.

[0063] Correlation comparison process: For multiple input images, first obtain their histograms {H1, H2, ..., H}. nThen, using the relevance comparison formula, each image of the same entity is compared pairwise with the histograms of all other images to obtain the histogram similarity score d. If the histogram similarity between an image and the other images is less than the set similarity threshold, the image is considered to have low relevance to the entity and is removed.

[0064] For example, if we define the histograms of the two images to be compared as {H1, H2}, then the formula for comparing the correlation between the two histograms is:

[0065]

[0066] Where N is the total number of bins in the histogram. H is the histogram mean of the k-th image. k (J) represents the value of the Jth bin of the histogram of the kth image, d(H1,H2) represents the similarity score of the histograms of the two images, and H1 and H2 are the two histograms to be compared.

[0067] After obtaining the similarity scores of all images, images that meet the conditions are selected according to the set threshold, and these images are combined into an image set related to the entity.

[0068] Obtain the image set {img1, img2, ..., img} corresponding to the entity. n}, where img i This represents the image set composed of images associated with the i-th entity.

[0069]

Obtaining Image Feature Vectors

[0070] After removing images with low relevance, the extraction of image feature vectors is then performed, referring to... Figure 3 The flowchart shown (entity images 1-4 represent four different images of the same entity) illustrates the implementation steps of the ResNet network, specifically including:

[0071] Input layer: Input the image corresponding to each entity. in, This represents the m-th image of the i-th entity;

[0072] The input image is normalized by scaling the pixel values ​​to the range of [0,1]. Based on the input size of ResNet, the input image is adjusted to the corresponding size. The processed image has a shape of (C,H,W), where C is the number of channels, H is the height of the image, and W is the width of the image.

[0073] Initial convolutional layer: Spatial downsampling and feature extraction are performed on the input image (shape (C,H,W)). This layer consists of a convolutional layer of size k. h ×kw The convolutional kernel consists of a batch normalization layer and a ReLU activation function, resulting in an image with shape (C,H1,W1).

[0074] Residual block set: The structure of the basic residual block is as follows Figure 4 As shown, a residual block group consists of multiple residual blocks. Each residual block group is responsible for extracting higher-level features from the features extracted from the previous group. Through residual linking, each residual block group can learn a complex nonlinear mapping between the input and the output.

[0075] The operation of a residual block group is as follows:

[0076] Convolution: The first convolutional layer uses a convolutional kernel, and the output shape is C×H×W;

[0077] Batch normalization: Perform batch normalization on the output after convolution;

[0078] ReLU activation function: applies ReLU activation to the output;

[0079] The second convolutional layer: The second convolutional layer outputs a shape of C×H×W;

[0080] Residual connection: The input is added to the output after convolution to form a residual connection. The output obtained after the residual connection still has the shape of C×H×W.

[0081] Average Pooling Layer: After feature extraction and non-linear mapping through a series of residual blocks, the ResNet network uses a global average pooling layer as the last convolutional layer. In this layer, each feature map is reduced to a single numerical value, significantly reducing model parameters and computational cost. The input is the output from the last residual block, with a shape of C. f ×H f ×W f Global average pooling is applied to the feature map of each channel to calculate the average value for each channel. The spatial dimension (H) is then... f ×W f Compressed to 1, the output shape is C. f ×1;

[0082] Fully connected layer: The pooled feature vectors are input into the fully connected layer, which maps them to a space representing the number of classes. The output of the fully connected layer has the shape of output_class_number. size ×1 (keeping the same size as the entity text feature vector);

[0083] Through the processing of the above layer structure, the feature vector of each image of the entity is obtained, referring to... Figure 5 Then, the vectors of multiple images of the entity are concatenated into an m×output array. sizeThe feature matrix of the dimensional vector (where m is the total number of entity images of the i-th entity), and then this feature matrix (with shape m×output) size The two-dimensional vector is average-pooled along the m-dimensional axis to obtain a one-dimensional vector with a dimension of 1×output. size This feature vector fuses features from all images for each entity.

[0084]

Obtaining Text Feature Vectors

[0085] Furthermore, in this embodiment, the BERT model in PyTorch is used to obtain text feature vectors. The text descriptions of entities are obtained from annotation files in the WordNet corpus and input into the BERT model to obtain all word vectors {x1, x2, ..., x...} corresponding to the entity text descriptions. n Then, the output of the BERT model is input into a fully connected layer to obtain a one-dimensional embedding vector representing the entity text features;

[0086] Reference Figure 2 Specifically, it includes the following steps:

[0087] Extracting entity definitions from WordNet: Use the `wn.synsets` method in the NLTK library to find the entity's synonym set (Synset) and extract the definition text from it. For example, for the entity "dog", find the entity's synonym set {dog.n.01, dog.n.02, ...}, where n.01 and n.02 are two different synonym sets, corresponding to the meaning of "dog" in different contexts. The `synset.definition()` method provides the text description of the synonym set, and the most suitable text description is selected as the entity's text description using professional knowledge (e.g., "A dog is a domesticated carnivorous mammal.").

[0088] Load the pre-trained BERT model and tokenizer: Using the Transformers library, load the BERT model and the corresponding BertTokenizer, input the obtained entity description text into the BERT model for tokenization, and obtain word vectors;

[0089] The segmented input is passed to the BERT model: the BERT model returns the word vector for each token (a one-dimensional vector of the same length);

[0090] Concatenate the n word vectors into an n×hidden size The two-dimensional vector is used as the entity text description vector (assuming there are n words in the sentence, and the word vector dimension of each word is hidden). size);

[0091] The two-dimensional vector describing the entity text is then input into the fully connected layer to obtain a one-dimensional entity text feature vector; the calculation formula for the fully connected layer is:

[0092] Y = XW T +b

[0093] Where X is the input matrix with shape (n, hidden) size ), W T Let be the transpose of the weight matrix, with the shape (hidden). size ,output size b is the bias matrix with shape (1, output). size Y is the output of the fully connected layer, with shape (n, output). size );

[0094] The output of the fully connected layer (with shape (n, output)) size A two-dimensional vector is average-pooled along the n-dimensional axis to obtain a one-dimensional vector with dimension 1×output. size This one-dimensional vector is used as the feature vector of the entity text.

[0095] [2D Convolution Knowledge Image Embedding]

[0096] It should be noted that structure-based representation of knowledge graphs refers to representing entities and relations in a knowledge graph by utilizing the structural features of the graph. The structure-based representation method focuses on the position of entities and relations in the graph, the connection method, and their interrelationships, rather than directly relying on external semantic information (such as text and images).

[0097] The 2D convolutional knowledge graph embedding proposed in this embodiment makes up for the shortcomings of knowledge graph representation based solely on structure. It integrates the text and image representations of entities into the knowledge graph embedding for knowledge graph learning, thereby improving the expressiveness and generalization ability of the embedding model.

[0098] In knowledge graphs, triples representing entity structure features are represented as (H s R, T s The triplet representation using entity text features is (H) t R, T t The triplet representation of entity image features is (H) i R, T i ), H s T represents the structural representation of the head entity of a triple in a knowledge graph. sH represents the structural representation of the tail entity of a triple in a knowledge graph. t T represents the text representation of the head entity of the triples obtained from the text feature vector. t H represents the text representation of the tail entity of the triple obtained from the text feature vector. i T represents the image representation of the head entity of the triples obtained from the image feature vector. i R represents the image representation of the tail entity of the triple obtained from the image feature vector, and R represents the relationship between the head and tail entities. All different entity representations share the corresponding relationship vector.

[0099] Preferably, the 2D convolutional model used in this embodiment is a ConvE model, and the ConvE model structure is as follows: Figure 6 As shown, according to the model architecture diagram, the vectors of the head entity and relation are stacked and then reshaped into a two-dimensional tensor (similar to an image representation). After convolution, a feature map is obtained, which is then projected into a k-dimensional space through a fully connected layer. In the inner product layer, it is matched with the embedding of the candidate target (using a scoring function) to finally obtain the entity / relation representation that fuses text and image.

[0100] As an example, in a 2D morphological neural network model, the scoring function for entity structure representation is:

[0101]

[0102] Among them, R r ∈R k， It is a relation parameter that depends on the relationship R between the head and tail entities. H represents s and R r The two-dimensional structure, correspondingly, if H s R r ∈R k ,So At this time, k = k w k h * indicates a convolution operation, w is the filter of the 2D convolutional layer, W is the fully connected projection matrix, and H s T represents the structural representation of the head entity of a triple in a knowledge graph. s The structural representation of the tail entity of a triple in a knowledge graph;

[0103] To integrate entity text features, the scoring function for entity text feature representation in the 2D morphological neural network model is:

[0104]

[0105] in, Both the head and tail entities in the text are entity text features, while The head entity in the diagram represents the entity's structural features, while the tail entity represents the entity's textual features. In this context, the head entity represents the entity's textual features, while the tail entity represents the entity's structural features. t The text representation of the head entity of a triple, T t The text representation of the tail entity of a triple.

[0106] As an example, to fuse entity image features, in a 2D morphological neural network model, the scoring function for entity image feature representation is defined as:

[0107]

[0108] in, The head and tail entities in the preceding part are both entity image features, while In this context, the head entity represents the entity's structural features, while the tail entity represents the entity's image features. In the diagram, the head entity represents entity image features, while the tail entity represents entity structural features. i The image representation of the head entity of a triple, T i A visual representation of the tail entity of a triple.

[0109] In an optional implementation, the 2D shape neural network model needs to be trained, including:

[0110] A scoring function combining entity structure representation, entity text feature representation, and entity image feature representation is used to score the entity;

[0111] Constructing a binary cross-entropy loss function:

[0112]

[0113] Where N is the sample size, t i For feature labels, s i To predict the score, for a 1-1 rating, t is a dimension R 1×1 The label vector is denoted by R, and for scores of 1-N, the dimension is represented as R. 1×N ;

[0114] Then, the Adam optimizer is used to optimize and train the loss function.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text-image enhanced multi-modal knowledge graph embedding method, characterized in that, The method comprises the following steps: image preprocessing is performed on all entity image sets by using an image filtering algorithm to obtain a high-quality image set corresponding to each entity; based on the high-quality image set, image feature extraction is performed by using a deep residual network ResNet, and entity image feature representation is obtained by global average pooling; text description of all entities is input into a BERT model to obtain a text feature vector, and entity text feature representation is obtained by a fully connected layer; entity structure representation, text feature representation and image feature representation of all entities are input into a 2D modal neural network model, the 2D modal neural network model adopts a ConvE model structure, is used for multi-modal fusion and knowledge graph embedding, and obtains entity / relationship representation fused with text and image; in the 2D modal neural network model, the scoring function of the entity structure representation is: ； wherein R r ∈R k is a relation parameter depending on the relation R between the head and tail entities, , denotes the two-dimensional structure of H s and R r , and correspondingly, if H s , R r ∈R k , then , ∈R kw×kh , at this time k=k w k h , * denotes the convolution operation, w is the filter of the 2D convolution layer, W is the full connection projection matrix, H s denotes the structural representation of the head entity of the triple in the knowledge graph, and T s denotes the structural representation of the tail entity of the triple in the knowledge graph. in the 2D modal neural network model, the scoring function of the entity text feature representation is: ； wherein, the head entity and the tail entity in the are both entity text features, while the head entity in the is an entity structure feature, while the tail entity is an entity text feature, the head entity in the is an entity text feature, while the tail entity is an entity structure feature, H t denotes a text representation of the head entity of the triple, T t denotes a text representation of the tail entity of the triple; in the 2D modal neural network model, the scoring function of the entity image feature representation is defined as: ； wherein, both the head entity and the tail entity in the front part of the head entity is an entity structure feature and the tail entity is an entity image feature, the head entity is an entity image feature and the tail entity is an entity structure feature, i denotes an image representation of the head entity of the triple, i denotes an image representation of the tail entity of the triple. 2.The text-image enhanced multi-modal knowledge graph embedding method of claim 1, wherein, the image preprocessing comprises: unification and color space conversion are performed on all entity images; histogram data of the image is calculated and normalized; histogram correlation comparison method is used to calculate the similarity score between images; according to the set similarity threshold, a high-quality image set related to the entity is obtained. 3.The text-image enhanced multi-modal knowledge graph embedding method of claim 2, wherein, the calculation formula of the histogram correlation comparison method is: ； ； where N is the total number of bins in the histogram, Hk is the mean of the histogram of the kth image, k (J) is the value of the Jth bin of the histogram of the kth image, Hsim is the similarity score of two histograms, H1 and H2 are two histograms to be compared. 4.The text-image enhanced multi-modal knowledge graph embedding method of claim 1, wherein, the structure of the deep residual network ResNet comprises an input layer, an original curved surface layer, a plurality of residual block groups, a global average pooling layer and a fully connected layer; wherein, the plurality of residual block groups realize nonlinear mapping of features through residual connection, and the global average pooling layer is used for compressing the spatial dimension of the feature map. 5.The text-image enhanced multi-modal knowledge graph embedding method of claim 1, wherein, the 2D modal neural network model needs to be trained, comprising: combined with the scoring functions of the entity structure representation, the entity text feature representation and the entity image feature representation, the sigmoid function is used for scoring; a binary cross entropy loss function is constructed: ； where N is the number of samples, is the feature label, is the predicted score, for scoring 1-1, t is a label vector of dimension R 1×1 ; for scoring 1-N, the dimension is denoted as R 1×N ; and then the Adam optimizer is used to optimize and train the loss function.

Images