A method and device for constructing a multimodal knowledge graph based on multi-level knowledge association

Abstract

The present invention discloses a method and device for constructing a multimodal knowledge graph based on multi-level knowledge association. The method includes the following steps: S1, obtaining image and text data; S2, constructing an image node based on a single image; constructing a recursive tree of a single paragraph based on the word co-occurrence between sentences in a single paragraph in a single text data, calculating the distance between nodes at the same level in the two recursive trees, thereby calculating the overall distance between paragraphs, and constructing a text node for the single text data; S3, calculating the association score R between each text node and each image node, and connecting edges if the association score R is greater than a first threshold, otherwise not connecting edges, thereby constructing a multimodal knowledge graph. This method starts with raw Internet data, extracts paragraph text, and constructs a single text node by merging similar paragraph texts, thereby constructing a lightweight multimodal knowledge graph and reducing construction complexity.

Description

Technical Field

[0001] The present invention relates to multimodal knowledge graph construction, and in particular to a method and device for constructing a multimodal knowledge graph based on multi-level knowledge association. Background Art

[0002] The construction of traditional multimodal knowledge graphs generally uses a single entity as a node. This construction method requires processes such as entity extraction and linking, multimodal relationship mapping, etc. The process is complicated and relies on multiple models.

[0003] If a single text node is constructed from paragraphs or merged paragraphs, a distance metric between paragraphs is required. Existing work includes: 1) methods based on character or word co-occurrence frequencies, such as edit distance and Jaccard coefficient. These methods are computationally efficient but insensitive to semantic relationships. 2) Word embedding-based methods map words in the text into dense vectors and use optimal transfer methods to calculate the shortest distance, such as Words2Vec and WMD. 3) Methods represent the entire paragraph as a single vector and measure similarity using vector distances (such as cosine similarity). Examples include USE and SBERT. However, these methods perform poorly when the length of paragraphs varies significantly. Furthermore, in the field of long text summarization, RSB divides long book text into multiple paragraphs and collects human summaries of adjacent paragraphs. This method continues recursively until the entire book is summarized, generating a book-specific summary tree. However, this method fails to model fine-grained semantic relationships between sentences.

[0004] Existing work includes: 1) extracting entity objects from images using object recognition models and mapping them to entities in text, such as GAIA. 2) mapping images and text into a shared semantic space and calculating the distance between embedding vectors, such as LCIP and CLIP. 3) encoding images and text into binary hash codes and efficiently calculating similarity using Hamming distance, such as MRDH and SGRN. However, when establishing associations between long texts and images, these works often encounter situations where semantically relevant content may be scattered across multiple parts of the long text, requiring abstract summarization of the long text before associating it with the image.

[0005] In summary, the existing technology cannot effectively handle the problem of how to measure the distance between paragraphs when the text lengths of the paragraphs differ greatly. Summary of the Invention

[0006] This method starts from raw Internet data, extracts paragraph text, and merges similar paragraphs to construct a single text node, building a lightweight multimodal knowledge graph and reducing construction complexity.

[0007] This method addresses the problem of measuring distances between paragraphs when text lengths vary significantly. It proposes constructing each paragraph as a recursive tree containing multi-level knowledge and performing distance measurement based on this recursive tree. Specifically, sentences are used as leaf nodes, and a fine-grained distance measurement is constructed between them. Then, a large text model is used to summarize similar sentences, merging sentences with similar content. This merging process continues until the root node is reached. Finally, the distance between paragraphs is calculated using all nodes in the corresponding layers of the recursive tree.

[0008] The present invention provides the following technical solution: a method for constructing a multimodal knowledge graph based on multi-level knowledge association, comprising the following steps:

[0009] S1. Obtain image and text data;

[0010] S2. Construct an image node based on a single image; construct a recursive tree for a single paragraph based on the word co-occurrence between sentences in a single paragraph in a single text data, calculate the distance between nodes at the same level in the two recursive trees, and thus calculate the overall distance between paragraphs, thereby constructing a text node for the single text data;

[0011] S3. Calculate the association score R between each text node and each image node. If the association score R is greater than a first threshold, connect them; otherwise, do not connect them, thereby constructing a multimodal knowledge graph.

[0012] Furthermore, the text node of a single text data is constructed in step S2 as follows:

[0013] (1) Divide a single paragraph into sentences based on punctuation and number them, using the sentences as leaf nodes of the recursive tree;

[0014] (2) Use the word segmentation tool to segment the individual sentences in all leaf nodes, and then count the word co-occurrences between each sentence and other sentences. The formula is defined as follows:

[0015] C(i,j)=(Number(Wi∩Wj)) / (Number(Wi));

[0016] Where C(i,j) represents the word co-occurrence of sentence number i relative to sentence number i, Wi represents the set of all words after sentence number i is segmented; Wj represents the set of all words after sentence number j is segmented; Number() represents the number of elements in the set;

[0017] (3) Use the large text model to determine the logical correlation O(i, j) between sentence number i and sentence number j. If there is a correlation, O(i, j) is the second threshold; if there is no correlation, O(i, j) is the third threshold.

[0018] (4) The distance between two sentences is defined as C(i,j)+C(j,i)+O(i,j);

[0019] (5) Cluster all leaf nodes according to the distance between two sentences, and then use the text model to summarize each category after clustering, and merge each category into a parent node;

[0020] (6) Using the large text model, the parent nodes are combined into nodes on the next level until they are combined into the root node, thereby constructing a recursive tree for the paragraph;

[0021] (7) Repeat steps (1) to (6) to construct recursive trees for all paragraphs;

[0022] (8) Calculate the distance between nodes at the same level in two recursive trees, thereby calculating the overall distance between paragraphs;

[0023] (9) The overall distance is input into the clustering algorithm to cluster the paragraphs and construct text nodes of individual text data.

[0024] Furthermore, the steps for calculating the distance between nodes at the same level in two recursive trees and thus calculating the overall distance between paragraphs are as follows:

[0025] The sentence sets of the corresponding layer nodes of the two recursive trees are defined as X={x1,…xi…,xm}, Y={y1,…yj…,yn}, where x1 represents the content of the first node of a layer of a recursive tree, and xm represents the content of the last node of the layer, that is, the content of the mth node; y1 represents the content of the first node of a layer of the other recursive tree, and yn represents the content of the last node of the layer;

[0026] The formula for calculating the distance between nodes at the same level of two recursive trees using the optimal transmission method is as follows:

[0027] ;

[0028] Where ∏(X,Y) represents all possible paths from sentence set X to sentence set Y; P(i,j) is an m×n transmission matrix, each element of which represents the probability of matching xi to yi. The optimal P(i,j) is found through a linear programming algorithm;

[0029] The overall distance between two recursive trees is defined as:

[0030] ;

[0031] Where k represents the number of layers, and L is the number of layers of the recursive tree with the larger number of layers in the two recursive trees.

[0032] Furthermore, step S3 is specifically as follows:

[0033] (1) Calculate the relevance score R between a single text node and a single image node, as shown below:

[0034] ;

[0035] Where tree is the number of the recursive tree, image is the number of the image, and T is the total number of recursive trees in the text node. R(tree,image) is the relationship score between a single recursive tree and a single image node, calculated as follows:

[0036] ;

[0037] Among them, Normalization() represents the normalization operation; IS a is the correlation score between the a-th layer node and the image node obtained by the large language model, b represents the total number of layers in the recursive tree; I(image, tree) is a logical function. If a single recursive tree and a single image node have the same source, it is the fourth threshold; otherwise, it is the fifth threshold;

[0038] The image nodes and text nodes whose correlation scores R are greater than the first threshold are connected to construct a multimodal knowledge graph.

[0039] A multimodal knowledge graph construction device based on multi-level knowledge association, comprising:

[0040] Acquisition module, used to obtain image and text data;

[0041] The first construction module constructs an image node based on a single image; constructs a recursive tree based on the word co-occurrence between sentences in a single paragraph in a single text data, calculates the distance between nodes at the same level in two recursive trees, and thus calculates the overall distance between paragraphs, thereby constructing a text node for a single text data;

[0042] The second construction module calculates the association scores of text nodes and image nodes. If the association score is greater than a threshold, the edges are connected; otherwise, no edges are connected, thereby constructing a multimodal knowledge graph.

[0043] An electronic device, comprising:

[0044] one or more processors;

[0045] a memory for storing one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method.

[0047] A computer-readable storage medium stores computer instructions, which implement the steps of the above method when executed by a processor.

[0048] The beneficial effects of the present invention are as follows:

[0049] (1) This method starts with raw Internet data, extracts paragraph text, and constructs a single text node by merging similar paragraph texts, thereby building a lightweight multimodal knowledge graph and reducing the construction complexity.

[0050] (2) Based on the paragraphs, a recursive tree containing multiple levels of knowledge is constructed, and the distance between paragraphs is calculated based on this tree. This can better model the semantic relationship between paragraphs at multiple levels.

[0051] (3) When a text node contains a lot of text, multi-level semantic association between the paragraph-based recursive tree and the image can effectively alleviate the problem of difficulty in associating long texts and images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of a method for constructing a multimodal knowledge graph based on multi-level knowledge associations according to the present invention;

[0053] Figure 2 This is a module diagram of a multimodal knowledge graph construction device based on multi-level knowledge associations of the present invention. DETAILED DESCRIPTION

[0054] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0055] The process of building a lightweight multimodal knowledge graph based on multi-level knowledge association is as follows: 1) First, clean the internet data and distinguish paragraphs and images; 2) Perform multi-level knowledge association between paragraphs and merge similar paragraphs to form a single text node; 3) Based on the multi-level knowledge in 2), associate single image nodes with single text nodes; 4) Form a lightweight multimodal knowledge graph.

[0056] The embodiments of the present invention are further described below with reference to a number of embodiments.

[0057] Example 1

[0058] A method for constructing a multimodal knowledge graph based on multi-level knowledge association includes the following steps:

[0059] S1. Obtain image and text data;

[0060] This method uses Python crawler tools to search the Internet for words such as drone, drone detection, flight, and field, and obtains Internet articles as raw data for building a lightweight multimodal knowledge graph. Then, the raw data is cleaned to remove words such as <xml>< / xml> , The raw data typically consists of multiple fragmented paragraphs and images. These fragmented paragraphs may belong to the same online article, or they may be a paragraph on one webpage and an unrelated advertisement on another.

[0061] S2. Construct an image node based on a single image; construct a recursive tree for a single paragraph based on the word co-occurrence between sentences in a single paragraph in a single text data, calculate the distance between nodes at the same level in the two recursive trees, and thus calculate the overall distance between paragraphs, thereby constructing a text node for the single text data;

[0062] The specific steps of constructing a text node of a single text data in step S2 are as follows:

[0063] (1) Divide a single paragraph into sentences based on punctuation and number them, using the sentences as leaf nodes of the recursive tree;

[0064] (2) Use a word segmentation tool to segment individual sentences in all leaf nodes. In this embodiment, the Jieba word segmentation tool is used, but other word segmentation tools can also be used. Then, the word co-occurrence between each sentence and other sentences is counted. The formula is defined as follows:

[0065] C(i,j)=(Number(Wi∩Wj)) / (Number(Wi));

[0066] Where C(i,j) represents the word co-occurrence of sentence number i relative to sentence number i, Wi represents the set of all words after sentence number i is segmented; Wj represents the set of all words after sentence number j is segmented; Number() represents the number of elements in the set;

[0067] (3) Use the large text model to determine the logical correlation O(i, j) between sentence number i and sentence number j. If there is a correlation, O(i, j) is the second threshold; if there is no correlation, O(i, j) is the third threshold. In this embodiment, the second threshold is 1 and the third threshold is 0, but other values ​​are also possible.

[0068] (4) The distance between two sentences is defined as C(i,j)+C(j,i)+O(i,j);

[0069] (5) Cluster all leaf nodes according to the distance between two sentences, and then use the text model to summarize each category after clustering, and merge each category into a parent node;

[0070] (6) Using the large text model, the parent nodes are combined into nodes on the next level until they are combined into the root node, thereby constructing a recursive tree for the paragraph;

[0071] (7) Repeat steps (1) to (6) to construct recursive trees for all paragraphs;

[0072] (8) Calculate the distance between nodes at the same level in two recursive trees, thereby calculating the overall distance between paragraphs;

[0073] (9) The overall distance is input into the clustering algorithm to cluster the paragraphs and construct text nodes of individual text data.

[0074] The steps to calculate the distance between nodes at the same level in two recursive trees, and thus the overall distance between paragraphs, are as follows:

[0075] The sentence sets of the corresponding layer nodes of the two recursive trees are defined as X={x1,…xi…,xm}, Y={y1,…yj…,yn}, where x1 represents the content of the first node of a layer of a recursive tree, and xm represents the content of the last node of the layer, that is, the content of the mth node; y1 represents the content of the first node of a layer of the other recursive tree, and yn represents the content of the last node of the layer;

[0076] The formula for calculating the distance between nodes at the same level of two recursive trees using the optimal transmission method is as follows:

[0077] ;

[0078] Where ∏(X,Y) represents all possible paths from sentence set X to sentence set Y; P(i,j) is an m×n transmission matrix, each element of which represents the probability of matching xi to yi. The optimal P(i,j) is found through a linear programming algorithm;

[0079] The overall distance between two recursive trees is defined as:

[0080] ;

[0081] Where k represents the number of layers, and L is the number of layers of the recursive tree with the larger number of layers in the two recursive trees.

[0082] S3. Calculate the association score R between each text node and each image node. If the association score R is greater than a first threshold, connect them; otherwise, do not connect them, thereby constructing a multimodal knowledge graph.

[0083] The image nodes and text nodes whose correlation scores R are greater than a first threshold are connected to construct a multimodal knowledge graph. In this embodiment, the first threshold is 0.8, but it can also be other values.

[0084] Step S3 is as follows:

[0085] (1) Calculate the relevance score R between a single text node and a single image node, as shown below:

[0086] ;

[0087] Where tree is the number of the recursive tree, image is the number of the image, and T is the total number of recursive trees in the text node. R(tree,image) is the relationship score between a single recursive tree and a single image node, calculated as follows:

[0088] ;

[0089] Among them, Normalization() represents the normalization operation; IS a is the correlation score between the a-th layer node and the image node obtained by the large language model, b represents the total number of layers in the recursive tree; I(image, tree) is a logical function. If a single recursive tree and a single image node have the same source, it is the fourth threshold; otherwise, it is the fifth threshold;

[0090] A multimodal knowledge graph construction device based on multi-level knowledge association, comprising:

[0091] Acquisition module, used to obtain image and text data;

[0092] The first construction module constructs an image node based on a single image; constructs a recursive tree based on the word co-occurrence between sentences in a single paragraph in a single text data, calculates the distance between nodes at the same level in two recursive trees, and thus calculates the overall distance between paragraphs, thereby constructing a text node for a single text data;

[0093] The second construction module calculates the association scores of text nodes and image nodes. If the association score is greater than a threshold, the edges are connected; otherwise, no edges are connected, thereby constructing a multimodal knowledge graph.

[0094] An electronic device, comprising:

[0095] one or more processors;

[0096] a memory for storing one or more programs;

[0097] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method.

[0098] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the methods and systems, methods, and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0100] On the other hand, a computer-readable storage medium stores computer instructions thereon, which implement the steps of the above-mentioned method when executed by a processor. When the computer program is executed by the processor, it implements the method as described in any one of the first aspects above. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory 101 (ROM), a random access memory 101 (RAM), a magnetic disk or an optical disk.

[0101] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for constructing a multimodal knowledge graph based on multi-level knowledge association, characterized in that: The steps include: S1. Obtain image and text data; S2. Construct an image node based on a single image; construct a recursive tree for a single paragraph based on the word co-occurrence between sentences in a single paragraph in a single text data, calculate the distance between nodes at the same level in the two recursive trees, and thus calculate the overall distance between paragraphs, thereby constructing a text node for the single text data; S3. Calculate the correlation score R between each text node and each image node. If the correlation score R is greater than a first threshold, connect the edges; otherwise, do not connect the edges, thereby constructing a multimodal knowledge graph. The specific steps of constructing a text node of a single text data in step S2 are as follows: (1) Divide a single paragraph into sentences based on punctuation and number them, using the sentences as leaf nodes of the recursive tree; (2) Use the word segmentation tool to segment the individual sentences in all leaf nodes, and then count the word co-occurrences between each sentence and other sentences. The formula is defined as follows: C(i,j)=(Number(Wi∩Wj)) / (Number(Wi)); Where C(i,j) represents the word co-occurrence of sentence number i relative to sentence number i, Wi represents the set of all words after sentence number i is segmented; Wj represents the set of all words after sentence number j is segmented; Number() represents the number of elements in the set; (3) Use the large text model to determine the logical correlation O(i, j) between sentence number i and sentence number j. If there is a correlation, O(i, j) is the second threshold; if there is no correlation, O(i, j) is the third threshold. (4) The distance between two sentences is defined as C(i,j)+C(j,i)+O(i,j); (5) Cluster all leaf nodes according to the distance between two sentences, and then use the text model to summarize each category after clustering, and merge each category into a parent node; (6) Using the large text model, the parent nodes are combined into nodes on the next level until they are combined into the root node, thereby constructing a recursive tree for the paragraph; (7) Repeat steps (1) to (6) to construct recursive trees for all paragraphs; (8) Calculate the distance between nodes at the same level in two recursive trees, thereby calculating the overall distance between paragraphs; (9) The overall distance is input into the clustering algorithm to cluster the paragraphs and construct text nodes of individual text data.

2. The method according to claim 1, characterized in that The steps to calculate the distance between nodes at the same level in two recursive trees, and thus the overall distance between paragraphs, are as follows: The sentence sets of the corresponding layer nodes of the two recursive trees are defined as X={x1,…xi…,xm}, Y={y1,…yj…,yn}, where x1 represents the content of the first node of a layer of a recursive tree, and xm represents the content of the last node of the layer, that is, the content of the mth node; y1 represents the content of the first node of a layer of the other recursive tree, and yn represents the content of the last node of the layer; The formula for calculating the distance between nodes at the same level of two recursive trees using the optimal transmission method is as follows: ； Where ∏(X,Y) represents all possible paths from sentence set X to sentence set Y; P(i,j) is an m×n transmission matrix, each element of which represents the probability of matching xi to yi. The optimal P(i,j) is found through a linear programming algorithm; The overall distance between two recursive trees is defined as: ； Where k represents the number of layers, and L is the number of layers of the recursive tree with the larger number of layers in the two recursive trees.

3. The method according to claim 1, characterized in that Step S3 is as follows: (1) Calculate the correlation score R between a single text node and a single image node, as shown below: ； Where tree is the number of the recursive tree, image is the number of the image, and T is the total number of recursive trees in the text node. R(tree,image) is the relationship score between a single recursive tree and a single image node, calculated as follows: ； Among them, Normalization() represents the normalization operation; IS a is the correlation score between the a-th layer node and the image node obtained by the large language model, b represents the total number of layers in the recursive tree; I(image, tree) is a logical function. If a single recursive tree and a single image node have the same source, it is the fourth threshold; otherwise, it is the fifth threshold; The image nodes and text nodes whose correlation scores R are greater than the first threshold are connected to construct a multimodal knowledge graph.

4. A multimodal knowledge graph construction device based on multi-level knowledge association, characterized in that: include: Acquisition module, used to obtain image and text data; The first building module builds an image node based on a single picture; A recursive tree is constructed based on the word co-occurrence between sentences in a single paragraph in a single text data. The distance between nodes at the same level in two recursive trees is calculated, thereby calculating the overall distance between paragraphs and constructing the text node of the single text data. The second construction module calculates the correlation scores between text nodes and image nodes. If the correlation score is greater than a threshold, the edges are connected; otherwise, no edges are connected, thus constructing a multimodal knowledge graph. The specific steps to construct a text node for a single text data are as follows: (1) Divide a single paragraph into sentences based on punctuation and number them, using the sentences as leaf nodes of the recursive tree; (2) Use the word segmentation tool to segment the individual sentences in all leaf nodes, and then count the word co-occurrences between each sentence and other sentences. The formula is defined as follows: C(i,j)=(Number(Wi∩Wj)) / (Number(Wi)); Where C(i,j) represents the word co-occurrence of sentence number i relative to sentence number i, Wi represents the set of all words after sentence number i is segmented; Wj represents the set of all words after sentence number j is segmented; Number() represents the number of elements in the set; (3) Use the large text model to determine the logical correlation O(i, j) between sentence number i and sentence number j. If there is a correlation, O(i, j) is the second threshold; if there is no correlation, O(i, j) is the third threshold. (4) The distance between two sentences is defined as C(i,j)+C(j,i)+O(i,j); (5) Cluster all leaf nodes according to the distance between two sentences, and then use the text model to summarize each category after clustering, and merge each category into a parent node; (6) Using the large text model, the parent nodes are combined into nodes on the next level until they are combined into the root node, thereby constructing a recursive tree for the paragraph; (7) Repeat steps (1) to (6) to construct recursive trees for all paragraphs; (8) Calculate the distance between nodes at the same level in two recursive trees, thereby calculating the overall distance between paragraphs; (9) The overall distance is input into the clustering algorithm to cluster the paragraphs and construct text nodes of individual text data.

5. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to claims 1 to 3.

6. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to claims 1 to 3 are implemented.

Images