An autonomous construction method of a multi-modal common sense knowledge graph

Abstract

The application provides a kind of self-construction method of multi-modal common sense knowledge graph, belongs to knowledge graph technical field.The application uses large-scale general text common sense knowledge graph ConceptNet, guarantees the high quality and diversity of entity;Using the picture data set ImageNet21K with accurate annotation, it is guaranteed that the text knowledge graph can effectively land as a multi-modal knowledge graph;The double-path contrast learning algorithm used, compared with the previous single-path contrast learning algorithm-contrast the similarity of entity and picture, there is the contrast of entity and picture text label, make full use of the high-quality annotation of ImageNet21K data set, make the landing of entity more accurate;Use large language model to conduct common sense test on the constructed multi-modal knowledge graph, guarantee the accuracy of the constructed multi-modal knowledge graph;The method does not need to be trained, can be calculated in batches, can be quickly expanded and applied to other fields.

Description

Technical Field

[0001] This invention provides an autonomous construction method for a multimodal common sense knowledge graph, specifically involving a dual-path comparison learning algorithm and a common sense verification algorithm, which effectively transforms a textual common sense knowledge graph into a multimodal (text-image) common sense knowledge graph, belonging to the field of knowledge graph technology in artificial intelligence. Background Technology

[0002] Knowledge graphs are typically composed of triples r(h,t) consisting of entities (head node h and tail node t) and relations (r), such as the capital city (China, Beijing). Knowledge graphs in which entities (head and tail nodes) include images are called multimodal knowledge graphs.

[0003] Publicly available multilingual resources, such as Wikipedia, play a crucial role in constructing knowledge graphs. However, the rich visual information in Wikipedia is often difficult to utilize, which limits the ability of models to incorporate rich multimodal data. Although some structured knowledge graphs containing images are publicly available, they either only cover a limited range of domains [1], or although they span multiple domains, the image quality varies greatly [2], or a single image constitutes a scene graph and lacks universality [3]. Therefore, researching how to quickly construct a general, high-quality, multimodal commonsense knowledge graph is of great significance.

[0004] [1] Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied knowledge representation learning. In Proceedings of the 26th International Joint Conference on Artificial Intelli-gence, IJCAI'17, page 3140–3146. AAAIPress.

[0005] [2]Roberto Navigli and Simone Paolo Ponzetto. 2012. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193: 217–250.

[0006] [3]Krishna, Ranjay, et al. "Visual genome: Connecting language and vision using crowdsourced dense image annotations." International journal of computervision 123(2017):32-73. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention proposes an autonomous construction method for a multimodal common sense knowledge graph. Utilizing dual-path comparative learning and common sense verification, this method effectively transforms textual common sense knowledge graphs into multimodal common sense knowledge graphs. It covers general common sense knowledge, features high-quality images, and has a large data scale, making it a rapid method for constructing general, high-quality multimodal common sense knowledge graphs.

[0008] For convenience, the following lists the names and meanings of relevant terms in this invention:

[0009] CLIP-ViT: is a deep neural network that can be used as a text encoder and image encoder to convert text and images into vector representations;

[0010] ConceptNet is a general-purpose textual commonsense knowledge graph containing millions of concepts and relationships, covering a wide range of topics including science, culture, history, geography, and more.

[0011] ImageNet21K is a massive image dataset collected from the internet, containing over 20,000 classes of accurately labeled images. This ensures that text knowledge graphs can be effectively translated into visual knowledge graphs. Each image in ImageNet21K has a text label, which is crucial for dual-path contrastive learning.

[0012] The technical solution of the present invention is as follows:

[0013] A self-construction method for a multimodal commonsense knowledge graph is proposed. This method uses the general text commonsense knowledge graph ConceptNet as the base text commonsense knowledge graph; it uses images from the ImageNet21K image dataset to match entities in ConceptNet, effectively transforming the text knowledge graph into a visual knowledge graph; it uses CLIP-ViT as a text and image encoder to convert the original text or image into vectors; it uses a large language model to determine whether triples conform to commonsense; and for each commonsense triple in ConceptNet, it performs the following steps to transform it into a multimodal commonsense triple:

[0014] 1) Extract all labels (label 1, label 2, ..., label m) and images (image 1, image 2, ..., image m) from the ImageNet21K image dataset, and convert them into embedding representations (T1, T2, ..., Tm) using the CLIP-ViT text encoder and image encoder. m (I1, I2, ..., I) m );

[0015] 2) For two entities in a common sense triple: entity 1 and entity 2, use CLIP-ViT's text encoder to convert their entities into embedded representations C1 and C2;

[0016] 3) Calculate C1 and (T1, T2, ..., T... m Similarity (S) 11 S 12 S 1m ), calculate C2 and (T1, T2, ..., T m Similarity (S) 21 S 22 S 2m );

[0017] 4) Calculate C1 and (I1, I2, ..., I... m The similarity (S') 11 S' 12 ..., S' 1m ), calculate C2 and (I1, I2, ..., I m The similarity (S') 21 S' 22 ..., S' 2m );

[0018] 5) (S) 11 S 12 S 1m ) and (S' 11 S' 12 ..., S' 1m Calculate the average value, select the image with the highest similarity value, indicating that entity 1 is mapped to label 1-image 1, and then calculate (S). 21 S 22 S 2m ) and (S' 21 S' 22 ..., S' 2m Calculate the average value and select the image with the highest similarity value, indicating that entity 2 is mapped to label 2-image 2. Thus, the triple of relation 1 (entity 1, entity 2) is mapped to a new triple: relation 1 (label 1-image 1, label 2-image 2), which is a multimodal commonsense triple.

[0019] 6) Use a large language model to perform common sense testing. By manually constructing prompt words that include positive and negative examples, the large language model can understand the task type and judge whether the new triplet obtained in step 5) conforms to common sense according to the examples of the prompt words. If it conforms to common sense, the new triplet is retained; if it does not conform to common sense, the new triplet is deleted.

[0020] For all triples in the general text common sense knowledge graph ConceptNet, perform steps 2) to 6) to complete the landing and filtering of all text common sense triples into multimodal common sense triples, and realize the autonomous construction of multimodal common sense knowledge graph.

[0021] Furthermore, the similarity calculation in steps 3) and 4) uses cosine similarity calculation.

[0022] Furthermore, the large language model used in step 6) is Google's latest open-source Gemma large language model.

[0023] The technical effects of this invention are as follows:

[0024] This invention presents an autonomous construction method for a multimodal commonsense knowledge graph. It utilizes the large-scale, general-purpose text-based commonsense knowledge graph ConceptNet, ensuring the high quality and diversity of triples composed of entities and relations. It employs the accurately labeled image dataset ImageNet21K, guaranteeing the effective transformation of the text knowledge graph into a multimodal (text-image) knowledge graph. A dual-path contrastive learning algorithm is used, which, compared to the previous single-path contrastive learning algorithm (comparing the similarity between entities and images), adds the comparison of entity and image text labels, fully leveraging the high-quality annotations of the ImageNet21K dataset for more accurate entity identification. A large language model is used to perform commonsense checks on the constructed multimodal knowledge graph, removing entity triples that do not conform to commonsense, thus ensuring the accuracy of the constructed multimodal knowledge graph. This method requires no training, can be performed in batches, and can be quickly expanded and applied to other fields. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the autonomous construction method of a multimodal commonsense knowledge graph according to the present invention. Detailed Implementation

[0026] The present invention will be further clearly and completely described below with reference to the accompanying drawings and specific embodiments.

[0027] This invention relates to an autonomous construction method for a multimodal commonsense knowledge graph, the method flow of which is as follows: Figure 1 As shown.

[0028] Figure 1 (Corresponding to step 1 on the left), CLIP-ViT is used to process the ImageNet21K labels and images into embedded representations (T1, T2, ..., T...). m (I1, I2, ..., I) m ); Figure 1 In step 2), entities 1 and 2 in ConceptNet are converted into embedded representations C1 and C2. Figure 1 In the middle, corresponding to steps 3) and 4), the similarity of C1, C2, and all images and labels is calculated respectively; Figure 1 The following corresponds to step 5), taking the label and image with the highest average similarity and their relationship with the original to form a new triplet "relationship 1 (label 1-image 1, label 2-image 2)"; Figure 1 On the right side, corresponding to step 6), the large language model Gemma is used to determine whether the new triplet conforms to common sense.

[0029] Example 1:

[0030] First, obtain all approximately 20,000 images of objects from different categories in the ImageNet21K dataset, along with their text labels, including images of different categories such as "vegetables" and "refrigerators". Input all the ImageNet21K object images into the CLIP-ViT image encoder to obtain the embedding representations of all images, which are then saved locally for later use. Next, input all the ImageNet21K text labels into the CLIP-ViT text encoder to obtain the embedding representations of all text labels, which are also saved locally for later use.

[0031] To further illustrate this invention, we select "located in (fresh produce, refrigerator)" from the ConceptNet triples. "Fresh produce" and "refrigerator" are referred to as two entities in ConceptNet, and "located in" represents the relationship between these two entities. "Fresh produce" and "refrigerator" are input into the CLIP-ViT text encoder to obtain embedding representations C1 and C2. Cosine similarity is calculated between C1 and C2 and all text labels and image embedding representations from ImageNet21K. The label-image pair with the highest average similarity is selected. The label with the highest similarity for "fresh produce" is "vegetables," so "fresh produce" is matched with "vegetables." Similarly, "refrigerator" is matched with "refrigerator," resulting in the new triple "located in (vegetables-vegetable image, refrigerator-refrigerator image)."

[0032] Next, a commonsense check is performed. Clue words containing both positive and negative examples are manually constructed for the Gemma large language model. The new triple "located in (vegetables-vegetable image, refrigerator-refrigerator image)" is then input into the Gemma large language model, which determines whether it conforms to common sense based on the examples in the clue words. Since vegetables being located in a refrigerator is common sense, Gemma outputs "correct," and this triple is retained in the final knowledge graph.

[0033] The above process is performed on all triples in ConceptNet, and all triples that pass the commonsense test together constitute the final multimodal commonsense knowledge graph.

[0034] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for autonomously constructing a multimodal commonsense knowledge graph, characterized in that, This method uses the general text commonsense knowledge graph ConceptNet as the basic text commonsense knowledge graph, uses images from the ImageNet21K image dataset to match entities in ConceptNet, uses CLIP-ViT as a text and image encoder to convert the original text or image into vectors, and uses a large language model to determine whether the triples conform to commonsense. For each commonsense triple in the ConceptNet, the following steps are performed to convert it into a multimodal commonsense triple: 1) Extract all labels (label 1, label 2, ..., label m) and images (image 1, image 2, ..., image m) from the ImageNet21K image dataset, and convert them into embedding representations (T1, T2, ..., Tm) using the CLIP-ViT text encoder and image encoder. m (I1, I2, ..., I) m ); 2) For two entities in a common sense triple: entity 1 and entity 2, use the text encoder of CLIP-ViT to convert their entities into embedded representations C1 and C2; 3) Calculate C1 and (T1, T2, ..., T... m Similarity (S) 11 S 12 S 1m ), calculate C2 and (T1, T2, ..., T m Similarity (S) 21 S 22 S 2m ); 4) Calculate C1 and (I1, I2, ..., I... m Similarity (S) 11 S 12 S 1m ), calculate C2 and (I1, I2, ..., I m Similarity (S) 21 S 22 S 2m ); 5) (S) 11 S 12 S 1m ) and (S 11 S 12 S 1m Calculate the average value, select the image with the highest similarity value ''', indicating that entity 1 is mapped to label 1-image 1, and then (S 21 S 22 S 2m ) and (S 21 S 22 S 2m Calculate the average value and select the image with the highest similarity value, indicating that entity 2 is mapped to label 2-image 2; thus, the triple of relation 1 (entity 1, entity 2) is mapped to a new triple: relation 1 (label 1-image 1, label 2-image 2), which is a multimodal common sense triple; 6) Use the large language model to perform common sense testing. By manually constructing prompt words that include positive and negative examples, the large language model can be helped to understand the task type and judge whether the new triplet obtained in step 5) conforms to common sense according to the examples of the prompt words. If it conforms to common sense, the new triplet is retained; if it does not conform to common sense, the new triplet is deleted. For all triples in the general text common sense knowledge graph ConceptNet, perform steps 1) to 6) to complete the landing and filtering of all text common sense triples into multimodal common sense triples, and realize the autonomous construction of multimodal common sense knowledge graph.

2. The method for autonomously constructing a multimodal commonsense knowledge graph as described in claim 1, characterized in that, The similarity calculation in steps 3) and 4) uses cosine similarity calculation.

3. The method for autonomously constructing a multimodal commonsense knowledge graph as described in claim 1, characterized in that, The large language model used in step 6) is Google's latest open-source Gemma large language model.

Images