Multi-modal knowledge graph construction method and system supporting image-text mutual query

Abstract

The invention relates to the technical field of knowledge engineering, and discloses a multi-modal knowledge graph construction method and system supporting image-text mutual query, and the method comprises the steps: obtaining an image and an associated text in a multi-modal data source; creating image nodes for the images, and extracting text entities such as diseases and symptoms from associated texts; establishing an accurate semantic relationship between the image node and the text entity by adopting a presentation and belonging double-relationship model; meanwhile, the associated text is vectorized, and a graph structure, an entity, a relationship and a semantic vector are integrally stored in a single database instance, so that the atomicity and the consistency of data are ensured. The invention further provides a mixed retrieval method, graph structure retrieval based on entities and semantic similarity retrieval based on query vectors are executed in parallel, the results are fused and sorted, and the knowledge graph is retrieved by combining the structural advantages of the knowledge graph and the semantic generalization ability of vector retrieval. And the accuracy and recall ratio of image-text mixed retrieval are obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge engineering, in particular to a multi-modal knowledge graph construction method and system supporting image-text mutual search. BACKGROUND

[0002] In the professional fields of veterinary, medical, industrial detection, etc., images as a kind of information carrier are increasingly valued. A large number of professional image documents provide valuable resources for the inheritance and utilization of knowledge. How to quickly and accurately retrieve target images from these unstructured image-text materials has become a key technical challenge to improve industry efficiency.

[0003] Current image retrieval technology largely depends on the text information associated with the image, such as file name, caption, metadata or context paragraph around the image. However, the effectiveness of this method is highly dependent on the quality and completeness of the text description. In many practical scenarios, the text annotation may be very short, ambiguous or even missing, making it difficult to retrieve relevant images through keyword matching alone.

[0004] In addition, there is often a semantic gap between the user's query intent and the text description of the image, for example, the user may query based on the disease name, while the direct description of the image is a specific symptom caused by the disease. Traditional search engines cannot understand the deep association between the disease and the symptom, thus missing a large number of relevant images, resulting in low recall rate of the search. Although knowledge graph technology provides the possibility to bridge this semantic gap by constructing a relationship network between entities, existing graph construction methods usually treat images as a separate black box node, and their relationship is still dependent on incomplete text, failing to fully exploit the semantic content of the image itself, and the retrieval dimension is relatively single.

[0005] On the other hand, with the development of deep learning technology, semantic vector-based similarity retrieval can cross the literal difference and find conceptually similar content, but this method, while providing semantic generalization capability, may also return some conceptually related but specifically incorrect results due to the lack of structured constraints, sacrificing the accuracy of the search. Existing technical solutions often treat graph structure retrieval and semantic vector retrieval as two independent technical paths, lacking a unified framework that can combine the advantages of both and achieve integrated data management and collaborative retrieval efficiency. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a multi-modal knowledge graph construction method and system supporting image-text mutual search, which solves the problem of low image retrieval dimension, low precision and low recall rate caused by the inability to understand the deep semantic association between images and text by simply relying on text labels when processing mixed image-text professional materials.

[0007] To achieve the above object, the first aspect of the present application provides a multi-modal knowledge graph construction method supporting cross-checking between images and texts, which comprises the following steps:

[0008] Obtaining a multi-modal data source containing images and associated texts;

[0009] Creating a dedicated image node for at least one image in the multi-modal data source;

[0010] Extracting at least one text entity from the associated texts;

[0011] Establishing a semantic relationship between the image node and the text entity;

[0012] Storing the image node, the text entity and the semantic relationship between them in a database supporting graph structure to form a multi-modal knowledge graph.

[0013] In a specific implementation process, the step of creating a dedicated image node is realized by calculating the hash value of the binary data stream of the image, which is used as the unique identifier of the image node. This process can be expressed by the following formula:

[0014] id I =Hash(B(I));

[0015] In the formula, id I represents the unique identifier of the image node, B(I) represents the binary data stream of the image, and Hash(·) represents the hash function, such as SHA-256.

[0016] The image node can also contain attribute fields such as the identifier of the source document and the page number of the location in the source document.

[0017] In a specific implementation process, the semantic relationship includes a presentation relationship and a belonging relationship. The presentation relationship is used to represent the image corresponding to the image node, which reflects the typical phenotype of the disease as a text entity as a whole. The relationship direction is from the image node to the text entity. The belonging relationship is used to represent the symptom as a text entity appearing in the image corresponding to the image node. The relationship direction is from the text entity to the image node.

[0018] In a specific implementation process, the method further comprises the step of vectorizing the associated texts of the image, which converts the associated texts into semantic vectors through a text embedding model. This process can be expressed by the following formula:

[0019] V T =femb (T);

[0020] wherein: V T denotes the semantic vector; T denotes the associated text; f emb denotes the text embedding model.

[0021] In a specific implementation process, the step of storing in the database supporting the graph structure is specifically adopting an integrated storage architecture, which uniformly stores the graph structure data of the multi-modal knowledge graph, the metadata associated with the image node, and the semantic vector in the same database instance, and the database instance is integrated with a graph data management component for managing nodes and edges, and a vector data management component for storing and indexing the semantic vector.

[0022] In a specific implementation process, the method further includes a retrieval step of supporting graph-text mutual search, which, after receiving a user query request, can perform graph traversal retrieval in the multi-modal knowledge graph based on the semantic relationship to obtain a first retrieval result, and when the multi-modal knowledge graph contains the semantic vector, the retrieval step can further convert the user query request into a query vector and perform vector retrieval based on semantic similarity to obtain a second retrieval result, and fuse the first retrieval result and the second retrieval result to generate a final query result.

[0023] The second aspect of the present application provides a multi-modal knowledge graph construction system supporting graph-text mutual search, which comprises:

[0024] A data acquisition module is configured to acquire a multi-modal data source containing images and associated texts.

[0025] An image node creation module is configured to create a dedicated image node for at least one image in the multi-modal data source.

[0026] A text entity extraction module is configured to extract at least one text entity from the associated text.

[0027] A semantic relationship establishment module is configured to establish a semantic relationship between the image node and the text entity.

[0028] An integrated storage module is configured to store the image node, the text entity, and the semantic relationship therebetween in a database supporting a graph structure to form a multi-modal knowledge graph.

[0029] The present application provides a multi-modal knowledge graph construction method and system supporting graph-text mutual search. The present application has the following beneficial effects:

[0030] 1、The present application creates a dedicated image node, establishes a presentation relationship from the image node to a disease entity, and a belonging relationship from a symptom entity to the image node, so that the traversal of the knowledge graph can be initiated bidirectionally from a text entity or an image node, and the structure enables the system to perform operations such as querying associated images from a text entity and querying the disease presented or the symptoms contained from an image, realizing bidirectional retrieval between images and texts.

[0031] 2、The present application integrates storage architecture to uniformly manage graph structure data, vector data and metadata in a single database instance, which avoids data synchronization delay or inconsistency caused by cross-database operations by completing update operations on different modal data within a single transaction, and guarantees the data consistency and integrity of the knowledge base during incremental update.

[0032] 3、The present application uses the hash value of the image content as the unique identifier of the image node, which can perform existence check before knowledge entry, avoids repeated modeling of the same image from the mechanism, ensures the uniqueness of the knowledge, and the source literature and page number recorded in the image node provide direct technical basis for the use and source tracing of image data. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The method flowchart of the present application;

[0034] Figure 2 The system architecture diagram of the present application;

[0035] Figure 3 The double relationship semantic model schematic diagram of the present application;

[0036] Figure 4 The multi-modal data acquisition and preprocessing schematic diagram of the present application;

[0037] Figure 5 The integrated storage architecture schematic diagram of the present application;

[0038] Figure 6 The mixed retrieval flowchart of the present application.

[0039] Among them, 10, data acquisition module; 20, image node creation module; 30, text entity extraction module; 40, semantic relationship establishment module; 50, integrated storage module; 60, mixed retrieval module. DETAILED DESCRIPTION

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example:

[0042] Please see the appendix Figure 1 - Appendix Figure 6 This invention provides a method for constructing a multimodal knowledge graph that supports text-image cross-referencing, comprising the following steps:

[0043] S101 is for multimodal data acquisition and preprocessing.

[0044] This step is performed by the data acquisition module 10 in the system. The input to this step is an unstructured or semi-structured document containing images and text, such as an electronic document in PDF format or a web page file in HTML format. The output is a set of data pairs, each containing a binary data stream of an image and one or more pieces of text associated with the image.

[0045] Specifically, when the input data source is a PDF document, this step first traverses the document page by page. On each page, the system identifies and extracts two basic types of objects: image objects and text block objects. For each image object, it extracts its position coordinates on the page and its uncompressed raw binary data stream. For each text block object, it similarly extracts its position coordinates on the page and its contained text content.

[0046] After obtaining all images and text blocks on the page and their location information, it is necessary to determine the associated text for each image. This process is achieved through a location-based association algorithm. For each image on the page, the system defines a preset two-dimensional search area based on its bounding box. This search area can be set to extend outward by a fixed pixel distance in each of the four directions of the image bounding box (up, down, left, and right), or by a percentage distance related to the page size.

[0047] Subsequently, the system traverses all text blocks on the page and determines whether the bounding box of each text block has a spatial overlap or proximity relationship with the search area of ​​the current image. All text blocks whose bounding boxes fall into or touch the search area are initially identified as text related to the image.

[0048] To further differentiate between different types of associated text, the system can perform content analysis on initially associated text blocks. For example, by using regular expression matching, it can find text blocks that match a specific pattern and mark them as the title or caption of the image. Other text blocks that fall into the search area but do not match a specific pattern are classified as the contextual description paragraphs of the image.

[0049] After determining the relationships, all related text content is preprocessed, including:

[0050] Remove headers, footers, page numbers, and other characters from the document that are not related to the content;

[0051] Merge text lines that are broken due to typesetting;

[0052] The text is segmented into individual sentences based on punctuation marks such as periods and semicolons. After preprocessing, clean and structured text data is obtained.

[0053] Finally, step S101 combines the binary data stream of each image with its corresponding preprocessed associated text into a data pair, which will be used as inputs for subsequent steps S102, S103 and S105 respectively.

[0054] S102. Creation of image nodes and generation of unique identifiers.

[0055] This step is performed by the image node creation module 20 in the system. The input of this step is the image binary data stream B(I) in the data pair output by step S101. Its main task is to calculate a globally unique identifier in the knowledge graph for each unique image and use the identifier to create or locate a dedicated image node.

[0056] This step first applies a deterministic hash function to the input image binary data stream B(I) to generate its unique identifier id. I In this embodiment, the hash function used is SHA-256, and the calculation process is represented by the following formula:

[0057] id I =SHA-256(B(I));

[0058] In the formula, B(I) is the complete binary data stream of the image, and SHA-256(·) is a standard hash algorithm. The characteristic of this algorithm guarantees that as long as the input binary data stream B(I) is exactly the same, its output hash value id will be the same. I (A fixed-length string of 256 characters) must also be exactly the same.

[0059] Obtain the unique identifier ID IThen, the system performs a check and creation operation. Specifically, the system uses this ID... I Used as the query key, this searches the target database to determine if a unique identifier for an image node exists that matches the given id. I Matching.

[0060] If the search result is yes, it means that a database already contains an ID with the same id. I If an image node is found to be already entered, it indicates that the image has already been entered. In this case, the image node creation module 20 will not create a new node, but will directly return the identifier of the existing image node for use in subsequent steps. This operation ensures that images with completely identical content will not generate redundant nodes in the map.

[0061] If the search result is negative, it indicates that the image is being processed for the first time. The image node creation module 20 will create a new image node in the database. This image node is a structured data unit with specific attributes, and its core attributes are set as follows:

[0062] image id The value of this attribute is set to the unique identifier id obtained in this calculation. I This attribute is the primary key of the node.

[0063] source doc The value of this attribute is set to the source document identifier passed in from step S101 along with the image.

[0064] page num The value of this attribute is set to the page number in the source document that was passed in from step S101 along with the image.

[0065] caption: The value of this attribute is set to the original, unvectorized caption or title text that is passed in from step S101 along with the image.

[0066] After the above attributes are created and populated, the newly created image node is submitted to the database.

[0067] Ultimately, whether an existing node is found through retrieval or a new node is created, the output of step S102 is a unique identifier (id) for that image node. I This identifier will be passed to step S104 as one of the endpoints for establishing semantic relationships.

[0068] S103, Entity extraction from related text.

[0069] This step is performed by the text entity extraction module 30 in the system, the input of this step is the pre-processed associated text in the data pair output by step S101, and the output of this step is one or more structured text entities, each entity contains its text content and a predefined entity category.

[0070] In this embodiment, the text entity extraction module 30 is configured with a named entity recognition (NER) model trained on a professional domain corpus, and the goal of the model is to identify and extract two types of predefined entities: disease entities and symptom entities.

[0071] The specific execution process of this step is as follows:

[0072] First, the text entity extraction module 30 receives the associated text in sentences from step S101, and for each sentence, the NER model processes it to assign a predefined label to each word unit in the sentence, and the label system uses the BIO format, for example, the starting word unit of a disease name is labeled as B-DISEASE, the subsequent word unit of the disease name is labeled as I-DISEASE, and the starting word unit of a symptom name is labeled as B-SYMPTOM. Word units that do not belong to any predefined entity category are uniformly labeled as O.

[0073] After completing the word unit labeling of all sentences, the text entity extraction module 30 reconstructs the complete entity according to the label sequence, specifically, it traverses the label sequence, concatenates the word units corresponding to a B-DISEASE label and all subsequent continuous I-DISEASE labels to form a complete disease entity text, and similarly, it reconstructs the text of all symptom entities in the same way.

[0074] To ensure the standardization and consistency of knowledge in the knowledge graph, the extracted entity text will be standardized, and the text entity extraction module 30 is internally configured with a domain dictionary that stores the standard names of various entities and their corresponding synonyms, abbreviations or aliases. The module matches each extracted entity text with the dictionary, and if a match is found, the entity will use its corresponding standard name in the dictionary as its final representation, for example, if the "Foot and Mouth Disease" and "FMD" entities are extracted from the text, they will both be mapped to the same standard name "Foot and Mouth Disease" by querying the domain dictionary.

[0075] After the above processing, step S103 finally outputs a list containing all unique, standardized text entities identified from the associated text, each item in the list explicitly records the standard name of an entity and its category, and the list will be passed to step S104 to establish semantic relationships with image nodes.

[0076] S104, double relationship semantic modeling.

[0077] This step is executed by the semantic relationship establishment module 40 in the system. The input of this step is: the unique identifier id of the image node from step S102 I , and the text entity list containing entity standard name and entity category from step S103. The output of this step is one or more sets of triples describing the relationship between nodes.

[0078] The core task of this step is to establish deterministic semantic relationships between image nodes and text entities according to the category of text entities. In this embodiment, two types of semantic relationships are defined: presentation relationship and belonging relationship. The semantic relationship establishment module 40 is internally configured with a set of relationship establishment rules.

[0079] Rule 1: When receiving a text entity with a disease category from the input list, trigger the creation of a presentation relationship, which represents the image node it is connected to, and macroscopically presents the typical phenotype of the disease entity. This relationship is constructed as a directed edge from the image node to the disease entity, with a triple structure of: (image node, depicts, disease entity).

[0080] Rule 2: When receiving a text entity with a symptom category from the input list, trigger the creation of a belonging relationship, which represents the specific disease that the symptom entity refers to, and appears in the image content of the image node it is connected to. This relationship is constructed as a directed edge from the symptom entity to the image node, with a triple structure of: (symptom entity, belongs to , image node).

[0081] The following is a specific example to illustrate the execution process of this step. Assume that step S102 generates an image node with a unique identifier of for a cow hoof image, and at the same time, step S103 extracts and standardizes two text entities from the associated text of the image: entity one {standard name: foot and mouth disease, category: disease}, and entity two {standard name: hoof blister, category: symptom}.

[0082] The semantic relationship establishment module 40 receives and the above two entities. First, the module processes entity one, as its category is disease, the module applies rule one to generate a presentation relationship triple: (image node depicts, disease entity); then, the module processes entity two, as its category is symptom, the module applies rule two to generate a belonging relationship triple: (symptom entity, belongs to , image node ).

[0083] After processing all input text entities, step S104 outputs all generated relationship data conforming to the triple structure, which will be passed to step S106 for persistent storage.

[0084] S105, semantic vectorization of the associated text.

[0085] The execution logic of this step can be integrated in the text processing flow, and its output is finally stored by the integrated storage module 50. The input of this step is the preprocessed associated text T from step S101, and the purpose of this step is to convert the unstructured text content into a fixed-length real number vector, i.e., a semantic vector, to support subsequent retrieval based on semantic similarity.

[0086] The specific implementation process of this step is as follows: the system calls the preloaded text embedding model f emb (·), which is a pre-trained language model based on the Transformer architecture in this embodiment, such as the m3e-base model.

[0087] First, the input associated text T is sent to a special tokenizer paired with the text embedding model. The tokenizer divides the text T into a token sequence according to its internal vocabulary and maps each token to a unique integer ID, thereby generating an integer ID sequence.

[0088] Next, the integer ID sequence is provided as input to the main body of the text embedding model, and the model calculates the sequence to output a matrix, where each row corresponds to a high-dimensional context embedding vector of a token in the input sequence.

[0089] In order to obtain a single vector representing the entire associated text T, the system performs a pooling operation on the token embedding matrix output by the model. In this embodiment, the average pooling strategy is adopted, and the specific operation is as follows:

[0090] The embedding vectors of all tokens in the matrix are accumulated according to the corresponding dimensions, and then each dimension of the accumulation result is divided by the total number of tokens. The result of this operation is a single, fixed-dimensional vector, which is the semantic vector V T of the associated text T.

[0091] The complete process of semantic vectorization can be represented by the following formula:

[0092] V T =f emb (T);

[0093] In the formula: V Tdenotes the final generated semantic vector, which is a 768-dimensional float vector if m3e-base model is used, T denotes the input associated text, f emb denotes the whole text embedding process including tokenization, model computation and pooling operations.

[0094] Finally, step S105 outputs the computed semantic vector V T , which will be passed to step S106 and stored in the database together with the association of the image node corresponding to the associated text generating the vector.

[0095] S106, integrated storage of multi-modal knowledge.

[0096] This step is performed by the integrated storage module 50, and the input of this step is all the data generated by the previous steps to be persisted, including image nodes and their attributes, text entities, semantic relationship triples and semantic vectors. The output of this step is a multi-modal knowledge graph that contains all the above information and has consistent data structure.

[0097] In this embodiment, the integrated storage architecture is implemented by a PostgreSQL database instance configured with specific extensions, which internally integrates both graph data management components and vector data management components.

[0098] Specifically, the PostgreSQL instance loads a graph data management component, which is Apache AGE extension in this embodiment. This component creates a graph-specific namespace in the database, and the image nodes generated by step S102 and the text entities generated by step S103 are both stored as nodes in this graph namespace. The attributes of the nodes, such as the unique identifier image id , source doc , page num , and the standard name of the text entity, are all stored as attributes in the corresponding node data structure. The depicts and belongs to semantic relationships generated by step S104 are stored as directed edges connecting the corresponding nodes.

[0099] At the same time, the PostgreSQL instance also loads a vector data management component, which is pgvector extension in this embodiment. This extension provides a vector data type for the database to store high-dimensional vectors, and provides functions for vector similarity calculation and indexes for accelerating retrieval.

[0100] When storing data, the integrated storage module 50 stores the semantic vector V TThe image node corresponding thereto is stored in association, and the specific implementation manner is that, in the underlying data table used by Apache AGE to store the image node, a column is extended, the data type of the column is defined as the vector type provided by pgvector, and then the semantic vector V T is stored in the same row as the corresponding image node in the column, in this way, the image node in the graph and the semantic vector of the associated text thereof directly correspond at the physical storage layer.

[0101] Through the above configuration, the topological structure data of the graph, the attribute metadata of the node, and the high-dimensional semantic vector data are all stored under the unified management of the same database instance, and all operations of creating, updating or deleting the same knowledge unit can be completed in a single database transaction, thereby guaranteeing the atomicity and consistency of data operation.

[0102] S107, image-text hybrid retrieval based on the knowledge graph.

[0103] This step is performed by the hybrid retrieval module 60, the input of this step is the natural language query request provided by the user, and the output is a sorted list containing image and associated text information.

[0104] This step first processes the query request input by the user, and the processing includes two parallel operations:

[0105] First, the same named entity recognition (NER) model as in step S103 is used to identify key entities such as diseases or symptoms from the query request, and second, the same text embedding model f emb () as in step S105 is used to convert the entire query request string into a query vector.

[0106] After the query request processing is completed, the hybrid retrieval module 60 starts two independent retrieval modes.

[0107] The first mode is a graph structure-based retrieval, which is activated when the entity is successfully identified in the query request, and the hybrid retrieval module 60 generates a piece of graph query language code according to the identified entity and executes it on the graph stored in the integrated storage module 50, for example, if the user query is a picture of foot-and-mouth disease, the system identifies foot-and-mouth disease as a disease entity, then a query is executed, the logic of which is: find the disease entity node named foot-and-mouth disease, then traverse all the image nodes connected to it along all the depicts relationships pointing to it, the result returned by this retrieval mode is all image nodes having a direct and precise structured association with the query entity, this result set is defined as the first retrieval result.

[0108] The second mode is semantic vector based retrieval, which utilizes the query vector generated in the previous step to perform a vector similarity search in the database. Specifically, the hybrid retrieval module 60 issues a command to the database to compute the cosine similarity between the query vector and the semantic vectors associated with all image nodes stored in the database. The database utilizes the index structure provided by its vector data management component to accelerate this computation process. This retrieval mode returns a list of image nodes ordered by similarity score from high to low, typically taking the top K results. This result set is defined as the second retrieval result.

[0109] After obtaining the first retrieval result and the second retrieval result, the hybrid retrieval module 60 performs a fusion operation on these two result sets, the steps of which are as follows:

[0110] First, assign the highest score, e.g. 1.0, to all image nodes that appear in the first retrieval result. Then, merge the first retrieval result list with the second retrieval result list. For image nodes that appear in both lists, retain the higher score value (i.e. 1.0). Finally, sort the merged list in descending order of score and remove duplicate nodes, thereby generating the final query result list.

[0111] Finally, the hybrid retrieval module 60 extracts the image corresponding to each image node from the database according to the sorted list, as well as metadata such as source and caption, and combines them into a structured entry to return to the user.

[0112] Referring to the accompanying drawings Figure 2 A multi-modal knowledge graph construction system supporting image-text mutual search, comprising:

[0113] A data acquisition module 10, an image node creation module 20, a text entity extraction module 30, a semantic relationship establishment module 40, an integrated storage module 50, and a hybrid retrieval module 60.

[0114] The data acquisition module 10 is used to perform multi-modal data source acquisition and preprocessing. This module is configured to parse the input document format, identify and extract image objects and their binary data streams, as well as text blocks and their position coordinates in the document. The module contains a location-based association algorithm inside, which is used to determine the associated text for each image by calculating the spatial proximity between the image bounding box and the text block bounding box.

[0115] The image node creation module 20 is connected to the output of the data acquisition module 10. This module is used to create a dedicated image node for each image. It generates a unique identifier by applying a hash function to the binary data stream of the image. The module is also configured to use this unique identifier to query the integrated storage module 50 to check whether the node already exists, and to create a new node containing the imageid , source doc new nodes with attributes such as

[0116] a text entity extraction module 30, the input of which is connected to the output of the data acquisition module 10, and inside which a named entity recognition (NER) model is deployed to process the associated text and identify predefined entity categories such as disease entities and symptom entities, and the module further includes an entity normalization unit that uses a domain dictionary to map the extracted entity text to its standard name to ensure the consistency of the entity.

[0117] a semantic relationship establishment module 40, the input of which is connected to the output of the image node creation module 20 and the text entity extraction module 30, and the module establishes semantic relationships between the image nodes and the text entities according to a preset rule set, and when the received entity category is a disease, it generates a depicts relationship from the image node to the disease entity, and when the entity category is a symptom, it generates a belongs to relationship from the symptom entity to the image node.

[0118] an integrated storage module 50, which serves as a central storage unit and is connected to the image node creation module 20, the text entity extraction module 30, and the semantic relationship establishment module 40, and the module manages a database instance that integrates a graph data management component and a vector data management component, and it is responsible for persisting the received nodes, entities, relationships, and semantic vectors into the corresponding structures of the database, for example, storing the nodes and edges into the graph structure and storing the semantic vectors into the vector type field associated with the nodes.

[0119] a hybrid retrieval module 60, which is connected to the integrated storage module 50 and is used to execute queries, and the module includes a query processor for entity recognition and vectorization of the user's natural language input, and the module further includes two retrieval execution units:

[0120] a graph retrieval unit for generating and executing a graph traversal query according to the identified entities;

[0121] a vector retrieval unit for performing a similarity search based on the query vector, and the output of the module is further connected to a result fusion unit that merges the results from the two retrieval units according to preset scoring and ranking rules to generate a final graph-text hybrid retrieval list.

Claims

1. A method for constructing a multimodal knowledge graph that supports text-image cross-referencing, characterized in that, Includes the following steps: Acquire multimodal data sources containing images and associated text; For at least one image from the multimodal data source, create a dedicated image node; Extract at least one text entity from the associated text; Establish the semantic relationship between the image nodes and the text entities; The image nodes, text entities, and their semantic relationships are stored together in a graph-supporting database to form a multimodal knowledge graph.

2. The method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 1, characterized in that, The steps for creating dedicated image nodes specifically include: Calculate the hash value of the binary data stream of the image to generate a unique identifier for the image node; Among them, the unique identifier id I Generate using the following formula: id I =Hash(B(I)); Among them, id I B(I) represents the unique identifier of the image node; B(I) represents the binary data stream of the image; Hash(·) represents the hash function.

3. The method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 1, characterized in that, The semantic relationship includes at least one of the following: Presentation relationship: This means that the image corresponding to the image node as a whole reflects the typical phenotype of the disease referred to by the text entity, and the presentation relationship is from the image node to the text entity; Belongs to: indicates that the symptom referred to by the text entity appears in the image corresponding to the image node, and the belonging relationship is that the text entity points to the image node.

4. The method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 2, characterized in that, The attribute fields included in the image node also include: Record the identifier of the source document for the image, and record the page number of the image's location in the source document.

5. The method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 1, characterized in that, The method further includes: The associated text of the image is converted into a semantic vector using a text embedding model; The semantic vector is generated using the following formula: V T =f emb (T); Where: V T The semantic vector is represented by T; the associated text is represented by f. emb (·) represents the text embedding model.

6. A method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 5, characterized in that, The step of storing the data in a database that supports graph structures is as follows: An integrated storage architecture is adopted, which stores the graph structure data of the multimodal knowledge graph, the metadata associated with the image nodes, and the semantic vectors in the same database instance.

7. A method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 6, characterized in that, The unified storage architecture is specifically implemented by integrating the following components into the database instance: A graph data management component for managing nodes and edges; A vector data management component for storing and indexing the semantic vectors.

8. The method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 1, characterized in that, The method also includes a retrieval step that supports cross-referencing between text and images, the retrieval step including: Receive user query requests; based on the semantic relationships, perform graph traversal retrieval in the multimodal knowledge graph to obtain the first retrieval result.

9. A method for constructing a multimodal knowledge graph supporting text-image cross-referencing according to claim 8, characterized in that, When the multimodal knowledge graph contains semantic vectors, the retrieval step further includes: The user query request is converted into a query vector, and a vector retrieval based on semantic similarity is performed to obtain a second retrieval result; The first search result and the second search result are combined to generate the final query result.

10. A multimodal knowledge graph construction system supporting text-image cross-referencing, and a multimodal knowledge graph construction method supporting text-image cross-referencing according to any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire multimodal data sources containing images and associated text; An image node creation module is used to create a dedicated image node for at least one image from the multimodal data source; A text entity extraction module is used to extract at least one text entity from the associated text; A semantic relationship establishment module is used to establish semantic relationships between the image nodes and the text entities; An integrated storage module is used to store the image nodes, text entities and their semantic relationships into a database that supports graph structures to form a multimodal knowledge graph.

Images