Cross-modal knowledge graph construction method

Abstract

The invention discloses a method for constructing a cross-modal knowledge graph, and relates to the technical field of knowledge graphs, and the method comprises the steps: obtaining multi-source heterogeneous data, and carrying out the collection, structural analysis, modal recognition and classification, cleaning and standardization processing, so as to form structured multi-modal data; extracting entities and relationships of the identified modals from the structured multi-modal data, and summarizing the entities and relationships to form a multi-modal knowledge element set; mapping different modal entities in the multi-modal knowledge element set to a unified semantic space, and generating a unified entity relationship set through semantic matching, alignment and fusion; and normalizing the data into knowledge triples, and storing and organizing the knowledge triples through a graph database to form a cross-modal knowledge graph. According to the method, the problems of difficulty in multi-modal heterogeneous information alignment and difficulty in entity relationship extraction can be relieved, semantic association is enhanced, and knowledge graph integrity and multi-scene adaptability are improved.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, and in particular to a method for constructing a cross-modal knowledge graph. Background Technology

[0002] With the development of artificial intelligence technology, knowledge graphs, as a structured semantic knowledge representation, have been widely applied in various scenarios. Traditional knowledge graph construction methods mainly rely on text data, constructing a semantic network composed of "entity-relationship-entity" triples through information extraction techniques. However, information in the real world often comes from multiple modalities such as text, images, videos, and audio. Relying on a single modality to construct a knowledge graph cannot fully capture the semantic relationships between multimodal information, resulting in an incomplete knowledge acquisition chain.

[0003] In existing technologies, although multimodal large models can achieve some cross-modal semantic mapping, it is difficult to align multimodal heterogeneous information in a unified space, and the different granularity of information expression between modalities makes it difficult to extract entities and relationships. Summary of the Invention

[0004] To address the technical challenges of existing multimodal large models, which can achieve partial cross-modal semantic mapping but struggle to align heterogeneous multimodal information in a unified space and suffer from difficulties in entity and relation extraction due to different information expression granularities between modalities, this invention provides a method for constructing cross-modal knowledge graphs.

[0005] The technical solution adopted in this invention is:

[0006] A method for constructing a cross-modal knowledge graph includes the following steps:

[0007] Step 1: Acquire multi-source heterogeneous data, and perform data collection, structure analysis, modality recognition and classification, cleaning and standardization on the multi-source heterogeneous data to form structured multimodal data.

[0008] Step 2: Extract the entities, entity attributes, and relationships between entities for each identified modality from the structured multimodal data, and summarize them to form a multimodal knowledge element set.

[0009] Step 3: Map entities of different modalities in the multimodal knowledge element set to a unified semantic space, and achieve entity alignment and fusion through semantic matching to generate a unified entity relationship set.

[0010] Step 4: Standardize the unified entity relationship set into knowledge triples, and store and organize them in a structured manner through a graph database to form a cross-modal knowledge graph.

[0011] Preferably, step 1 includes the following sub-steps:

[0012] Step 1.1: Collect raw data units from multiple heterogeneous sources. The raw data units include web page content, documents, images, videos, and audio. Mark each raw data unit with its source path, collection timestamp, and unique identifier.

[0013] Step 1.2: Perform structural analysis on the original data unit to separate the text content unit, image content unit, video frame content unit, and audio track content unit.

[0014] Step 1.3: Perform modal recognition and classification labeling on the text content unit, image content unit, video frame content unit, and audio track content unit, encapsulate them into multimodal data objects containing intermodal mapping relationship identifiers, store them in a multimodal data warehouse, and establish a modal cross-index.

[0015] Step 1.4: Clean each modal content unit in the multimodal data object to generate cleaned multimodal content units.

[0016] Step 1.5: Semantically extract the video frame content unit and audio track content unit in the multimodal content unit, generate a textual description, and store it in association with the original modal features to form a multimodal content unit with attached textual semantics.

[0017] Step 1.6: Perform language detection, timestamp annotation, and content fingerprint generation on multimodal content units with accompanying textual semantics to form semantically traceable structured multimodal data.

[0018] Preferably, step 2 includes the following sub-steps:

[0019] Step 2.1: For the text content units in the structured multimodal data, use the trained large language model to perform named entity recognition and relation extraction, and generate entity pairs with type labels and relation descriptions.

[0020] Step 2.2: For the image content units in the structured multimodal data, identify objects and scenes through a visual model, convert them into entity and semantic relationship text, and associate them with image location information.

[0021] Step 2.3: For the video frame content units in the structured multimodal data, perform entity recognition by combining the keyframe entity recognition model, action recognition model, and audio track transcription text, extract event-level entities and dynamic relationships, and associate time intervals.

[0022] Step 2.4: For the audio track content units in the structured multimodal data, after transcribing the audio into text, extract entities and relations using an entity and relation extraction model, and mark the speaker affiliation using a voiceprint recognition model.

[0023] Step 2.5: Summarize the processing results from Steps 2.1 to 2.4, record the modal type of each entity's source and the extracted confidence level, and form a multimodal knowledge element set.

[0024] Preferably, step 2.5 includes the following sub-steps:

[0025] Step 2.5.1: Evaluate the confidence level of the entities and relationships extracted in Steps 2.1 to 2.4, and remove entity pairs and relationship descriptions with confidence levels below the set threshold.

[0026] Step 2.5.2: Mark the frequently occurring entities with unclear referents among the entities and relationships remaining after the filtering in Step 2.5.1, and clarify the entity's referent by combining the contextual semantics or associated modal information.

[0027] Step 2.5.3: Randomly select some entity and relation samples processed in Step 2.5.2 for manual review, and feed back the review results to the large language model used in Step 2.1, the visual model used in Step 2.2, and the entity recognition and relation extraction related models involved in Steps 2.3 and 2.4.

[0028] Step 2.5.4: After processing in steps 2.5.1 to 2.5.3, record the modal type of each entity's source and extract confidence level to form a multimodal knowledge element set.

[0029] Preferably, step 3 includes the following sub-steps:

[0030] Step 3.1: Encode the entities of each modality in the multimodal knowledge element set using a multimodal pre-trained model to generate equal-dimensional semantic vectors.

[0031] Step 3.2: Calculate the semantic similarity of entities of different modalities based on the equal-dimensional semantic vectors, and determine entity equivalence by combining time context factors and location context factors, and generate entity equivalence determination results.

[0032] Step 3.3: Based on the entity equivalence determination results, merge entities that are determined to be equivalent, unify entity identifiers, aggregate multimodal attributes, and establish a modality mapping table.

[0033] Step 3.4: For entities with the same name but different semantics, use a disambiguation model to distinguish them and update the modality mapping table.

[0034] Step 3.5: Based on the updated modal mapping table, integrate entities and relationships between entities to generate a unified entity relationship set.

[0035] Preferably, step 4 includes the following sub-steps:

[0036] Step 4.1: Normalize the unified entity relation set into knowledge triples with the structure <head entity, relation, tail entity>.

[0037] Step 4.2: Store the knowledge triples in a graph database and construct an association structure containing entity nodes and relation edges, where the content units of the original modality associated with the entity nodes serve as evidence.

[0038] Step 4.3: Establish a retrieval index based on the entity nodes, relation edges, and semantic vectors, and integrate them to form a cross-modal knowledge graph with multimodal association and traceability; wherein, the retrieval index includes entity type index, relation type index, and semantic vector index; the cross-modal knowledge graph supports visual organization and query operations.

[0039] Preferably, step 5 is also included: introducing a graph structure retrieval enhancement generation mechanism based on cross-modal knowledge graph, constructing a retrieval index and performing community segmentation and summary generation on cross-modal knowledge graph, planning the retrieval path according to the query intent, combining the structure generation enhancement prompts of cross-modal knowledge graph, and realizing knowledge completion and semantic reasoning through language model.

[0040] Preferably, step 5 includes the following sub-steps:

[0041] Step 5.1: Use a community discovery algorithm to divide the cross-modal knowledge graph into topic communities. Based on the entity type index and relation type index described in Step 4.3, extract the core entities, main relations and context information of each community to generate a community knowledge summary.

[0042] Step 5.2: Identify the query intent type, plan a single-hop search, multi-hop path search, or community search path according to the intent type, and obtain a graph support fragment containing core entities, relational structures, and modal evidence based on the search index described in Step 4.3 and the community knowledge summary described in Step 5.1.

[0043] Step 5.3: Integrate the graph support fragments and query intent described in Step 5.2 into structural enhancement prompts, input them into the language model to generate knowledge content that conforms to the cross-modal knowledge graph structure. The knowledge content includes entity relation completion results, multimodal association reasoning conclusions, and natural language answers, thereby realizing knowledge completion and semantic reasoning.

[0044] Preferably, it also includes step 6: dynamically updating the cross-modal knowledge graph through incremental data access, integrating new entities with the existing cross-modal knowledge graph, mining potential relationships by combining graph neural networks and language models, recording update logs and maintaining version chains, and supporting the iterative evolution and knowledge expansion of the cross-modal knowledge graph.

[0045] Preferably, step 6 includes the following sub-steps:

[0046] Step 6.1: Monitor the sources of multi-source heterogeneous data in real time, obtain newly added original data units, and sequentially perform the structured processing of Step 1 and the entity and relation extraction process of Step 2 on the newly added original data units to generate an incremental entity relation set.

[0047] Step 6.2: Calculate the semantic similarity between the entities in the incremental entity relationship set and the entities in the existing cross-modal knowledge graph, and perform entity fusion or addition operations based on the similarity threshold: merge multimodal attributes and update the existing knowledge triples for entities with similarity exceeding the threshold, and create new knowledge triples for entities with similarity below the threshold; generate an updated knowledge triple set by handling relationship conflicts through evidence weighting, multi-source verification, or manual intervention.

[0048] Step 6.3: Use a graph neural network to perform representation learning on the updated knowledge triple set to predict the relationships between potential entities; combine the language model to generate candidate relationship descriptions for high-frequency co-occurring but unrelated entity pairs to form a potential relationship set.

[0049] Step 6.4: Based on the updated set of knowledge triples and the set of potential relations, record the timestamp, update content and operation type of each update, maintain the version chain of the cross-modal knowledge graph, and generate the updated cross-modal knowledge graph; the updated cross-modal knowledge graph serves as the base graph for subsequent execution of Step 5, supporting the dynamic iteration of retrieval and reasoning.

[0050] The beneficial effects of the present invention are at least one of the following:

[0051] By performing structured processing on multi-source heterogeneous data, multimodal information such as text, images, videos, and audio can be transformed into standardized structured multimodal data. This helps alleviate the problem of messy formats and difficulty in uniform processing of multimodal heterogeneous information, and provides a regular foundation of data for subsequent entity and relation extraction.

[0052] Extracting and summarizing entities and relationships between entities for each identified modality, and then processing them in a targeted manner based on the information expression characteristics of different modalities, helps to reduce the difficulty of entity and relationship extraction caused by different information expression granularities between modalities and improves the completeness of the multimodal knowledge element set.

[0053] Mapping entities of different modalities to a unified semantic space and achieving alignment and fusion provides a unified semantic association basis for multimodal heterogeneous information, which helps to alleviate the problem of difficulty in aligning multimodal information in a unified space and enhances the semantic association between entities of different modalities. Attached Figure Description

[0054] Figure 1This is a schematic diagram of the method flow of Embodiment 1 of the present invention. Detailed Implementation

[0055] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] Example 1

[0057] In existing technologies, while large multimodal models can achieve partial cross-modal semantic mapping, it is difficult to align heterogeneous multimodal information in a unified space. The different granularities of information expression between modalities lead to difficulties in entity and relation extraction. To address this technical problem, this embodiment provides a method for constructing a cross-modal knowledge graph, including the following steps:

[0058] Step 1: Acquire multi-source heterogeneous data, and perform data collection, structure analysis, modality recognition and classification, cleaning and standardization on the multi-source heterogeneous data to form structured multimodal data.

[0059] In one possible implementation, step 1 includes the following sub-steps:

[0060] Step 1.1: Collect raw data units from multiple heterogeneous sources. The raw data units include web page content, documents, images, videos, and audio. Mark each raw data unit with its source path, collection timestamp, and unique identifier.

[0061] For example, raw data units to be processed are obtained from multiple data source systems. These raw data units include, but are not limited to, the following modalities: Text modality: including web page content, PDF documents, news corpora, product descriptions, social media posts, book summaries, etc. Image modality: including image screenshots, news illustrations, product images, video keyframes, etc. Video modality: including short videos, instructional videos, meeting recordings, surveillance videos, etc., which may come from Bilibili, Douyin, enterprise knowledge bases, security systems, etc. Speech modality: including interview audio, voice calls, broadcast content, etc., which can be transcribed into text through a speech recognition system. During the acquisition process, each type of modal data is automatically marked with its source path, acquisition timestamp, modality type, and unique identifier, forming a structured index.

[0062] Step 1.2: Perform structural analysis on the original data unit to separate the text content unit, image content unit, video frame content unit, and audio track content unit.

[0063] For example, the collected raw data (such as web pages, PDF documents, presentations, video files, etc.) undergoes structural analysis to determine whether it belongs to a composite modal data source based on its content type. Specifically, this includes:

[0064] For web pages (HTML, JSON, PDF, or PPT), parse text paragraphs, embedded images, tables, and other content elements to build their document tree structure. For video files, extract keyframes using inter-frame differencing or scene detection techniques, and separate image frame sequences, audio tracks, and subtitle tracks (if present). For content containing mixed text and images, extract images and associate them with their corresponding text context, recording their document location information.

[0065] Step 1.3: Perform modal recognition and classification labeling on the text content unit, image content unit, video frame content unit, and audio track content unit, encapsulate them into multimodal data objects containing intermodal mapping relationship identifiers, store them in a multimodal data warehouse, and establish a modal cross-index.

[0066] For example, based on the content's structural attributes, file metadata, and content type, each content unit undergoes modality recognition and is labeled as one of the following categories: text modality, image modality, video modality, audio modality, or mixed modality. Specific judgment rules include:

[0067] If the content consists of natural language characters and can be parsed by a language processing model, it is labeled as text modality. If the content is a static image, bitmap data, or an image extracted from a document, it is labeled as image modality. If the content contains video frame sequences, timecodes, and image frame streams, it is labeled as video modality. If it is a speech waveform file, audio stream, or audio track extracted from video, it is labeled as audio modality. If the content contains two or more modalities of data that are nested or closely related, it is labeled as mixed modality.

[0068] The above classification process supports automated identification and can assist the system in adopting different subsequent processing strategies for data of different modalities.

[0069] For content objects with identified modal categories, modal structure separation and semantic time synchronization mapping operations are performed. Specifically, for video content, keyframe images are extracted at set frame intervals, and their corresponding timecodes are recorded for subsequent alignment with subtitles and audio content.

[0070] The timestamps of the voice modal and video frames are synchronized, and the video frame offset is corrected by matching the audio waveform; entities are extracted by voice semantic role labeling (such as speaker A's action instructions) for pure voice data (without visual association), such as identifying the relationship in the call recording where Zhang San approved Li Si's application.

[0071] For the audio track in the video, automatic speech recognition (ASR) technology is used to transcribe the speech into text and add a timestamp to form a time-segment transcribed text.

[0072] For document content containing images, extract the images and their contextual text (title, description, paragraph text) to form semantic fragments.

[0073] All image frames, text paragraphs, and audio texts are synchronously mapped to their original content to ensure that the same semantic entity can be traced and located in multiple modalities.

[0074] Through the above mapping process, semantic anchoring between text, images, audio, and video is achieved, providing a foundation for subsequent entity extraction and alignment.

[0075] The processed modal content is encapsulated into a unified format multimodal data object. Each object contains the following fields: a unique data identifier (ObjectID); a modal type identifier (text, image, video, audio); the original content or path reference; the timestamp, source path, and acquisition method of the content; a mapping relationship identifier with other modalities (e.g., anchor ID for images and text, corresponding time periods for video frames and transcribed segments); a cleaning status marker; and structural integrity verification information. This encapsulation structure can be persistently stored using JSON, XML, or database fields to meet the system's unified scheduling, indexing, and retrieval requirements.

[0076] All encapsulated data objects are stored in a unified multimodal data warehouse; indexes are created according to metadata such as modality type, collection time, and associated scenario; a digest hash value is generated for each data object for data consistency verification and version tracking; and a modal cross-index table is established for quickly finding the corresponding item of the same semantic entity in different modalities. This storage structure ensures that subsequent processing can call the required modal content on demand, avoiding repeated parsing and redundant storage, and provides underlying data support for multimodal alignment and graph construction.

[0077] Step 1.4: Clean each modal content unit in the multimodal data object to generate cleaned multimodal content units.

[0078] Specifically, cleaning each modal content unit in the multimodal data object includes denoising text content units, deduplication of image content units, keyframe extraction of video frame content units, and demute of audio track content units; and format unification and quality verification of each cleaned modal content unit.

[0079] Record the processing logs during the cleaning process, including quality assessment indicators such as the number of items removed and the proportion of abnormal segments; for some data that cannot pass the cleaning process (such as semantic missing, unclear, damaged audio tracks, etc.), mark them as unusable and exclude them from the knowledge graph construction process.

[0080] Step 1.5: Semantically extract the video frame content unit and audio track content unit in the multimodal content unit, generate a textual description and store it in association with the original modal features to form a multimodal content unit with attached textual semantics.

[0081] Specifically, image recognition is performed on video frame content units in the multimodal content unit to generate textual semantic information such as scene description, object tags, and action description; speech recognition is performed on audio track content units to generate transcribed text and combine it with voiceprint recognition to generate textual semantic information of speaker tags; the textual semantic information is bound and stored with the corresponding modal content units through a unique identifier to form a multimodal content unit with textual semantics.

[0082] Step 1.6: Perform language detection, timestamp annotation, and content fingerprint generation on multimodal content units with accompanying textual semantics to form semantically traceable structured multimodal data.

[0083] It should be noted that the purpose of this step is to perform unified semantic annotation and time alignment processing on multimodal content units with attached textual semantics, so as to ensure that cross-modal content has traceability and structural consistency in terms of semantic understanding, time sequence and version recognition, and lay the foundation for entity alignment and semantic fusion.

[0084] This step involves uniformly writing the cleaned and labeled multimodal content units with accompanying textual semantics into the data management system of a multimodal data warehouse or graph construction platform. This completes standardized data entry, version management, and scalable index configuration, providing a structurally complete and traceable data foundation for subsequent processes.

[0085] To support fast scheduling and cross-modal retrieval, the following multi-dimensional index structure is established:

[0086] A composite index is built based on fields such as modality type, timestamp, language, and source path; vector indexes are built for images, video frames, and text segments based on multimodal embedding vectors, supporting similarity matching and semantic retrieval; a cross-reference table between modalities is established to achieve fast mapping and querying of text, image-text, and image-video. To achieve data traceability and controllability, a version control mechanism is established, mainly including: automatically recording data change records after each cleaning, annotation, or structural update, including operation time, operation content, and modifier; maintaining a historical version chain for each data object, supporting backtracking, comparison, and recovery; using content hash values ​​to verify whether the current data is consistent with historical versions, and triggering an update flag if differences exist. After the data is successfully entered into the database, the system automatically assigns an entry status label to each data object, such as successful entry, missing modality, incomplete semantics, or pending supplementation; if entry fails or the structure is incomplete, it automatically feeds back to the cleaning process for rollback or supplementation; all status information can be used by the system scheduler for task tracking, failure retries, and data quality assessment.

[0087] Step 2: Extract entities, entity attributes, and relationships between entities for each identified modality from the structured multimodal data, and summarize them to form a multimodal knowledge element set.

[0088] It should be noted that this step is used to identify semantic entities and relationships between entities from cleaned and labeled multimodal data, and is one of the core steps in knowledge graph construction. The entity and relationship extraction covers modalities such as text, image, and video, and is processed using appropriate models and processes, and a unified candidate entity pool is constructed for subsequent semantic alignment and fusion.

[0089] In one possible implementation, step 2 includes the following sub-steps:

[0090] Step 2.1: For the text content units in the structured multimodal data, use the trained large language model to perform named entity recognition and relation extraction, and generate entity pairs with type labels and relation descriptions.

[0091] For example, text is the most common knowledge-carrying modality. This invention employs a large language model or a dedicated pre-trained language model for entity recognition and semantic relation extraction from text data. The main processing flow is as follows: Syntactic analysis and sentence segmentation are performed on the input text to identify named entity boundaries, semantic context, and relational connection structures. Using a trained language model T, words in the text paragraphs are labeled, outputting labeled entity items. Contextual semantic analysis is performed on the identified entity pairs to determine whether a relation exists and to identify the relation type.

[0092] The extracted entities are standardized, such as by uniformly merging synonyms and variant expressions to ensure the uniqueness of graph nodes. The text extraction results are then encapsulated into a standard structure.

[0093] Entity: Entity ID, Entity Name, Type, Context, Source Paragraph.

[0094] Relationships: Head entity, tail entity, relationship type, evidence text, confidence level.

[0095] Step 2.2: For the image content units in the structured multimodal data, identify objects and scenes through a visual model, convert them into entity and semantic relationship text, and associate them with image location information.

[0096] For example, in order to extract entity and semantic information from image modalities, the present invention adopts a multi-level visual semantic understanding model and combines image-text joint modeling to achieve semantic output. The processing flow is as follows: the object detection model YOLOv11 is used to identify perceptible objects in the image and return the bounding box, label and confidence score.

[0097] Image label recognition: A multi-label classification model is used on the entire image to identify the scene category or event theme contained within the image. A semantic description of the image is generated using the BLIP-2 image-text alignment model. Key phrases in the image description are extracted as entities, classified, and labeled, while their image location and modality source are recorded. Entities in the image are compared with entities appearing in the text modality for similarity; if semantic consistency is found, they are recorded as fusionable candidate entities.

[0098] Step 2.3: For the video frame content units in the structured multimodal data, perform entity recognition by combining the keyframe entity recognition model, action recognition model, and audio track transcription text, extract event-level entities and dynamic relationships, and associate time intervals.

[0099] For example, video modalities are characterized by strong temporal sequence and high information density. This invention extracts entities and relationships in dynamic events through comprehensive analysis of video image frames and audio track content. The processing flow is as follows: Keyframe extraction: Keyframes are extracted from the video at fixed time intervals or based on visual change rates to reduce redundant data. Object detection and scene recognition algorithms are applied to the keyframe images to identify people, objects, scene backgrounds, etc., in the same way as image modal processing. The video understanding model I3D is used to identify actions and behavioral events in the video, supplementing the behavioral type relationships in the graph. The video audio track is transcribed into text using the Whisper speech recognition model, and entities and relationships are extracted using a semantic model, in the same way as text modal processing. The frame image recognition results, transcribed text, and action tags are integrated into event segments. Each event segment corresponds to a specific time interval and includes relevant participating entities and their actions and states. For the same entity appearing in multiple consecutive frames, it is merged into a unified entity node through visual alignment algorithms and text semantic judgment.

[0100] Step 2.4: For the audio track content units in the structured multimodal data, after transcribing the audio into text, extract entities and relations using an entity and relation extraction model, and mark the speaker affiliation using a voiceprint recognition model.

[0101] Step 2.5: Summarize the processing results from Steps 2.1 to 2.4, record the modal type of each entity's source and the extracted confidence level, and form a multimodal knowledge element set.

[0102] For example, all entities extracted from different modalities are uniformly aggregated into a candidate entity pool, and the following operations are performed: the original name, modality, semantic description, frequency of occurrence, and modal feature vector of each entity are recorded; the entity source and its context information are marked; synonymous entities with consistent sources and semantics are initially merged to prepare data support for subsequent cross-modal alignment and fusion. Speech entities need to be associated with speaker feature vectors, and speaker consistency weights are added when aligning with text / image entities.

[0103] To improve the quality of the spectrum, in one possible implementation, step 2.5 includes the following sub-steps:

[0104] Step 2.5.1: Evaluate the confidence level of the entities and relationships extracted in Steps 2.1 to 2.4, and remove entity pairs and relationship descriptions with confidence levels below the set threshold.

[0105] Step 2.5.2: Mark the frequently occurring entities with unclear referents among the entities and relationships remaining after the filtering in Step 2.5.1, and clarify the entity's referent by combining the contextual semantics or associated modal information.

[0106] Step 2.5.3: Randomly select some entity and relation samples processed in Step 2.5.2 for manual review, and feed back the review results to the large language model used in Step 2.1, the visual model used in Step 2.2, and the entity recognition and relation extraction related models involved in Steps 2.3 and 2.4.

[0107] Step 2.5.4: After processing in steps 2.5.1 to 2.5.3, record the modal type of each entity's source and extract confidence level to form a multimodal knowledge element set.

[0108] For example, a confidence threshold is set to filter out recognition results with low confidence; high-frequency but semantically ambiguous entity items are marked or removed; a portion of the recognition samples are randomly selected for manual review and used for model tuning and effect evaluation; and an extraction quality evaluation report is output, with indicators including: recognition accuracy, recall, F1 score, etc.

[0109] Step 3: Map entities of different modalities in the multimodal knowledge element set to a unified semantic space, and achieve entity alignment and fusion through semantic matching to generate a unified entity relationship set.

[0110] It should be noted that this step maps entities and contextual semantics identified from data objects of different modalities (such as text, images, videos, and audio) to a shared semantic space through a unified encoding strategy, constructing cross-modal semantic equivalence relations and achieving entity alignment and fusion. This step is a core component in the construction of a multimodal knowledge graph, achieving modality fusion and semantic consistency.

[0111] In one possible implementation, step 3 includes the following sub-steps:

[0112] Step 3.1: Encode the entities of each modality in the multimodal knowledge element set using a multimodal pre-trained model to generate equal-dimensional semantic vectors.

[0113] For example, to achieve comparability and structural consistency of content across different modalities, a multimodal pre-trained model is used to perform vectorized embedding representations for each modality. Specifically, this includes: for text entities and context, the BERT language model is used for embedding encoding, outputting the semantic vector of each entity and the sentence-level context vector. For entities or whole-image descriptions identified in images, the BLIP image-text joint pre-trained model is used for image embedding generation, obtaining semantic vector representations at the image region level or whole-image level. Video segments are composed of keyframe image sequences and audio track semantics: keyframe images are semantically encoded separately; synchronized speech-to-text transcription is encoded using a language model; finally, the frame sequence and text representation are fused to generate a semantic representation of the video event. For speech segments not transcribed into text, an audio-text joint model (such as Wav2Vec2.0+TextDecoder) can be used to extract speech semantic embeddings for alignment with other modalities.

[0114] All encoding results are standardized into vectors of equal dimensions and normalized before being stored in a unified vector space for calculating semantic similarity between modalities.

[0115] Step 3.2: Calculate the semantic similarity of entities of different modalities based on the equal-dimensional semantic vectors, and determine entity equivalence by combining time context factors and location context factors, and generate entity equivalence determination results.

[0116] For example, to achieve semantic consistency judgment and alignment operations between entities of different modalities, the following strategy is adopted for similarity calculation and entity fusion:

[0117] Cosine similarity is used to measure the similarity between semantic vectors of entities from different modalities. When the similarity between two entity vectors exceeds a set threshold (e.g., 0.85), they are determined to be semantically equivalent entities, and a mapping relationship is established. The cosine similarity threshold can be dynamically adjusted according to the modality type. The threshold adjustment logic is determined through offline training on a validation set and supports user-defined configuration.

[0118] A contrastive learning strategy is introduced, using positive and negative samples to train the model on the same data object (such as an image and its descriptive text), thereby improving the model's accuracy in cross-modal semantic understanding. Entities with high similarity but different contexts are further analyzed. If they have significant differences in contextual factors such as time, location, and participating objects, they are retained as independent entities; otherwise, they are merged.

[0119] Step 3.3: Based on the entity equivalence determination results, merge entities that are determined to be equivalent, unify entity identifiers, aggregate multimodal attributes, and establish a modality mapping table;

[0120] For example, for entity pairs that have completed semantic alignment, entity node merging and uniqueness operations are performed. Specifically, this includes: assigning a primary entity identifier (such as the first-appearing modal entity or an entity with high confidence) as the primary ID, and attaching the remaining synonymous entities as equivalent mappings. Aggregating attribute information provided by each modality (such as name, modality source, location of occurrence, contextual description, etc.) enriches the entity description. Establishing a modality mapping table records the correspondence between each merged entity and its source modal entity, for use in graph node creation and inference.

[0121] Step 3.4: For entities with the same name but different semantics, use a disambiguation model to distinguish them and update the modality mapping table.

[0122] For example, if two entities have the same name but different modal origins, contexts, or entity types (e.g., an apple can be a company or a fruit), a disambiguation model is used to make a determination to ensure semantic consistency and accuracy of entity alignment.

[0123] Step 3.5: Based on the updated modal mapping table, integrate entities and relationships between entities to generate a unified entity relationship set.

[0124] A graphical visual review interface (such as entity clustering graphs and vector heatmaps) is provided for alignment results, allowing for manual verification and model feedback. For entity items that fail to align but are semantically similar, entity pairs to be confirmed are generated for subsequent rule supplementation or human-computer interaction decision-making. Through the above semantic encoding and alignment process, this step achieves the mapping of multimodal content to a unified semantic space and completes the fusion connection between equivalent entities, laying a cross-modal semantic consistency foundation for subsequent knowledge triple generation and structural graph construction.

[0125] Step 4: Standardize the unified entity relationship set into knowledge triples, and store and organize them in a structured manner through a graph database to form a cross-modal knowledge graph.

[0126] In one possible implementation, step 4 includes the following sub-steps:

[0127] Step 4.1: Normalize the unified entity relation set into knowledge triples with the structure <head entity, relation, tail entity>;

[0128] Step 4.2: Store the knowledge triples in a graph database and construct an association structure containing entity nodes and relation edges, wherein the entity nodes are associated with the content units of the original modality as evidence;

[0129] Step 4.3: Establish a retrieval index based on the entity nodes, relation edges, and semantic vectors, and integrate them to form a cross-modal knowledge graph with multimodal association and traceability; wherein, the retrieval index includes entity type index, relation type index, and semantic vector index; the cross-modal knowledge graph supports visual organization and query operations.

[0130] For example, after completing multimodal entity extraction and semantic alignment, the system generates a preliminary cross-modal knowledge graph: each entity node includes: entity identifier, modal source (such as text, image, video), context description, frequency of occurrence, and modal vector representation. Each relation edge includes: entity pair, relation type, modal evidence fragment, and extraction confidence; the cross-modal knowledge graph is organized using a graph database or graph storage model.

[0131] After the cross-modal knowledge graph is established, the following indexing mechanisms are initialized: Entity Index Table: Records the unique IDs and modal types of all entity nodes. Relationship Index Table: Stores all triple structures for path retrieval; Modality-Content Lookup Table: Establishes the reference relationships between structural nodes and corresponding original data (image paths, frame numbers, text segments, etc.); Semantic Vector Index Library: Used for semantic similarity calculation between entities, supporting semantic search and matching.

[0132] Example 2

[0133] Building upon Example 1, this invention presents a retrieval enhancement mechanism that combines a knowledge graph structure with a generative language model. Based on the entity-relationship network structure of the knowledge graph, this mechanism utilizes techniques such as community segmentation, semantic summarization, and structure-guided retrieval to provide more accurate, interpretable, and highly recall semantic enhancement context for complex semantic tasks in multimodal scenarios, enabling downstream functions such as knowledge generation, semantic question answering, and triple completion.

[0134] like Figure 1 As shown, this embodiment also includes step 5: based on the cross-modal knowledge graph, an enhanced generation mechanism of graph structure retrieval (GraphRAG) is introduced to construct a retrieval index and perform community segmentation and summary generation on the cross-modal knowledge graph. The retrieval path is planned according to the query intent, and the structure generation enhancement prompts of the cross-modal knowledge graph are combined to realize knowledge completion and semantic reasoning through a language model.

[0135] It should be noted that in this embodiment, step 5 can be set after step 4 to carry out retrieval enhancement and reasoning based on the formed cross-modal knowledge graph; or it can be adjusted to be implemented before step 4. That is, before the unified entity relationship set is standardized into knowledge triples, a graph structure retrieval enhancement generation mechanism is introduced based on the multimodal knowledge element set or the unified entity relationship set. Through community segmentation, path planning and other technologies, potential relationships between entities are explored in advance to provide semantic guidance for the subsequent standardization of knowledge triples and graph construction. Finally, the cross-modal knowledge graph is still formed through step 4.

[0136] It should be noted that the GraphRAG architecture has the following significant improvements compared to the conventional RAG (Retrieval-Augmented Generation): it supports structured knowledge sources (i.e., graphs) as retrieval control conditions; it can obtain topic aggregation and path constraints through semantic community partitioning; and it supports the retrieval and construction of cross-modal semantic content in a multimodal context.

[0137] In one possible implementation, step 5 includes the following sub-steps:

[0138] Step 5.1: Use a community discovery algorithm to divide the cross-modal knowledge graph into topic communities. Based on the entity type index and relation type index described in Step 4.3, extract the core entities, main relations and context information of each community to generate a community knowledge summary.

[0139] It should be noted that community segmentation refers to dividing the graph based on the graph topology and edge density features, using community discovery algorithms such as Leiden and Louvain; each community represents a semantic topic or subdomain knowledge fragment, and has strong local entity aggregation (such as a people-organization-activity structure cluster).

[0140] Summary generation refers to the structural analysis of the core nodes, main paths, and key relationships in each community; extracting the relational context (such as third-order adjacency paths) of the community's central entity; and using LLM to generate a brief natural language description of the community, including the main entities, event relationships, and background description. The generated summary is both readable and structurally compact, and can be used for model prompt construction or to assist human-computer interaction.

[0141] Step 5.2: Identify the query intent type, plan a single-hop search, multi-hop path search, or community search path according to the intent type, and obtain a graph support fragment containing core entities, relational structures, and modal evidence based on the search index described in Step 4.3 and the community knowledge summary described in Step 5.1.

[0142] For example, when a semantic query is received, the system automatically determines the type of query intent and designs a structured retrieval path accordingly: Type identification: Identify whether the query is: factual question and answer, event tracing, multi-hop reasoning, modal verification (such as who is in the picture), relationship discovery, etc.; a classification model or intent identification module can be used to complete the identification by combining question keywords and structural features.

[0143] Structural path planning strategies: Single-hop retrieval: directly locates entities and their relation edges; Multi-hop path retrieval: performs K-order adjacency expansion based on the entity graph to search for entity chains; Community retrieval: retrieves the summary or adjacent entity information of the entity's community to enhance the language model's contextual understanding; Modal content retrieval: based on the query modal intent, prioritizes recalling entities with corresponding modal labels and their evidence sources. The result of path planning is an output graph support fragment, including core entities, entity relation structure, community summary, and related modal content.

[0144] Step 5.3: Integrate the graph support fragments and query intent described in Step 5.2 into structural enhancement prompts, input them into the language model to generate knowledge content that conforms to the cross-modal knowledge graph structure. The knowledge content includes entity relation completion results, multimodal association reasoning conclusions, and natural language answers, thereby realizing knowledge completion and semantic reasoning.

[0145] For example, combining graph support fragments with natural language queries to form structural prompts guides the language model in knowledge generation or answer generation. The processing flow includes:

[0146] Prompt Construction: The input segments are organized according to a template, consisting of graph summaries, entity triples, and raw text fragments; structural labels are injected; it can connect to large language models that support long contexts; and it executes instruction-driven generation. It controls the generated length, logical consistency, and citation accuracy; and validates the generated content, including entity name hit rate, factual consistency score, and contextual relevance.

[0147] For newly added knowledge items in the generated results, the following processing is performed: New entity identification: Determine whether it is a new entity that does not exist in the graph. If the confidence is high, add a node; New relation construction: Structure the generated statement into triples and update the relation edges; Knowledge verification mechanism: If there is an entity with the same name, call the semantic disambiguation model to determine whether it belongs to the original entity; If it is conflicting information, it can be recorded as a triple to be confirmed for manual confirmation; Knowledge evolution chain management: Mark the new knowledge generated by GraphRAG with the generation source tag and timestamp for subsequent version evolution and traceability.

[0148] Through the above steps, the GraphRAG retrieval enhancement mechanism of this invention achieves: efficient linkage between the graph structure and the large model; support for cross-modal, multi-hop, and multi-source fusion query tasks; and realization of the knowledge graph's scalability and knowledge generation closed loop. This mechanism is applicable to complex intelligent application scenarios such as question answering systems, intelligent reasoning, multimodal search, and content analysis, significantly improving the system's semantic depth, response accuracy, and knowledge completeness.

[0149] Example 3

[0150] Based on Examples 1 and 2, in order to achieve continuous optimization and intelligent evolution of the knowledge graph, and to ensure that the graph can achieve dynamic incremental updates, entity fusion and relationship completion when facing new data, modal expansion and knowledge discovery needs, and at the same time, to achieve deep semantic association, path reasoning and unexplicit knowledge discovery based on graph structure by using reasoning mechanisms such as graph neural networks and language models.

[0151] like Figure 1 As shown, this embodiment also includes step 6 after step 5: dynamically updating the cross-modal knowledge graph through incremental data access, fusing new entities with the existing cross-modal knowledge graph, mining potential relationships by combining graph neural networks and language models, recording update logs and maintaining version chains, and supporting the iterative evolution and knowledge expansion of the cross-modal knowledge graph.

[0152] In one possible implementation, step 6 includes the following sub-steps:

[0153] Step 6.1: Monitor the sources of multi-source heterogeneous data in real time, obtain newly added original data units, and sequentially perform the structured processing of Step 1 and the entity and relation extraction process of Step 2 on the newly added original data units to generate an incremental entity relation set.

[0154] For example, the system continuously monitors multimodal data sources (such as news streams, video streams, real-time audio, etc.). Once a new data item is detected and collected and cleaned, an automatic update process is triggered. Detecting that a new data item has been collected and cleaned means that the structured processing described in step 1 is performed sequentially on the newly added raw data unit.

[0155] Change comparison and version identification: Use content hash to determine whether the new data is duplicate data or enhanced content of an existing entity; if it is completely new data, it is included in the candidate processing flow; if it is an extended modality or added attribute of an existing entity, it is marked as a graph completion task.

[0156] The candidate processing flow includes the entity and relation extraction process in step 2, which ultimately generates an incremental entity and relation set.

[0157] It should be noted that the update trigger strategy can be configured to support three types of update mechanisms: time-based trigger (scheduled task), event-based trigger (new entity exceeds threshold), and manual trigger (expert annotation feedback); to ensure that the map remains fresh, timely, and contextually continuous.

[0158] Step 6.2: Calculate the semantic similarity between the entities in the incremental entity relationship set and the entities in the existing cross-modal knowledge graph, and perform entity fusion or addition operations based on the similarity threshold: merge multimodal attributes and update the existing knowledge triples for entities with similarity exceeding the threshold, and create new knowledge triples for entities with similarity below the threshold; generate an updated knowledge triple set by handling relationship conflicts through evidence weighting, multi-source verification, or manual intervention.

[0159] For example, the following update operation is performed on newly arrived data entity and relationship information:

[0160] Entity alignment and fusion judgment: Newly identified entities are semantically matched with existing graph nodes (based on vector similarity, name normalization, modal context, etc.). If the matching degree is higher than the threshold, an entity fusion operation is performed to merge its attributes and the context in which it appears. Otherwise, it is inserted into the graph as a new node and connected with its context entities. A hierarchical strategy is adopted for the entity fusion similarity threshold to avoid semantic distortion caused by excessive merging.

[0161] If an entity already has a text description, but new data provides image, video, or audio modal content, this modal information is attached as a supplementary attribute to the entity, improving the entity's multimodal coverage; cross-modal reference mapping updates are supported to enhance structural richness. Newly extracted triples are checked for conflicts with existing relation edges; if the relation types are the same but the description context or confidence level differs, it can be retained as a multi-relational edge or recorded as a version evolution trajectory; a three-level strategy is used to handle relation edge conflicts that are severe or logically contradictory:

[0162] Evidence weighting refers to calculating the overall confidence level of new and existing relationships based on the credibility of their sources, retaining items with high confidence levels. Multi-source verification means that if conflicting relationships come from different modalities, bimodal evidence of "text + image" (such as consistency between video subtitles and in-frame identifiers) is given priority. Human intervention refers to triggering an expert annotation interface for conflicts involving high-priority entities (such as medical terminology or legal entities), with the results fed back to the model training library.

[0163] Step 6.3: Use a graph neural network to perform representation learning on the updated knowledge triple set to predict the relationships between potential entities; combine the language model to generate candidate relationship descriptions for high-frequency co-occurring but unrelated entity pairs to form a potential relationship set.

[0164] To complete the relationships not explicitly labeled in the graph or to discover potential entity connection paths, structured reasoning techniques such as graph neural networks are introduced.

[0165] For example, the Relational Generative Neural Network (R-GCN) model is used to learn representations for graph nodes. Through multi-hop adjacency propagation, structurally related but not directly connected entity pairs are mined to construct latent semantic similarity relationships. An edge prediction task is performed on candidate entity pairs in the graph, outputting the possible relationship types and their confidence levels. If the prediction confidence exceeds a set threshold, it is added as an implicit triple to the completion candidate pool. All candidate completion edges record the prediction source and model version for knowledge verification and continuous training. Cross-level, multi-hop fact chains are identified based on the PathRanking Algorithm (PRA) or graph-structure-based logical rules (Horn rules).

[0166] Step 6.4: Based on the updated set of knowledge triples and the set of potential relations, record the timestamp, update content and operation type of each update, maintain the version chain of the cross-modal knowledge graph, and generate the updated cross-modal knowledge graph; the updated cross-modal knowledge graph serves as the base graph for subsequent execution of Step 5 to support the dynamic iteration of retrieval and reasoning.

[0167] For example, each graph update is labeled with a version number, update time, update method (addition / completion / replacement), and change log; historical version rollback, comparison, and recovery operations are supported to ensure the stability and controllability of the graph's long-term evolution. A graph growth evolution view is provided, displaying trends such as entity growth, relationship density changes, and modality coverage evolution; this is used for system review, user evaluation, and management decision-making. User interaction behaviors (such as clicks, selections, and corrections) are used as model feedback signals; entity classifiers, relationship recognizers, and GNN parameters are trained and updated on a batch or incremental basis to improve the system's adaptive capabilities.

[0168] Through this step, the system not only achieves dynamic updating capability of cross-modal knowledge graph, but also integrates structured reasoning capability and generative completion capability, which can be widely applied to complex intelligent application scenarios such as semantic search, multi-turn dialogue, knowledge question answering, and content recommendation.

[0169] In summary, the updated cross-modal knowledge graph has the following advantages: it supports dynamic addition, fusion, and knowledge evolution tracking; it combines graph neural networks to identify potential structural relationships; it integrates completion and generation: forming a knowledge loop through structure + large model reasoning; it continuously absorbs modal extension content such as text, images, and videos; and it has strong stability and controllability: ensuring graph quality through version control and conflict detection.

[0170] Compared with existing single-modal knowledge graph construction methods or traditional multimodal information extraction techniques, the cross-modal knowledge graph construction method proposed in this invention has the following significant advantages:

[0171] This invention supports the simultaneous extraction of knowledge information from multiple modalities such as text, images, videos, and audio, and performs semantic alignment through a multimodal large model. It integrates data from different sources and in different forms into a unified knowledge graph structure, significantly improving the semantic integrity and representation coverage of knowledge.

[0172] By introducing the GraphRAG mechanism, we can fully utilize the structural information of the knowledge graph and the semantic paths between entities to guide the language model to perform accurate generation tasks and complex question-answering reasoning. Compared with traditional RAG or vector retrieval methods, it has stronger reasoning ability and fact consistency guarantee.

[0173] This invention establishes a complete mechanism for incremental data access, entity fusion, triple completion, and version control. It can continuously absorb content from new data sources and automatically update graph nodes and relationships, ensuring the real-time performance, freshness, and semantic evolution capabilities of the graph, making it suitable for long-term operation scenarios.

[0174] The end-to-end automated processing is supported throughout the entire process: from multimodal data acquisition, entity recognition, and pair fusion to triple generation and graph construction, no extensive manual annotation or rule intervention is required, which significantly reduces construction costs and improves development efficiency and system maintainability.

[0175] Thanks to the collaborative reasoning module of graph neural network and language model, this invention can perform multi-hop, multimodal path combination and event association recognition in the graph, which is suitable for highly complex question answering, recommendation, and source tracing scenarios, and improves the intelligence of the system in unstructured big data environment.

[0176] This method adopts a modular structure design, and each processing stage (such as modality processing, graph construction, GraphRAG enhancement, and update mechanism) can be deployed and upgraded independently. It is compatible with mainstream databases, vector libraries and large model interfaces, and is easy to integrate and implement in actual systems.

[0177] All graph entities and relationships retain their source modalities, original evidence, and processing paths, allowing users or the system to trace the knowledge source at any time. This is suitable for fields with high requirements for accuracy and transparency, such as healthcare, finance, and government affairs. The comparative experimental results based on public datasets (COCO, WikiMultimodal) are shown in the table below:

[0178]

[0179] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for constructing a cross-modal knowledge graph, characterized in that, Includes the following steps: Step 1: Acquire multi-source heterogeneous data, and perform data collection, structure analysis, modality recognition and classification, cleaning and standardization on the multi-source heterogeneous data to form structured multimodal data; Step 2: Extract entities, entity attributes, and relationships between entities for each identified modality from the structured multimodal data, and summarize them to form a multimodal knowledge element set; Step 3: Map entities of different modalities in the multimodal knowledge element set to a unified semantic space, and achieve entity alignment and fusion through semantic matching to generate a unified entity relationship set; Step 4: Standardize the unified entity relationship set into knowledge triples, and store and organize them in a structured manner through a graph database to form a cross-modal knowledge graph.

2. The method for constructing a cross-modal knowledge graph according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: Collect raw data units from multiple heterogeneous sources. The raw data units include web page content, documents, images, videos, and audio. Mark each raw data unit with its source path, collection timestamp, and unique identifier. Step 1.2: Perform structural analysis on the original data unit to separate the text content unit, image content unit, video frame content unit, and audio track content unit; Step 1.3: Perform modal recognition and classification labeling on the text content unit, image content unit, video frame content unit, and audio track content unit, encapsulate them into multimodal data objects containing intermodal mapping relationship identifiers, store them in a multimodal data warehouse, and establish a modal cross-index; Step 1.4: Clean each modal content unit in the multimodal data object to generate cleaned multimodal content units; Step 1.5: Semantically extract the video frame content unit and audio track content unit in the multimodal content unit, generate a textual description and store it in association with the original modal features to form a multimodal content unit with attached textual semantics. Step 1.6: Perform language detection, timestamp annotation, and content fingerprint generation on multimodal content units with accompanying textual semantics to form semantically traceable structured multimodal data.

3. The method for constructing a cross-modal knowledge graph according to claim 2, characterized in that, Step 2 includes the following sub-steps: Step 2.1: For the text content units in the structured multimodal data, use the trained large language model to perform named entity recognition and relation extraction, and generate entity pairs with type labels and relation descriptions; Step 2.2: For the image content units in the structured multimodal data, identify objects and scenes through a visual model, convert them into entity and semantic relationship text, and associate them with image location information; Step 2.3: For the video frame content units in the structured multimodal data, perform entity recognition by combining the keyframe entity recognition model, perform action recognition by the action recognition model, and transcribe audio tracks into text to extract event-level entities and dynamic relationships, and associate time intervals; Step 2.4: For the audio track content units in the structured multimodal data, after transcribing the audio into text, extract entities and relations using an entity and relation extraction model, and mark the speaker's affiliation using a voiceprint recognition model; Step 2.5: Summarize the processing results from Steps 2.1 to 2.4, record the modal type of each entity's source and the extracted confidence level, and form a multimodal knowledge element set.

4. The method for constructing a cross-modal knowledge graph according to claim 3, characterized in that, Step 2.5 includes the following sub-steps: Step 2.5.1: Evaluate the confidence level of the entities and relationships extracted in Steps 2.1 to 2.4, and remove entity pairs and relationship descriptions with confidence levels below the set threshold; Step 2.5.2: Mark the frequently occurring entities with unclear referents among the entities and relationships remaining after filtering in Step 2.5.1, and clarify the entity's referent by combining the contextual semantics or associated modal information; Step 2.5.3: Randomly select some entity and relation samples processed in Step 2.5.2 for manual review, and feed back the review results to the large language model used in Step 2.1, the visual model used in Step 2.2, and the entity recognition and relation extraction related models involved in Steps 2.3 and 2.4; Step 2.5.4: After processing in steps 2.5.1 to 2.5.3, record the modal type of each entity's source and extract the confidence level to form a multimodal knowledge element set.

5. The method for constructing a cross-modal knowledge graph according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Encode the entities of each modality in the multimodal knowledge element set using a multimodal pre-trained model to generate equal-dimensional semantic vectors; Step 3.2: Calculate the semantic similarity of entities of different modalities based on the equal-dimensional semantic vectors, and determine entity equivalence by combining time context factors and location context factors, and generate entity equivalence determination results; Step 3.3: Based on the entity equivalence determination results, merge entities that are determined to be equivalent, unify entity identifiers, aggregate multimodal attributes, and establish a modality mapping table; Step 3.4: For entities with the same name but different semantics, use a disambiguation model to distinguish them and update the modality mapping table; Step 3.5: Based on the updated modal mapping table, integrate entities and relationships between entities to generate a unified entity relationship set.

6. The method for constructing a cross-modal knowledge graph according to claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Normalize the unified entity relation set into knowledge triples with the structure <head entity, relation, tail entity>; Step 4.2: Store the knowledge triples in a graph database and construct an association structure containing entity nodes and relation edges, wherein the entity nodes are associated with the content units of the original modality as evidence; Step 4.3: Establish a retrieval index based on the entity nodes, relation edges, and semantic vectors, and integrate them to form a cross-modal knowledge graph with multimodal association and traceability; wherein, the retrieval index includes entity type index, relation type index, and semantic vector index; the cross-modal knowledge graph supports visual organization and query operations.

7. The method for constructing a cross-modal knowledge graph according to claim 6, characterized in that, It also includes step 5: introducing a graph structure retrieval enhancement generation mechanism based on cross-modal knowledge graph, constructing a retrieval index and performing community segmentation and summary generation on cross-modal knowledge graph, planning the retrieval path according to the query intent, combining the structure generation enhancement prompts of cross-modal knowledge graph, and realizing knowledge completion and semantic reasoning through language model.

8. The method for constructing a cross-modal knowledge graph according to claim 7, characterized in that, Step 5 includes the following sub-steps: Step 5.1: Use a community discovery algorithm to divide the cross-modal knowledge graph into topic communities. Based on the entity type index and relation type index described in Step 4.3, extract the core entities, main relations and context information of each community to generate a community knowledge summary. Step 5.2: Identify the query intent type, plan a single-hop search, multi-hop path search, or community search path according to the intent type, and obtain a graph support fragment containing core entities, relational structures, and modal evidence based on the search index described in Step 4.3 and the community knowledge summary described in Step 5.

1. Step 5.3: Integrate the graph support fragments and query intent described in Step 5.2 into structural enhancement prompts, input them into the language model to generate knowledge content that conforms to the cross-modal knowledge graph structure. The knowledge content includes entity relation completion results, multimodal association reasoning conclusions, and natural language answers, thereby realizing knowledge completion and semantic reasoning.

9. The method for constructing a cross-modal knowledge graph according to claim 7, characterized in that, It also includes step 6: dynamically updating the cross-modal knowledge graph through incremental data access, integrating new entities with the existing cross-modal knowledge graph, mining potential relationships by combining graph neural networks and language models, recording update logs and maintaining version chains, and supporting the iterative evolution and knowledge expansion of the cross-modal knowledge graph.

10. The method for constructing a cross-modal knowledge graph according to claim 9, characterized in that, Step 6 includes the following sub-steps: Step 6.1: Monitor the sources of multi-source heterogeneous data in real time, obtain newly added original data units, and sequentially perform the structured processing of Step 1 and the entity and relation extraction process of Step 2 on the newly added original data units to generate an incremental entity relation set; Step 6.2: Calculate the semantic similarity between the entities in the incremental entity relationship set and the entities in the existing cross-modal knowledge graph, and perform entity fusion or addition operations based on the similarity threshold: merge multimodal attributes and update existing knowledge triples for entities with similarity exceeding the threshold, and create new knowledge triples for entities with similarity below the threshold; handle relationship conflicts through evidence weighting, multi-source verification, or manual intervention to generate an updated set of knowledge triples. Step 6.3: Use a graph neural network to perform representation learning on the updated knowledge triple set to predict the relationships between potential entities; combine the language model to generate candidate relationship descriptions for high-frequency co-occurring but unrelated entity pairs to form a potential relationship set; Step 6.4: Based on the updated set of knowledge triples and the set of potential relations, record the timestamp, update content and operation type of each update, maintain the version chain of the cross-modal knowledge graph, and generate the updated cross-modal knowledge graph; the updated cross-modal knowledge graph serves as the base graph for subsequent execution of Step 5, supporting the dynamic iteration of retrieval and reasoning.

Images