Multi-modal automatic knowledge graph construction method based on large language model

Abstract

According to the multi-modal automatic knowledge graph construction method based on the large language model, a multi-modal data stream is preprocessed, features are extracted, and the multi-modal data stream is mapped to a unified semantic space through a cross-modal alignment network after being processed through the large language model, a visual converter and a time sequence neural network. In the space, entities and categories are recognized based on a large language model, a triple is generated by combining a multi-modal feature judgment entity relationship, mapping fusion is performed through an ontology alignment algorithm driven by a graph neural network and a predefined domain ontology, finally knowledge is stored in a graph database, and dynamic updating is performed by means of incremental learning and online reasoning. Standardized APIs and visualization components are provided. According to the method, the construction efficiency and the automation degree of the knowledge graph are remarkably improved, the cross-modal information fusion and knowledge maintenance capability is enhanced, and the application requirements of intelligent retrieval, recommendation, decision support and the like are met.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph construction technology, specifically involving a multimodal automated knowledge graph construction method based on a large language model. It is particularly suitable for automatically extracting entities and relationships, fusing heterogeneous information, and dynamically updating knowledge graphs in environments with large-scale multimodal data such as text, images, audio, and video. Background Technology

[0002] Currently, knowledge graphs, as an important technical means of representing entities and their relationships, have been widely used in fields such as intelligent retrieval, recommendation systems, question answering, and decision support. However, existing knowledge graph construction methods suffer from insufficient automation, difficulties in cross-modal fusion, and high dynamic update costs when faced with the ever-increasing volume of multimodal data, making it difficult to meet the performance and accuracy requirements of large-scale, multi-source, and heterogeneous scenarios.

[0003] Traditional construction methods rely on experts to formulate rules or perform statistical analysis on large-scale corpora. Although they can achieve high accuracy in specific domains, they are poorly scalable and have high maintenance costs due to their heavy reliance on manual annotation and rule design. Even with the introduction of deep learning technology, existing solutions are mostly focused on text modality processing and lack systematic support for non-text data such as images, audio, and video, resulting in incomplete knowledge coverage and limited automation.

[0004] Multimodal information fusion faces challenges such as large differences in representation and difficulties in semantic alignment. Features of different modalities differ significantly in their expression, dimension, and distribution. Existing cross-modal alignment often relies on weak supervision or manual feature matching, resulting in a complex fusion process and unstable results. Meanwhile, dynamic data such as real-time video streams and social media images are constantly being introduced, making it difficult for existing systems to perform efficient incremental updates while ensuring consistency and accuracy.

[0005] Although large language models have demonstrated excellent semantic understanding and generation capabilities in large-scale text pre-training, providing new ideas for text knowledge extraction, existing applications are mainly concentrated in pure text scenarios. A systematic method has not yet been formed that deeply integrates large language models with multimodal features such as visual converters and temporal networks, unifies encoding, and extracts entity relationships in an integrated manner. Therefore, there is an urgent need for a complete technical solution that can fully leverage the semantic capabilities of large language models, combine visual and acoustic features, and achieve efficient multimodal automated knowledge graph construction and dynamic updates. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal automated knowledge graph construction method based on a large language model, which enables deep semantic understanding and unified representation of heterogeneous data such as text, images, audio, and video, automatically extracts entities and relationships, integrates domain ontology and dynamically updates incrementally, improves the efficiency and quality of knowledge graph construction, and meets the needs of various downstream applications such as intelligent retrieval, recommendation, question answering and decision support.

[0007] This invention is implemented as follows:

[0008] A method for constructing a multimodal automated knowledge graph based on a large language model includes the following steps:

[0009] Step 1, Multimodal Data Preprocessing:

[0010] 1) Perform noise reduction, format conversion, or segmentation on raw data streams such as text, images, audio, and video;

[0011] 2) The input text preprocessing module, visual preprocessing module, and acoustic preprocessing module are combined to generate a structured corpus, visual feature set, and acoustic feature set;

[0012] Step 2, Multimodal Feature Extraction:

[0013] 1) Utilize large language models to perform deep semantic encoding on text;

[0014] 2) Extract visual features from images using a visual converter;

[0015] 3) Utilize temporal neural networks to extract acoustic and temporal features from audio / video;

[0016] Step 3, cross-modal semantic alignment:

[0017] 1) By using a cross-modal alignment network, the features of each modality are linearly mapped to a unified semantic space, and affine transformations are performed on the text, visual, and acoustic vectors respectively;

[0018] 2) Multi-head self-attention fusion;

[0019] 3) Conduct two-way comparative learning and optimization;

[0020] Step 4, Entity and Relation Extraction:

[0021] 1) Entity recognition uses the Viterbi algorithm to obtain the optimal label sequence based on the aligned text vector sequence, fine-tuned large language model (LLM) and conditional random field (CRF);

[0022] 2) Relation extraction: Entities in the text are paired to generate candidate pairs. For each pair, the entity(e) is concatenated. i ,e jAlign vectors, input graph attention network classification, and jointly optimize entity recognition and relation classification through multi-task learning;

[0023] Step 5, Ontology matching and fusion:

[0024] 1) An ontology alignment algorithm driven by a graph neural network is adopted to automatically map and fuse the extracted entities and relations with predefined domain ontology, thereby solving the problems of synonymy, ambiguity and hierarchical relationship.

[0025] 2) The mapping results undergo consistency checks and conflict resolution to ensure the integrity and unambiguity of the fused map;

[0026] Step 6, Map Storage and Incremental Update:

[0027] The merged knowledge is stored in the form of a graph database, and entities, relationships and attributes are dynamically updated for new data based on incremental learning strategies and online reasoning mechanisms.

[0028] Step 7, Visualization and Interface Support:

[0029] It provides configurable visualization components and standardized APIs to support natural language querying, knowledge reasoning, and downstream application integration.

[0030] Furthermore, in step 1, the text data is further preprocessed by sentence segmentation, word segmentation, part-of-speech tagging, and named entity recognition.

[0031] Furthermore, in step 2, the large language model is a pre-trained model based on the Transformer architecture and fine-tuned for the domain, the visual converter is a pre-trained ViT (VisionTransformer) model, and the temporal neural network is a bidirectional long short-term memory network (Bi-LSTM).

[0032] Furthermore, in step 2, a large language model is used to perform deep semantic encoding on the text, and aggregation strategies are adopted to improve the quality of feature representation, including average pooling or CLS-token. Attention-weighted pooling can also be used to assign higher weights to more important words. The calculation formula is as follows:

[0033]

[0034] Among them, h t For the weighted aggregated text vector, α i Here, u represents the normalized attention weights, and u is the learnable context vector.

[0035] Furthermore, in step 3, the formula for calculating the affine transformation is:

[0036]

[0037] Among them, h m Let t represent the feature vectors for different modalities, v for the text modality, a for the audio modality, and W for the audio modality. m This is the weight matrix. Ensure that the output dimension is consistently d.

[0038] Furthermore, in step 3, the cross-modal alignment employs a multi-head self-attention mechanism combined with a contrastive loss function for cross-modal representation alignment. The loss function used is a simplified version of the contrastive loss, and its calculation formula is as follows:

[0039]

[0040] Where δ is the safety interval threshold, which encourages positive example similarity and suppresses negative example similarity.

[0041] Furthermore, in step 4, the calculation formula for the Viterbi algorithm is as follows:

[0042]

[0043] in, Let A be the score given by LLM for the i-th word, and let A be the label transition score matrix.

[0044] Furthermore, in step 4, the formula for calculating the splicing alignment vector is:

[0045]

[0046] in, This is the pooling vector of the text fragments between the two entities.

[0047] The graph attention network dynamically adjusts the edge weights based on the semantic relationships between nodes in the graph. Finally, it obtains the relationship probability distribution through a fully connected layer combined with a Softmax function. The calculation formula is as follows:

[0048]

[0049] Among them, b r For the bias term, h ij For node e i With e j The joint feature vector, r is the relation type, and W is the weight vector.

[0050] Furthermore, in step 6, the incremental learning strategy includes two modes: sample replay and online fine-tuning, and automatically switches between them according to data changes.

[0051] Furthermore, in step 7, the visualization component supports multi-level interactive display based on nodes, edges, and subgraphs, and can be dynamically rendered according to user-defined filtering conditions; the standardized API includes multiple interfaces for entity query, relationship query, path search, and pattern matching, and supports both RESTful and GraphQL protocols; it provides user behavior log collection and analysis functions to provide feedback on the quality of knowledge graph construction and usage effectiveness.

[0052] The beneficial effects of this invention are as follows: A multimodal automated knowledge graph construction method based on a large language model employs a large language model and graph neural network methods, eliminating the need for extensive manual annotation and rule design, and achieving end-to-end automatic extraction and fusion of multimodal data; through a unified semantic space and contrastive learning mechanism, it eliminates the semantic gap between different modalities, improving the accuracy of entity recognition and relation extraction; it supports multiple modalities such as text, image, audio, and video, and can flexibly integrate new preprocessing and feature extraction components to adapt to different domain needs; based on incremental learning and online inference mechanisms, it can quickly respond to new data, achieving real-time or near real-time graph updates; it provides rich visualization and standardized API support, significantly improving the efficiency and automation of knowledge graph construction, strengthening cross-modal information fusion and knowledge maintenance capabilities, meeting the application needs of intelligent retrieval, recommendation, and decision support, facilitating seamless integration with existing business systems and intelligent applications, and improving the availability of knowledge services and user experience.

[0053] The present invention will be further explained in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall process of the multimodal automated knowledge graph construction method based on a large language model according to the present invention;

[0055] Figure 2 This is a schematic diagram of the cross-modal alignment and relation extraction sub-process of the present invention. Detailed Implementation

[0056] A method for constructing a multimodal automated knowledge graph based on a large language model, such as Figure 1 , Figure 2 As shown, it includes the following steps:

[0057] Step 1: Multimodal preprocessing:

[0058] 1. Text preprocessing:

[0059] (1) Sentence segmentation and word segmentation: The original text is first segmented into sentences according to punctuation marks, and then word segmentation (WordPiece or BPE) is performed on each sentence to generate a word index sequence {w1,…,w n}

[0060] (2) Part-of-speech tagging and named entity recognition: Use existing tools to tag part-of-speech and entity types so that subsequent models can utilize lexical and preliminary entity clues.

[0061] (3) Noise filtering: delete sentences that are too short or invalid, and map or filter common stop words and special symbols in a unified manner to improve the training efficiency of the model.

[0062] 2. Image preprocessing:

[0063] (1) Size normalization: Scaling or cropping the image to a fixed resolution (e.g., 224×224) to ensure consistent visual model input;

[0064] (2) Denoising and normalization: Apply Gaussian filtering or median filtering to remove interference noise and normalize the pixel values ​​to [0,1] or standardize the mean and variance.

[0065] (3) Slicing and data augmentation: Divide the image into P overlapping or non-overlapping blocks, with a common size of 16×16 pixels, and use augmentation strategies such as random cropping, rotation, and color dithering.

[0066] 3. Audio preprocessing:

[0067] (1) Endpoint detection: Detect silence on continuous speech or video audio tracks and segment effective speech segments;

[0068] (2) Frame segmentation: Divide the segment into frames of fixed duration (e.g., 25ms), and calculate the Mel-spectrogram or MFCC features for each frame;

[0069] (3) Normalization and noise reduction: Perform logarithmic transformation on the spectrum and standardize it, or use algorithms such as spectral subtraction to remove background noise in advance.

[0070] After the above preprocessing is completed, the three modal data are converted into vector sequences or tensors that are easy for deep models to process. These are then input into the text preprocessing module, visual preprocessing module, and acoustic preprocessing module, respectively, to generate a structured corpus, visual feature set, and acoustic feature set. These are then cached in a distributed file system or memory for subsequent high-concurrency access.

[0071] Step 2: Multimodal feature extraction:

[0072] 1. Text feature extraction:

[0073] (1) Model selection: Large-scale language models (LLM) based on Transformer, such as GPT, BERT, and RoBERTa, can be selected. After fine-tuning with domain corpora (such as medicine, finance, and law), more accurate semantic representations can be obtained.

[0074] (2) Forward computation: The sentence {w i Input the model to obtain the hidden vectors of the intermediate layers.

[0075] (3) Aggregation strategy: In addition to average pooling or CLS-token, attention-weighted pooling can also be used to assign higher weights to more important words. The calculation formula is as follows:

[0076]

[0077] Where u is a learnable context vector.

[0078] 2. Visual Feature Extraction:

[0079] (1) Pre-trained ViT (VisionTransformer): ViT pre-trained using large-scale datasets such as ImageNet and COCO, with input image patch sequences {p j}, output the corresponding features

[0080] (2) Position encoding: Position encoding is added to each block vector to preserve spatial position information;

[0081] (3) Feature fusion: Local attention modules can be inserted before pooling to enhance the representation of key regions (such as people, faces, and signs).

[0082] 3. Acoustic Feature Extraction:

[0083] (1) Bi-LSTM architecture: The frame features {x} are then processed. k Input a bidirectional LSTM to obtain the forward hidden state. and reverse hidden state

[0084] (2) Stitching and dimensionality reduction: Usually, the last frame is stitched together in both forward and reverse states. If the dimensionality is too high, it can be reduced to a length consistent with other modalities through a fully connected layer.

[0085] (3) Alternative solutions: The latest timing Transformer or Conv1D-Pooling structures can also be used, and the best one can be selected according to the requirements of real-time performance and accuracy.

[0086] The modality extraction steps described above can be deployed in parallel on a GPU cluster and coordinated through an asynchronous queue mechanism to maximize throughput and reduce latency.

[0087] Step 3: Cross-modal semantic alignment:

[0088] 1. A cross-modal alignment network is used to linearly map features from each modality to a unified semantic space. Affine transformations are then performed on text, visual, and acoustic vectors respectively. The calculation formula is as follows:

[0089]

[0090] in Ensure that the output dimension is consistently d.

[0091] 2. Multi-head self-attention fusion:

[0092] Multi-head self-attention fusion captures multi-dimensional associations of cross-modal features, integrates cross-modal dependencies from different perspectives, and generates a unified representation space by splicing to achieve semantic alignment and complementarity among modalities such as text, image, and audio.

[0093] Will Treating it as a sequence of "words" and inputting it into a multi-head self-attention layer enables the model to automatically learn complementary information between modalities;

[0094] Multiple attention layers can be stacked, or a feedforward network can be added to the top layer for nonlinear transformation.

[0095] 3. Two-way comparative learning optimization:

[0096] By employing a bidirectional contrastive learning mechanism, positive sample pairs are brought closer together while negative sample pairs are pushed further away, a discriminative feature space that is compact within classes and separate between classes is constructed, thereby enhancing semantic representation capabilities and minimizing the representational differences of the same entity in different modalities.

[0097] Positive and negative sample construction: For the same entity, a set of vectors is generated in text, image and audio to form a positive sample pair; other entity vectors form a negative sample pair.

[0098] Loss function: A simplified version of the contrastive loss is used, and the calculation formula is as follows:

[0099]

[0100] Where δ is the safety interval threshold, which encourages positive example similarity and suppresses negative example similarity.

[0101] This alignment module can be deployed as a standalone service, providing a Batch-API interface that supports online or offline training.

[0102] Step 4: Entity Recognition and Relation Extraction

[0103] Within the unified semantic space, the entity and relation extraction task is based on the prompting engineering and fine-tuning technology of the large language model, which automatically identifies entity boundaries and categories, and combines multimodal features such as text, images, audio, and video to generate triples through semantic association calculation using a relation classifier.

[0104] 1. Entity recognition:

[0105] Input: Aligned text vector sequence

[0106] Model: Based on fine-tuned LLM+CRF, with the CRF layer used to capture dependencies between tags;

[0107] Decoding: The Viterbi algorithm is used to find the optimal label sequence. The calculation formula is as follows:

[0108]

[0109] in Let A be the score given by LLM for the i-th word, and let A be the label transition score matrix.

[0110] 2. Relation extraction:

[0111] Candidate pair generation: Pair all entities in the same text together to generate a list of candidate entity pairs;

[0112] Feature splicing: for each pair (e) i ,e j ), concatenate the two aligned vectors, and calculate using the following formula:

[0113]

[0114] in This is the pooling vector of the text fragment between the two entities;

[0115] The relation classifier employs a graph attention network (GAT)-based structure to improve the accuracy of entity relation identification. It dynamically adjusts edge weights based on graph context, and finally obtains the relation probability distribution through a fully connected layer combined with a softmax function. The calculation formula is as follows:

[0116]

[0117] During model training, multi-task learning can be used to jointly optimize entity recognition and relationship classification, so as to share the underlying representation and improve the overall performance.

[0118] Step 5: Ontology matching and fusion:

[0119] Ontology matching and fusion are technologies that solve semantic heterogeneity and achieve knowledge interoperability and integration. Through semantic standardization and knowledge integration, machines can understand the common meaning behind different data, providing an interpretable and reasonable knowledge foundation for the model.

[0120] An ontology alignment algorithm driven by a graph neural network is used to automatically map and fuse the extracted entities and relations with predefined domain ontology, solving the problems of synonymy, ambiguity and hierarchical relationship. The mapping results are checked for consistency and conflict resolution to ensure the integrity and unambiguity of the fused graph.

[0121] 1. Vector matching:

[0122] For extracting entity vector {h e} and the domain ontology node vector {h o Calculate the cosine similarity;

[0123] If the similarity of a pair exceeds the threshold δ o If they are synonyms or semantically similar, then they are considered to be synonyms or semantically similar.

[0124] 2. Rules and Conflict Resolution:

[0125] For fuzzy mappings or many-to-one relationships, priority rules can be applied based on the ontology hierarchy (parent-child relationship, synonym set). Those still unresolved can be marked as "pending review" and can be configured for manual verification.

[0126] Step 6: Knowledge Graph Storage and Incremental Updates:

[0127] The fused knowledge is stored in the form of a graph database, and entities, relations and attributes are dynamically updated for new data based on incremental learning strategies and online inference mechanisms. The incremental learning strategy includes two modes: sample replay and online fine-tuning, and automatically switches according to data changes.

[0128] 1. Initial write:

[0129] Triples (entity-relationship-entity) are written to a graph database that supports ACID transactions, and node and edge indexes are built to accelerate queries.

[0130] 2. Incremental Update:

[0131] Change detection: Perform deduplication and conflict detection on newly extracted triples and existing maps;

[0132] Online learning: Fine-tuning alignment and model extraction only for the changed parts, avoiding full retraining;

[0133] Synchronous write: New or modified nodes and edges are efficiently written through batch interfaces, triggering downstream cache or search index updates.

[0134] Step 7, Visualization and Interface Support:

[0135] It provides configurable visualization components and standardized APIs, supports natural language querying, knowledge reasoning and downstream application integration, and provides user behavior log collection and analysis functions to provide feedback on the quality of knowledge graph construction and usage effectiveness.

[0136] The visualization component supports multi-level interactive display based on nodes, edges, and subgraphs, and can be dynamically rendered according to user-defined filtering conditions; the standardized API includes multiple interfaces for entity query, relationship query, path search, and pattern matching, and supports both RESTful and GraphQL protocols.

[0137] 1. Visual Interface:

[0138] Supports multi-level expansion and scaling, and allows filtering by entity type, relationship type, time, and other dimensions;

[0139] It provides functions such as subgraph export and chart analysis (degree distribution, community discovery results).

[0140] 2. API Service:

[0141] Implemented using both RESTful and GraphQL protocols, with interfaces including entity retrieval, relationship retrieval, path search, subgraph extraction, and aggregation analysis.

[0142] Supports batch and real-time queries, and returns multiple formats such as JSON and GraphSON;

[0143] Integrate identity authentication and access control to ensure data security and compliance of data collection.

[0144] This invention presents a method for constructing a multimodal automated knowledge graph based on a large language model. It fully reveals how to achieve automated, refined, and scalable knowledge graph construction and dynamic updates in a large-scale multimodal environment using large language models and graph neural network technology. Users can build a complete system based on this method, or introduce modules or replace sub-algorithms in stages to meet the needs of different scenarios.

[0145] An automated knowledge graph construction system based on a multimodal automated knowledge graph construction method using a large language model includes:

[0146] The data preprocessing module is used to perform multimodal data preprocessing steps;

[0147] The feature extraction module is used to perform the multimodal feature representation step;

[0148] The alignment module is used to perform cross-modal alignment steps;

[0149] The extraction module is used to perform entity and relation extraction steps;

[0150] The ontology fusion module is used to perform ontology matching and fusion steps;

[0151] The storage update module is used to perform the map storage and incremental update steps;

[0152] The visualization interface module is used to perform visualization and interface support steps.

[0153] The data preprocessing module includes a distributed task scheduler and microservice nodes to support large-scale parallel data processing.

[0154] In the feature extraction module, the visual converter is a pre-trained ViT model, and the temporal neural network is a bidirectional long short-term memory network (Bi-LSTM).

[0155] The alignment module employs a multi-head self-attention mechanism combined with a contrastive loss function for cross-modal representation alignment.

[0156] In the storage update module, the graph database supports transactional operations and distributed consistency protocols, and provides an event subscription interface to trigger online inference.

[0157] The visualization interface module supports drag-and-drop subgraph construction and user-defined view templates, and provides real-time collaboration functionality.

[0158] The system is deployed in a containerized environment with a microservice architecture and uses a service mesh to achieve load balancing and elastic scaling.

[0159] Finally, it should be noted that the above is only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention (such as the application of various formulas, the order of steps, etc.) without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for constructing a multimodal automated knowledge graph based on a large language model, characterized in that, Includes the following steps: Step 1, Multimodal Data Preprocessing: 1) Perform noise reduction, format conversion, or segmentation on raw data streams such as text, images, audio, and video; 2) The input text preprocessing module, visual preprocessing module, and acoustic preprocessing module are combined to generate a structured corpus, visual feature set, and acoustic feature set; Step 2, Multimodal Feature Extraction: 1) Utilize large language models to perform deep semantic encoding on text; 2) Extract visual features from images using a visual converter; 3) Utilize temporal neural networks to extract acoustic and temporal features from audio / video; Step 3, cross-modal semantic alignment: 1) By using a cross-modal alignment network, the features of each modality are linearly mapped to a unified semantic space, and affine transformations are performed on the text, visual, and acoustic vectors respectively; 2) Multi-head self-attention fusion; 3) Conduct two-way comparative learning and optimization; Step 4, Entity and Relation Extraction: 1) Entity recognition uses the Viterbi algorithm to obtain the optimal label sequence based on the aligned text vector sequence, fine-tuned large language model (LLM) and conditional random field (CRF); 2) Relation extraction: Entities in the text are paired to generate candidate pairs. For each pair, the entity(e) is concatenated. i ,e j Align vectors, input graph attention network classification, and jointly optimize entity recognition and relation classification through multi-task learning; Step 5, Ontology matching and fusion: 1) An ontology alignment algorithm driven by a graph neural network is adopted to automatically map and fuse the extracted entities and relations with predefined domain ontology, thereby solving the problems of synonymy, ambiguity and hierarchical relationship. 2) The mapping results undergo consistency checks and conflict resolution to ensure the integrity and unambiguity of the fused map; Step 6, Map Storage and Incremental Update: The merged knowledge is stored in the form of a graph database, and entities, relationships and attributes are dynamically updated for new data based on incremental learning strategies and online reasoning mechanisms. Step 7, Visualization and Interface Support: It provides configurable visualization components and standardized APIs to support natural language querying, knowledge reasoning, and downstream application integration.

2. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 1, the text data is further preprocessed by sentence segmentation, word segmentation, part-of-speech tagging, and named entity recognition.

3. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 2, the large language model is a pre-trained model based on the Transformer architecture and fine-tuned for the domain, the visual converter is a pre-trained ViT (VisionTransformer) model, and the temporal neural network is a bidirectional long short-term memory network (Bi-LSTM).

4. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 2, a large language model is used to perform deep semantic encoding on the text. Aggregation strategies are adopted to improve the quality of feature representation, including average pooling or CLS-token. Attention-weighted pooling can also be used to assign higher weights to more important words. The calculation formula is as follows: Among them, h t For the weighted aggregated text vector, α i Here, u represents the normalized attention weights, and u is the learnable context vector.

5. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 3, the formula for calculating the affine transformation is: Among them, h m Let t represent the feature vectors for different modalities, v for the text modality, a for the audio modality, and W for the audio modality. m This is the weight matrix. Ensure that the output dimension is uniformly d.

6. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 3, the cross-modal alignment employs a multi-head self-attention mechanism combined with a contrastive loss function for cross-modal representation alignment. The loss function used is a simplified version of the contrastive loss, and the calculation formula is as follows: Where δ is the safety interval threshold, which encourages positive example similarity and suppresses negative example similarity.

7. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 4, the calculation formula for the Viterbi algorithm is as follows: in, Let A be the score given by LLM for the i-th word, and let A be the label transition score matrix.

8. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 4, the formula for calculating the spliced ​​alignment vector is: in, This is the pooling vector of the text fragment between the two entities; The graph attention network dynamically adjusts the edge weights based on the semantic relationships between nodes in the graph. Finally, it obtains the relationship probability distribution through a fully connected layer combined with a Softmax function. The calculation formula is as follows: Among them, b r h is the bias term. ij For node e i With e j The joint feature vector, r is the relation type, and W is the weight vector.

9. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 6, the incremental learning strategy includes two modes: sample replay and online fine-tuning, and automatically switches between them according to data changes.

10. The multimodal automated knowledge graph construction method according to claim 1, characterized in that, In step 7, the visualization component supports multi-level interactive display based on nodes, edges, and subgraphs, and can be dynamically rendered according to user-defined filtering conditions; the standardized API includes multiple interfaces for entity query, relationship query, path search, and pattern matching, and supports both RESTful and GraphQL protocols; it provides user behavior log collection and analysis functions to provide feedback for the knowledge graph.

Images