Knowledge graph construction method and system for intelligently fusing multi-modal data

Abstract

The invention provides a knowledge graph construction method and system for intelligently fusing multi-modal data, and the system employs a layered architecture, and comprises a data input layer, a mode special extractor, a multi-modal aligner, a graph construction / mining device, and a multi-modal retrieval service layer. The modular design enables the system to have good expandability and maintainability, and each module can be independently developed and optimized. The characteristics of different modal data are fully utilized, and the efficiency and accuracy of feature extraction are improved. Through the accurate alignment processing, the accuracy of entities and relationships in the knowledge graph can be effectively improved, and the alignment error is reduced. The deep association relationship among the entities in the basic knowledge graph is mined through the configured deep association mining engine, so that potential knowledge hidden in data can be found, and the content of the knowledge graph is enriched. A user can obtain knowledge from multiple angles, different application requirements are met, and the utilization efficiency of the knowledge is improved.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph application technology, and in particular to a method for constructing a knowledge graph that intelligently integrates multimodal data, as well as a knowledge graph construction system, electronic device, and computer-readable storage medium that intelligently integrates multimodal data. Background Technology

[0002] In existing technologies, multimodal knowledge graphs are constructed in the following ways:

[0003] Data collection: Collecting data from multiple modalities (such as text, images, audio, video, etc.). These data sources are diverse and may include online text, social media images, voice recordings, etc.

[0004] Feature extraction: For different modalities of data, appropriate feature extraction methods are used. For example, for text data, semantic features are extracted using methods such as word embedding and topic modeling; for image data, visual features are extracted using convolutional neural networks (CNNs); and for audio data, acoustic features are extracted using audio feature extraction algorithms (such as MFCC).

[0005] Alignment processing: Aligning features extracted from different modalities to achieve a unified semantic representation. Common alignment methods include deep learning-based cross-modal mapping models, which train neural networks to map features from different modalities to the same semantic space, thereby finding the correspondence between data from different modalities.

[0006] Knowledge graph construction: Based on the aligned features, entities and the relationships between them are determined, and a knowledge graph is constructed. A graph database (such as Neo4j) can be used to store and manage the knowledge graph.

[0007] However, the above steps have the following technical drawbacks:

[0008] 1. Data Processing Challenges: The heterogeneity and diversity of multimodal data complicate data preprocessing and feature extraction. Different modalities exhibit significant differences in data format, size, and quality; for example, image data has varying resolutions, while text data may contain noise, further increasing the difficulty of data processing.

[0009] 2. Low alignment accuracy: Achieving accurate alignment of data from different modalities is a challenge. Due to the differences in semantic representation among different modalities, errors are prone to occur during the alignment process, leading to inaccurate alignment of entities and relations, which affects the quality of the knowledge graph.

[0010] 3. High computational resource requirements: Constructing a multimodal knowledge graph requires substantial computational resources, especially when training deep learning models and processing large-scale data. This not only increases the construction cost but also limits the system's scalability.

[0011] 4. Lack of universal standards: Currently, the construction of multimodal knowledge graphs lacks unified standards and specifications. Different research and applications use different methods and technologies, resulting in poor interoperability between knowledge graphs. Summary of the Invention

[0012] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0013] On the one hand, a knowledge graph construction system that intelligently integrates multimodal data is provided, the system comprising:

[0014] The data input layer is used to input multimodal data into the modality-specific extractor;

[0015] A modality-specific extractor is used to split the multimodal data, based on the deployed multimodal feature processing system to split the data features of each modality and transmit them to the multimodal aligner;

[0016] The multimodal aligner is used to perform unified semantic representation on the split modal data features, align the entities and relationships of the modal data features, and send them to the graph builder / miner;

[0017] The graph builder / miner is used to build a corresponding basic knowledge graph based on the entities and the relationships between them after alignment of the modal data features, and to mine the deep association relationships between the entities in the basic knowledge graph through a configured deep association mining engine.

[0018] The multimodal retrieval service layer is used to provide knowledge retrieval and question answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

[0019] Preferably, the multimodal feature processing system of the modality-specific extractor includes:

[0020] A text processor for identifying entities and extracting relationships from text based on nBERT / RoBERTa, including: locating entities in text and mining semantic relationships between entities;

[0021] Image processor for detecting targets in images based on YOLO / CLIP, including: identifying objects in images and analyzing spatial / semantic relationships between objects;

[0022] A video processor for extracting keyframes from videos and analyzing motion behavior within those keyframes using 3D-CNN / LSTM.

[0023] The structure processor is used to map table structures to graph nodes / relationships through the rule engine.

[0024] Preferably, the multimodal aligner achieves cross-modal semantic alignment through unified semantic representation learning (multimodal Transformer), aligning entities and relations in multimodal data and eliminating intermodal differences.

[0025] Preferably, the multimodal aligner includes:

[0026] The entity alignment module is used to align entities of different modalities through vector similarity calculation and attribute matching.

[0027] The Relationship Alignment Module is used to align relationships of different modalities through evidence fusion and confidence weighting.

[0028] Preferably, the map builder / miner includes:

[0029] The entity fusion module is used for disambiguation (distinguishing entities with the same name) and merging (integrating different modal information of the same entity) of aligned entities.

[0030] The relationship fusion module is used to complete (supplement missing relationships) and verify (validate relationship validity) the aligned relationships.

[0031] The attribute fusion module is used to resolve attribute conflicts between entities and relations input by the entity fusion module and the relation fusion module respectively, and to fuse the differences in descriptions of the same attribute by different modalities.

[0032] The graph construction module is used to construct the basic knowledge graph based on the entities, relationships, and attributes fused and output by the attribute fusion module.

[0033] A deep association mining engine for mining deep associations between entities in a graph based on GNN (Graph Neural Network), TransE (Embedding Algorithm), and / or rule-based reasoning models.

[0034] Preferably, the multimodal retrieval service layer includes:

[0035] A graph database is used to store the basic knowledge graph and provide knowledge retrieval services based on vector indexes (to accelerate semantic retrieval);

[0036] The graph traversal query module is used to query related information by traversing nodes / relationships based on the graph structure of the basic knowledge graph.

[0037] The semantic vector search module is used to retrieve semantically related entities / relationships from the basic knowledge graph through semantic vector matching;

[0038] The hybrid condition retrieval module is used to retrieve multimodal association information from the basic knowledge graph by combining with the multimodal feature processing system;

[0039] The intelligent question-answering service module is used to understand user questions based on LLM and automatically input the above-mentioned graph database, graph traversal query module, semantic vector search module and / or mixed condition retrieval module for knowledge retrieval. Based on the retrieval results, it generates corresponding intelligent knowledge-based answer dialogues and outputs them.

[0040] On the other hand, a knowledge graph construction system for intelligently fusing multimodal data is provided. This system is used to implement the aforementioned knowledge graph construction method for intelligently fusing multimodal data. The system includes:

[0041] S1. Input multimodal data into the modality extractor;

[0042] S2. The modality-specific extractor splits the multimodal data, and based on the deployed multimodal feature processing system, splits the data features of each modality and transmits them to the multimodal aligner;

[0043] S3, the multimodal aligner performs unified semantic representation on the features of each split modal data, aligns the entities and relationships of each modal data feature, and sends them to the graph builder / miner;

[0044] S4. The graph builder / miner constructs a corresponding basic knowledge graph based on the entities and their relationships among the aligned modal data features, and mines the deep association relationships among the entities in the basic knowledge graph through the configured deep association mining engine.

[0045] S5. The multimodal retrieval service layer provides knowledge retrieval and question-answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

[0046] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when executed by the processor, the computer-readable instructions implement any of the methods described above for constructing a knowledge graph using intelligent fusion of multimodal data.

[0047] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for constructing a knowledge graph using intelligent fusion of multimodal data.

[0048] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0049] This system has the following technical advantages in the construction and application of multimodal knowledge graphs:

[0050] 1. Modular Design: The system adopts a layered architecture, including a data input layer, a dedicated modality extractor, a multimodal aligner, a graph builder / miner, and a multimodal retrieval service layer. This modular design gives the system good scalability and maintainability, allowing each module to be developed and optimized independently.

[0051] 2. Efficient Feature Processing: The dedicated modality extractor, based on a deployed multimodal feature processing system, can selectively break down data features for each modality. This approach fully leverages the characteristics of different modalities, improving the efficiency and accuracy of feature extraction.

[0052] 3. Accurate Alignment Mechanism: The multimodal aligner can provide a unified semantic representation for the split modal data features, achieving entity and relation alignment across modal data features. This precise alignment process effectively improves the accuracy of entities and relations in the knowledge graph and reduces alignment errors.

[0053] 4. Deep Relationship Mining: The graph builder / miner can not only construct the basic knowledge graph, but also mine the deep relationships between entities in the basic knowledge graph through a configured deep relationship mining engine. This helps to discover potential knowledge hidden in the data and enrich the content of the knowledge graph.

[0054] 5. Powerful Retrieval Services: The multimodal retrieval service layer, based on a fundamental knowledge graph and a deep association mining engine, provides knowledge retrieval and question-answering services across various dimensions. Users can acquire knowledge from multiple perspectives, meeting diverse application needs and improving the efficiency of knowledge utilization. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the topology of a knowledge graph construction system for intelligent fusion of multimodal data provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0059] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0060] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0061] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0062] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0063] This invention provides a method for constructing a knowledge graph by intelligently fusing multimodal data. This method can be implemented by an electronic device, which can be a terminal or a server.

[0064] like Figure 1 The diagram shows the topology of a knowledge graph construction system that integrates intelligent multimodal data. The system includes:

[0065] The data input layer is used to input multimodal data into the modality-specific extractor;

[0066] A modality-specific extractor is used to split the multimodal data, based on the deployed multimodal feature processing system to split the data features of each modality and transmit them to the multimodal aligner;

[0067] The multimodal aligner is used to perform unified semantic representation on the split modal data features, align the entities and relationships of the modal data features, and send them to the graph builder / miner;

[0068] The graph builder / miner is used to build a corresponding basic knowledge graph based on the entities and the relationships between them after alignment of the modal data features, and to mine the deep association relationships between the entities in the basic knowledge graph through a configured deep association mining engine.

[0069] The multimodal retrieval service layer is used to provide knowledge retrieval and question answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

[0070] The system works as follows: Starting with multimodal data input, the data is split by a dedicated modality extractor, its features are parsed by a modality processor, and its semantic representation is unified by a multimodal aligner. Finally, a knowledge graph is generated through a graph builder / miner, and application services are provided based on a multimodal retrieval service layer. The overall logic is: data input → modality parsing → semantic alignment → graph construction → application services, realizing the fusion, understanding, and value release of multimodal data.

[0071] The specific system components are as follows:

[0072] 1. Data Input Layer

[0073] Multimodal data input: Receives multimodal raw data such as text, images, videos, and structured data to provide a data source for subsequent processing.

[0074] 2. Modal Extractor Layer

[0075] Multimodal data is split according to type and then sent to the corresponding modality processor. For example:

[0076] Text: Passed to a text processor;

[0077] Image: Passed to the image processor;

[0078] Video: Transferred to the video processor;

[0079] Structured data: Passed to the structured processor.

[0080] For different modalities of data, dedicated algorithms are used to extract features and information. The multimodal feature processing system of the modality-specific extractor includes:

[0081] (1) A text processor for identifying entities and extracting relationships in text based on nBERT / RoBERTa, including: locating entities in text and mining semantic relationships between entities;

[0082] Entity recognition: nBERT (Neural Bidirectional Encoder Representations from Transformers) and RoBERTa (Robustly Optimized BERT Pretraining Approach) are both pre-trained language models based on the Transformer architecture. nBERT may have been optimized for specific domains or tasks based on BERT, while RoBERTa improves the BERT pre-training process, making it more efficient and stable. In entity recognition tasks, text is input into a finely tuned nBERT / RoBERTa model. The model uses the language knowledge learned during pre-training and the task-related knowledge learned during fine-tuning to classify each word in the text and determine whether it belongs to a certain entity type, such as a person's name, place name, or organization name.

[0083] Relation Extraction: After entity recognition, the model further explores the semantic relationships between entities. This is typically done by analyzing the contextual information of entities in the text and using the model's attention mechanism to capture the interaction information between entities. The model learns feature representations between different entity pairs and maps them to predefined relation categories, thereby determining the semantic relationships between entities, such as "works at" or "place of birth".

[0084] For example, in news text processing: For a news report, a text processor can locate entities such as people, places, and events. For instance, in the sentence "Steve Jobs founded Apple in California in 1976," the processor can identify entities such as "Steve Jobs," "Apple," and "California," and extract relationships such as "Steve Jobs - founded - Apple" and "Steve Jobs - founding location - California."

[0085] The nBERT / RoBERTa models are pre-trained on large-scale corpora, enabling them to learn rich linguistic knowledge and semantic information, thus performing excellently in entity recognition and relation extraction tasks. The Transformer architecture's attention mechanism allows the model to fully consider the contextual information of the text, accurately handling texts with complex semantics and long-range dependencies. Furthermore, the model can be fine-tuned to adapt to the needs of different domains and tasks, requiring only domain-specific labeled data for training.

[0086] (2) Image processor, used to detect targets in images based on YOLO / CLIP, including: identifying objects in images and analyzing spatial / semantic relationships between objects;

[0087] YOLO (You Only Look Once) is a real-time object detection algorithm that treats object detection as a regression problem, predicting the bounding box and category of objects directly on an image. YOLO divides the image into multiple grids, with each grid responsible for predicting the object within its own grid. CLIP (Contrastive Language - Image Pretraining) is a contrastive learning-based model that learns the association between images and text, enabling image understanding and object detection using textual cues. In image processors, YOLO can quickly locate objects in an image, while CLIP leverages its cross-modal capabilities, combined with textual descriptions, to more accurately identify object categories. After identifying objects in an image, the processor analyzes the spatial relationships between objects, such as vertical, horizontal, and containment relationships. Simultaneously, it utilizes the semantic information learned by CLIP to analyze semantic associations between objects, such as the predator-prey relationship in a cat-mouse game.

[0088] For example, in security monitoring scenarios, image processors can detect targets such as people and vehicles in the monitoring screen in real time and analyze the spatial relationships between them, such as whether people have entered a specific area or whether vehicles are parked in a designated location.

[0089] The high efficiency of the YOLO algorithm enables image processors to complete object detection tasks quickly, making it suitable for scenarios with high real-time requirements. The introduction of CLIP allows image processors to utilize textual information for more accurate object recognition and handle cross-modal image understanding tasks with complex semantics. The appropriate YOLO version and CLIP usage can be selected and adjusted according to different application scenarios and needs to achieve optimal performance.

[0090] (3) Video processor, used to extract video keyframes and analyze the action behavior in the video keyframes based on 3D-CNN / LSTM;

[0091] Videos consist of a series of consecutive frames. Keyframe extraction aims to select representative frames from the video to reduce data volume while retaining the main information. Content-based methods are commonly used, such as calculating the similarity between frames and selecting frames with significant variations in similarity as keyframes. 3D-CNN (3D Convolutional Neural Networks) and LSTM (Long Short-Term Memory) are two models commonly used for video motion analysis. 3D-CNN can process the three-dimensional data (spatial and temporal dimensions) of a video, extracting spatiotemporal features through convolutional operations. LSTM, on the other hand, is a recurrent neural network capable of processing sequential data, making it suitable for analyzing the temporal series information of actions in videos. In a video processor, the extracted keyframes are input into the 3D-CNN / LSTM model, which learns the motion patterns and features in the video, thereby recognizing actions such as running, dancing, and falling.

[0092] For example, in sports event videos, video processors can extract keyframes, such as moments when athletes start running, shoot, or hit the goal, and then use motion behavior analysis to evaluate the athletes' technical movements and performance, providing data support for coaches and athletes.

[0093] 3D-CNN can process both spatial and temporal information in video simultaneously, while LSTM can capture the temporal sequence features of actions, enabling video processors to accurately analyze complex actions in videos.

[0094] Keyframe extraction reduces the amount of data processed in video, improves processing efficiency, and preserves the main information of the video. By appropriately selecting keyframe extraction methods and model parameters, high accuracy in action recognition can be achieved while ensuring real-time processing.

[0095] (4) Structured processor, used to perform table structure mapping through the rule engine, mapping structured data to graph nodes / relationships.

[0096] Table structure mapping in a structured data processor: A rule engine is a rule-based reasoning system that contains a series of rules and conditions. In a structured data processor, the rule engine maps fields and records in structured data (such as database tables, spreadsheets, etc.) to graph nodes and relationships according to predefined rules. For example, defining a rule: "If one column in the table represents a person's name and another column represents a company name, and there is some business relationship, then map the person's name to a person node, the company name to an organization node, and establish a 'work-to' relationship between them." After completing the table structure mapping, the mapped nodes and relationships are combined into a knowledge graph. Nodes in the graph represent entities, and edges represent relationships between entities. Knowledge graphs provide a more intuitive representation of the relationships and semantic information between structured data.

[0097] For example, in enterprise data management, a company's database contains employee information tables, department information tables, etc. A structured data processor can map employee information to person nodes and department information to organization nodes. Through a rule engine, it can establish the "affiliation" relationship between employees and departments, build an organizational structure knowledge graph for the enterprise, and facilitate personnel management and information retrieval.

[0098] Therefore, the rule engine allows users to define flexible mapping rules based on different business needs and data characteristics, adapting to various complex structured data. It integrates structured data from different sources and formats into the knowledge graph, breaking down data silos and facilitating global data analysis and mining. The knowledge graph can intuitively express the semantic relationships between structured data, improving data understandability and utilization value.

[0099] 4. Multimodal Aligner Layer

[0100] Cross-modal semantic alignment is achieved through unified semantic representation learning (multimodal Transformer), resolving the issue of semantic differences between modalities. The following alignment module aligns entities and relations in multimodal data, eliminating intermodal differences.

[0101] (1) Entity alignment module methods: vector similarity calculation, attribute matching;

[0102] Function: Align entities of different modalities (such as text entities and image entities).

[0103] Vector similarity calculation: Entities are represented as vectors, which can be obtained through pre-trained models (such as word embedding models) that can capture the semantic information of entities. In the vector space, the similarity between different entity vectors is calculated. Common similarity calculation methods include cosine similarity, Euclidean distance, etc. The higher the similarity, the closer the two entities are semantically, thus determining whether they are the same entity. For example, in the text, the vector similarity between "Apple Inc." and "苹果公司" is very high, so they can be considered the same entity.

[0104] Attribute matching: Each entity has its specific attributes, such as name, description, category, etc. By comparing the attributes of different entities, if most of the attributes match, these entities are considered aligned. For example, for two movie entities, compare their attributes such as name, director, leading actors, release time, etc. If these attributes are basically the same, then these two entities can be aligned.

[0105] For example, an application of aligning text entities with image entities: In a multimedia news database, there are both news texts and related pictures. For the entity "姚明" mentioned in the text, there is also a person in the picture. First, convert the text "姚明" into a vector through a word embedding model, and at the same time extract the features of the person in the picture (such as using a convolutional neural network) to obtain a vector. Calculate the similarity between these two vectors. If the similarity exceeds the set threshold, it is considered that "姚明" in the text and the person in the picture are the same entity. Additionally, attribute matching can also be used. For example, the attributes such as Yao Ming's height and occupation mentioned in the text are used to identify information such as the approximate height and clothing of the person in the picture (judging whether it is basketball player clothing) through image recognition technology and match it with the text attributes to complete the alignment.

[0106] The methods of vector similarity calculation and attribute matching can be applied to various types of entities, regardless of the modality in which the entities are located. Entities in text, image, or other modalities can all be processed. Vector similarity calculation can capture the semantic information of entities. Even if the expression forms of entities are different, it can determine whether they are the same entity through semantic similarity, improving the accuracy of alignment. The vector representation and attribute extraction methods can be continuously optimized based on AI technology to improve the alignment effect. For example, more powerful pre-trained models can be used to generate more accurate entity vectors. At the same time, entity-relationship synchronous alignment can also be achieved by combining AI. When aligning entities, the relationships between entities are also considered, further improving the accuracy and comprehensiveness of alignment.

[0107] (2) Methods of the relationship alignment module: Evidence fusion, confidence weighting;

[0108] Function: Align relationships in different modalities (such as text relationships and video relationships).

[0109] Evidence fusion: In different modalities, multiple pieces of evidence may exist to represent relationships between entities. These pieces of evidence from different sources are fused to form a comprehensive evidentiary representation. For example, a text might describe a "friends" relationship between two people, while a video might show these two people engaging in intimate interactions (such as hugging or talking). This is also evidence of a "friends" relationship. Fusing the evidence from the text and the video allows us to determine whether the relationship between the two people is consistent.

[0110] Confidence weighting: This involves assigning different confidence levels to evidence from different sources. Confidence levels can be determined based on factors such as the reliability of the evidence and the credibility of the source. For example, relationships described in texts published by authoritative media may have a higher confidence level, while relationship descriptions in user comments may have a lower confidence level. When aligning relationships, different pieces of evidence are weighted according to their confidence levels to obtain the final relationship judgment.

[0111] An application of aligning textual and video relationships can be illustrated as follows: On a film review website, a textual review mentions that two characters in a movie are "antagonistic," while a video clip shows a fight scene between these two characters. First, semantic analysis is performed on the text to extract the "antagonistic" relationship. Then, behavioral recognition and scene analysis are performed on the video clip to determine if antagonistic behavior exists. These two pieces of evidence are then fused, and their confidence levels are assigned based on factors such as the source of the text review (e.g., a well-known film critic or an ordinary user) and the clarity and completeness of the video. Finally, a weighted confidence level is used to comprehensively determine whether the relationship between the two characters in the text and video is consistent.

[0112] Therefore, by employing evidence fusion and confidence weighting, multiple pieces of evidence from different modalities are comprehensively considered, avoiding the limitations of single evidence and improving the accuracy of relation alignment. In practical applications, relation representations in different modalities can be complex and diverse. Evidence fusion and confidence weighting methods can handle this complexity and better accomplish relation alignment tasks. The confidence weighting approach makes the relation alignment results somewhat interpretable, clearly showing the contribution of each piece of evidence to the final result. Similarly, AI-based methods can further optimize evidence fusion and confidence allocation, simultaneously achieving entity and relation alignment. By considering entity alignment while aligning relations, the entire alignment process becomes more comprehensive and accurate.

[0113] 6. Knowledge Graph Construction / Mining Tool: Based on aligned entities and relationships, constructs a knowledge graph and mines deep connections. It mainly includes the following functions:

[0114] (1) Entity fusion module, used for disambiguation (distinguishing entities with the same name) and merging (integrating different modal information of the same entity) of aligned entities.

[0115] The entity fusion module is designed to handle the disambiguation and merging of aligned entities. In the data, entities with the same name may represent different actual objects. This necessitates accurately distinguishing these entities by analyzing their contextual information, attribute features, and relationships—a process known as disambiguation. Furthermore, for the same entity, information may exist in different modalities (such as text, images, and audio). This module integrates this information across different modalities to achieve comprehensive and unified entity information. Specifically, it uses attribute vectors to represent entities and calculates the similarity between vectors to determine if entities are the same. Entities with high similarity are merged. For disambiguation, feature vectors of entities are constructed and then combined with machine learning classification algorithms for differentiation.

[0116] In a knowledge graph construction scenario, consider two entity records: "Apple," one described as "fruit, rich in Vitamin C," and the other as "technology company, which launched products such as the iPhone." The entity fusion module extracts the feature vectors of these two "apples," analyzes their features to identify them as belonging to different categories (fruit and technology company), and thus distinguishes them, achieving disambiguation. Furthermore, for the same person entity, information such as name and occupation is recorded in the text, while its physical features are extracted from the image. The entity fusion module integrates this information from different modalities under the same person entity, achieving merging.

[0117] Therefore, disambiguation avoids data errors caused by confusion of entities with the same name, making the entity information in the knowledge graph more accurate. Merging information from different modalities makes the entity descriptions more comprehensive, providing richer information for subsequent knowledge graph applications. After accurately distinguishing and integrating entities, the efficiency and accuracy of subsequent operations such as querying and reasoning based on the knowledge graph will be improved.

[0118] (2) Relationship fusion module, used for completing (supplementing missing relationships) and verifying (validating relationship validity) the aligned relationships.

[0119] The relationship fusion module is primarily responsible for completing and validating the aligned relationships. During knowledge graph construction, due to data incompleteness, some relationships may be missing. This module analyzes existing relationships between entities, entity attributes, and contextual information, using machine learning models or rule-based reasoning methods to fill in these missing relationships. Simultaneously, for existing relationships, their validity needs to be verified by checking for logical consistency and conflicts with other known information to ensure their correctness.

[0120] For example, in a knowledge graph of interpersonal relationships, it is known that "Zhang San" and "Li Si" are colleagues, and "Li Si" and "Wang Wu" are colleagues, but the relationship between "Zhang San" and "Wang Wu" is missing. The relationship fusion module, based on the existing colleague relationship information, infers that "Zhang San" and "Wang Wu" may also be colleagues, thus completing the relationship. For the existing relationship "Zhang San is Li Si's father," the module checks the age information of Zhang San and Li Si. If it finds that Li Si is older than Zhang San, it determines that the relationship is invalid, completing the relationship verification.

[0121] Therefore, completing missing relationships makes the knowledge graph more complete, reflecting the connections between entities more comprehensively and providing a richer information foundation for its applications. Verifying the validity of relationships avoids interference from erroneous relationships, improving the reliability and accuracy of the knowledge graph. Complete and accurate relationship information facilitates more complex knowledge reasoning and uncovers more potential knowledge.

[0122] (3) Attribute fusion module, used to resolve the attribute conflicts of entities and relations input by the entity fusion module and the relation fusion module respectively, and to fuse the differences in description of the same attribute by different modalities;

[0123] The primary task of the attribute fusion module is to resolve attribute conflicts between entities and relations input to the entity fusion and relation fusion modules. Data from different modalities may have different descriptions of the same attribute. This module analyzes and processes these descriptions, fusing them into a unified attribute value by defining attribute fusion rules and using machine learning algorithms. For example, for the entity's "age" attribute, it might be recorded as "30 years old" in one text source and as "approximately 30 years old" in another. The module will fuse these two descriptions to determine a reasonable attribute value.

[0124] For example, in a product knowledge graph, the "screen size" attribute of a certain mobile phone might be recorded as "6.5 inches" in the product manual and described as "approximately 6.4-6.6 inches" in user reviews. The attribute fusion module will, based on the fusion rules, consider the manual to be more accurate and determine the "screen size" attribute value as "6.5 inches".

[0125] Therefore, it eliminates discrepancies in the description of the same attribute across different modalities, making the attribute information in the knowledge graph more unified and standardized, facilitating subsequent queries and analysis. It resolves attribute conflicts, ensuring the consistency of entity and relation attributes in the knowledge graph and avoiding erroneous reasoning and decisions caused by attribute inconsistencies. The unified and standardized attribute information makes the knowledge graph easier to understand and use, enhancing its value in practical applications.

[0126] (4) Graph construction module, used to construct the basic knowledge graph based on the entities, relations and attributes fused by the attribute fusion module;

[0127] The knowledge graph construction module builds a basic knowledge graph based on the entities, relationships, and attributes fused from the attribute fusion module. This module organizes and stores the fused entities as nodes, relationships as edges, and attributes as additional information for nodes and edges, following a graph data structure. During construction, a graph database is used to store the knowledge graph, and the data is indexed and optimized to improve the efficiency of subsequent queries and reasoning.

[0128] Suppose the attribute fusion module outputs a set of entities (such as people, companies, products, etc.), their relationships (such as employment relationships, production relationships, etc.), and corresponding attributes (such as a person's name, a company's address, etc.). The graph construction module represents these entities as nodes in the graph, relationships as edges connecting the nodes, and attributes are added to the nodes and edges. For example, using "Zhang San" and "ABC Company" as nodes, "Zhang San works at ABC Company" as an edge, attributes such as "Zhang San's" name and age are added to the "Zhang San" node, and attributes such as "ABC Company's" address and business scope are added to the "ABC Company" node, ultimately constructing a complete knowledge graph.

[0129] By integrating entities, relationships, and attributes into a unified knowledge graph, a complete knowledge system is formed, thus comprehensively reflecting the knowledge and information within the domain. The knowledge graph is stored using a graph data structure, leveraging the efficient query capabilities of graph databases to make management and retrieval of the knowledge graph more convenient and efficient. The constructed knowledge graph provides a foundation for subsequent applications such as knowledge reasoning, intelligent question answering, and decision support, enabling the extraction of more valuable information.

[0130] (5) Deep association mining engine, used to mine deep associations between entities in the graph based on GNN (Graph Neural Network), TransE (Embedding Algorithm) and / or rule reasoning model.

[0131] The deep association mining engine leverages GNNs (Graph Neural Networks), TransE (embedding algorithms), and / or rule-based reasoning models to uncover deep relationships between entities in a knowledge graph. GNNs automatically learn the features of nodes and edges in the graph structure, propagating information through message passing to discover potential relationships between entities. The TransE algorithm embeds entities and relationships into a low-dimensional vector space, calculating the distance between vectors to determine the strength of relationships and uncover hidden connections. Rule-based reasoning models, based on predefined rules, infer information in the knowledge graph to discover relationships that conform to the rules.

[0132] Built upon a social network knowledge graph, a Generative Neural Network (GNN) model can learn the social relationship features between user nodes, discovering potential social circles and indirect connections between users. For example, by analyzing users' friend relationships and interaction records, it can be discovered that two seemingly unrelated users may have a potential social relationship through mutual friends. The TransE algorithm can embed users and social behaviors (such as likes and comments) into a vector space, and by calculating vector distance, it can discover users' preferences for different types of content. Rule-based reasoning models can infer new friend relationships based on the rule that "a friend's friend may also be a friend."

[0133] Therefore, it is possible to discover hidden, deep relationships within knowledge graphs that are not easily observed directly, bringing more valuable information to the knowledge graph. Through machine learning and rule-based reasoning, knowledge graphs can acquire stronger intelligent analytical capabilities, enabling more complex knowledge reasoning and decision support. Models such as GNNs can effectively handle the complex graph structure of knowledge graphs, fully utilizing the node and edge information to uncover more accurate relationships.

[0134] 7. Multimodal Retrieval Service Layer: Based on the constructed knowledge graph, it provides multi-dimensional retrieval and question-answering services. It mainly provides the following retrieval service functions:

[0135] (1) Atlas Database

[0136] Data storage: Graph databases, such as Neo4j and JanusGraph, are used as the underlying storage structure. These graph databases store data in the form of nodes and edges, where nodes represent entities and edges represent relationships between entities. Simultaneously, vector databases (such as Faiss) are used to store vector representations of entities and relationships, facilitating vector indexing.

[0137] Vector indexing: Deep learning models (such as TransE, RotatE, etc.) are used to map entities and relations in the graph to a low-dimensional vector space. Then, an approximate nearest neighbor search (ANN) algorithm (such as HNSW, KD-Tree, etc.) is used to construct the vector index, which accelerates semantic retrieval.

[0138] Query Interface: Provides interfaces for SQL-like or graph query languages ​​(such as Cypher and Gremlin) to facilitate knowledge retrieval for users.

[0139] When a knowledge retrieval request is received, the query statement is first converted into a vector representation. Then, the vector index is used to quickly locate the entity and relation vectors most similar to the query vector in the vector database. Next, based on the entities and relations corresponding to these vectors, relevant detailed information is retrieved from the graph database. Finally, the query results are returned to the user.

[0140] Suppose there is a medical knowledge graph that stores entities such as diseases, symptoms, and drugs, along with their relationships. When a user queries "What medications are available to treat a cold?", the system converts the query into a vector. It then uses vector indexes to find vectors related to "cold" and "medications," and subsequently retrieves specific drug information from the graph database, such as "ibuprofen" or "acetaminophen."

[0141] Therefore, vector indexes enable rapid location of semantically relevant knowledge, significantly improving retrieval efficiency. Graph databases effectively represent complex relationships between entities, facilitating knowledge storage and management. New entities and relationships can be easily added to adapt to ever-growing amounts of knowledge data.

[0142] (2) Graph traversal query module

[0143] Depth-first search (DFS) and breadth-first search (BFS) are two classic graph traversal algorithms that form the basis for graph traversal queries. In practical applications, the appropriate traversal algorithm can be selected based on specific requirements.

[0144] The path planning algorithms used for traversal, such as Dijkstra's algorithm and A* algorithm, are used to find the shortest or optimal path in the graph.

[0145] Graph query languages: Use graph query languages ​​such as Cypher and Gremlin to write traversal query statements.

[0146] When a query request based on a graph structure is received, the starting node and target node are determined according to the query conditions. Then, using the selected graph traversal algorithm, starting from the starting node, the nodes in the graph are traversed along the edges until a node or path that satisfies the query conditions is found. Finally, the traversal result is returned to the user.

[0147] For example, in a social network knowledge graph, the friend relationships between users are stored. When a user queries "Who are user A's third-degree friends?", the system starts from the user A node, uses a breadth-first search algorithm, and traverses user A's friend nodes layer by layer until it finds the third-degree friend nodes, and then returns the information of these nodes to the user.

[0148] Graph traversal can delve into the relationships between entities in a graph, uncovering hidden connections. The traversal algorithm and query conditions can be flexibly adjusted to suit different query needs. The graph query language is simple and intuitive, allowing users to easily write complex queries.

[0149] (3) Semantic Vector Search Module

[0150] Employing pre-trained language models: Using pre-trained language models such as BERT and GPT to convert textual descriptions of entities and relationships into semantic vectors.

[0151] Alternatively, similarity calculation can be performed: methods such as cosine similarity and Euclidean distance can be used to calculate the similarity between the query vector and the entity and relation vectors in the graph.

[0152] Alternatively, vector retrieval engines such as Elasticsearch and Milvus can be used to store and retrieve semantic vectors.

[0153] First, the query statement and the entity and relation text in the graph are converted into vector representations using a pre-trained language model. Then, the similarity between the query vector and all vectors in the graph is calculated. Finally, based on the similarity scores, the entities and relations with the highest similarity are selected as the search results and returned to the user.

[0154] The technological advantage lies in its ability to perform matching based on semantic vectors, ignoring surface-level differences in text and improving retrieval accuracy. Pre-trained language models can process text in multiple languages, enabling cross-lingual semantic search. They can understand the contextual information of query statements, more accurately locating relevant knowledge.

[0155] (4) Hybrid conditional search module

[0156] Hybrid retrieval methods first employ different models to extract features from multimodal data such as text, images, and audio. For example, convolutional neural networks like ResNet can be used to extract image features, while models like WaveNet can be used to extract audio features. The extracted multimodal features are then fused, commonly using methods such as concatenation and weighted summation. Based on the fused features, a multimodal index structure, such as a multimodal hash table or a multimodal tree, is constructed to facilitate the retrieval of multimodal data.

[0157] When a multimodal association information retrieval request is received, the system first extracts and fuses features from the multimodal data in the query. Then, it uses a multimodal index to search the multimodal database for multimodal data most similar to the query features. Finally, based on the entities and relationships corresponding to these data, it queries the knowledge graph for relevant detailed information and returns the results to the user. For example, a movie knowledge graph stores multimodal information such as movie posters (images), plot summaries (text), and theme songs (audio). When a user queries "What are some suspense movies with a dog?", the system converts the query text into a feature vector. It can also combine the features of movie poster images or theme song audio that the user might provide to find relevant movies through a multimodal index, such as "Hachiko: A Dog's Tale".

[0158] Therefore, hybrid retrieval combines multimodal data for searching, providing more comprehensive and richer knowledge information. The fusion of multimodal features can improve retrieval accuracy and reduce false positives. This meets users' query needs across different modalities and enhances the user experience.

[0159] (5) Intelligent Question Answering Service Module

[0160] Large-scale language models (LLMs), such as GPT-3 and ChatGLM, are employed to understand user questions and generate natural language answers. A graph database, graph traversal query module, semantic vector search module, and hybrid conditional retrieval module are integrated into the intelligent question-answering system to achieve automatic knowledge retrieval. Dialogue management technologies, such as state machines and rule-based methods, are used to manage the context and flow of the dialogue.

[0161] When a user asks a question, the LLM first understands and analyzes it. Then, based on the question type and requirements, it automatically calls the graph database, graph traversal query module, semantic vector search module, and / or mixed conditional retrieval module to perform knowledge retrieval. Next, the retrieval results are fed back to the LLM, which generates a natural language answer based on these results and outputs the answer to the user. For example, when a user asks "What planets are in the solar system?", the LLM understands the question, calls the graph database to perform knowledge retrieval, and obtains information on planets such as "Mercury," "Venus," "Earth," "Mars," "Jupiter," "Saturn," "Uranus," and "Neptune." Then, the LLM integrates this information into a natural language answer: "There are eight planets in the solar system: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune," and outputs it to the user.

[0162] Therefore, users can ask questions using natural language, and the system can understand and provide natural language answers, improving the user's interactive experience. It can automatically select the appropriate search module based on the question, improving the efficiency and accuracy of knowledge retrieval. Through dialogue management technology, it can understand the context of the dialogue and provide more coherent and accurate answers.

[0163] On the other hand, a knowledge graph construction system for intelligently fusing multimodal data is provided. This system is used to implement the aforementioned knowledge graph construction method for intelligently fusing multimodal data. The system includes:

[0164] S1. Input multimodal data into the modality extractor;

[0165] S2. The modality-specific extractor splits the multimodal data, and based on the deployed multimodal feature processing system, splits the data features of each modality and transmits them to the multimodal aligner;

[0166] S3, the multimodal aligner performs unified semantic representation on the features of each split modal data, aligns the entities and relationships of each modal data feature, and sends them to the graph builder / miner;

[0167] S4. The graph builder / miner constructs a corresponding basic knowledge graph based on the entities and their relationships among the aligned modal data features, and mines the deep association relationships among the entities in the basic knowledge graph through the configured deep association mining engine.

[0168] S5. The multimodal retrieval service layer provides knowledge retrieval and question-answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

[0169] Please implement and understand this method in conjunction with the system functions and application principles described above; it will not be elaborated upon here.

[0170] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 2 As shown, optionally, electronic device 410 may include a first processor 2001.

[0171] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0172] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0173] The following is combined Figure 2 A detailed description of each component of electronic device 410 is provided below:

[0174] The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0175] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0176] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 are shown in the diagram.

[0177] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 2 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0178] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0179] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected via the interface circuit of the electronic device 410. Figure 2 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0180] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0181] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 2 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0182] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 410. Figure 2 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0183] It should be noted that, Figure 2 The structure of the electronic device 410 shown does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0184] Furthermore, the technical effects of the electronic device 410 can be referenced from the technical effects of the knowledge graph construction method for intelligent fusion of multimodal data described in the above method embodiments, and will not be repeated here.

[0185] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0186] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0187] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0188] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0189] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0190] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0191] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0192] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, systems, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0193] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0194] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0196] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A knowledge graph construction system that intelligently integrates multimodal data, characterized in that, The system includes: The data input layer is used to input multimodal data into the modality-specific extractor; A modality-specific extractor is used to split the multimodal data, based on the deployed multimodal feature processing system to split the data features of each modality and transmit them to the multimodal aligner; The multimodal aligner is used to perform unified semantic representation on the split modal data features, align the entities and relationships of the modal data features, and send them to the graph builder / miner; The graph builder / miner is used to build a corresponding basic knowledge graph based on the entities and the relationships between them after alignment of the modal data features, and to mine the deep association relationships between the entities in the basic knowledge graph through a configured deep association mining engine. The multimodal retrieval service layer is used to provide knowledge retrieval and question answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

2. The knowledge graph construction system for intelligent fusion of multimodal data according to claim 1, characterized in that, The multimodal feature processing system of the modality-specific extractor includes: A text processor for identifying entities and extracting relationships from text based on nBERT / RoBERTa, including: locating entities in text and mining semantic relationships between entities; Image processor for detecting targets in images based on YOLO / CLIP, including: identifying objects in images and analyzing spatial / semantic relationships between objects; A video processor for extracting keyframes from videos and analyzing motion behavior within those keyframes using 3D-CNN / LSTM. The structure processor is used to map table structures to graph nodes / relationships through the rule engine.

3. The knowledge graph construction system for intelligent fusion of multimodal data according to claim 1, characterized in that, The multimodal aligner achieves cross-modal semantic alignment through unified semantic representation learning (multimodal Transformer), aligning entities and relations in multimodal data and eliminating intermodal differences.

4. The knowledge graph construction system for intelligent fusion of multimodal data according to claim 3, characterized in that, The multimodal aligner includes: The entity alignment module is used to align entities of different modalities through vector similarity calculation and attribute matching. The Relationship Alignment Module is used to align relationships of different modalities through evidence fusion and confidence weighting.

5. The knowledge graph construction system for intelligent fusion of multimodal data according to claim 1, characterized in that, The map builder / miner includes: The entity fusion module is used for disambiguation (distinguishing entities with the same name) and merging (integrating different modal information of the same entity) of aligned entities. The relationship fusion module is used to complete (supplement missing relationships) and verify (validate relationship validity) the aligned relationships. The attribute fusion module is used to resolve attribute conflicts between entities and relations input by the entity fusion module and the relation fusion module respectively, and to fuse the differences in descriptions of the same attribute by different modalities. The graph construction module is used to construct the basic knowledge graph based on the entities, relationships, and attributes fused and output by the attribute fusion module. A deep association mining engine for mining deep associations between entities in a graph based on GNN (Graph Neural Network), TransE (Embedding Algorithm), and / or rule-based reasoning models.

6. The knowledge graph construction system for intelligent fusion of multimodal data according to claim 1, characterized in that, The multimodal retrieval service layer includes: A graph database is used to store the basic knowledge graph and provide knowledge retrieval services based on vector indexes (to accelerate semantic retrieval); The graph traversal query module is used to query related information by traversing nodes / relationships based on the graph structure of the basic knowledge graph. The semantic vector search module is used to retrieve semantically related entities / relationships from the basic knowledge graph through semantic vector matching; The hybrid condition retrieval module is used to retrieve multimodal association information from the basic knowledge graph by combining with the multimodal feature processing system; The intelligent question-answering service module is used to understand user questions based on LLM and automatically input the above-mentioned graph database, graph traversal query module, semantic vector search module and / or mixed condition retrieval module for knowledge retrieval. Based on the retrieval results, it generates corresponding intelligent knowledge-based answer dialogues and outputs them.

7. A knowledge graph construction system for intelligent fusion of multimodal data, wherein the knowledge graph construction system for intelligent fusion of multimodal data is used to implement the knowledge graph construction method for intelligent fusion of multimodal data as described in any one of claims 1-6, characterized in that, The system includes: S1. Input multimodal data into the modality extractor; S2. The modality-specific extractor splits the multimodal data, and based on the deployed multimodal feature processing system, splits the data features of each modality and transmits them to the multimodal aligner; S3, the multimodal aligner performs unified semantic representation on the features of each split modal data, aligns the entities and relationships of each modal data feature, and sends them to the graph builder / miner; S4. The graph builder / miner constructs a corresponding basic knowledge graph based on the entities and their relationships among the aligned modal data features, and mines the deep association relationships among the entities in the basic knowledge graph through the configured deep association mining engine. S5. The multimodal retrieval service layer provides knowledge retrieval and question-answering services in various dimensions based on the basic knowledge graph and the deep association mining engine.

8. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 1.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images