Intelligent visualization and text association method for multi-modal knowledge graph

Abstract

The invention discloses an intelligent visualization and text association method for a multi-modal knowledge graph, and relates to the technical field of information processing. According to the intelligent visualization and text association method for the multi-modal knowledge graph, multi-modal data in an internal database of an enterprise is obtained, and a multi-modal feature vector is generated by using a data preprocessing and feature extraction technology; the feature vectors are mapped to a unified vector space through a cross-modal embedding algorithm, semantic alignment, association and fusion among multi-modal data are realized, and a complete multi-modal knowledge graph is constructed. Through bidirectional association and multi-modal joint query, the information acquisition efficiency is improved, and visual display of the knowledge graph is realized by using the graph neural network. Meanwhile, the knowledge graph is dynamically updated through a graph reasoning algorithm, and the timeliness and accuracy of the knowledge graph are ensured. A more efficient solution is provided for an enterprise in the aspects of management and application of multi-modal data, and the intelligent process of knowledge management is promoted.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and specifically to an intelligent visualization and text association method of a multimodal knowledge graph. Background Art

[0002] With the rapid development of information technology, enterprises have accumulated a large amount of multimodal data in their daily operations, including text, images, videos and other forms of data. These data not only contain rich knowledge information, but also play a vital role in the decision-making and management of enterprises. However, traditional data processing methods often fail to fully explore and utilize the potential value of these multimodal data. In order to solve this problem, it is particularly important to build an intelligent multimodal knowledge graph. By integrating data of different modalities, enterprises can achieve efficient information management, intelligent decision support and visual display of knowledge, thereby improving operational efficiency and market competitiveness.

[0003] Traditional knowledge graph construction methods mainly rely on single-modal data, usually based only on structured or semi-structured text information. Data of different modalities are often stored in separate systems, resulting in the inability to effectively share information, affecting the comprehensive analysis and utilization of data. In the process of processing multimodal data, traditional methods are usually unable to effectively align and fuse data features of different modalities, resulting in information loss and redundancy. Traditional query methods can often only process single-modal data, making it difficult to achieve linkage queries between multimodal data, affecting the comprehensiveness and accuracy of the query. Existing methods are often unable to update the knowledge graph in a timely manner when faced with new data, resulting in obsolescence of the knowledge base and lagging information. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an intelligent visualization and text association method for a multimodal knowledge graph, which solves the problems of the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for intelligent visualization and text association of a multimodal knowledge graph, comprising the following steps: S1. Obtain multimodal data in the internal database of the target enterprise and perform data preprocessing and feature extraction to generate a multimodal feature vector; S2. Map the multimodal feature vector to a unified vector space through a cross-modal embedding algorithm, align, associate and fuse the semantics between the multimodal data, and construct a multimodal knowledge graph; S3. Perform bidirectional association and multimodal joint query on the multimodal knowledge graph and the input text to obtain the corresponding multimodal data, and visualize the multimodal knowledge graph and the queried multimodal data through a graph neural network; S4. According to the newly added multimodal data, analyze the association relationship through a graph reasoning algorithm, and dynamically update the multimodal knowledge graph.

[0006] Furthermore, the multimodal data in the target enterprise database includes technical documents, product design drawings, and test videos; the multimodal feature vector includes a semantic feature vector, a visual feature vector, and a video dynamic feature vector.

[0007] Furthermore, the specific process of multimodal data preprocessing and feature extraction is as follows: the semantic information of the text is captured through a preset natural language processing model, and the technical documents are converted into semantic feature vectors; the key visual elements in the image are captured using a convolutional neural network, and the visual feature vectors of the product design drawings are extracted; the temporal dynamic changes in the video are captured through a 3D convolutional neural network, and the video dynamic feature vectors of the test video are extracted.

[0008] Furthermore, the mapping logic of mapping the multimodal feature vector to a unified vector space through a cross-modal embedding algorithm is as follows: the embedding layer of the semantic feature vector, the visual feature vector and the video dynamic feature vector is mapped through the cross-modal embedding algorithm to generate a mapped embedding vector, including a semantic embedding vector, a visual embedding vector and a video dynamic embedding vector; the embedding vectors of different modalities are feature aligned through a contrast loss function, and the aligned semantic embedding vector, visual embedding vector and video dynamic embedding vector are fused.

[0009] Furthermore, the specific process of constructing a multimodal knowledge graph is as follows: by aligning and fusing semantic embedding vectors, visual embedding vectors and video dynamic embedding vectors, a multimodal node embedding representation is generated; the node embeddings are feature propagated through a graph convolutional network, the associated edges between nodes are constructed, and the global structure of the multimodal knowledge graph is constructed based on cross-modal semantic associations; the weights of nodes and edges are optimized through a contrast loss function to generate a multimodal knowledge graph.

[0010] Furthermore, the multimodal knowledge graph is bidirectionally associated with the input text and multimodal joint query is performed to obtain the corresponding multimodal data. The specific process is as follows: the input text is semantically vectorized through the text embedding model to generate a text embedding vector; based on the node representation of the multimodal knowledge graph, the multimodal semantic matching index between the text embedding vector and the multimodal node embedding representation is calculated through the semantic matching algorithm; based on the multimodal semantic matching index, bidirectional semantic association is performed to screen multimodal data nodes that are highly relevant to the input text; through the multimodal joint query mechanism, the multimodal data set associated with the text is obtained, and the corresponding semantic matching result is returned.

[0011] Furthermore, the specific process of obtaining the multimodal semantic matching index is as follows: the semantic similarities between the text embedding vector and the semantic embedding vector, the visual embedding vector and the video dynamic embedding vector are calculated respectively through the semantic matching algorithm; the similarities of different modalities are weighted and comprehensively analyzed to obtain the multimodal semantic matching index.

[0012] Furthermore, based on the newly added multimodal data, the specific process of analyzing the association relationship through the graph reasoning algorithm is as follows: the newly added multimodal data is expanded through the graph reasoning algorithm to identify the existing nodes associated with the newly added data; the nodes and edges in the multimodal knowledge graph are aggregated through the graph convolutional network to update the node representation; the reasoning rules and relationship models are applied to derive potential implicit relationships, enhance the structure and semantic information of the knowledge graph, and dynamically update the multimodal knowledge graph.

[0013] The present invention has the following beneficial effects: (1) A method for intelligent visualization and text association of multimodal knowledge graphs. Through unified preprocessing and feature extraction of multimodal data, it ensures that data from different sources and formats can be represented in the same feature space, thereby improving the efficiency and accuracy of subsequent processing. It avoids the format incompatibility problem that may be caused by directly using the original data, and feature extraction can retain the core information of the data. It uses a cross-modal embedding algorithm to map feature vectors of different modalities to the same vector space, solves the problem of semantic inconsistency of multimodal data, realizes semantic alignment and data fusion, and constructs a unified multimodal knowledge graph.

[0014] (2) A method for intelligent visualization and text association of multimodal knowledge graphs. Through bidirectional association and multimodal joint query mechanisms, the input text and the data in the multimodal knowledge graph can be semantically matched and associated; a graph neural network is used to visualize the knowledge graph. This ensures that the input text and the data in the knowledge graph can be efficiently matched, significantly improving the accuracy of the query. At the same time, through visualization by the graph neural network, users can intuitively understand the complex relationship between multimodal data, improving their insight into the data and interactive experience. The graph reasoning algorithm is used to analyze the association relationship of the newly added data, dynamically update the existing multimodal knowledge graph, and maintain the timeliness and accuracy of the knowledge graph. The knowledge graph can dynamically adapt to the addition of new data, avoiding the limitations of traditional static knowledge graphs in data updating, ensuring the system's real-time reflection of multimodal data, and improving the flexibility and scalability of the system.

[0015] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for intelligent visualization and text association of a multimodal knowledge graph according to the present invention. DETAILED DESCRIPTION

[0017] The embodiments of the present application solve the problems of information islands, insufficient feature fusion, low query efficiency and insufficient dynamic update capability in the traditional multimodal data processing process through an intelligent visualization and text association method of a multimodal knowledge graph.

[0018] The overall idea of ​​the problem in the embodiment of this application is as follows: Acquire multimodal data from the target enterprise's internal database and perform data preprocessing and feature extraction to generate multimodal feature vectors.

[0019] Through the cross-modal embedding algorithm, the multimodal feature vectors are mapped to a unified vector space, the semantics between multimodal data are aligned, associated and fused, and a multimodal knowledge graph is constructed.

[0020] The multimodal knowledge graph is bidirectionally associated with the input text and multimodal joint query is performed to obtain the corresponding multimodal data, and the multimodal knowledge graph and query multimodal data are visualized through the graph neural network.

[0021] Based on the newly added multimodal data, the association relationship is analyzed through the graph reasoning algorithm, and the multimodal knowledge graph is dynamically updated.

[0022] Example 1: Please refer to Figure 1 , an embodiment of the present invention provides a technical solution: a method for intelligent visualization and text association of a multimodal knowledge graph, comprising the following steps: S1. Acquire multimodal data in the internal database of a target enterprise and perform data preprocessing and feature extraction to generate a multimodal feature vector; S2. Map the multimodal feature vector to a unified vector space through a cross-modal embedding algorithm, align, associate and fuse the semantics between the multimodal data, and construct a multimodal knowledge graph; S3. Perform bidirectional association and multimodal joint query on the multimodal knowledge graph and the input text to obtain the corresponding multimodal data, and visualize the multimodal knowledge graph and the queried multimodal data through a graph neural network; S4. According to the newly added multimodal data, analyze the association relationship through a graph reasoning algorithm, and dynamically update the multimodal knowledge graph.

[0023] In this implementation scheme, S1. obtains various types of data from the internal database of the enterprise, which are data of different modalities such as text, images, and videos. Then preprocess these data, extract the data features of each modality, and generate corresponding multimodal feature vectors. These feature vectors will be used to represent the processing basis of each modal data in subsequent steps. Multimodal data: refers to different types of data sources, such as text, images, videos, audio, etc., and data of different modalities usually have different features and structures. Feature extraction: extract core information from the original data through a specific algorithm, such as word vectors of text, feature points of images, etc., to form a feature representation that is easy for machine processing. S2. Use a cross-modal embedding algorithm to uniformly map the feature vectors of different modalities to a common vector space. The key to this step is to ensure that data of different modalities can be aligned and compared under the same semantic dimension. Through this alignment and fusion, a multimodal knowledge graph representing the semantic association between multimodal data can be constructed. Cross-modal embedding algorithm: an algorithm for mapping features of different modalities (such as text, images, and videos) to a unified space to ensure that data of different modalities can be semantically aligned and compared. Vector space: A mathematical representation used to map complex data features into vectors with uniform dimensions so that distance calculations and similarity comparisons can be performed in this space. Multimodal knowledge graph: A structured knowledge representation used to represent different modal data and their semantic associations. S3. The system receives the text input by the user and performs bidirectional association and multimodal joint query with the constructed multimodal knowledge graph to find multimodal data semantically related to the input text. Then, the graph neural network is used to intelligently visualize these knowledge graph data to display the multimodal data associated with the input text and the relationship between them. Bidirectional association: refers to the system's ability to find multimodal data related to the input text based on the input text, and can also reversely query related text from the knowledge graph. Multimodal joint query: Through cross-modal semantic associations, joint queries are performed between different types of data (such as text, images, videos, etc.) to obtain multimodal data related to the input text. Graph Neural Network (GNN): A neural network used to process graph structured data, which can learn the representation of nodes in the graph and the relationships between them, and is used for visualization and query of knowledge graphs. S4. When new multimodal data is added to the system, use graph reasoning algorithms to analyze the association between the new data and existing data, and dynamically update the existing multimodal knowledge graph to ensure that the knowledge graph always reflects the latest data and its relationship. Graph reasoning algorithm: an algorithm that performs reasoning calculations on graph structured data, used to analyze the association between nodes in the graph, especially when new data is added, re-evaluate its semantic association with existing data. Dynamic update: through automated algorithm analysis, the structure and content of the knowledge graph are updated in real time to maintain the timeliness of the knowledge graph for data relationships.

[0024] Specifically, the multimodal data in the target enterprise database includes technical documents, product design drawings, and test videos; the multimodal feature vectors include semantic feature vectors, visual feature vectors, and video dynamic feature vectors.

[0025] In this implementation scheme, technical documents: This is data in text mode, usually including technical specifications, patent documents, research and development reports, standard operating procedures (SOPs), etc., which provide technical details of enterprise products or projects. For technical documents, semantic feature vectors are mainly extracted, and text data is converted into vector form that can be calculated and compared through natural language processing (NLP) technology. Product design drawings: This is data in visual mode, usually showing the design of enterprise products in the form of images or drawings, such as CAD drawings, product schematics, structural diagrams, etc. These data are extracted through computer vision technology. Visual feature vectors, such as edge detection, shape analysis, color information, etc., are used to represent the core information in the design drawing. Test video: This is data in video mode, usually used to record product testing processes or performance verification, such as mechanical testing, stress testing, user interaction experiments, etc. The test video extracts video dynamic feature vectors through video processing technology, including dynamic features such as motion information, time series changes, scene changes, etc. in the video, which are used to describe the video content. Semantic feature vector: The feature vector extracted from text data such as technical documents is mainly generated through natural language processing technology, which can capture the core semantic information of the text, such as keywords, phrases and their contextual associations. These feature vectors are used to represent the semantic connotation of text content, facilitating cross-modal alignment with data from other modalities. Visual feature vectors: Feature vectors extracted from visual data such as product design drawings, generated using image processing or computer vision algorithms, can describe information such as structure, shape, texture, etc. in the image. These vectors are used to represent data features of the visual modality, facilitating comparison with modalities such as text and video. Video dynamic feature vectors: Feature vectors extracted from dynamic data such as test videos, using video processing algorithms to capture temporal changes, motion trajectories, and scene dynamics in the video. These features are used to represent dynamic information in video content and to perform cross-modal associations with other modalities (such as text descriptions or design drawings).

[0026] Specifically, the specific process of multimodal data preprocessing and feature extraction is as follows: the semantic information of the text is captured through a preset natural language processing model, and the technical documents are converted into semantic feature vectors; the key visual elements in the image are captured using a convolutional neural network, and the visual feature vectors of the product design drawings are extracted; the temporal dynamic changes in the video are captured through a 3D convolutional neural network, and the video dynamic feature vectors of the test video are extracted.

[0027] In this implementation scheme, the technical document is converted into a semantic feature vector: the technical document is processed by a preset natural language processing model (BERT, Word2Vec). The model analyzes the words, phrases and their contextual relationships in the text to extract the semantic information of the text. Natural language processing (NLP) technology is used to analyze and understand the meaning of text data, converting complex language information into feature vectors in numerical form so that computers can process and understand it. These semantic feature vectors reflect the core themes and semantic structures of the technical document, and are easy to align with data from other modalities. Extracting visual feature vectors of product design drawings: Using convolutional neural networks (CNNs) to extract features from product design drawings. CNNs can automatically learn and identify key visual elements in images, such as edges, shapes and textures, and generate visual feature vectors through multi-layer convolution and pooling operations. Convolutional neural networks (CNNs) are a deep learning model that specializes in processing image data, and effectively extracts image features through local connections and weight sharing. Its application in image recognition and processing enables the key information in product design drawings to be accurately captured, thereby generating visual feature vectors with representation capabilities. Extracting video dynamic feature vectors from test videos: The test videos are processed by 3D convolutional neural networks (3D-CNN) to capture the temporal dynamic changes in the videos. 3D-CNN expands the concept of traditional CNN, using three-dimensional convolution kernels to analyze data in the time and space dimensions and extract temporal dynamic feature vectors. 3D convolutional neural networks (3D-CNN) are designed specifically for processing video data and can effectively capture the dynamic relationship and changes between video frames. By considering the time dimension, it not only analyzes the features of a single frame, but also identifies the actions, events, and their temporal features that occur in the video, thereby generating a video feature vector containing dynamic information.

[0028] Specifically, the mapping logic of mapping the multimodal feature vector to a unified vector space through a cross-modal embedding algorithm is as follows: the embedding layer of the semantic feature vector, the visual feature vector and the video dynamic feature vector is mapped through the cross-modal embedding algorithm to generate a mapped embedding vector, including a semantic embedding vector, a visual embedding vector and a video dynamic embedding vector; the embedding vectors of different modalities are feature aligned through the contrast loss function, and the aligned semantic embedding vector, visual embedding vector and video dynamic embedding vector are fused.

[0029] In this implementation scheme, a cross-modal embedding algorithm is used to input the generated semantic feature vectors, visual feature vectors, and video dynamic feature vectors into the embedding layer. The task of this layer is to map feature vectors of different modalities to the same vector space, so as to achieve mutual comparison and association between features. The cross-modal embedding algorithm is a technology that maps features from different modalities (such as text, images, and videos) to the same space by learning. This process enables data of different modalities to be effectively compared and analyzed in the same vector space, thereby facilitating subsequent processing and fusion. Generate mapped embedding vectors: Through the cross-modal embedding algorithm, the generated embedding vectors include semantic embedding vectors, visual embedding vectors, and video dynamic embedding vectors. Each embedding vector represents the features of the corresponding modality and is expressed in a unified form. The embedding vectors are feature representations after mapping, which not only retain the data features of the original modality, but also form a unified expression form in the same space. This consistency makes it easier for features of different modalities to interact and fuse. Feature alignment: The embedding vectors of different modalities are feature aligned through the contrastive loss function. This loss function aims to maximize the distance between similar modal embedding vectors and minimize the distance between different modal embedding vectors to achieve better alignment. Contrastive loss function is a commonly used loss function used to train models to distinguish similar and dissimilar samples. In multimodal embedding, the use of this loss function can ensure that different modal features with the same semantics can be clustered together, thereby improving the accuracy and robustness of the model. Embedding vector fusion: The aligned semantic embedding vector, visual embedding vector and video dynamic embedding vector are fused to form a comprehensive feature representation. Fusion can be performed in a variety of ways, such as splicing, weighted summation, etc. The specific choice depends on the application requirements. Embedding vector fusion is to merge the features of different modalities into a unified representation for subsequent processing and analysis. Through fusion, the advantages of each modality can be fully utilized, the performance of the model can be enhanced, and it can better adapt to the complexity of multimodal data. The semantic feature vector, visual feature vector and video dynamic feature vector are mapped to a unified embedding layer through a cross-modal embedding algorithm. The generated mapped embedding vector is represented as: ; Parameter explanation: : The generated unified embedding vector includes semantic embedding, visual embedding and video dynamic embedding. : Input semantic feature vector. : Input visual feature vector. : Input video dynamic feature vector. : The cross-modal embedding algorithm is a neural network used to align and fuse features of different modalities. Contrastive loss function: In order to achieve feature alignment, the formula for feature alignment of embedding vectors of different modalities through contrastive loss function is as follows: ; Parameter explanation: : Contrastive loss, which measures the similarity between different modalities and the preset margin. : Total number of samples. similarity : Used to calculate the similarity between embedded vectors, usually cosine similarity or Euclidean distance. Margin: The threshold set, indicating the minimum similarity distance between different modalities. Embedded vector fusion: The aligned embedded vectors are fused, expressed as: ; Parameter explanation: : The fused embedding vector, : Semantic embedding vector after alignment, : The aligned visual embedding vector, : Dynamic embedding vector after alignment, : represents the semantic embedding vector The weight coefficient in the fusion process reflects the importance of semantic information in the construction of multimodal knowledge graphs. For example, if semantic information is crucial for the target task (such as text understanding or semantic query), The value may be larger. : represents the visual embedding vector The weight coefficient in the fusion process reflects the importance of visual features in the construction of multimodal knowledge graphs. For example, for tasks involving image content recognition, The value of may be larger, indicating that the visual information contributes more significantly to the model. : represents a dynamic embedding vector The weight coefficient in the fusion process reflects the importance of video dynamic features in the construction of multimodal knowledge graphs. For example, in tasks involving time series analysis or dynamic behavior recognition, The value of may be large, indicating the importance of video dynamic information.

[0030] Specifically, the specific process of constructing a multimodal knowledge graph is as follows: generate a multimodal node embedding representation by aligning and fusing semantic embedding vectors, visual embedding vectors and video dynamic embedding vectors; propagate features of node embeddings through a graph convolutional network, construct associated edges between nodes, and construct the global structure of the multimodal knowledge graph based on cross-modal semantic associations; optimize the weights of nodes and edges through a contrast loss function to generate a multimodal knowledge graph.

[0031] In this implementation scheme, a multimodal node embedding representation is generated: a multimodal node embedding representation is generated by aligning and fusing semantic embedding vectors, visual embedding vectors, and video dynamic embedding vectors. These node embedding representations are used to represent each node in the knowledge graph, covering feature information of multiple modalities. The multimodal node embedding representation is an integration of features of different modalities, and through alignment and fusion steps, a representation that can reflect the diversity and complexity of nodes is formed. This representation allows each node to contain not only information of a single modality, but also integrate knowledge of multiple modalities. Feature propagation and construction of associated edges: feature propagation of node embedding is performed using a graph convolutional network (GCN). GCN updates node embeddings through the features of nodes and their neighbors, enhancing information exchange between nodes. At the same time, associated edges between nodes are constructed based on cross-modal semantic associations to represent semantic connections between nodes. Graph convolutional networks are an effective deep learning algorithm that can perform feature learning and propagation in graph structured data. Through feature propagation, GCN can capture local structural information of nodes in the graph and improve the richness of the embedded representation of nodes. In addition, building associated edges helps to clarify the semantic relationship between nodes, thereby enhancing the expressiveness of the knowledge graph. Optimize the weights of nodes and edges: The weights of nodes and edges are optimized through the contrast loss function to ensure that similar nodes and edges in the knowledge graph are given higher weights, while dissimilar nodes and edges are given lower weights. This process helps to improve the accuracy and effectiveness of the multimodal knowledge graph. The contrast loss function is used to train the model to distinguish between similar and dissimilar samples. Through the optimization of this loss function, the weights of nodes and edges can be dynamically adjusted to make the connections in the knowledge graph more reasonable, thereby improving the performance of query and reasoning. Generate a multimodal knowledge graph: Based on the optimized node and edge weights, a complete multimodal knowledge graph is generated. The graph not only contains the multimodal features of the nodes, but also reflects the semantic relationship between them through the associated edges between the nodes. The generated multimodal knowledge graph can effectively integrate information from different modalities and provide a rich knowledge representation. This graph not only supports multimodal query and visualization, but also provides a basis for subsequent knowledge reasoning and decision support.

[0032] Specifically, the specific process of bidirectionally associating the multimodal knowledge graph with the input text and performing multimodal joint query to obtain the corresponding multimodal data is as follows: semantically vectorizing the input text through the text embedding model to generate a text embedding vector; based on the node representation of the multimodal knowledge graph, the multimodal semantic matching index between the text embedding vector and the multimodal node embedding representation is calculated through the semantic matching algorithm; bidirectional semantic association is performed based on the multimodal semantic matching index to screen multimodal data nodes that are highly relevant to the input text; through the multimodal joint query mechanism, the multimodal data set associated with the text is obtained, and the corresponding semantic matching result is returned.

[0033] In this implementation scheme, text embedding vectorization: the input text is processed through a preset text embedding model (BERT, Word2Vec) to convert the text into a semantic vector. This step can capture the semantic information and contextual relationship of the text and generate a text embedding vector. Convert the text into a form that can be compared with the node representation in the multimodal knowledge graph. Calculate the multimodal semantic matching index: use the semantic matching algorithm to calculate the similarity between the generated text embedding vector and the embedding representation (semantic embedding vector, visual embedding vector and video dynamic embedding vector) of each node in the multimodal knowledge graph. Evaluate the relevance of the input text to each node in the knowledge graph so that the most relevant nodes can be subsequently screened. Bidirectional semantic association: According to the calculated multimodal semantic matching index, the multimodal data nodes are screened. Multimodal data nodes that are highly relevant to the input text are screened out according to the threshold. Bidirectional association means that not only queries are performed from text to knowledge graph nodes, but also reverse matching can be performed from nodes to text. Ensure that nodes that are highly semantically relevant to the input text are screened out to enhance the accuracy and relevance of the query results. Multimodal joint query mechanism: Use the multimodal joint query mechanism to associate the selected relevant multimodal data nodes with the input text to obtain the corresponding multimodal dataset. This process may involve complex queries on the knowledge graph and joint access to multiple data formats. Obtain the multimodal dataset associated with the text and return the semantic matching results containing relevant content for subsequent analysis and application.

[0034] Specifically, the specific process of obtaining the multimodal semantic matching index is as follows: the semantic similarity between the text embedding vector and the semantic embedding vector, the visual embedding vector and the video dynamic embedding vector is calculated respectively through the semantic matching algorithm; the similarity of different modalities is weighted and comprehensively analyzed to obtain the multimodal semantic matching index.

[0035] In this implementation, cosine similarity is used to calculate the similarity between the text embedding vector and each modality embedding vector; the specific formula is: semantic similarity: ; Visual similarity; ; Video dynamic similarity: ; The similarity of each modality is weighted and integrated to obtain the multimodal semantic matching index: ;in, : Similarity between text and semantic modality. : The embedding vector of the semantic modality, which represents the semantic features related to the text, : text embedding vector, : Similarity between text and visual modalities. : Embedding vector of the visual modality, representing the visual features associated with the text. : Similarity between text and video dynamic modalities. : The embedding vector of the video dynamics modality, representing the video features related to the text. : Multimodal semantic matching index, which indicates the comprehensive relevance between the text and all modalities. , : Represents the weight coefficients of semantic, visual and video dynamic modes, respectively, controls the influence of each mode in the comprehensive analysis, and satisfies the condition .

[0036] Specifically, according to the newly added multimodal data, the specific process of analyzing the association relationship through the graph reasoning algorithm is as follows: the newly added multimodal data is expanded through the graph reasoning algorithm to identify the existing nodes associated with the newly added data; the nodes and edges in the multimodal knowledge graph are aggregated through the graph convolutional network to update the node representation; the reasoning rules and relationship models are applied to derive potential implicit relationships, enhance the structure and semantic information of the knowledge graph, and dynamically update the multimodal knowledge graph.

[0037] In this implementation scheme, node expansion: In a multimodal knowledge graph, each node represents a different data entity (such as technical documents, product design drawings, etc.). Through a graph reasoning algorithm, the process of node expansion of newly added multimodal data involves identifying the relationship between the newly added data and existing nodes. Associated existing node identification: Evaluate the similarity between the newly added node and the existing node through similarity calculation or an algorithm based on semantic matching. The similarity can be calculated based on semantic, visual or video dynamic features. After identifying the existing nodes associated with the newly added node, it provides a basis for subsequent feature updates and reasoning. Graph convolutional network (GCN): A graph convolutional network is used to aggregate features of nodes and edges in a multimodal knowledge graph. Graph convolutional network is a deep learning model that can effectively process graph structured data. In this process, GCN updates the representation of each node by aggregating the features of adjacent nodes. This feature aggregation can take into account the relationship between nodes and further enhance the expressiveness of node representation through a multi-layer convolutional structure. Update the representation of the node: Through feature aggregation, the representation of the node will be updated to reflect the newly added multimodal data and its relationship with the existing nodes. The updated node representation will contain richer structural and semantic information, thereby improving the accuracy and completeness of the knowledge graph. Apply inference rules and relationship models: After updating the node representation, apply specific inference rules and relationship models to identify potential implicit relationships. These inference rules may include logical reasoning, statistical reasoning, etc., which can derive new relationships from existing nodes and edges. For example, if a technical document has a relationship with a product design drawing, and the design drawing is associated with a test video, it can be deduced that there may be an implicit relationship between the technical document and the test video. Enhance the structure and semantic information of the knowledge graph: Through the derived implicit relationships, the structure of the knowledge graph will be enhanced, further enriching its semantic information. This not only improves the expressive power of the knowledge graph, but also provides users with deeper insights. Dynamically update the multimodal knowledge graph: Based on the updated node representations and implicit relationships obtained in the above steps, the multimodal knowledge graph is dynamically updated. This means that whenever new data enters the system, the knowledge graph can be adjusted and optimized in time to adapt to the latest data changes and maintain its accuracy and effectiveness.

[0038] In summary, this application has at least the following effects: A method for intelligent visualization and text association of multimodal knowledge graphs ensures that data from different sources and formats can be represented in the same feature space through unified preprocessing and feature extraction of multimodal data, thereby improving the efficiency and accuracy of subsequent processing. It avoids the format incompatibility problem that may be caused by directly using the original data, and feature extraction can retain the core information of the data. The cross-modal embedding algorithm is used to map the feature vectors of different modalities to the same vector space, solves the problem of semantic inconsistency of multimodal data, realizes semantic alignment and data fusion, and constructs a unified multimodal knowledge graph. Through bidirectional association and multimodal joint query mechanism, the input text and the data in the multimodal knowledge graph can be semantically matched and associated; the graph neural network is used for visualization of the knowledge graph. It ensures that the input text and the data in the knowledge graph can be efficiently matched, significantly improving the accuracy of the query. At the same time, through visualization by graph neural network, users can intuitively understand the complex relationship between multimodal data, and enhance their insight into the data and interactive experience. The graph reasoning algorithm is used to analyze the association relationship of the newly added data, dynamically update the existing multimodal knowledge graph, and maintain the timeliness and accuracy of the knowledge graph. It enables the knowledge graph to dynamically adapt to the addition of new data, avoids the limitations of traditional static knowledge graphs in data updating, ensures the system's real-time reflection of multimodal data, and improves the flexibility and scalability of the system.

[0039] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0040] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0041] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0042] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0043] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0044] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for intelligent visualization and text association of multimodal knowledge graphs, characterized in that: The following steps are involved: S1. Obtain multimodal data from the target enterprise's internal database and perform data preprocessing and feature extraction to generate a multimodal feature vector; S2. Use a cross-modal embedding algorithm to map multimodal feature vectors into a unified vector space, align, associate, and fuse the semantics between multimodal data, and construct a multimodal knowledge graph; S3. Perform bidirectional association and multimodal joint query on the multimodal knowledge graph and the input text to obtain the corresponding multimodal data, and visualize the multimodal knowledge graph and the queried multimodal data through the graph neural network; S4. Based on the newly added multimodal data, the association relationship is analyzed through the graph reasoning algorithm, and the multimodal knowledge graph is dynamically updated; The mapping logic of mapping the multimodal feature vector to a unified vector space through the cross-modal embedding algorithm is as follows: An embedding layer that maps a semantic feature vector, a visual feature vector, and a video dynamic feature vector through a cross-modal embedding algorithm generates a mapped embedding vector, including a semantic embedding vector, a visual embedding vector, and a video dynamic embedding vector; The feature alignment of embedding vectors of different modalities is performed through contrast loss function, and the aligned semantic embedding vector, visual embedding vector and video dynamic embedding vector are fused; The specific process of constructing the multimodal knowledge graph is as follows: Generate multimodal node embedding representation by aligning and fusing semantic embedding vectors, visual embedding vectors, and video dynamic embedding vectors; The feature propagation of node embedding is performed through graph convolutional networks, the associated edges between nodes are constructed, and the global structure of the multimodal knowledge graph is constructed based on cross-modal semantic associations; The weights of nodes and edges are optimized by contrasting loss functions to generate a multimodal knowledge graph.

2. The method for intelligent visualization and text association of a multimodal knowledge graph according to claim 1, characterized in that: The multimodal data in the target enterprise database includes technical documents, product design drawings, and test videos; The multimodal feature vector includes a semantic feature vector, a visual feature vector and a video dynamic feature vector.

3. The method for intelligent visualization and text association of a multimodal knowledge graph according to claim 2, characterized in that: The specific process of multimodal data preprocessing and feature extraction is as follows: Capture the semantic information of the text through a preset natural language processing model and convert the technical documents into semantic feature vectors; Use convolutional neural networks to capture key visual elements in images and extract visual feature vectors of product design images; The temporal dynamic changes in the video are captured through a 3D convolutional neural network, and the video dynamic feature vector of the test video is extracted.

4. The method for intelligent visualization and text association of a multimodal knowledge graph according to claim 3, characterized in that: The specific process of bidirectionally associating the multimodal knowledge graph with the input text and performing multimodal joint query to obtain the corresponding multimodal data is as follows: The input text is semantically vectorized through the text embedding model to generate a text embedding vector; Based on the node representation of the multimodal knowledge graph, the multimodal semantic matching index between the text embedding vector and the multimodal node embedding representation is calculated through the semantic matching algorithm; Based on the multimodal semantic matching index, bidirectional semantic association is performed to filter multimodal data nodes that are highly relevant to the input text; Through the multimodal joint query mechanism, the multimodal dataset associated with the text is obtained, and the corresponding semantic matching results are returned.

5. The method for intelligent visualization and text association of a multimodal knowledge graph according to claim 4, characterized in that: The specific process of obtaining the multimodal semantic matching index is as follows: The semantic similarity between the text embedding vector and the semantic embedding vector, the visual embedding vector and the video dynamic embedding vector is calculated by the semantic matching algorithm; A weighted comprehensive analysis is performed on the similarities of different modalities to obtain the multimodal semantic matching index.

6. The method for intelligent visualization and text association of a multimodal knowledge graph according to claim 5, characterized in that: Based on the newly added multimodal data, the specific process of analyzing the association relationship through the graph reasoning algorithm is as follows: Expand nodes of newly added multimodal data through graph reasoning algorithms and identify existing nodes associated with the newly added data; The graph convolutional network is used to aggregate the features of nodes and edges in the multimodal knowledge graph and update the representation of the nodes. Apply inference rules and relational models to derive potential implicit relationships, enhance the structure and semantic information of the knowledge graph, and dynamically update the multimodal knowledge graph.

Images