A Distributed Multimodal Data Retrieval Method and System Based on Knowledge Graph and Vector Matching

Abstract

This invention discloses a distributed retrieval method and system for multimodal data based on knowledge graphs and vector matching. It stores and organizes multimodal data using a distributed database, establishes a unified multimodal mapping model to embed multimodal data into vectors and align them into the same vector subspace, and then builds an index through multi-level dynamic clustering. Simultaneously, a multimodal knowledge graph is established to support knowledge graph-based multimodal retrieval, and the two methods are combined to achieve multimodal data retrieval functionality. The introduction of distributed technology ensures the accuracy and speed of the retrieval process. This invention can organize and store multimodal data, and embed and cluster it. It also establishes a multimodal knowledge graph to achieve effective association across modal data, and combines vector similarity matching to realize multimodal retrieval functionality. Furthermore, the introduction of a distributed mechanism enables parallel retrieval, ensuring high availability and efficiency of the system.

Description

Technical Field

[0001] This invention belongs to the field of big data technology and relates to information retrieval technology in massive multimodal data, particularly to a distributed retrieval method and system for multimodal data based on knowledge graphs and vector matching. Background Technology

[0002] In the context of the big data era, the explosive growth of information has made data processing and retrieval an extremely challenging task. Past data retrieval systems, whether based on traditional inverted indexes, more generalizable vector similarity matching, or knowledge graphs, have mostly been applied only to single-modal data retrieval scenarios. This has resulted in the inability to fully explore the richness of information in multimodal data scenarios, and the difficulty in achieving effective connections between different modalities, failing to fully utilize multimodal data in terms of comprehensiveness and accuracy.

[0003] Therefore, some scholars have begun to focus on building multimodal data retrieval systems. Current multimodal data retrieval methods can be broadly categorized into three types. One is based on keywords and inverted indexes. This method has the advantage of high accuracy in small datasets or specific domains, but its disadvantages include the need for manual extraction of keywords for different modalities in multimodal retrieval and poor generalization ability. Therefore, this method is less commonly used.

[0004] The second approach is multimodal retrieval based on deep learning and vector matching. One such approach, as seen in patent CN115563316A, involves mapping the multimodal data to be retrieved into a unified vector subspace using deep learning methods. Then, matching data is obtained by calculating the similarity distance between vectors (usually Euclidean distance or cosine similarity). Alternatively, a deep learning model can be used to perform the matching process, as in patent CN110019652A, which uses a trained matching model to retrieve similar vectors. This retrieval method has strong generalization ability and high accuracy across domains. However, its accuracy relies excessively on the mapping model's performance and has poor interpretability.

[0005] Thirdly, there is retrieval based on multimodal knowledge graphs, such as patent CN114741466A. This method stores multimodal data in triplet format and builds a multimodal knowledge graph, then transforms multimodal retrieval into a graph search problem within the knowledge graph. This approach offers high accuracy and interpretability, but building a complete and accurate multimodal knowledge graph is costly.

[0006] Subsequently, some scholars have constructed multimodal retrieval systems that combine multiple methods, such as patent CN117573882A, which can support multiple retrieval methods such as knowledge graph retrieval, text vector retrieval, and image retrieval. However, these are limited to a single domain, and each retrieval method operates independently without efficient integration.

[0007] In summary, the shortcomings and deficiencies of existing multimodal retrieval patent technologies are mainly reflected in the following aspects:

[0008] Multimodal unified mapping models are unstable in performance: Most current multimodal unified mapping models can only handle image and text modal data, lacking support for temporal series, video, and other modal data. Furthermore, the vectors embedded in multimodal data often fail to effectively eliminate ambiguity, resulting in unstable mapping performance.

[0009] Multimodal knowledge graphs are incomplete and inaccurate: Multimodal knowledge graphs need to integrate data from different modalities, thus requiring the matching of entities from different modalities. Currently, entity alignment in multimodal knowledge graphs presents significant challenges.

[0010] The system failed to efficiently integrate multiple search methods: a unified model was not established to weigh and integrate the relevance of results from different search methods. Furthermore, the search process was not executed in parallel, preventing the efficient integration of multiple search methods.

[0011] Long retrieval time and low efficiency: Most current multimodal retrieval systems do not utilize the advantages of distributed computing, resulting in long retrieval times and low efficiency. Summary of the Invention

[0012] To overcome the shortcomings of the existing technologies, the present invention aims to provide a distributed multimodal data retrieval method and system based on knowledge graphs and vector matching. This method can organize and store multimodal data, and embed and cluster it. Simultaneously, it establishes a multimodal knowledge graph to achieve effective cross-modal data association, and combines vector similarity matching to realize multimodal retrieval functionality. Furthermore, a distributed mechanism is introduced to achieve parallel retrieval, ensuring high availability and efficiency of the system.

[0013] To achieve the above objectives, this invention stores and organizes multimodal data using a distributed database, establishes a unified multimodal mapping model to embed multimodal data into vectors and align them into the same vector subspace, and then builds an index through multi-level dynamic clustering. Simultaneously, a multimodal knowledge graph is established to support knowledge graph-based multimodal retrieval, and the two methods are combined to achieve multimodal data retrieval functionality. The introduction of distributed technology ensures the accuracy and speed of the retrieval process. Furthermore, the system provides a visual display, allowing users to intuitively understand the multimodal data retrieval results and perform comparison and filtering operations.

[0014] Specifically, in a first aspect, the present invention provides a multimodal data distributed retrieval method based on knowledge graphs and vector matching, comprising the following steps:

[0015] Step 1: Clean and store the multimodal data from several multimodal database nodes;

[0016] Step 2: Embed the stored multimodal data into vectors using the multimodal data unified mapping model, and form a tree-shaped retrieval structure from the vectors through multi-level dynamic clustering. Establish a doubly linked list for the data in each cluster, and query the specific data in the cluster by using a pointer to the cluster center vector of the cluster to which it belongs.

[0017] Step 3: Store the stored multimodal data in triplet format and build a multimodal knowledge graph;

[0018] Step 4: Perform multimodal data retrieval based on the user-input search criteria. The method is as follows: embed the search criteria into a condition vector, making it in the same space as the cluster center vector. Then, perform vector retrieval and multimodal knowledge graph retrieval in parallel. After scoring the data retrieved by the two methods, integrate and sort them, return the top n data items with the highest scores, and display them.

[0019] A second aspect of the present invention provides a multimodal data distributed retrieval system based on knowledge graphs and vector matching, capable of implementing the multimodal data distributed retrieval method based on knowledge graphs and vector matching described in the first aspect. The system includes:

[0020] The data storage and organization module cleans the multimodal data and stores it according to the corresponding data structure or database, while introducing a distributed mechanism to improve storage capacity and system retrieval efficiency.

[0021] The multimodal data representation and fusion module embeds multimodal data into vectors through a unified multimodal data mapping model, and maps the initial vectors to the same vector space through feature fusion and alignment.

[0022] The multi-level dynamic clustering and indexing module forms a tree-like retrieval structure from the embedded vectors through multi-level dynamic clustering. At the same time, it establishes a doubly linked list for the data in each cluster, and queries the specific data in the cluster by using a pointer to the cluster center vector of the cluster to which it belongs.

[0023] Multimodal knowledge graphs are constructed by storing and building multimodal data in triplet format to support subsequent retrieval on the multimodal knowledge graph.

[0024] The multimodal retrieval module embeds the user-input search conditions into a condition vector, placing it in the same space as the cluster center vector. Then, it performs vector retrieval and multimodal knowledge graph retrieval in parallel. Afterward, it scores the data retrieved by the two methods, integrates and sorts them, and returns a list of the top n data items with the highest scores.

[0025] Therefore, compared with the prior art, the main advantages of the present invention are:

[0026] 1. By introducing distributed technology to store and organize multimodal data, and combining it with multi-level dynamic clustering to establish a flexible and efficient index, we can improve data storage capacity and retrieval efficiency, adapt to the dynamic distribution of multimodal data, and ensure the accuracy and speed of the retrieval process.

[0027] 2. A unified mapping model for multimodal data is established using deep learning methods, enabling multimodal data to be embedded in the same vector space. In addition to images and text, temporal modal data such as video and audio are added, establishing associations between different modalities to achieve multimodal data retrieval.

[0028] 3. Most existing multimodal knowledge graph construction methods are based on traditional entity and relation extraction methods. This invention introduces a large model to help build a multimodal knowledge graph. By utilizing the large model's ability to understand semantic and image data, ambiguity between entities of different modalities is eliminated, significantly improving the completeness and accuracy of the multimodal knowledge graph.

[0029] 4. Existing multimodal retrieval methods do not combine vector matching and multimodal knowledge graph retrieval. This invention efficiently integrates vector matching and multimodal knowledge graph retrieval methods to improve retrieval accuracy and generalization ability. Attached Figure Description

[0030] Figure 1 This is a system framework diagram of the present invention.

[0031] Figure 2 This is a structural diagram of the multimodal data representation and fusion module.

[0032] Figure 3 This is a structural diagram of the Bert encoding module.

[0033] Figure 4 This is a structural diagram of the Linear encoding module.

[0034] Figure 5 This is a structural diagram of the ResNet-LSTM encoding module.

[0035] Figure 6 This is an index structure diagram formed by two-level dynamic clustering.

[0036] Figure 7Word cloud generated from clustered clusters.

[0037] Figure 8 This is a flowchart of the multimodal retrieval module.

[0038] Figure 9 This displays the corresponding multimodal data retrieval results. Detailed Implementation

[0039] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0040] The system structure of the present invention is as follows: Figure 1 As shown, the data storage and organization module manages a distributed multimodal database to store and organize large amounts of data in different modalities, such as text, graphs, tables, and images. The multimodal data representation and fusion module embeds multimodal data into vector form using a unified multimodal mapping model. Simultaneously, it establishes a multimodal knowledge graph to structurally connect the multimodal data. The multi-level dynamic clustering and indexing module processes the embedded vectors, establishing a multimodal data retrieval index through multi-level dynamic clustering and inverted indexing. The data retrieval module automatically embeds search conditions into vectors and extracts key information from the conditions through entity and relation extraction. It then performs vector similarity matching and multimodal knowledge graph retrieval, integrates the search results, and returns a suitable list of the top n data entries. The visualization module displays the retrieved multimodal data results in various graphical and intuitive ways.

[0041] The implementation steps of this invention are as follows:

[0042] I. Data Storage and Organization

[0043] This step mainly involves cleaning and storing the multimodal data from several multimodal database nodes. The specific process and further optimization measures can be described as follows:

[0044] (1) Data cleaning requires standardization according to certain rules, such as date format and numerical format, and data format standardization by conversion. Null values, duplicate values, and outliers in the data should be cleaned to ensure data integrity and accuracy.

[0045] (2) After data cleaning, for different modal data, the sharding key is determined according to its specific requirements, and the multimodal data is routed to the corresponding storage shard through the sharding algorithm in the data sharding engine.

[0046] (3) Determine the corresponding database and data storage method based on the different modes and structures of the data, and store the data in a distributed database.

[0047] This step can be done Figure 1 The data storage and organization module in the system architecture shown is implemented. In this module, multimodal data is cleaned and stored according to a suitable data structure or database. Furthermore, a distributed mechanism can be introduced to improve storage capacity and system retrieval efficiency.

[0048] II. Multimodal Data Representation and Fusion

[0049] This step primarily involves embedding the stored multimodal data into vectors using a unified multimodal data mapping model. Further feature fusion and alignment can then be used to map the initial vectors to the same vector space. The specific process and further optimization measures are described below:

[0050] (1) Different deep learning methods are used to embed different modal data in the combination, i.e., embedding into feature vectors.

[0051] For time-series data, the features are represented by an LSTM encoding module as E1 = LSTM(M_waveform). In this process, the input to the LSTM module is the time-series data. The method is as follows: First, the time-series data is normalized, scaling it to the range of 0-1 to make it easier for the LSTM model to learn. Then, the data is formed into an array of shape (n_samples, look_back, n_features), where n_samples is the number of samples, look_back indicates how many consecutive time points to use as input (set to 10), and n_features is the number of features at each time point. Finally, the generated array data is processed through an LSTM layer and a fully connected layer (linear layer) to output a feature vector E1. Further, in this embodiment, the input size of the LSTM layer is the same as the number of features in the data, and the hidden layer size is 512. The fully connected layer converts the output of the LSTM layer into a feature vector of size (1, 1000), i.e., E1.

[0052] For image data, the feature representation is obtained through the ResNet-LSTM encoding module as E2 = ResNet-LSTM(M_image). The ResNet-LSTM encoding module is as follows: Figure 5As shown, the method is as follows: First, the image is preprocessed and adjusted to an RGB image; then, a pre-trained ResNet50 model is loaded, and the last fully connected layer and average pooling layer are removed to extract features from the image; next, the output of the ResNet50 model is flattened into a vector and input into a fully connected layer, which transforms it into a vector of the required size; the transformed vector is then input into an LSTM layer, which processes it as a time step of a sequence and outputs a final LSTM state; finally, the LSTM state is input into a fully connected layer, which outputs a feature vector E2. Further, in an embodiment of the present invention, the RGB image size adjusted for the image data is 3*224*224, the flattened output vector of the ResNet50 model is (1, 1, 100352), and it is input into a fully connected layer to transform it into a vector of the required size (1, 1, 512), finally outputting a feature vector of size (1, 512), i.e., E2.

[0053] For text data, the feature representation is E3 = Bert(M_text) using the Bert encoding module. The structure of the Bert encoding module is as follows: Figure 3 As shown, the method is as follows: First, the input text is converted into numerical form. For example, the WordPiece word segmenter can be used to segment the input text and map each word to a corresponding number in the BERT model's vocabulary (vocab_size = 21128). Then, the word segments are positionally encoded. For example, an encoding formula for positionally encoding word segments is as follows:

[0054]

[0055]

[0056] Where t represents the position of the word segment in the sentence, i represents the index of each value in the positional encoding, k = 0, 1, 2, 3, ..., and d represents the number of hidden units in the BERT model, i.e., d = 768; next, the obtained encoded sequence is input into the BERT model for forward propagation, and after self-attention mechanism, fully connected and pooling operations, it is transformed into a series of vectors, where each vector represents a token in the input text. For example, the self-attention mechanism is set to 12 layers; finally, the feature vector E3 is output by averaging these vectors. In the embodiment of the present invention, when d = 768, the size of the output feature vector E3 is (1, 768).

[0057] For relational data, the feature representation is E4 = Linear(index) using the Linear encoding module. The Linear encoding module is as follows: Figure 4As shown, the method is as follows: First, take the given LET value as input, and set the index num_bins to be divided and the width bin. width For example, setting num_bins = 10, where let max and let min These are the upper and lower bounds of the LET range, respectively. Then, the LET is assigned to a bin. The lower bound of the LET range is subtracted from the LET value, and then divided by the bin width to obtain the index of the bin containing that value. If the calculated index exceeds the range of 0 to the partitioned index (e.g., num_bins-1), it needs to be limited to this range. Next, through a loop, the center values ​​of each bin are used to form a discrete vector. According to the previous example, the length of the discrete vector formed by the center values ​​of each bin is 10. Finally, feature extraction is performed through a fully connected layer linear transformation operation, and the feature vector E4 is output. According to the parameters taken in the previous example, the size of E4 is (1, 30).

[0058] (2) Since the feature representations of each modality have different dimensions, linear transformations are used to transform the dimensions of the feature vectors output by each modality, so that they can be represented in the same feature vector space as the performance indicators. For example, the linear transformation method is as follows: let the dimension of linear transformation W1 be (1000, 10), the dimension of linear transformation W2 be (512, 10), the dimension of linear transformation W3 be (768, 10), and the dimension of linear transformation W4 be (30, 10). The transformed feature vectors are E'1 = E1*W1, E'2 = E2*W2, E'3 = E3*W3, and E'4 = E4*W4, and their size is (1, 10).

[0059] (3) Two Transformer layers are used to perform multiple nonlinear transformations on the linearly transformed feature vectors to extract richer features, thereby obtaining the feature vector G. According to the parameters taken in the previous example, the size of G is (1, 768). In this network structure, the first Transformer layer can extract features of different types of feature vectors to obtain more information representation. The second Transformer layer can further enhance the representation ability of features to improve the accuracy of the embedded vector.

[0060] This step can be completed via... Figure 1 The multimodal data representation and fusion module in the system architecture shown is implemented as follows: The specific working principle of this module is as follows... Figure 2 As shown, multimodal data is embedded into vectors through a unified multimodal data mapping model, and the initial vectors are mapped to the same vector space through feature fusion and alignment to support subsequent retrieval operations.

[0061] III. Multi-level Dynamic Clustering and Indexing

[0062] This step primarily uses multi-level dynamic clustering to form a tree-like retrieval structure from the vectors. A doubly linked list is established for the data in each cluster, and the specific data within that cluster is retrieved via a pointer to the cluster's centroid vector. The specific process and further optimization measures can be described as follows:

[0063] (1) Perform multi-level dynamic clustering on the vectors after embedding the multimodal embedding module to form a tree-type index structure.

[0064] Multi-level dynamic clustering is performed using the incremental Gaussian Mixture Model (GMM) algorithm to form a tree-like retrieval structure from the vectors. The basic clustering process can be described as follows: First, the GMM is initialized, selecting a subset of data as initial Gaussian components Ci, and assigning an initial mean μi and covariance matrix Σi to each component. Then, using incremental learning, new data points are gradually added to the existing GMM, and the clustering results are dynamically updated based on the model parameters. The expectation-maximization algorithm is used to update the mean μi and covariance matrix Σi of each Gaussian component, as well as the weight wi of each component, until clustering is complete.

[0065] When the number of clusters exceeds the maximum cluster capacity n, the above clustering process is repeated to perform incremental dynamic clustering at the next level for that cluster, ultimately forming a multi-level tree-structured vector index. The index structure diagram formed by two-level dynamic clustering in this embodiment can be found in the reference diagram. Figure 6 As shown.

[0066] (2) For each cluster in the tree-structured index, extract its features and keywords to generate a word cloud representing the features of that cluster. The specific method is as follows:

[0067] First, all data within each cluster is preprocessed, including removing stop words and useless features, and stemming, to extract keywords that represent the content of that cluster. Then, the frequency of keywords in each cluster is calculated, and these frequencies are used as the basis for generating the word cloud. Next, the frequency of each word in the preprocessed text is calculated. This step is crucial for generating the word cloud because the size of each word in the cloud will be based on its frequency. Let W be the set of words in the text, w... i It is a single word, n i It is the word w i The number of times word w appears, where N is the total number of words in the text. i Frequency of occurrence:

[0068]

[0069] After calculating the word frequencies, these words need to be displayed at different sizes to form a word cloud, and laid out using the following linear table: Select a starting point, arrange the words horizontally or vertically according to their frequency, and perform the following size mapping:

[0070] size(w i )=k·freq(w i )

[0071] That is, the word w i Font size (w) in word cloud i ) is directly mapped from its frequency freq(w) i (), where k is the scaling factor. Then, the word cloud is drawn according to the calculated size and layout. A word cloud diagram generated in this embodiment of the invention can be referenced. Figure 7 As shown.

[0072] (3) Create a doubly linked list for the leaf nodes of the index tree and the specific data contained in the node.

[0073] A doubly linked list is used to ensure that each entity with the same cluster center has a pointer to its own cluster center vector, and the database is indexed according to this pointer to achieve fast retrieval of data within a cluster based on the cluster center. After forming the incremental learning Gaussian mixture model in (1), the mean of each Gaussian component is used to represent the cluster center, and a bidirectional pointer to the cluster center vector is established. For each Gaussian component, the cluster center it represents, all data points contained in the cluster, and their position information in the original dataset are recorded. When it is necessary to quickly retrieve data within a cluster based on a given cluster center, all data points represented by the cluster center can be quickly found by using the bidirectional pointer. This retrieval method can greatly improve the retrieval speed and also facilitate the management and processing of data within the cluster.

[0074] This step can be completed via... Figure 1 The multi-level dynamic clustering and indexing module in the system structure shown implements this. This module forms a tree-like retrieval structure by multi-level dynamic clustering of the embedded vectors. At the same time, it establishes a doubly linked list for the data in each cluster, and queries the specific data in the cluster by using a pointer to the cluster center vector of its own cluster.

[0075] IV. Establishing a Multimodal Knowledge Graph

[0076] This step primarily involves storing the multimodal data in triplet format and building a multimodal knowledge graph. The multimodal knowledge graph is constructed using a document-based knowledge graph generation method based on LLM (Large Language Model). By leveraging the large language model, structured knowledge representations are created from the multimodal data without manual intervention. These knowledge representations are continuously expanded into the relevant domain knowledge graph, building knowledge connections within that domain. Named entity recognition, relation extraction, and entity disambiguation techniques are used to transform unstructured text into structured representations that capture entities, their relationships, and associated attributes.

[0077] First, data augmentation-based methods are needed to fine-tune a large model using multiple modalities of data within a specific domain, increasing data diversity. This allows for fine-tuning the LLM (Low-Range Model) training. Then, the fine-tuned LLM is used to generate triples from the original (unlabeled) data. This alleviates the problem of insufficient labeled data in deep learning, increases the diversity of training data, and thus improves the model's generalization ability.

[0078] After fine-tuning, a large model is used to generate entity and relation triples. First, multimodal data parsing is performed: the model parses the input multimodal data, understanding its text structure, image structure, and temporal sequence structure, thereby identifying key information. Then, entity recognition is performed: based on the identified key information, the model identifies entities in the multimodal data. This step is achieved through training the model on a large-scale corpus. Simultaneously, relation recognition is performed: while recognizing entities, the model attempts to understand the relationships between entities. This involves understanding linguistic elements such as verbs and prepositions in the text, and how they connect different entities. Finally, triple construction is performed: combining entities and their relationships, the model can construct entity-relation triples, thus establishing a complete multimodal knowledge graph.

[0079] This step can be completed via... Figure 1 The multimodal knowledge graph in the system structure shown demonstrates how storing multimodal data in triple format and building a multimodal knowledge graph enables subsequent retrieval on the multimodal knowledge graph.

[0080] V. Multimodal Retrieval

[0081] Based on the preceding steps, this step can perform multimodal data retrieval according to the user-input search criteria. The retrieval method and principle are as follows:

[0082] The multimodal embedding module embeds the user-input search conditions into a condition vector, making it in the same space as the cluster center vector. Then, vector retrieval and multimodal knowledge graph retrieval are performed in parallel. The data retrieved by the two methods are scored, integrated and sorted, and a list of the top n data items with the highest scores is returned.

[0083] The parallel execution of vector retrieval and multimodal knowledge graph retrieval is as follows:

[0084] (1) Entity and relation identification is performed on the search condition text, and the entity and relation keywords involved are temporarily stored.

[0085] (2) Perform multi-level similarity matching between the conditional vectors and the cluster centers in the multi-level dynamic clustering and indexing module. Gradually refine to more specific cluster center vectors and determine the final cluster. At the same time, perform conditional mapping on the entities and relations temporarily stored in (1): map the entities they contain to the corresponding nodes in the graph, and map the relations to the edges or attributes related to the node. After the entities and relations of the retrieval conditional text are mapped, convert the mapping results into the graph query language Cypher to obtain the specific entities and relations stored from the multimodal knowledge graph.

[0086] (3) Retrieve all multimodal data within a cluster using bidirectional pointers pointing from the cluster center to the cluster vector, and perform similarity matching with the conditional vector to obtain result S1. Simultaneously, perform specific relation queries on the multimodal knowledge graph based on the entities and relations mapped by the conditions to obtain result S2.

[0087] (4) Integrate S1 and S2 and sort them according to similarity, and return the list S of the top n data items with the highest similarity.

[0088] This step can be completed via... Figure 1 The multimodal retrieval module in the system architecture shown is implemented as follows: The working process of this module is as follows... Figure 8 As shown, it is responsible for multimodal data retrieval based on the search criteria input by the user. First, the user-input search criteria are embedded into a condition vector through the multimodal embedding module, making it in the same space as the cluster center vector. Then, vector retrieval and multimodal knowledge graph retrieval are performed in parallel. After that, the data retrieved by the two methods are scored, integrated and sorted, and a list of the top n data items with the highest scores is returned.

[0089] VI. Visual Display

[0090] This step is not essential in this invention. Its main purpose is to present highly similar multimodal data to the user in an intuitive and easy-to-use manner, such as through text, lists, line charts, and bar charts, facilitating subsequent processing and analysis. This can be achieved through... Figure 1 The visualization module in the system structure shown allows users to filter, sort, and perform other operations on the results according to their needs.

[0091] In one embodiment of the present invention, a retrieval embodiment is provided, the steps of which are described in sequence as follows:

[0092] Step 1: Input the search criteria: "Devices and chips that meet general radiation resistance performance". Embed this criterion into a criterion vector E, and perform similarity matching with the multi-level cluster center vectors obtained after multi-level clustering in the database to obtain the final cluster. At the same time, extract the entities "device", "chip" and "radiation resistance performance", with the relationship "meet", and perform retrieval through the knowledge graph.

[0093] Step 2 involves repeating similarity matching within the final cluster to obtain the top n multimodal data points that meet the criteria. Simultaneously, these are integrated with the results retrieved from the knowledge graph, and duplicate data is removed.

[0094] Step 3: Differentiate by mode and sort by similarity from top to bottom. The text data with the highest similarity is "Radiation Resistance Analysis and Design of CMOS Integrated Circuits.pdf", and the text with the lowest similarity is "Research Status and Application Analysis of Bidirectional DC-DC Converters for DC Microgrids". The table data with the highest similarity is "Influence of Different Radiation Dose on the Output of Radiation-Resistant Bandgap Reference Source.xlsx", and the image with the lowest similarity is "Measured Results After Bandgap Reference Radiation.xlsx". The image data with the highest similarity is "Radiation-Resistant Level Conversion Circuit.png", and the image with the lowest similarity is "12V Boost Converter Circuit.png".

[0095] Step 4: Display them separately according to their different modalities (text, image, table, etc.), such as Figure 9 As shown in the figure, the correlation between multimodal data is displayed using a correlation graph, making it convenient for users to view relevant data.

Claims

1. A method for distributed retrieval of multi-modal data based on knowledge graph and vector matching, characterized in that, Comprising the following steps: Step 1, after data cleaning, store the multi-modal data from several multi-modal database nodes; Step 2, embed the stored multi-modal data into vectors using a multi-modal data unified mapping model, and form a tree-shaped retrieval structure through multi-level dynamic clustering, establish a double-linked list for the data in each cluster, and query the specific data in the cluster through the pointer to the cluster center vector of the cluster to which it belongs; Step 3, store the stored multi-modal data in triple format and establish a multi-modal knowledge graph; Step 4, according to the retrieval condition input by the user, perform multi-modal data retrieval, the method is as follows: embed the retrieval condition into a condition vector, so that it is in the same space as the cluster center vector, then perform vector retrieval and multi-modal knowledge graph retrieval in parallel, integrate and sort the data retrieved by the two methods after scoring, and return the top n data list with high scores and display it; Wherein: In step 2, use the incremental GMM algorithm for multi-level dynamic clustering to form a tree-shaped retrieval structure, and the basic clustering process is as follows: First, initialize the Gaussian mixture model, select part of the data as the initial Gaussian component Ci, and assign an initial mean μi and covariance matrix Σi to each component; Then, use incremental learning, gradually add new data points to the existing Gaussian mixture model, and dynamically update the clustering results according to the model parameters, use the expectation maximization algorithm to update the mean μi and covariance matrix Σi of each Gaussian component, and the weight wi of each component, until the clustering is completed; When the number of a cluster is greater than the maximum capacity n of the cluster, repeat the above clustering process to perform the next level of incremental dynamic clustering on the cluster, and finally form a multi-level tree vector index; The method for establishing a double-linked list for the data in each cluster and querying the specific data in the cluster through the pointer to the cluster center vector of the cluster to which it belongs is as follows: For each cluster in the tree index structure, extract the features and keywords therein to generate a word cloud representing the characteristics of the cluster; For the leaf nodes of the index tree and the specific data contained in the nodes, a double-linked list is established, that is, the mean of each Gaussian component represents the cluster center, and a double-direction pointer to the cluster center vector is established, for each Gaussian component, the cluster center it represents, all data points contained in the cluster, and their position information in the original data set are recorded; When it is necessary to quickly retrieve the data in the cluster according to the given cluster center, all data points represented by the cluster center can be found according to the double-direction pointer. 2.The method of claim 1, wherein, In step 1, after data cleaning, for different modal data, determine the shard key according to its corresponding requirements, route the multi-modal data to the corresponding storage shard through the shard algorithm in the data shard engine, and determine the corresponding database and data storage method according to the different modal and structure of the data, and store it in the distributed database. 3.The method of claim 1, wherein, In step 2, after embedding the stored multi-modal data into vectors using a multi-modal data unified mapping model, map the initial vectors to the same vector space through feature fusion and alignment. 4.The method of claim 1, wherein, The step of embedding the stored multi-modal data into vectors using a multi-modal data unified mapping model comprises: For time series data, feature representation is performed by an LSTM encoding module as follows: first, normalize the time series data to scale it to the range of 0-1; then, form an array with shape (n_samples, look_back, n_features), where n_samples is the number of samples, look_back is the parameter indicating how many consecutive time points are used as input, and n_features is the number of features at each time point; finally, the generated array data is processed by an LSTM layer and a fully connected layer to output a feature vector E1; For image data, feature representation is performed by a ResNet-LSTM encoding module as follows: first, adjust the image to an RGB image after preprocessing; then, load a pre-trained ResNet50 model and remove the last fully connected layer and average pooling layer; next, flatten the output of the ResNet50 model into a vector and input it into a fully connected layer to convert it to the desired size; continue to input the transformed vector into an LSTM layer, which processes it as a sequence of time steps and outputs a final LSTM state; finally, input the LSTM state into a fully connected layer to output a feature vector E2; For text data, feature representation is performed by a Bert encoding module as follows: first, convert the input text into digital form; then, perform position encoding on the tokenization of the text; next, input the obtained encoding sequence into the BERT model for forward propagation, and convert it into a series of vectors through self-attention mechanism, fully connected and pooling operation, where each vector represents a token in the input text; finally, output a feature vector E3 by averaging these vectors; For relational data, feature representation is performed by a Linear encoding module as follows: first, take the given LET value as input, set the index to be divided and the width of the bin; then, assign the LET to a bin, subtract the lower limit of the LET range, and divide by the bin width to get the index of the bin where the value is located; if the calculated index exceeds the range of 0 to the divided index, limit it within this range; next, construct a discrete vector by looping through the center value of each bin; finally, perform feature extraction by linear transformation operation through a fully connected layer to output a feature vector E4. 5.The method of claim 1, wherein, The feature vectors output by each modality data are transformed in dimension by linear transformation, so that they can be represented in the same feature vector space as the performance indicators; two Transformer layers are used to perform multiple non-linear transformations on the linearly transformed feature vectors to extract more rich features and obtain a feature vector G; For time series data, the input size of the LSTM layer is the same as the number of features of the data, and the size of the hidden layer is 512; the fully connected layer converts the output of the LSTM layer into a feature vector E1 with size (1, 1000); For picture data, the adjusted RGB image size is 3*224*224, the output flattened vector size of the ResNet50 model is (1, 1, 100352), and the output is input into a fully connected layer to convert it into a vector (1, 1, 512) with a required size, and finally a feature vector E2 with a size of (1, 512) is output; For text data, the number of hidden units in the BERT model is d = 768, and finally a feature vector E3 with a size of (1, 768) is output; For relational data, set the index to be partitioned width of bin where and are the upper and lower limits of the LET range; the length of the discrete vector composed of the center values of each bin is 10; the final output is a feature vector E4 of size (1, 30); The linear transformation, assuming that the linear transformation W1 has a dimension of (1000, 10), the linear transformation W2 has a dimension of (512, 10), the linear transformation W3 has a dimension of (768, 10), and the linear transformation W4 has a dimension of (30, 10), the converted feature vectors are E'1 = E1* W1, E'2 = E2*W2, E'3 = E3*W3, and E'4 = E4*W4, all with a size of (1, 10); and the feature vector G has a size of (1, 768). 6.The method of claim 1, wherein, The method for extracting features and keywords from each cluster in the tree index structure to generate a word cloud representing the features of the cluster is as follows: First, pre-process all the data in each cluster to extract keywords that can represent the content of the cluster; Then, count the frequency of each keyword in the cluster, and use these frequencies as the basic data for generating the word cloud; Next, calculate the frequency of each word in the pre-processed text; Finally, display each word in different sizes to form a word cloud, and use the following linear layout: select a starting point, arrange the words horizontally or vertically in order of word frequency size, and perform the following size mapping: That is, the word Font size in the word cloud Directly mapped from its frequency where is a scaling factor. 7.The method of claim 1, wherein, Step 4, perform vector retrieval and multi-modal knowledge graph retrieval in parallel as follows: Perform entity and relationship recognition on the search condition text, and temporarily store the involved entity and relationship keywords; Perform multi-level similarity matching between the condition vector and the cluster center, gradually refining to the cluster center vector, and determine the final cluster; at the same time, perform condition mapping on the temporarily stored entities and relationships: map the entities temporarily stored in the previous step to the corresponding nodes in the multi-modal knowledge graph, and map the relationships to the edges or attributes related to the nodes; after the entity and relationship mapping of the search condition text is complete, convert the mapping results into the graph query language Cypher to obtain the stored specific entities and relationships from the multi-modal knowledge graph; Retrieve all multi-modal data in the cluster through the bidirectional pointer from the cluster center vector, and perform similarity matching with the condition vector to obtain the result S1; at the same time, perform specific relationship queries on the multi-modal knowledge graph based on the mapped entities and relationships to obtain the result S2.

8. A distributed retrieval system for multi-modal data based on knowledge graph and vector matching, used to implement the method for distributed retrieval of multi-modal data based on knowledge graph and vector matching in claim 1, characterized in that, It includes: A data storage and organization module that cleanses multi-modal data and stores them according to corresponding data structures or databases, while introducing a distributed mechanism to improve storage capacity and system retrieval efficiency; A multi-modal data representation and fusion module that embeds multi-modal data into vectors through a multi-modal data unified mapping model, and maps the initial vectors to the same vector space through feature fusion and alignment. The multi-level dynamic clustering and indexing module forms a tree retrieval structure through multi-level dynamic clustering of embedded vectors, and establishes a bidirectional linked list for data in each cluster, so that specific data in the cluster is queried through a pointer to a cluster center vector of the cluster to which the specific data belongs; The multi-modal knowledge graph stores and establishes multi-modal data in a triple format to support subsequent retrieval on the multi-modal knowledge graph; The multi-modal retrieval module embeds the retrieval condition input by the user into a condition vector, so that the condition vector is in the same space as the cluster center vectors, and then performs vector retrieval and multi-modal knowledge graph retrieval in parallel, integrates and sorts the data retrieved in the two ways after scoring, and returns a list of the top n data with the highest scores.

Images