Archive knowledge base construction and retrieval method and system based on multi-modal data fusion

Abstract

The invention discloses an archive knowledge base construction and retrieval method and system based on multi-modal data fusion. The method comprises the steps that heterogeneous archive data are cleaned, image features are extracted through CNN, text features are extracted through Transform, audio is converted into text and then subjected to similarity, a unified feature vector is generated, and metadata is constructed according to archive code association; creating a graph database instance, defining nodes and relationship types, importing entities and relationships, and storing feature vectors and metadata; the features are mapped to a high-dimensional shared semantic space, positive and negative sample pairs are constructed to update embedded layer parameters, self-attention is used in modalities, a shared attention mechanism is used between modalities, weights are adjusted according to archive features, and unified knowledge representation is generated; segmenting the steering quantity of the multi-modal data, storing the steering quantity into a database, and adopting hierarchical indexing and optimizing as required; related document fragments are retrieved through RAG technology vectors, answers are generated with the help of a large language model, and session feedback is provided. The file retrieval efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of archive management and data processing, and particularly relates to a method and system for constructing and retrieving an archive knowledge base based on multi-modal data fusion. Background Art

[0002] With the rapid development of information technology, the field of archive management is gradually transforming from traditional paper archives to digital and multi-modal archive resources. Currently, the archive knowledge base has become the core of the construction of smart archives, which not only needs to process text data, but also needs to integrate various modal data such as images, audio, and video. However, existing archive management systems face many challenges in processing multi-modal data, such as data heterogeneity, insufficient semantic relevance, and lack of efficient fusion and retrieval mechanisms. In addition, existing systems still have imperfect functions in knowledge graph construction and intelligent retrieval.

[0003] In recent years, multi-modal data fusion technology has made remarkable progress in the field of artificial intelligence. Through deep learning and neural network technologies, different modal data can be effectively integrated to achieve more comprehensive information processing and understanding. For example, using deep learning models to extract features and perform semantic alignment on text, images, audio and other data can significantly improve the fusion effect of multi-modal data. In addition, the development of large language models provides new technical support for semantic analysis and intelligent retrieval of archive knowledge bases.

[0004] In the field of archive management, although existing research has explored the integrated management of multi-modal archive resources, there are still deficiencies in the existing technology. Most research focuses on the processing of single-modal data and lacks an overall fusion scheme for multi-modal data. Existing archive knowledge bases mostly adopt traditional coding and classification methods (such as time, classification number, serial number, etc.), which cannot reflect the semantic relationships between archive contents, making the implicit connections between archives (such as the associations of events, people, and locations) not effectively mined and utilized, resulting in insufficient semantic relevance between archives. For multi-modal data (such as text, pictures, audio), existing archive knowledge bases lack efficient natural language processing (NLP) and computer vision technologies and are difficult to accurately extract knowledge entities from archive contents; for multi-modal data with inconsistent data formats and structures, existing archive knowledge bases lack unified data standards and cleaning tools and are difficult to achieve standardized processing of data. These problems lead to insufficient multi-modal data processing capabilities of existing solutions. Due to the above problems, existing archive knowledge bases are difficult to extract knowledge entities and their relationships from multi-modal data and cannot effectively construct knowledge graphs, and the difficulty in extracting semantic relationships also leads to the fact that existing archive knowledge bases can only perform retrieval based on traditional keyword matching, and intelligent retrieval is difficult to achieve.

[0005] Therefore, to meet the requirements of the archival knowledge base in multi-modal data fusion and intelligent retrieval, an innovative technical solution is urgently needed, which can achieve the efficient fusion, unified representation, and intelligent retrieval of multi-modal data, thereby improving the intelligent level of archival management. Summary of the Invention

[0006] Object of the Invention: To solve the above technical problems, the present invention provides a method and system for constructing and retrieving an archival knowledge base based on multi-modal data fusion, enabling multiple types of institutions to directly call through this system to achieve the efficient fusion of multi-modal data, improving the accuracy and efficiency of archival retrieval, constructing a structured knowledge graph, and deeply developing and utilizing multi-modal data, so as to meet the complex needs of users for multi-modal archival data and promote the technological progress in the field of archival management.

[0007] Technical Solution: In the first aspect, a method for constructing and retrieving an archival knowledge base based on multi-modal data fusion, the method comprises the following steps:

[0008] Construct a multi-modal archival knowledge base: Perform cleaning operations on heterogeneous archival data, extract image features using a convolutional neural network (CNN), extract text features using a Transformer network, convert audio data into text through speech recognition and then extract text features in the same way, generate a unified feature vector, and based on archival coding association, construct metadata to represent the association between the feature vector of archival data and archival entities; Create an instance of a graph database, define node types and relationship types, import entities and relationships into the graph database, and store the feature vector and metadata as node attributes;

[0009] Multi-modal data fusion and indexing: Map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss and update the parameters of the embedding layer; Use the self-attention mechanism within each modality, introduce a shared attention mechanism between different modalities, use the unique identifier of the archival coding as the modality alignment benchmark, and dynamically adjust the attention weight distribution according to the archival characteristics to generate a unified multi-modal knowledge representation; Segment and convert the multi-modal data into vector representations, store them in a vector database, adopt a hierarchical indexing HNSW structure and dynamically optimize the indexing structure according to data distribution and query requirements;

[0010] Intelligent retrieval and knowledge generation: Utilize the Retrieval-Augmented Generation (RAG) technology, obtain the most relevant document fragments to the user query through vector retrieval, generate answers using a large language model after obtaining the candidate vectors, and provide session-level feedback retrieval; Support complex queries based on the knowledge graph through the query language of the graph database, use various graph reasoning algorithms to mine potential relationships from the knowledge graph, and regularly check and update the knowledge graph.

[0011] Furthermore, the metadata includes: unique identifier, content type, original format, creation date, and version number.

[0012] Furthermore, when constructing the knowledge graph, the file encoding is structured and defined as an independent node in the knowledge graph, with its structure being <unique identifier, content type, original format, creation date, version number>, and the feature vectors of text, image, and audio modal data are imported.

[0013] Extract text, image, and audio entity modal data from the original file set, and extract the relationships between the modal data.

[0014] Use the file encoding as the central node and connect it to the entity nodes of text, image, and audio modal data through relationship edges.

[0015] Furthermore, after the knowledge graph is constructed, it is optimized, including: entity disambiguation, including context information disambiguation and external knowledge base DBpedia disambiguation; and using a pre-trained model to extract potential entity relationships from text files and integrate them with the existing knowledge graph.

[0016] Furthermore, map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss, and update the parameters of the embedding layer, including:

[0017] Map the extracted features to a high-dimensional shared semantic space through the embedding layer, and the parameters of the embedding layer are optimized through contrastive learning, so that the features of different modalities have similar distributions in the semantic space. The implementation method of contrastive learning is as follows:

[0018] Construct positive and negative sample pairs and calculate the embedding vectors: Samples in the same group of multimodal data form positive sample pairs, and samples in different groups of multimodal data form negative sample pairs. Map the samples of each modality to the shared semantic space through the corresponding embedding layer to obtain their embedding vectors.

[0019] Calculate the contrastive loss: Calculate the InfoNCE contrastive loss function.

[0020] Update the parameters of the embedding layer: Starting from the loss function, calculate the gradient of the loss function with respect to the parameters of the embedding layer through the backpropagation algorithm, and use the Adam optimization algorithm to update the parameters of the embedding layer according to the gradient.

[0021] Furthermore, the following specific strategies are introduced in the self-attention and shared attention mechanisms to improve the processing effect of archival data: using the unique identifier of the archival encoding as the benchmark for modality alignment, and through sharing the attention weights of the archival encoding, aligning the text, image, and audio modalities of the same archive in the semantic space; dynamically adjusting the attention weight allocation according to the content type and original format of the archive; and achieving temporal alignment of different versions of the archive through the version number and creation date in the archival encoding.

[0022] Furthermore, the multi-modal data is segmented and converted into vector representations, including:

[0023] For text data, it is segmented by paragraph or sentence; for image data, it is segmented by region; for the segmented data fragments, a pre-trained model is used to convert them into vector representations.

[0024] In a second aspect, an archival knowledge base construction and retrieval system based on multi-modal data fusion includes

[0025] A multi-modal archival knowledge base construction module: performing cleaning operations on heterogeneous archival data, using a convolutional neural network (CNN) to extract image features, using a Transformer network to extract text features, converting audio data into text through speech recognition and then processing it in the same way to extract text features, generating unified feature vectors, and constructing metadata based on archival encoding to represent the association between the feature vectors of archival data and archival entities; creating an instance of a graph database, defining node types and relationship types, importing entities and relationships into the graph database, and storing the feature vectors and metadata as node attributes;

[0026] A multi-modal data fusion and indexing module: mapping the extracted features to a high-dimensional shared semantic space, constructing positive and negative sample pairs, calculating the contrastive loss and updating the parameters of the embedding layer; using the self-attention mechanism within each modality and introducing a shared attention mechanism between different modalities, using the unique identifier of the archival encoding as the modality alignment benchmark, dynamically adjusting the attention weight allocation according to the characteristics of the archive, and generating a unified multi-modal knowledge representation; segmenting and converting the multi-modal data into vector representations, storing them in a vector database, adopting a hierarchical index HNSW structure and dynamically optimizing the index structure according to data distribution and query requirements;

[0027] An intelligent retrieval and knowledge generation module: using the retrieval-augmented generation (RAG) technology, obtaining the most relevant document fragments to the user query through vector retrieval, generating answers using a large language model after obtaining candidate vectors, and providing session-level feedback retrieval; supporting complex queries based on the knowledge graph through the query language of the graph database, using a variety of graph reasoning algorithms to mine potential relationships from the knowledge graph, and regularly checking and updating the knowledge graph.

[0028] In a third aspect, a computer device includes: one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the method for constructing and retrieving an archive knowledge base based on multimodal data fusion as described in the first aspect of the present invention are implemented.

[0029] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for constructing and retrieving an archive knowledge base based on multimodal data fusion as described in the first aspect of the present invention are implemented.

[0030] Advantageous effects: Compared with the prior art, the present invention has the following advantageous effects:

[0031] (1) By extracting features from multimodal data (text, images, audio, archive codes), and combining contrastive learning and cross-modal attention mechanisms, the feature representations of different modalities are mapped to a unified semantic space, thereby achieving cross-modal semantic alignment. This technical means solves the problem of inaccurate retrieval caused by modality heterogeneity in existing archive retrieval systems, and significantly improves the relevance and accuracy of retrieval results. For example, users can retrieve relevant image or audio archives through natural language descriptions, or retrieve relevant text descriptions through images.

[0032] (2) Taking archive codes as independent nodes in the knowledge graph and connecting them with modal data such as text, images, and audio through relational edges to form a structured knowledge representation. This not only solves the problem that traditional archive management systems are difficult to effectively integrate multimodal data, but also provides basic support for in-depth association and semantic retrieval of archives. For example, through archive codes, the associated multimodal archive data can be quickly located, realizing efficient archive management and retrieval.

[0033] (3) Using knowledge graph technology, a semantic retrieval function based on archive codes, metadata, and graph retrieval is realized. Compared with traditional keyword matching methods, the present invention can support more complex query requirements, such as retrieving multimodal archives through natural language descriptions, or retrieving all related entities and relationships through archive codes. It significantly improves the flexibility and intelligence level of archive retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of the method for constructing and retrieving an archive knowledge base based on multimodal data fusion in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the following provides a more detailed description of the present invention through specific embodiments and in conjunction with the accompanying drawings.

[0036] Referring to Figure 1 , in an embodiment of the present invention, a method for constructing and retrieving an archive knowledge base based on multimodal data fusion includes the following steps:

[0037] S1. Construct a multimodal archive knowledge base

[0038] Hardware and software preparation: To construct a multimodal archive knowledge base platform, high-performance computing resources need to be prepared, including servers, storage devices, and network facilities. In the present invention, an Intel Xeon server based on the x86 architecture is selected, and a solid-state drive is used as the storage device to meet the storage requirements of multimodal data. In addition, the present invention configures virtualization software KVM and management tool VMware vCenter Server to achieve efficient resource management and dynamic allocation. An ultra-converged database (MatrixOne) with a cloud-native architecture is adopted, which supports the efficient storage and retrieval of multimodal data. It supports the Serverless computing mode, realizes on-demand allocation and elastic expansion of resources, and reduces operation and maintenance costs.

[0039] The multi-source heterogeneous archive data involved in the present invention mainly includes:

[0040] Structured data: Such as archive codes, metadata (unique identifiers, content types, original formats, creation dates, version numbers) in databases, which are directly accessed through SQL query interfaces.

[0041] Semi-structured data: Such as archive description files in JSON or XML formats, which are parsed and extracted through parsers.

[0042] Unstructured data: Such as images (scanned copies, photos), audio (recorded files), etc., which are accessed through file system interfaces or streaming media APIs.

[0043] S1.1: First, perform data processing, feature extraction, and generation of metadata: The sources of multimodal archive data are extensive and the formats are diverse. Therefore, data cleaning, format conversion, and metadata generation are required. The main steps include:

[0044] 1.1.1 Data cleaning: Perform operations such as removing noise, filling missing values, and unifying timestamps on the heterogeneous archive data.

[0045] 1.1.2 Feature extraction: Select appropriate models according to different data modalities. In the present invention, a convolutional neural network (CNN) is used to process image data, a Transformer architecture is used to process text data, and for audio data, it is converted into text using speech recognition and processed in the same way as text.

[0046] 1.1.3 Metadata Generation: Through the feature extraction in the previous step, the different modality data form unified feature vectors. After these feature vectors are stored, based on the archival encoding association, metadata is constructed to represent the association between the feature vectors of the archival data and the archival entities. The metadata includes a unique identifier, content type, original format, creation date, and version number.

[0047] S1.2, Knowledge Graph Construction: Store the extracted entities and relationships into the graph database Neo4j. The following steps are included:

[0048] 1.2.1 Graph Database Initialization: Create an instance of the graph database and define the types of nodes and relationships. Define node types such as documents, keywords, authors, etc., and relationship types such as "contains", "associates with", "created in" in Neo4j. Note that the archival encoding should be used as an independent node in the knowledge graph and connected to modality data such as text, images, and audio through relationship edges.

[0049] 1.2.2 Data Import: Use the Cypher language to import entities and relationships into the graph database.

[0050] 1.2.3 Multi-modal Data Storage: Store the feature vectors and metadata of data such as text, images, and audio as node attributes to support the unified management of multi-modal data.

[0051] S1.3, Knowledge Graph Optimization: Optimize to improve the performance and usability of the knowledge graph. The optimization steps are as follows:

[0052] 1.3.1 Entity Disambiguation: In archival data, the same entity may appear under different names or forms, or different entities may have the same name. These ambiguities will affect the accuracy and retrieval efficiency of the knowledge graph. The disambiguation method of the present invention is context information disambiguation and external knowledge base DBpedia disambiguation. Context information disambiguation distinguishes homonymous entities with different features by comparing the context features of entities (such as time, location, associated persons, etc.) to determine the uniqueness of the entity. Ensure the uniqueness of entities in the knowledge graph. For ambiguity problems that cannot be solved by context information, the present invention introduces the external knowledge base DBpedia for disambiguation. The specific steps include: matching the entities in the archives with the entries in the external knowledge base. For example, find the corresponding unique entity in DBpedia through information such as the name, time, and location of the entity. Subsequently, use the entity descriptions and association relationships in the external knowledge base to compare the entity information to further confirm the uniqueness of the entity in the archives.

[0053] 1.3.2 Relationship Enhancement: Use a pre-trained model to extract explicit entity relationships from text archives, and combine the metadata encoded in the archives to construct a preliminary knowledge graph. For implicit relationships in the archives (such as event associations, person relationships, etc.), the pre-trained model performs semantic analysis on multi-modal archive data to generate potential entity relationships and verify the potential relationships. The main methods include context consistency matching and external knowledge base matching. Finally, the newly generated relationships are integrated with the existing knowledge graph to achieve the update and optimization of the knowledge graph.

[0054] S2, Multi-modal Data Fusion and Indexing

[0055] S2.1 Feature Alignment and Fusion:

[0056] 2.1.1 Embedding Mapping: The features extracted by the present invention are mapped to a high-dimensional shared semantic space through an embedding layer. The parameters of the embedding layer are optimized through contrastive learning to ensure that the features of different modalities have similar distributions in the semantic space. The implementation method of contrastive learning in the present invention is as follows:

[0057] Construct positive and negative sample pairs and calculate embedding vectors: Samples in the same set of multi-modal data (such as a picture of a cat and the text "This is a cat") form positive sample pairs, and samples in different sets of multi-modal data (such as a picture of a cat and the text "This is a dog") form negative sample pairs. The samples of each modality are mapped to the shared semantic space through the corresponding embedding layer to obtain their embedding vectors.

[0058] Calculate the contrastive loss: Calculate the InfoNCE contrastive loss function. The smaller the loss value, the closer the feature distances of the positive sample pairs, the farther the feature distances of the negative sample pairs, and the stronger the feature representation ability of the model.

[0059] Update the parameters of the embedding layer: Starting from the loss function, use the backpropagation algorithm to calculate the gradient of the loss function with respect to the parameters of the embedding layer. The gradient represents the direction and degree of change of the loss function value with respect to the parameters. Finally, use the Adam optimization algorithm to update the parameters of the embedding layer according to the gradient.

[0060] 2.1.2 Multi-modal Fusion Model: The present invention uses a self-attention mechanism within each modality to capture long-range dependencies within the modality, enhance the expressive ability of features, and calculate an attention weight matrix for the feature representation of each modality. The specific formula is as follows:

[0061]

[0062] Among them, Q, K, and V represent the Query, Key, and Value matrices respectively, and d kis the dimensionality scaling factor. Through the self-attention mechanism, each feature position within a modality can be weighted and combined based on the features of other positions, thereby capturing long-range dependencies and enhancing the expressive power of the modality features.

[0063] Introduce a shared attention mechanism between different modalities. In the present invention, by sharing attention weights, different modalities share the same attention weight matrix, and the formula is as follows:

[0064]

[0065] where Q m is the query matrix of modality m, K m and V m are the key and value matrices of modality m'. By sharing attention weights, the information interaction between modalities is more efficient, and the computational complexity is significantly reduced. For weak modality data (such as low-quality images or low-clarity audio), the shared attention mechanism can supplement information through the attention weights of strong modalities (such as high-quality text), which can enhance the feature expression ability of weak modality data.

[0066] Regarding the specific processing strategy for archive features, the present invention introduces the following specific strategies in the self-attention and shared attention mechanisms to improve the processing effect of archive data: Use the unique identifier of the archive encoding as the benchmark for modality alignment. By sharing the attention weights of the archive encoding, ensure that modalities such as text, images, and audio of the same archive are aligned in the semantic space; Dynamically adjust the attention weight allocation according to the content type (such as text, image, audio) and original format of the archive. For example, for image data, the attention weights pay more attention to the visual feature regions, and for text data, the attention weights pay more attention to keywords and semantic information; Achieve temporal alignment of different versions of the archive through the version number and creation date in the archive encoding. For example, different versions of the same archive can be aligned in the semantic space through the shared attention mechanism to ensure the timeliness and version consistency of the retrieval results.

[0067] 2.1.3 Unified multi-modal knowledge representation: Through the above feature alignment and fusion steps, the present invention generates a unified multi-modal knowledge representation. This representation can capture the semantic associations between different modalities and provide a basis for subsequent retrieval and knowledge generation. For example, for an archive containing an image and a text description, the generated unified knowledge representation can simultaneously reflect the visual features of the image and the semantic content of the text.

[0068] S2.2 Index construction and optimization:

[0069] 2.2.1 Multimodal Data Segmentation and Vectorization: The present invention segments multimodal data into appropriately sized fragments. For text data, it is segmented by paragraphs or sentences; for image data, it is segmented by regions. For the segmented data fragments, the present invention uses a pre-trained model to convert them into vector representations. These vectors can capture the semantic information of the data and provide a basis for subsequent similarity retrieval.

[0070] 2.2.2 Selection and Integration of Vector Database: The present invention selects MilvusStore as the vector database, which has the ability to efficiently store and quickly retrieve multimodal vector data. MilvusStore supports multiple data types (such as floating-point numbers, binary vectors) and index structures, and can meet the storage and retrieval requirements of large-scale data.

[0071] 2.2.3 Index Structure Construction: The present invention adopts a hierarchical index (HNSW) structure. The construction method is as follows: First, determine the number of nodes and connection density at each level, ensuring that the high-level graph is sparse enough to accelerate retrieval, and the low-level graph is dense enough to ensure accuracy. Subsequently, construct the low-level graph, which contains the adjacency relationships of all data points. Each node uses a greedy search algorithm to ensure that its neighbor nodes are its nearest neighbors, forming a dense graph structure. On the basis of the low-level graph, construct the high-level graph layer by layer. The high-level graph is a sparsified version of the low-level graph, and the number of nodes decreases layer by layer. The construction of the high-level graph is achieved through hierarchical sampling, and each node is promoted to a higher level with a certain probability.

[0072] 2.2.4 Index Structure Optimization: To further improve the retrieval performance, the present invention dynamically adjusts the index parameters (such as the number of levels, connection density) of the HNSW structure according to the data distribution and query requirements to achieve the optimal retrieval performance.

[0073] The adjustment of the number of levels is mainly based on the data distribution density and user query frequency: For high-density data regions and high-frequency query data, the system increases the number of levels to improve the retrieval accuracy and recall rate; for sparse data regions and low-frequency query data, the number of levels is reduced to reduce the computational complexity and improve the retrieval efficiency.

[0074] The adjustment of the connection density is mainly based on the distribution density of data points, and dynamically adjusts the number of connections of each node: For dense regions, increase the number of connections to improve the retrieval accuracy; for sparse regions, reduce the number of connections to reduce the computational complexity.

[0075] Through the hierarchical index structure and dynamic index parameter adjustment technology, the problem of low retrieval efficiency of existing archival retrieval systems in processing large-scale multimodal data is solved, ensuring efficient retrieval performance in different scenarios.

[0076] S3, Intelligent Retrieval and Knowledge Generation

[0077] The present invention utilizes knowledge graph technology to implement a semantic retrieval function based on file encoding, metadata, and graph retrieval. Compared with traditional keyword matching methods, the present invention can support more complex query requirements. For example, it can retrieve multimodal files through natural language descriptions, or retrieve all related entities and relationships through file encoding. This significantly improves the flexibility and intelligence level of file retrieval.

[0078] S3.1 Application of Retrieval-Augmented Generation (RAG) technology:

[0079] 3.1.1 Vector retrieval and re-ranking: In the retrieval stage, the present invention utilizes Retrieval-Augmented Generation (RAG) technology to obtain the document fragments most relevant to the user query through vector retrieval. The present invention uses the EmbeddingRetriever module to convert the user query and document fragments into vector representations and perform retrieval in the HNSW graph. The retrieval starts from the top layer of the graph and quickly locates the approximate nearest neighbors of the query vector through greedy search. As the retrieval process progresses to the lower layers of the graph, the search scope is gradually refined until the candidate set closest to the query vector is found. After the retrieval reaches the bottom layer of the graph, an exact search is performed to find the candidate vector most similar to the query vector. The retrieval results are re-ranked according to the relevance of the candidate vectors to ensure that the most relevant fragments are returned first.

[0080] 3.1.2 Generating answers relying on large language models: After the relevant candidate vectors are retrieved in the file knowledge base implemented by the present invention, these vectors come from multiple heterogeneous data sources. The text generation ability of the large language model is used to generate semantically complete and structurally clear answers from a semantic perspective for these information, which further facilitates the user to use the retrieved information. The knowledge base of the present invention provides session-level feedback retrieval, and the user continuously asks questions to the large model to obtain more accurate and rich results.

[0081] S3.2 Knowledge graph query and reasoning:

[0082] The present invention makes implicit knowledge in file data explicit through multimodal data fusion and knowledge graph construction, supporting the in-depth development and utilization of files. For example, by mining the potential associations between entities and relationships in the knowledge graph, implicit patterns and knowledge in file data can be discovered, providing data support for fields such as historical research and cultural inheritance.

[0083] 3.2.1 Complex queries based on the knowledge graph: The present invention supports query functions based on the knowledge graph and realizes the retrieval of complex relationships through the query language of the graph database (such as Cypher). Users can query entities and relationships in the knowledge graph through natural language or structured query language. For example, users can query all documents related to a certain keyword, or query all entities and their relationships mentioned in a certain document.

[0084] 3.2.2 Knowledge Graph Reasoning Function: The present invention also supports the knowledge graph reasoning function, which can derive new knowledge based on the entities and relationships in the existing knowledge graph. The present invention uses a variety of graph reasoning algorithms (such as path finding, rule reasoning, embedding reasoning, etc.) to mine potential relationships from the knowledge graph. The specific algorithms include:

[0085] Path Finding Algorithm: By finding the paths between entities, potential association relationships are discovered. For example, by finding the path between "Zhang San" and "a certain project", it is found that "Zhang San" is associated with "a certain project" through the "participation" relationship, and at the same time, "a certain project" is associated with "1980" through the "founded in" relationship. Such paths can discover implicit timelines and event chains.

[0086] Rule Reasoning Algorithm: Based on predefined rules, new relationships are derived from the knowledge graph (such as "if A participates in B, and B is founded in C, then there is a time association between A and C"). These rules can be defined manually or automatically generated through learning algorithms.

[0087] Embedding Reasoning Algorithm: Utilize graph embedding technology to map the entities and relationships in the knowledge graph to a vector space, and predict potential relationships through vector operations (such as vector addition, similarity calculation).

[0088] 3.2.3 Dynamic Update and Maintenance: To ensure the timeliness and accuracy of the knowledge graph, the present invention supports the dynamic update and maintenance of the knowledge graph. The system will regularly check and update the entities and relationships in the knowledge graph, and make adjustments according to the latest data and user feedback. In addition, the present invention also provides detailed update logs and operation guides to ensure that users can timely understand the changes in the knowledge graph.

[0089] Through automated data processing, feature extraction, and knowledge graph construction technologies, the present invention reduces the dependence on manual annotation and management, and can significantly reduce the cost of file management. In addition, the modular design of the present invention supports dynamic expansion, can adapt to the file management needs of different scales and types, and improves the scalability and adaptability of the system.

[0090] Another embodiment of the present invention provides an archive knowledge base construction and retrieval system based on multi-modal data fusion, including:

[0091] Multi-modal Archive Knowledge Base Construction Module: Perform cleaning operations on heterogeneous archive data, use a Convolutional Neural Network (CNN) to extract image features, use a Transformer network to extract text features, convert audio data through speech recognition into text and then extract text features in the same way, generate a unified feature vector, and based on archive coding association, construct metadata to represent the association between the feature vector of archive data and archive entities; create an instance of a graph database, define node types and relationship types, import entities and relationships into the graph database, and store the feature vector and metadata as node attributes;

[0092] Multi-modal Data Fusion and Indexing Module: Map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss and update the parameters of the embedding layer; use the self-attention mechanism within each modality and introduce a shared attention mechanism between different modalities, use the unique identifier of the archive coding as the modality alignment benchmark, and dynamically adjust the attention weight allocation according to the archive characteristics to generate a unified multi-modal knowledge representation; segment and convert the multi-modal data into vector representations, store them in a vector database, adopt a hierarchical index HNSW structure and dynamically optimize the index structure according to data distribution and query requirements;

[0093] Intelligent Retrieval and Knowledge Generation Module: Use the Retrieval-Augmented Generation (RAG) technology, obtain the document fragments most relevant to the user query through vector retrieval, generate answers using a large language model after obtaining candidate vectors, and provide session-level feedback retrieval; support complex queries based on the knowledge graph through the query language of the graph database, use a variety of graph inference algorithms to mine potential relationships from the knowledge graph, and regularly check and update the knowledge graph.

[0094] It should be understood that the archive knowledge base construction and retrieval system based on multi-modal data fusion in the embodiments of the present invention can implement all the technical solutions in the above method embodiments. The functions of its various components can be specifically implemented according to the method of constructing and retrieving an archive knowledge base based on multi-modal data fusion in the above method embodiments. The specific implementation process can refer to the relevant descriptions in the above embodiments and will not be elaborated here.

[0095] The present invention also provides a computer device, including: one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the program is executed by the processor, it implements the steps of the method for constructing and retrieving an archive knowledge base based on multi-modal data fusion as described above.

[0096] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method for constructing and retrieving an archive knowledge base based on multi-modal data fusion as described above.

[0097] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, an apparatus (system), a computer device, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0098] The present invention is described with reference to the flowchart of the method according to the embodiments of the present invention. It should be understood that each process in the flowchart and the combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes.

[0099] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes.

Claims

1. A method for constructing and retrieving an archive knowledge base based on multi-modal data fusion, characterized in that It includes the following steps: Construct a multi-modal archive knowledge base: perform cleaning operations on heterogeneous archive data, use a Convolutional Neural Network (CNN) to extract image features, use a Transformer network to extract text features, convert audio data into text through speech recognition and then extract text features in the same way, generate unified feature vectors, and based on archive coding association, construct metadata to represent the association between the feature vectors of archive data and archive entities; Create a graph database instance, define node types and relationship types, import entities and relationships into the graph database, and store feature vectors and metadata as node attributes; Multi-modal data fusion and indexing: map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss and update the parameters of the embedding layer; use the self-attention mechanism within each modality and introduce a shared attention mechanism between different modalities, use the unique identifier of the archive coding as the modality alignment benchmark, dynamically adjust the attention weight distribution according to archive characteristics, and generate a unified multi-modal knowledge representation; segment and convert multi-modal data into vector representations, store them in a vector database, adopt a hierarchical index HNSW structure and dynamically optimize the index structure according to data distribution and query requirements; Intelligent retrieval and knowledge generation: use Retrieval-Augmented Generation (RAG) technology, obtain the most relevant document fragments to the user query through vector retrieval, generate answers using a large language model after obtaining candidate vectors, and provide session-level feedback retrieval; support complex queries based on the knowledge graph through the query language of the graph database, use various graph reasoning algorithms to mine potential relationships from the knowledge graph, and regularly check and update the knowledge graph.

2. The method according to claim 1, wherein The metadata includes: Unique identifier, content type, original format, creation date, version number.

3. The method according to claim 1, characterized in that When constructing the knowledge graph, perform structured processing on the archive coding, define it as an independent node in the knowledge graph, and its structure is <unique identifier, content type, original format, creation date, version number>, and import the feature vectors of text, image, and audio modality data; Extract text, image, and audio entity modality data from the original archive set, and extract the relationships between modality data; Use the archive coding as the central node and connect it to the entity nodes of text, image, and audio modality data through relationship edges.

4. The method according to claim 3, wherein Optimize after the knowledge graph is constructed, including: entity disambiguation, including context information disambiguation and external knowledge base DBpedia disambiguation; and use a pre-trained model to extract potential entity relationships from text archives and integrate them with the existing knowledge graph.

5. The method according to claim 1, wherein Map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss and update the parameters of the embedding layer, including: Map the extracted features to a high-dimensional shared semantic space through the embedding layer, and optimize the parameters of the embedding layer through contrastive learning, so that the features of different modalities have similar distributions in the semantic space. The implementation method of contrastive learning is as follows: Construct positive and negative sample pairs and calculate embedding vectors: Samples in the same set of multimodal data form positive sample pairs, and samples in different sets of multimodal data form negative sample pairs. The samples of each modality are mapped to the shared semantic space through the corresponding embedding layer to obtain their embedding vectors; Calculate the contrastive loss: Calculate the InfoNCE contrastive loss function; Update the parameters of the embedding layer: Starting from the loss function through the backpropagation algorithm, calculate the gradient of the loss function with respect to the parameters of the embedding layer, and use the Adam optimization algorithm to update the parameters of the embedding layer according to the gradient.

6. The method according to claim 2, wherein The following specific strategies are introduced in the self-attention and shared attention mechanisms to improve the processing effect of archival data: Use the unique identifier of the archival encoding as the benchmark for modality alignment. Through the sharing of attention weights with the archival encoding, the text, image, and audio modalities of the same archive are aligned in the semantic space; Dynamically adjust the attention weight distribution according to the content type and original format of the archive; Achieve time alignment of different versions of the archive through the version number and creation date in the archival encoding.

7. The method according to claim 1, characterized in that, Segment and convert multimodal data into vector representations, including: For text data, segment it by paragraph or sentence; for image data, segment it by region; for the segmented data fragments, use a pre-trained model to convert them into vector representations.

8. An archive knowledge base construction and retrieval system based on multimodal data fusion, characterized in that, Include Multimodal archival knowledge base construction module: Perform cleaning operations on various heterogeneous archival data, use a convolutional neural network (CNN) to extract image features, use a Transformer network to extract text features, convert audio data into text through speech recognition and then perform the same processing to extract text features, generate unified feature vectors, and construct metadata based on archival encoding to represent the association between the feature vectors of archival data and archival entities; Create a graph database instance, define node types and relationship types, import entities and relationships into the graph database, and store the feature vectors and metadata as node attributes; Multimodal data fusion and indexing module: Map the extracted features to a high-dimensional shared semantic space, construct positive and negative sample pairs, calculate the contrastive loss and update the parameters of the embedding layer; Use the self-attention mechanism within each modality and introduce a shared attention mechanism between different modalities. Use the unique identifier of the archival encoding as the modality alignment benchmark, dynamically adjust the attention weight distribution according to the archival characteristics, and generate a unified multimodal knowledge representation; Segment and convert multimodal data into vector representations, store them in a vector database, adopt a hierarchical index (HNSW) structure, and dynamically optimize the index structure according to data distribution and query requirements; Intelligent retrieval and knowledge generation module: Use retrieval-augmented generation (RAG) technology to obtain the most relevant document fragments to the user query through vector retrieval. After obtaining the candidate vectors, use a large language model to generate answers and provide session-level feedback retrieval; Support complex queries based on the knowledge graph through the query language of the graph database, use various graph reasoning algorithms to mine potential relationships from the knowledge graph, and regularly check and update the knowledge graph.

9. A computer device, characterized in that, The computer device includes: one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, and when the program is executed by the processor, it implements the steps of the method for constructing and retrieving an archive knowledge base based on multimodal data fusion as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for constructing and retrieving an archive knowledge base based on multimodal data fusion as described in any one of claims 1-7.

Images