Archive data intelligent retrieval method and system based on multi-modal large model

Abstract

The invention belongs to the technical field of intelligent retrieval, and discloses an archive data intelligent retrieval method and system based on a multi-modal large model. The method comprises the following steps that a cloud data center builds an archive data intelligent retrieval platform, deploys a block chain network and builds an archive data intelligent retrieval engine; the cloud data center is used for carrying out data processing on the real-time multi-modal file big data; the cloud data center is used for generating a knowledge graph according to a plurality of real-time retrieval themes; the cloud data center is used for carrying out data clustering; the cloud data center uses a block chain network to carry out distributed storage; the user terminal is used for uploading real-time query data; the cloud data center is used for performing intelligent retrieval by using an archive data intelligent retrieval engine; and the cloud data center visually displays the target archive data. According to the method, the problems of low retrieval efficiency, insufficient accuracy, weak multi-modal data processing capability and insufficient security in the prior art are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent retrieval, and particularly relates to an intelligent retrieval method and system for archival data based on a multimodal large model. Background Art

[0002] With the rapid development of information technology, the digitization of archives has become an irreversible trend. Digitization not only improves the storage, retrieval, and utilization efficiency of archives, but also makes archival information easier to share and disseminate. With the explosive growth of archival data, higher requirements are put forward for the retrieval technology of archival data to ensure more efficient, convenient, and accurate archival information services for users.

[0003] The existing archival data retrieval technologies have the following defects:

[0004] 1) Low retrieval efficiency: Traditional archival data retrieval mainly relies on keyword matching. Facing a large amount of archival data, the retrieval speed is slow and the efficiency is low, making it difficult to quickly locate specific archival materials, especially in the case of a large amount of data;

[0005] 2) Insufficient accuracy: Keyword retrieval is easily affected by factors such as lexical diversity and semantic ambiguity, resulting in inaccurate retrieval results and difficulty in effectively dealing with complex language phenomena such as polysemy, synonyms, and antonyms, causing retrieval errors;

[0006] 3) Weak multimodal data processing ability: Most of the existing technologies are aimed at single-modal data (such as text or images), lacking the comprehensive processing ability for multimodal data (such as text, images, audio, etc.), making it difficult to fully utilize the rich information in multimodal data and restricting the in-depth understanding and mining of archival data;

[0007] 4) Insufficient security: The existing technologies are vulnerable to security threats such as data leakage and tampering during data storage and transmission, lacking effective data security guarantee mechanisms and making it difficult to ensure the integrity and credibility of archival data. Summary of the Invention

[0008] In order to solve the problems of low retrieval efficiency, insufficient accuracy, weak multimodal data processing ability, and insufficient security existing in the prior art, the purpose of the present invention is to provide an intelligent retrieval method and system for archival data based on a multimodal large model.

[0009] The technical solution adopted by the present invention is as follows:

[0010] An intelligent retrieval method for archival data based on a multimodal large model includes the following steps:

[0011] The cloud data center builds an intelligent retrieval platform for archival data, deploys a blockchain network, uses a multi-modal large model algorithm to construct an intelligent retrieval engine for archival data, and connects it to the intelligent retrieval platform for archival data;

[0012] The cloud data center uses the intelligent retrieval engine for archival data to process the collected real-time multi-modal archival big data, and obtains a number of standard real-time multi-modal archival data with real-time retrieval tags set;

[0013] The cloud data center uses the intelligent retrieval engine for archival data to generate knowledge graphs according to a number of real-time retrieval topics, and obtains real-time topic knowledge graphs for a number of real-time retrieval topics;

[0014] The cloud data center performs data clustering on a number of standard real-time multi-modal archival data with real-time retrieval tags set according to a number of real-time topic knowledge graphs, and obtains a number of real-time topic clustering archival data clusters;

[0015] The cloud data center uses the blockchain network to perform distributed storage on a number of real-time topic clustering archival data clusters and the corresponding real-time retrieval topics;

[0016] The user terminal accesses the intelligent retrieval platform for archival data, and uploads the user's real-time query data to the cloud data center through the intelligent retrieval platform for archival data;

[0017] The cloud data center uses the intelligent retrieval engine for archival data to perform intelligent retrieval in a number of real-time topic clustering archival data clusters and the corresponding real-time retrieval topics of the blockchain network according to the real-time query data, and obtains the target archival data;

[0018] The cloud data center visually displays the target archival data on the user terminal through the intelligent retrieval platform for archival data.

[0019] Furthermore, the cloud data center builds an intelligent retrieval platform for archival data, deploys a blockchain network, uses a multi-modal large model algorithm to construct an intelligent retrieval engine for archival data, and connects it to the intelligent retrieval platform for archival data, including the following steps:

[0020] The cloud data center builds the framework of the intelligent retrieval platform for archival data, and constructs a user login module, a query data input module, and an archival data visualization module in the framework of the intelligent retrieval platform for archival data to obtain the intelligent retrieval platform for archival data;

[0021] Distributively connect a number of data servers in the cloud data center, and set up an IPFS system and a smart contract to obtain a blockchain network;

[0022] Based on the big data of multimodal training samples, use the multimodal large model algorithm to construct an intelligent retrieval engine for archival data, and connect the intelligent retrieval engine for archival data to the intelligent retrieval platform for archival data.

[0023] Furthermore, based on the big data of multimodal training samples, use the multimodal large model algorithm to construct an intelligent retrieval engine for archival data, and connect the intelligent retrieval engine for archival data to the intelligent retrieval platform for archival data, including the following steps:

[0024] Collect the big data of multimodal training samples and a number of historical knowledge data of different preset retrieval topics, and perform preprocessing to obtain a number of preprocessed multimodal training samples and a number of preprocessed historical knowledge data;

[0025] According to a number of preprocessed multimodal archival data, use the multimodal data processing algorithm to construct a multimodal data processing model, and generate a number of standard multimodal training samples;

[0026] According to a number of preprocessed historical knowledge data, use the algorithm that fuses natural language processing and deep learning to construct a knowledge graph generation model;

[0027] According to a number of standard multimodal training samples, use the large model algorithm to construct a multimodal data feature extraction model, and generate a number of multimodal training sample features;

[0028] According to a number of multimodal training sample features, use the multimodal feature fusion algorithm to construct an archival data retrieval label generation model, and generate the historical retrieval labels of each standard multimodal training sample;

[0029] According to the historical retrieval labels of a number of standard multimodal training samples and randomly generated historical query data, use the deep learning algorithm to construct an intelligent retrieval model for archival data;

[0030] Integrate the multimodal data processing model, the multimodal data feature extraction model, the knowledge graph generation model, the archival data retrieval label generation model, and the intelligent retrieval model for archival data to obtain an intelligent retrieval engine for archival data, and connect the intelligent retrieval engine for archival data to the intelligent retrieval platform for archival data.

[0031] Furthermore, the multimodal data processing model is constructed based on the PPO-DPA algorithm;

[0032] The knowledge graph generation model is constructed based on the BERT-CRF-SVM-cGAN algorithm;

[0033] The multimodal data feature extraction model is constructed based on the BERT-FPN-CNN-Transformer algorithm;

[0034] The file data retrieval label generation model is constructed based on the RF-MLP algorithm;

[0035] The intelligent file data retrieval model is constructed based on the LSTM-DBN-MLP algorithm.

[0036] Furthermore, the cloud data center uses the intelligent file data retrieval engine to process the collected real-time multi-modal file big data, and obtains a number of real-time standard multi-modal file data with real-time retrieval labels, including the following steps:

[0037] Collect real-time multi-modal file big data, and preprocess the real-time multi-modal file big data to obtain a number of preprocessed real-time multi-modal file data;

[0038] Use the multi-modal data processing model of the intelligent file data retrieval engine to process each preprocessed real-time multi-modal file data to obtain the corresponding standard real-time multi-modal file data;

[0039] Use the multi-modal data feature extraction model of the intelligent file data retrieval engine to extract the real-time multi-modal file data features of the standard real-time multi-modal file data;

[0040] According to the real-time multi-modal file data features, use the file data retrieval label generation model of the intelligent file data retrieval engine to generate intelligent retrieval labels, and obtain the corresponding real-time retrieval labels;

[0041] Traverse all preprocessed real-time multi-modal file data to obtain a number of real-time standard multi-modal file data with real-time retrieval labels.

[0042] Furthermore, the cloud data center uses the intelligent file data retrieval engine to generate a knowledge graph according to a number of real-time retrieval topics, and obtains the real-time topic knowledge graph of a number of real-time retrieval topics, including the following steps:

[0043] The cloud data center collects a number of real-time knowledge data of different real-time retrieval topics, and preprocesses the number of real-time knowledge data to obtain a number of preprocessed real-time knowledge data of different real-time retrieval topics;

[0044] According to a number of preprocessed real-time knowledge data of the same real-time retrieval topic, use the knowledge graph generation model of the intelligent file data retrieval engine to generate a knowledge graph, and obtain the real-time topic knowledge graph;

[0045] Traverse all preprocessed real-time knowledge data of a number of real-time retrieval topics to obtain the real-time topic knowledge graph of a number of real-time retrieval topics.

[0046] Further, the cloud data center clusters the data of a number of standard real-time multi-modal archive data with real-time retrieval tags according to a number of real-time topic knowledge graphs, and obtains a number of real-time topic clustering archive data clusters, including the following steps:

[0047] The cloud data center obtains the similarity between the real-time retrieval tag of each standard real-time multi-modal archive data and all real-time topic knowledge graphs;

[0048] The standard real-time multi-modal archive data is divided into the real-time topic knowledge graph with the highest similarity to the real-time retrieval tag, and a number of standard real-time multi-modal archive data belonging to different real-time topic knowledge graphs are obtained;

[0049] A number of standard real-time multi-modal archive data belonging to the same real-time topic knowledge graph are used as the real-time topic clustering archive data cluster of the corresponding real-time topic knowledge graph.

[0050] Further, the cloud data center uses the blockchain network to perform distributed storage on a number of real-time topic clustering archive data clusters and the corresponding real-time retrieval topics, including the following steps:

[0051] A number of real-time topic clustering archive data clusters are stored in the IPFS system of the blockchain network to obtain a number of real-time data hash values;

[0052] According to the real-time data hash value and real-time timestamp of each real-time topic clustering archive data cluster, the smart contract of the blockchain network is used to generate corresponding real-time transaction data;

[0053] A number of data servers with distributed connections of the blockchain network are used to perform consensus on a number of real-time transaction data. After successful consensus, a number of real-time consensus records and corresponding real-time storage records are generated;

[0054] The real-time consensus record, real-time storage record of each real-time topic clustering archive data cluster, a number of real-time retrieval tags, and the corresponding real-time retrieval topic are synchronized to the distributed ledger of the blockchain network.

[0055] Further, the cloud data center uses the archive data intelligent retrieval engine to perform intelligent retrieval in a number of real-time topic clustering archive data clusters and the corresponding real-time retrieval topics of the blockchain network according to the real-time query data, and obtains the target archive data, including the following steps:

[0056] The cloud data center obtains the similarity between the real-time query data and the real-time topic knowledge graph of a number of real-time retrieval topics;

[0057] The real-time retrieval topic corresponding to the real-time topic knowledge graph with the highest similarity is used as the target real-time retrieval topic of the real-time query data;

[0058] Extract several alternative real-time retrieval tags corresponding to the target real-time retrieval topic in the distributed ledger of the blockchain network;

[0059] Use the archive data intelligent retrieval model of the archive data intelligent retrieval engine to perform intelligent retrieval based on several alternative real-time retrieval tags and real-time query data to obtain the target real-time retrieval tag;

[0060] Extract the target real-time storage record of the target real-time retrieval tag in the distributed ledger of the blockchain network, and extract the target real-time data hash value from several data servers in the blockchain network according to the target real-time storage record;

[0061] Extract the corresponding target archive data from the IPFS system of the blockchain network according to the target real-time data hash value.

[0062] An archive data intelligent retrieval system based on a multi-modal large model for implementing an archive data intelligent retrieval method. The system includes a cloud data center and several user terminals. Several user terminals are all communicatively connected to the cloud data center. The cloud data center is provided with an archive data intelligent retrieval platform and an archive data intelligent retrieval engine, and the cloud data center includes an initialization unit, a data processing unit, a knowledge graph generation unit, a data clustering unit, a distributed storage unit, an intelligent retrieval unit, and a visualization display unit connected in sequence.

[0063] The beneficial effects of the present invention are as follows:

[0064] A method and system for intelligent retrieval of archive data based on a multi-modal large model provided by the present invention, through the multi-modal large model algorithm and efficient data processing mechanism of the archive data intelligent retrieval engine, realizes fast retrieval, greatly shortens the retrieval time, effectively deals with a large amount of archive data, quickly locates the target materials, and improves work efficiency; uses the multi-modal feature extraction and deep learning technology of the archive data intelligent retrieval engine to accurately understand the query intention, reduces the error caused by semantic ambiguity, effectively processes complex language phenomena such as polysemous words and synonyms, and improves the accuracy of retrieval results; through the archive data intelligent retrieval engine comprehensively processes various modal data such as text, images, and audio, improves the multi-modal data processing ability, makes full use of the rich content in the multi-modal information, deeply excavates the potential value of the archive data, and improves the data utilization efficiency; deploys a blockchain network to ensure the security and integrity of data during storage and transmission, and prevents data leakage and tampering through distributed ledger and hash value verification; the cloud data center provides an archive data intelligent retrieval platform to support real-time data update and processing, and meets the retrieval requirements for real-time archive data.

[0065] Other beneficial effects of the present invention will be further described in the specific implementation manner. Brief Description of the Drawings

[0066] Figure 1 It is a flowchart of the intelligent retrieval method for archive data based on a multimodal large model in the present invention.

[0067] Figure 2 It is a structural block diagram of the intelligent retrieval system for archive data based on a multimodal large model in the present invention. Detailed implementation manners

[0068] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

[0069] Embodiment 1:

[0070] As Figure 1 shown, this embodiment provides an intelligent retrieval method for archive data based on a multimodal large model, including the following steps:

[0071] S1: In the cloud data center, build an intelligent retrieval platform for archive data, deploy a blockchain network, use a multimodal large model algorithm to construct an intelligent retrieval engine for archive data, and connect it to the intelligent retrieval platform for archive data, including the following steps:

[0072] S1-1: In the cloud data center, build a framework for the intelligent retrieval platform for archive data, and construct a user login module, a query data input module, and an archive data visualization module in the framework of the intelligent retrieval platform for archive data to obtain the intelligent retrieval platform for archive data;

[0073] The user login module is used to collect the login information of the user, and based on the login information, verify the login of the user. After the login verification is passed, the query data input module and the archive data visualization module of the intelligent retrieval platform for archive data are enabled; the query data input module is used to collect the real-time query data of the user and upload the real-time query data of the user to the cloud data center; the archive data visualization module is used to visually display the target archive data sent by the cloud data center on the user terminal;

[0074] S1-2: Connect several data servers in the cloud data center distributively, and set up the InterPlanetary File System (IPFS) and smart contracts to obtain a blockchain network;

[0075] S1-3: According to the big data of multimodal training samples, use a multimodal large model algorithm to construct an intelligent retrieval engine for archive data, and connect the intelligent retrieval engine for archive data to the intelligent retrieval platform for archive data, including the following steps:

[0076] S1-3-1: Collect big data of multimodal training samples and a number of historical knowledge data of different preset retrieval topics, and perform preprocessing to obtain a number of preprocessed multimodal training samples and a number of preprocessed historical knowledge data;

[0077] The preprocessing includes data cleaning, error data screening, format unification, etc. on the original data, improving the data quality, and providing data support for subsequent model construction;

[0078] S1-3-2: According to a number of preprocessed multimodal archive data, use multimodal data processing algorithms to construct a multimodal data processing model, and generate a number of standard multimodal training samples;

[0079] The multimodal data processing model is constructed based on the Optimized Proximal Policy Optimization - Data Processing Algorithm (MOPPO - DPA) algorithm, and the multimodal data processing model includes a multimodal archive information parsing module, a data processing strategy generation module constructed based on the PPO algorithm, and a data processing algorithm library constructed based on a number of DPA algorithms connected in sequence. The data processing strategy generation module includes an experience replay pool, an Actor network, a Critic network, and an agent, and the agent is respectively connected to the experience replay pool, the Actor network, and the Critic network;

[0080] The multi-modal archive information parsing module is used to parse the multi-modal archive information in the multi-modal archive data to obtain the data information of all modalities included in the multi-modal archive data. For example, the data information of the image modality archive data, the data information of the audio modality archive data, and the data information of the text modality archive data; The Actor network is responsible for outputting the probability distribution of the actions that should be taken in a given state. The goal is to learn an optimal policy, that is, to maximize the long-term cumulative reward. In a continuous action space, the Actor network usually outputs a mean and an optional variance parameter to describe the probability distribution of the actions. The Critic network is responsible for evaluating the value of a given state, that is, predicting the expected return that can be obtained starting from this state and following the current policy, and usually outputs a scalar value representing the value of the state or the state-action value. The experience replay pool is used to store historical experiences for reuse during the training process. The agent is used to generate a data processing strategy based on the probability distribution of the actions (data processing algorithm call actions) that should be taken in a given state (the state of the multi-modal archive information) output by the Actor network; The data processing algorithm library stores several DPA algorithms, including grayscale processing, size processing, Gaussian denoising, etc. for the image modality archive data, windowing processing, filter denoising, etc. for the audio modality archive data, or magnitude normalization, data format conversion, etc. for the text modality archive data. The purpose is to convert the multi-modal raw data into a standard format so that the subsequent model can process it more effectively;

[0081] S1-3-3: According to a number of preprocessed historical knowledge data, use the natural language processing and deep learning fusion algorithm to construct a knowledge graph generation model;

[0082] The knowledge graph generation model is constructed based on the Bidirectional Encoder Representations from Transformers (BERT)-Conditional Random Fields (CRF)-Support Vector Machine (SVM)-Conditional Generative Adversarial Network (cGAN) algorithm. And the knowledge graph generation model includes a semantic feature extraction module constructed based on the BERT algorithm, a named entity recognition module constructed based on the CRF algorithm, an entity relationship recognition module constructed based on the SVM algorithm, and a knowledge graph generation module constructed based on the cGAN algorithm connected in sequence. The knowledge graph generation module includes a generator and a discriminator connected in sequence;

[0083] BERT in the semantic feature extraction module can capture the deep semantic information in the knowledge text, which is very useful for identifying different types of knowledge named entities. CRF in the named entity recognition module can consider the dependencies between adjacent knowledge named entity tags, which can help the model learn the sequence dependencies of entity tags, thereby improving the accuracy of named entity annotation and realizing the extraction of knowledge named entities. SVM in the entity relationship recognition module is mainly used to classify the relationships of the extracted named entity pairs, judge whether there is a specific relationship between them, and the type of the relationship, and convert the named entities and their context information into high-dimensional feature vectors, which can effectively represent the relationships between entities. By combining the deep semantic information extracted by BERT and the sequence dependencies considered by CRF, complex entity relationships can be processed more effectively. The task of the generator is to generate the corresponding topic knowledge graph according to the named entities recognized by the named entity recognition module and the entity relationships recognized by the entity relationship recognition module. The task of the discriminator is to judge whether the topic knowledge graph output by the generator is real, that is, whether it conforms to the real topic knowledge graph. Through this adversarial training, the generator continuously optimizes its generation ability until it can generate high-quality topic knowledge graphs;

[0084] S1-3-4: According to a number of standard multimodal training samples, use the large model algorithm to construct a multimodal data feature extraction model and generate a number of multimodal training sample features;

[0085] The multimodal data feature extraction model is constructed based on the BERT-Feature Pyramid Networks (FPN)-Convolutional Neural Networks (CNN)-Transformer algorithm, and the multimodal data feature extraction model includes a text feature extraction module constructed based on the BERT algorithm, an image feature extraction module constructed based on the FPN algorithm, an audio feature extraction module constructed based on the CNN algorithm, and a multimodal data feature combination module constructed based on the Transformer algorithm. The text feature extraction module, the image feature extraction module, and the audio feature extraction module are all connected to the multimodal data feature combination module;

[0086] The image feature extraction module can effectively extract image features at different scales by using the top-down and bottom-up paths of FPN, ensuring that various details in the archival images can be accurately detected, achieving the fusion of features at different levels, and enhancing the feature expression ability. By collecting external text big data from sources such as the Internet, databases, and file systems, which cover various topics, fields, and language styles, the text feature extraction module is pre-trained, enabling it to have strong generalization ability. The bidirectional attention mechanism is used to capture the context information in the text, making the extracted text features more rich and accurate, better adapting to different text data and tasks, and identifying key information in the text, such as keywords, phrases, sentence structures, etc., to form high-dimensional text feature vectors. The audio feature extraction module captures the time-frequency features in the audio, such as pitch, volume, rhythm, etc., through convolution and pooling operations. The convolution and pooling operations can effectively capture the time-frequency information in the audio, which is beneficial to the classification and retrieval of audio data. The multi-modal data feature combination module captures the correlation relationships between different modal features through the self-attention mechanism and the multi-head attention mechanism, enhancing the semantic understanding ability of the model. It unifies and correlates the archival data in different modalities such as text, image, and audio. The fused multi-modal features have richer information, which helps to improve the comprehensiveness and accuracy of archival data retrieval;

[0087] S1-3-5: According to the features of several multi-modal training samples, use the multi-modal feature fusion algorithm to construct an archival data retrieval label generation model and generate historical retrieval labels for each standard multi-modal training sample;

[0088] The archival data retrieval label generation model is constructed based on the Random Forest (RF)-Multilayer Perceptron (MLP) algorithm, and the archival data retrieval label generation model includes a key feature extraction module constructed based on the RF algorithm and a retrieval label generation module constructed based on the MLP algorithm, which are connected in sequence;

[0089] The key feature extraction module improves prediction accuracy by constructing multiple decision trees and aggregating their predictions. Each decision tree is trained on a random subset of the training set and can evaluate the importance of each feature for the prediction result, thereby identifying key features from the multi-modal fusion features, jointly encoding data such as images, texts, and audios, learning cross-modal semantic representations, and reducing the dimensionality of the data by extracting key features, simplifying the complexity of subsequent models. Moreover, the extraction of key features helps to remove noise and irrelevant information, improving the prediction accuracy of the model. The retrieval label generation module generates retrieval labels by learning the weights and biases between the input features and the output labels, and can adjust the number of layers and neurons of the MLP according to needs to adapt to different complexity requirements.

[0090] S1-3-6: According to the historical retrieval labels of a number of standard multi-modal training samples and the randomly generated historical query data, use deep learning algorithms to construct an intelligent retrieval model for archive data.

[0091] The intelligent retrieval model for archive data is constructed based on the Long Short-Term Memory (LSTM)-Deep Belief Network (DBN)-MLP algorithm. The intelligent retrieval model for archive data includes a query data feature extraction module constructed based on the LSTM algorithm, a retrieval label matrix feature extraction module constructed based on the DBN algorithm, and an intelligent retrieval module for archive data constructed based on the MLP algorithm. The query data feature extraction module and the retrieval label matrix feature extraction module are both connected to the intelligent retrieval module for archive data.

[0092] The query data feature extraction module uses the long short-term memory network to perform sequence feature extraction on the query data input by the user, captures the time dependence and sequence information in the query data, forms a high-dimensional query feature vector, is good at processing sequence data, can effectively capture the time dependence in the query data, ensure the integrity of the sequence information, and an accurate query feature representation helps the model better understand the user's query intention and improve the relevance of the retrieval. The retrieval label matrix feature extraction module can learn the deep features of the label matrix, enhance the expression ability of the features, support unsupervised pre-training, can initialize the network parameters using a large amount of unlabeled data, improve the generalization ability of the model, and an accurate label feature representation helps the model more accurately identify and match the retrieval labels, improving the retrieval efficiency. The intelligent retrieval module for archive data uses a multi-layer perceptron to fuse and make decisions on the query feature vector and the label feature vector, has a strong non-linear mapping ability, can learn the complex relationship between the query data and the retrieval labels, improve the retrieval accuracy, and output the retrieval label most relevant to the query data.

[0093] S1-3-7: Integrate the multimodal data processing model, the multimodal data feature extraction model, the knowledge graph generation model, the archival data retrieval tag generation model, and the archival data intelligent retrieval model to obtain an archival data intelligent retrieval engine, and connect the archival data intelligent retrieval engine to the archival data intelligent retrieval platform;

[0094] S2: Cloud data center. Use the archival data intelligent retrieval engine to process the collected real-time multimodal archival big data to obtain a number of standard real-time multimodal archival data with real-time retrieval tags set, including the following steps:

[0095] S2-1: Collect real-time multimodal archival big data and preprocess the real-time multimodal archival big data to obtain a number of preprocessed real-time multimodal archival data;

[0096] S2-2: Use the multimodal data processing model of the archival data intelligent retrieval engine to process each preprocessed real-time multimodal archival data to obtain the corresponding standard real-time multimodal archival data, including the following steps:

[0097] S2-2-1: Use the multimodal archival information parsing module of the multimodal data processing model of the archival data intelligent retrieval engine to parse the preprocessed real-time multimodal archival data to obtain real-time multimodal archival information;

[0098] S2-2-2: According to the real-time multimodal archival information, use the data processing strategy generation module of the multimodal data processing model to generate a data processing strategy to obtain a real-time data processing strategy;

[0099] S2-2-3: According to the real-time data processing strategy, extract a number of target DPA algorithms from the data processing algorithm library of the multimodal data processing model;

[0100] S2-2-4: According to a number of target DPA algorithms, process the preprocessed real-time multimodal archival data to obtain the corresponding standard real-time multimodal archival data;

[0101] S2-2-5: Traverse all preprocessed real-time multimodal archival data to obtain a number of standard real-time multimodal archival data;

[0102] S2-3: Use the multimodal data feature extraction model of the archival data intelligent retrieval engine to extract the real-time multimodal archival data features of the standard real-time multimodal archival data, including the following steps:

[0103] S2-3-1: Use the text feature extraction module of the multimodal data feature extraction model of the archival data intelligent retrieval engine to extract the real-time text features of the real-time text modal archival data in the standard real-time multimodal archival data;

[0104] S2-3-2: Use the image feature extraction module of the multi-modal data feature extraction model to extract the real-time image features of the real-time image modal archive data in the standard real-time multi-modal archive data;

[0105] S2-3-3: Use the audio feature extraction module of the multi-modal data feature extraction model to extract the real-time audio features of the real-time audio modal archive data in the standard real-time multi-modal archive data;

[0106] S2-3-4: Use the multi-modal data feature combination module of the multi-modal data feature extraction model to perform feature combination on the real-time text features, real-time image features, and real-time audio features to obtain the real-time multi-modal archive data features;

[0107] S2-4: According to the real-time multi-modal archive data features, use the archive data retrieval label generation model of the archive data intelligent retrieval engine to perform intelligent retrieval label generation to obtain the corresponding real-time retrieval labels, including the following steps:

[0108] S2-4-1: Use the key feature extraction module of the archive data retrieval label generation model of the archive data intelligent retrieval engine to extract several real-time key features of the real-time multi-modal archive data features;

[0109] S2-4-2: According to several real-time key features, use the retrieval label generation module of the archive data retrieval label generation model to perform intelligent retrieval label generation to obtain the corresponding real-time retrieval labels;

[0110] S2-5: Traverse all preprocessed real-time multi-modal archive data to obtain several real-time standard multi-modal archive data set with real-time retrieval labels;

[0111] S3: The cloud data center uses the archive data intelligent retrieval engine to generate a knowledge graph according to several real-time retrieval topics to obtain the real-time topic knowledge graph of several real-time retrieval topics, including the following steps:

[0112] S3-1: The cloud data center collects several real-time knowledge data of different real-time retrieval topics and preprocesses the several real-time knowledge data to obtain several preprocessed real-time knowledge data of different real-time retrieval topics;

[0113] The preprocessing includes data cleaning, error data screening, format unification, etc. of the original data to improve the data quality and provide data support for subsequent model construction;

[0114] S3-2: According to several preprocessed real-time knowledge data of the same real-time retrieval topic, use the knowledge graph generation model of the archive data intelligent retrieval engine to perform knowledge graph generation to obtain the real-time topic knowledge graph, including the following steps:

[0115] S3-2-1: The semantic feature extraction module of the knowledge graph generation model using the file data intelligent retrieval engine extracts the real-time semantic features of the preprocessed real-time knowledge data.

[0116] S3-2-2: According to the real-time semantic features, use the named entity recognition module of the knowledge graph generation model to perform named entity recognition and obtain a number of knowledge named entities.

[0117] S3-2-3: For the real-time semantic features of a number of knowledge named entities, use the entity relationship recognition module of the knowledge graph generation model to perform entity relationship recognition and obtain a number of knowledge entity relationships.

[0118] S3-2-4: Traverse a number of preprocessed real-time knowledge data of the same real-time retrieval topic to obtain a number of knowledge named entities and a number of knowledge entity relationships of a number of preprocessed real-time knowledge data.

[0119] S3-2-5: Use the generator of the knowledge graph generation module of the knowledge graph generation model to generate a knowledge graph based on a number of knowledge named entities and a number of knowledge entity relationships of a number of preprocessed real-time knowledge data, and obtain the real-time topic knowledge graph of the corresponding real-time retrieval topic.

[0120] S3-3: Traverse a number of preprocessed real-time knowledge data of all real-time retrieval topics to obtain the real-time topic knowledge graphs of a number of real-time retrieval topics.

[0121] S4: The cloud data center performs data clustering on a number of standard real-time multimodal archive data set with real-time retrieval tags according to a number of real-time topic knowledge graphs, and obtains a number of real-time topic clustering archive data clusters, including the following steps:

[0122] S4-1: The cloud data center obtains the similarity between the real-time retrieval tag of each standard real-time multimodal archive data and all real-time topic knowledge graphs.

[0123] S4-2: Divide the standard real-time multimodal archive data into the real-time topic knowledge graph with the highest similarity to the real-time retrieval tag to obtain a number of standard real-time multimodal archive data belonging to different real-time topic knowledge graphs.

[0124] S4-3: Use a number of standard real-time multimodal archive data belonging to the same real-time topic knowledge graph as the real-time topic clustering archive data cluster corresponding to the real-time topic knowledge graph.

[0125] S5: The cloud data center uses the blockchain network to perform distributed storage on a number of real-time topic clustering archive data clusters and the corresponding real-time retrieval topics, including the following steps:

[0126] S5-1: Cluster and store several real-time topic clustering archive data clusters into the IPFS system of the blockchain network to obtain several real-time data hash values;

[0127] S5-2: According to the real-time data hash value and real-time timestamp of each real-time topic clustering archive data cluster, use the smart contract of the blockchain network to generate corresponding real-time transaction data;

[0128] S5-3: Use several data servers with distributed connections in the blockchain network to conduct consensus on several real-time transaction data. After successful consensus, generate several real-time consensus records and corresponding real-time storage records;

[0129] S5-4: Synchronize the real-time consensus records, real-time storage records, several real-time retrieval tags, and corresponding real-time retrieval topics of each real-time topic clustering archive data cluster to the distributed ledger of the blockchain network;

[0130] S6: The user terminal accesses the archive data intelligent retrieval platform and uploads the user's real-time query data to the cloud data center through the archive data intelligent retrieval platform;

[0131] S7: The cloud data center uses the archive data intelligent retrieval engine to conduct intelligent retrieval in several real-time topic clustering archive data clusters and corresponding real-time retrieval topics of the blockchain network according to the real-time query data, and obtain the target archive data, including the following steps:

[0132] S7-1: The cloud data center obtains the similarity between the real-time query data and the real-time topic knowledge graph of several real-time retrieval topics;

[0133] S7-2: Use the real-time retrieval topic corresponding to the real-time topic knowledge graph with the highest similarity as the target real-time retrieval topic of the real-time query data;

[0134] S7-3: Extract several alternative real-time retrieval tags corresponding to the target real-time retrieval topic from the distributed ledger of the blockchain network;

[0135] S7-4: Use the archive data intelligent retrieval model of the archive data intelligent retrieval engine to conduct intelligent retrieval according to several alternative real-time retrieval tags and real-time query data to obtain the target real-time retrieval tag, including the following steps:

[0136] S7-4-1: Convert several alternative real-time retrieval tags into a real-time alternative retrieval tag matrix and input the real-time alternative retrieval tag matrix into the archive data intelligent retrieval model of the archive data intelligent retrieval engine;

[0137] S7-4-2: The retrieval label matrix feature extraction module using the intelligent retrieval model for archival data extracts the real-time retrieval label matrix features of the real-time alternative retrieval label matrix.

[0138] S7-4-3: The query data feature extraction module using the intelligent retrieval model for archival data extracts the real-time query data features of the real-time query data.

[0139] S7-4-4: According to the real-time retrieval label matrix features and the real-time query data features, the archival data intelligent retrieval module using the intelligent retrieval model for archival data performs intelligent retrieval to obtain the target real-time retrieval label.

[0140] S7-5: Extract the target real-time storage record of the target real-time retrieval label from the distributed ledger of the blockchain network, and according to the target real-time storage record, extract the target real-time data hash value from several data servers in the blockchain network.

[0141] S7-6: According to the target real-time data hash value, extract the corresponding target archival data from the IPFS system in the blockchain network.

[0142] S8: The cloud data center visualizes the target archival data on the user terminal through the archival data intelligent retrieval platform.

[0143] Embodiment 2:

[0144] As Figure 2 shown, this embodiment provides an intelligent archival data retrieval system based on a multimodal large model for implementing the intelligent archival data retrieval method. The system includes a cloud data center and several user terminals. All the user terminals are communicatively connected to the cloud data center. The cloud data center is provided with an archival data intelligent retrieval platform and an archival data intelligent retrieval engine, and the cloud data center includes an initialization unit, a data processing unit, a knowledge graph generation unit, a data clustering unit, a distributed storage unit, an intelligent retrieval unit, and a visualization display unit that are connected in sequence;

[0145] The archival data intelligent retrieval platform is provided with a user login module, a query data input module, and an archival data visualization module;

[0146] The archival data intelligent retrieval engine is provided with a multimodal data processing model, a multimodal data feature extraction model, a knowledge graph generation model, an archival data retrieval label generation model, and an archival data intelligent retrieval model;

[0147] The user terminal is used to access the archival data intelligent retrieval platform and upload the user's real-time query data to the cloud data center through the archival data intelligent retrieval platform.

[0148] An initialization unit for building an intelligent retrieval platform for archival data, deploying a blockchain network, using a multimodal large model algorithm to construct an intelligent retrieval engine for archival data, and connecting it to the intelligent retrieval platform for archival data;

[0149] A data processing unit for using the intelligent retrieval engine for archival data to process the collected real-time multimodal archival big data, and obtaining a number of standard real-time multimodal archival data with real-time retrieval tags set;

[0150] A knowledge graph generation unit for using the intelligent retrieval engine for archival data to generate a knowledge graph according to a number of real-time retrieval topics, and obtaining real-time topic knowledge graphs for a number of real-time retrieval topics;

[0151] A data clustering unit for clustering a number of standard real-time multimodal archival data with real-time retrieval tags set according to a number of real-time topic knowledge graphs, and obtaining a number of real-time topic clustering archival data clusters;

[0152] A distributed storage unit for using the blockchain network to perform distributed storage on a number of real-time topic clustering archival data clusters and the corresponding real-time retrieval topics;

[0153] An intelligent retrieval unit for performing intelligent retrieval on a number of real-time topic clustering archival data clusters and the corresponding real-time retrieval topics in the blockchain network according to real-time query data by using the intelligent retrieval engine for archival data, and obtaining target archival data;

[0154] A visualization display unit for visually displaying the target archival data on a user terminal through the intelligent retrieval platform for archival data.

[0155] An intelligent retrieval method and system for archival data based on a multimodal large model provided by the present invention realizes fast retrieval through the multimodal large model algorithm and efficient data processing mechanism of the intelligent retrieval engine for archival data, greatly shortens the retrieval time, effectively deals with a large amount of archival data, quickly locates target materials, and improves work efficiency; uses the multimodal feature extraction and deep learning technology of the intelligent retrieval engine for archival data to accurately understand the query intention, reduces the error caused by semantic ambiguity, effectively processes complex language phenomena such as polysemous words and synonyms, and improves the accuracy of retrieval results; comprehensively processes various modal data such as text, images, and audio through the intelligent retrieval engine for archival data, improves the multimodal data processing ability, makes full use of the rich content in the multimodal information, deeply excavates the potential value of archival data, and improves the data utilization efficiency; deploys a blockchain network to ensure the security and integrity of data during storage and transmission, and prevents data leakage and tampering through distributed ledgers and hash value verification; the cloud data center provides an intelligent retrieval platform for archival data, supports real-time data update and processing, and meets the retrieval requirements for real-time archival data.

[0156] The present invention is not limited to the above optional embodiments, and anyone can obtain other various forms of products under the inspiration of the present invention. The above specific embodiments should not be construed as limiting the protection scope of the present invention, and the protection scope of the present invention should be defined by the claims, and the description can be used to interpret the claims.

Claims

1. A method for intelligent retrieval of archive data based on a multimodal large model, characterized in that: The steps include: Cloud data center, build an intelligent retrieval platform for archival data, deploy blockchain network, use multimodal large model algorithm, build an intelligent retrieval engine for archival data, and connect to the intelligent retrieval platform for archival data; The cloud data center uses an intelligent archival data retrieval engine to process the collected real-time multimodal archival big data to obtain a number of standard real-time multimodal archival data with real-time retrieval tags. The cloud data center uses the intelligent retrieval engine for archive data to generate knowledge graphs based on a number of real-time search topics, and obtains real-time subject knowledge graphs for a number of real-time search topics; The cloud data center clusters a number of standard real-time multimodal archive data with real-time search tags according to a number of real-time subject knowledge graphs to obtain a number of real-time subject clustering archive data clusters; The cloud data center uses the blockchain network to distribute the storage of several real-time subject clustering archive data clusters and corresponding real-time retrieval topics; The user terminal accesses the archive data intelligent retrieval platform and uploads the user's real-time query data to the cloud data center through the archive data intelligent retrieval platform; The cloud data center uses the intelligent retrieval engine for archival data based on real-time query data to perform intelligent retrieval in several real-time topic clustering archival data clusters and corresponding real-time retrieval topics in the blockchain network to obtain target archival data; The cloud data center visualizes the target archival data on the user terminal through the archival data intelligent retrieval platform.

2. The method for intelligent archival data retrieval based on a multimodal large model according to claim 1, characterized in that: The cloud data center builds an intelligent archival data retrieval platform, deploys a blockchain network, uses a multimodal large model algorithm, builds an intelligent archival data retrieval engine, and connects to the intelligent archival data retrieval platform, including the following steps: The cloud data center builds an archive data intelligent retrieval platform framework, and constructs a user login module, a query data input module, and an archive data visualization module in the archive data intelligent retrieval platform framework to obtain an archive data intelligent retrieval platform; Distribute and connect several data servers in the cloud data center, set up the IPFS system and smart contracts, and obtain a blockchain network; Based on the multimodal training sample big data, using the multimodal large model algorithm, build an archival data intelligent retrieval engine, and connect the archival data intelligent retrieval engine to the archival data intelligent retrieval platform.

3. The method for intelligent archival data retrieval based on a multimodal large model according to claim 2 is characterized in that: According to the multimodal training sample big data, using the multimodal big model algorithm, constructing the archive data intelligent retrieval engine, and connecting the archive data intelligent retrieval engine to the archive data intelligent retrieval platform, including the following steps: Collecting multimodal training sample big data and a number of historical knowledge data of different preset search topics, and preprocessing them to obtain a number of preprocessed multimodal training samples and a number of preprocessed historical knowledge data; According to a number of pre-processed multimodal archival data, a multimodal data processing model is constructed using a multimodal data processing algorithm, and a number of standard multimodal training samples are generated; Based on some pre-processed historical knowledge data, a knowledge graph generation model is constructed using a fusion algorithm of natural language processing and deep learning; Based on several standard multimodal training samples, a multimodal data feature extraction model is constructed using a large model algorithm, and several multimodal training sample features are generated; According to the characteristics of several multimodal training samples, a multimodal feature fusion algorithm is used to build an archive data retrieval label generation model, and generate a historical retrieval label for each standard multimodal training sample; Based on the historical retrieval labels of several standard multimodal training samples and randomly generated historical query data, a deep learning algorithm is used to build an intelligent retrieval model for archival data; Integrate the multimodal data processing model, multimodal data feature extraction model, knowledge graph generation model, archival data retrieval label generation model and archival data intelligent retrieval model to obtain the archival data intelligent retrieval engine, and connect the archival data intelligent retrieval engine to the archival data intelligent retrieval platform.

4. The method for intelligent archival data retrieval based on a multimodal large model according to claim 3 is characterized in that: The multimodal data processing model is constructed based on the PPO-DPA algorithm; The knowledge graph generation model is built based on the BERT-CRF-SVM-cGAN algorithm; The multimodal data feature extraction model is built based on the BERT-FPN-CNN-Transformer algorithm; The archive data retrieval tag generation model is constructed based on the RF-MLP algorithm; The archival data intelligent retrieval model is constructed based on the LSTM-DBN-MLP algorithm.

5. The method for intelligent archival data retrieval based on a multimodal large model according to claim 4 is characterized in that: The cloud data center uses an intelligent archival data retrieval engine to process the collected real-time multimodal archival big data to obtain a number of real-time standard multimodal archival data with real-time retrieval tags, including the following steps: Collecting real-time multimodal archive big data, and preprocessing the real-time multimodal archive big data to obtain a number of preprocessed real-time multimodal archive data; Using the multimodal data processing model of the intelligent retrieval engine for archival data, data processing is performed on each pre-processed real-time multimodal archival data to obtain the corresponding standard real-time multimodal archival data; Use the multimodal data feature extraction model of the archival data intelligent retrieval engine to extract the real-time multimodal archival data features of the standard real-time multimodal archival data; According to the characteristics of real-time multimodal archival data, the archival data retrieval label generation model of the archival data intelligent retrieval engine is used to generate intelligent retrieval labels to obtain corresponding real-time retrieval labels; All pre-processed real-time multimodal archive data are traversed to obtain a number of real-time standard multimodal archive data with real-time retrieval tags.

6. The method for intelligent archival data retrieval based on a multimodal large model according to claim 5 is characterized in that: The cloud data center uses the intelligent retrieval engine for archive data to generate knowledge graphs based on a number of real-time search topics, and obtains real-time topic knowledge graphs for a number of real-time search topics, including the following steps: The cloud data center collects a number of real-time knowledge data of different real-time search topics, and pre-processes the real-time knowledge data to obtain a number of pre-processed real-time knowledge data of different real-time search topics; Based on a number of pre-processed real-time knowledge data of the same real-time search topic, a knowledge graph generation model of the archive data intelligent search engine is used to generate a knowledge graph to obtain a real-time topic knowledge graph; Traverse a number of preprocessed real-time knowledge data of all real-time retrieval topics to obtain real-time topic knowledge graphs of a number of real-time retrieval topics.

7. The method for intelligent archival data retrieval based on a multimodal large model according to claim 6 is characterized by: The cloud data center clusters a number of standard real-time multimodal archive data with real-time search tags according to a number of real-time subject knowledge graphs to obtain a number of real-time subject clustering archive data clusters, including the following steps: The cloud data center obtains the similarity between the real-time retrieval tags of each standard real-time multimodal archive data and all real-time subject knowledge graphs; Divide the standard real-time multimodal archive data into the real-time subject knowledge graph with the highest similarity to the real-time search tag, and obtain a number of standard real-time multimodal archive data belonging to different real-time subject knowledge graphs; Several standard real-time multimodal archival data belonging to the same real-time subject knowledge graph are used as real-time subject clustering archival data clusters of the corresponding real-time subject knowledge graph.

8. The method for intelligent archival data retrieval based on a multimodal large model according to claim 7 is characterized in that: The cloud data center uses the blockchain network to perform distributed storage of several real-time subject clustering archive data clusters and corresponding real-time retrieval subjects, including the following steps: Store several real-time subject clustering archive data clusters in the IPFS system of the blockchain network to obtain several real-time data hash values; According to the real-time data hash value and real-time timestamp of each real-time subject clustering archive data cluster, the corresponding real-time transaction data is generated using the smart contract of the blockchain network; Use several data servers with distributed connections in the blockchain network to reach consensus on several real-time transaction data. After the consensus is successful, several real-time consensus records and corresponding real-time storage records are generated; The real-time consensus record, real-time storage record, several real-time search tags and corresponding real-time search topics of each real-time subject clustering archive data cluster are synchronized to the distributed ledger of the blockchain network.

9. The method for intelligent archival data retrieval based on a multimodal large model according to claim 8, characterized in that: The cloud data center uses the intelligent retrieval engine for archive data based on real-time query data to perform intelligent retrieval in several real-time topic clustering archive data clusters and corresponding real-time retrieval topics in the blockchain network to obtain target archive data, including the following steps: The cloud data center obtains the similarity between the real-time query data and the real-time subject knowledge graphs of several real-time search subjects; The real-time search topic corresponding to the real-time topic knowledge graph with the highest similarity is used as the target real-time search topic of the real-time query data; Extracting several candidate real-time search tags corresponding to the target real-time search topic from the distributed ledger of the blockchain network; Using the archive data intelligent retrieval model of the archive data intelligent retrieval engine, intelligent retrieval is performed based on a number of candidate real-time retrieval tags and real-time query data to obtain a target real-time retrieval tag; Extracting the target real-time storage record of the target real-time retrieval tag in the distributed ledger of the blockchain network, and extracting the target real-time data hash value in several data servers of the blockchain network according to the target real-time storage record; According to the target real-time data hash value, the corresponding target archive data is extracted from the IPFS system of the blockchain network.

10. An intelligent retrieval system for archival data based on a multimodal large model, used to implement the intelligent retrieval method for archival data as claimed in any one of claims 1 to 9, characterized in that: The system includes a cloud data center and several user terminals, and the several user terminals are all communicatively connected to the cloud data center. The cloud data center is provided with an archive data intelligent retrieval platform and an archive data intelligent retrieval engine, and the cloud data center includes an initialization unit, a data processing unit, a knowledge graph generation unit, a data clustering unit, a distributed storage unit, an intelligent retrieval unit and a visualization display unit connected in sequence.

Images