Multi-mode RAG method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-mode RAG method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing modal type identification on original multi-modal data, respectively performing structured feature extraction, and storing an original multi-modal data copy into a hierarchical storage database; respectively carrying out vector coding processing and semantic enhancement processing on the mechanism features, and then storing the mechanism features into a semantic vector library and a knowledge graph library; retrieving in the updated hierarchical storage database, the semantic vector library and the knowledge graph library at the same time based on the retrieval intention of the user question, outputting a candidate document set after deduplication and combination of retrieval results, reordering the candidate document set, and finally generating an answer for the user question. According to the invention, accurate answering of complex questions can be realized.

Description

A multimodal RAG method, apparatus, electronic device, and storage medium Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and particularly relates to a multimodal RAG method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, large models still have many of the aforementioned problems in production import and vertical domains. In order to enable large language models in the system to answer professional domain questions and reduce information errors, the main approach is to use vector-based retrieval-augmented generation (RAG) technology.

[0003] Existing RAG technology focuses on semantic similarity. It pre-vectorizes the text content in the document library using vectorization tools and stores it in a vector database. Before the user generates an answer using a large model, the query statement is first vectorized and matched. Knowledge matching the intent is retrieved from the vector database, and the retrieved knowledge and the question are provided together to the large model to generate the answer. This approach has the following problems in practical applications:

[0004] Insufficient multimodal support: Traditional RAGs only process text data and cannot effectively integrate multimodal information such as images, audio, and video; Lagging knowledge base updates: Static knowledge bases are difficult to adapt to dynamically changing data needs, resulting in outdated search results; Weak ability to handle complex scenarios: Using a single agent architecture for retrieval, sorting, and question-answer generation makes it difficult to handle multi-hop reasoning and complex logic tasks; Limitations of a single technical path: Low accuracy in answering complex questions. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal RAG method, apparatus, electronic device, and storage medium to address the problem that existing RAG methods cannot provide accurate answers to complex questions.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention provides a multimodal RAG method, comprising the following steps:

[0008] The process involves acquiring raw multimodal data, identifying the modality type of the raw multimodal data, distributing each modality data to the corresponding modality data parser based on the identification results, extracting structured features from the received modality data by each modality data parser, and storing a copy of the raw multimodal data in a hierarchical storage database.

[0009] The unified semantic vectors obtained by performing vector encoding on the structured features are stored in the semantic vector library, and the knowledge graph triples obtained by semantic enhancement are stored in the knowledge graph library.

[0010] When new multimodal data is input or after a fixed time period, the LSTM-Attention model is used to predict the access probability of data in the hierarchical storage database, semantic vector library and knowledge graph library, and the storage location of the data is updated according to the access probability, data timeliness and storage cost.

[0011] The system receives user questions and parses the corresponding search intent. The retrieval agent searches the updated hierarchical storage database, semantic vector database, and knowledge graph database simultaneously based on the search intent. The three search results are deduplicated and merged, and a set of candidate documents is output to the re-ranking agent. The re-ranking agent re-ranks the set of candidate documents to obtain a ranked document list. The generation agent generates an answer based on the ranked document list and the user question.

[0012] The method for identifying the modality type of the original multimodal data is as follows: an intelligent router using a data type identification algorithm based on deep learning is used to identify the modality type of the original multimodal data. The identified modality types include: text, image, audio and video.

[0013] The process by which each modal data parser extracts structured features from the received modal data includes: using a dedicated pre-trained model for each modality to extract high-quality feature representations of the structured features.

[0014]

[0015]

[0016]

[0017]

[0018] in: For the structured features of the input text data, For text modal feature vectors, RoBERTa(*) is an improved BERT model; For the structured features of the input image data, For image modal feature vectors, ViT-L / 14(*) is a large visual Transformer with an image block size of 14×14; For the structured features of the input audio data, For audio modal feature vectors, Wav2Vec2.0 (*) is a self-supervised audio representation learning model; For the structured features of the input video data, is the video modal feature vector, and VideoMAE(*) is the video mask autoencoder.

[0019] The vector encoding process includes:

[0020] The high-quality feature representations of the structured features of multimodal data are fused and mapped to a unified high-dimensional semantic space:

[0021]

[0022] in: For high-dimensional semantic feature vectors, [;;;] represents feature concatenation operations, with a total dimension of 768+1024+768+768=3328 dimensions; LayerNorm(*) is the normalization layer; MLP(*) is the multilayer perceptron;

[0023] By enhancing the semantic alignment capability between modalities through contrastive learning, high-dimensional semantic feature vectors are transformed into 512-dimensional unified semantic vectors.

[0024] The semantic enhancement process includes: performing named entity recognition on structured features based on a hybrid model of conditional random fields and BERT, identifying semantic relationships between entities using bidirectional LSTM and attention mechanisms, and constructing knowledge graph triples of corresponding entities and relationships using a rotation embedding model.

[0025] The hierarchical storage database, knowledge graph database, and semantic vector database all employ a temperature-sensing-based distributed multi-layer storage architecture to store data.

[0026] The tiered storage database includes: hot storage layer A1, warm storage layer A2, and cold storage layer A3.

[0027] The semantic vector library includes: hot storage layer B1, warm storage layer B2, and cold storage layer B3;

[0028] The knowledge graph database includes: hot storage layer C1 - warm storage layer C2 - cold storage layer C3;

[0029] The hot storage layer is used to store data that has been accessed more than 100 times per hour within 7 days, the warm storage layer is used to store data that has been accessed between 10 and 100 times per hour within 7-30 days, and the cold data layer is used to store data that has been accessed for more than 30 days or has an access frequency of less than 10 times per hour.

[0030] The process of updating the data storage location based on access probability, data timeliness, and storage cost includes:

[0031]

[0032] in, For data d, a migration score is given. β represents the access probability weight, β represents the timeliness weight, and γ represents the storage cost weight. This represents the time-dependent decay function. This represents the reciprocal of the storage size; smaller data sets receive higher scores.

[0033] When the migration score is higher than the warm data storage threshold but lower than the hot data storage threshold, the data is migrated from the hot data layer to the warm data layer, or from the cold data layer to the warm data layer.

[0034] When the migration score is higher than the hot data storage threshold, the data is migrated from the warm data layer to the hot data layer.

[0035] When the migration score is below the hot data storage threshold, the data is migrated from the warm data layer to the cold data layer.

[0036] The process by which the retrieval agent performs a retrieval simultaneously in the updated hierarchical storage database, semantic vector database, and knowledge graph database based on the retrieval intent includes:

[0037] For hierarchical storage databases, full-text retrieval using the BM25+ algorithm is performed to capture the exact word match between the query terms in the user's question and the words in the document, and to calculate the BM25+ score.

[0038] For the semantic vector library, dense vector similarity retrieval is performed. A preset embedding model is used to match the user's question with the unified semantic vector and calculate the similarity score.

[0039] For the knowledge graph database, structured queries based on entities and relationships are performed. Entities are extracted from user questions through named entity recognition. The identified entities are then linked to the corresponding nodes in the knowledge graph storage. Next, a two-hop depth graph traversal is performed in the knowledge graph to discover extended entities and relationships related to the user questions, and the matching degree score of entities and relationships is calculated.

[0040] The process of deduplicating and merging the three search results includes:

[0041] Deduplication and merging of the three search results based on document ID;

[0042] The weighted fusion relevance score of the three search results is calculated based on the BM25+ score, similarity score, matching score and their corresponding preset weights;

[0043] A candidate document set is obtained by initially ranking the search results based on a weighted fusion relevance score.

[0044] The process by which the sorting agent reorders the candidate document set includes:

[0045] Verify the completeness of the candidate document set, build a document index, and generate a list of candidate documents to be sorted;

[0046] Extract features from each document in the candidate document list, calculate the relevance score between the document's multidimensional feature vector and the user's question, and obtain the Top-K candidate set sorted by relevance score;

[0047] Based on the multidimensional feature vectors of documents, the LambdaMART learning ranking algorithm is adopted, and the candidate set is finely ranked with NDCG as the optimization objective to obtain a finely ranked list of documents.

[0048] The MMR algorithm is applied to the finely sorted document list to increase the diversity of results while ensuring relevance, and outputs the final sorted document list.

[0049] The process of generating an agent based on a sorted list of documents and user questions to generate answer text includes:

[0050] Analyze user questions, determine generation goals and constraints, and generate a structured task description;

[0051] Based on the task description, the document content in the sorted document list is concatenated into a long text. The TextRank algorithm is used to extract the most important sentences in the long text, determine the number of sentences to retain, and output a concise context.

[0052] The simplified context and corresponding knowledge graph information are dynamically fused through a knowledge gating mechanism. The RAG-Token model is used to select the fused content, and the most relevant knowledge fragments in the fusion process are used to generate a draft answer.

[0053] The initial draft of the answer is optimized and controlled in terms of attributes, including length, style, and professionalism, and the optimized answer text is output.

[0054] The system performs fact verification on the response text, generates a final response based on the verification results, and provides feedback to the user.

[0055] The process of fact-verifying the response text includes extracting factual statements from the response text, using an NLI model to verify whether each statement is supported by the candidate document set, and calculating an overall truthfulness score. If the truthfulness score is greater than a preset threshold, the response text is fed back to the user as the final response. If the truthfulness score is less than the preset threshold, the steps of generating the response draft and the response text are iterated. If the truthfulness score of the response text is greater than the preset threshold within the maximum number of iterations, the corresponding final response is output to the user; otherwise, an error feedback is generated.

[0056] Secondly, the present invention provides a multimodal RAG device, comprising:

[0057] The multimodal data preprocessing module is used to convert raw multimodal data into unified semantic vectors and knowledge graph triples;

[0058] The data storage and management module is used to independently and hierarchically store copies of the original multimodal data, unified semantic vectors, and knowledge graph triples, and to update the storage location of the data based on access probability, data timeliness, and storage cost.

[0059] The multi-agent collaborative generation module is used to search the updated hierarchical storage database, semantic vector library, and knowledge graph library based on the user's question, and generate the corresponding answer to the user's question after reordering the search results.

[0060] The multimodal data preprocessing module includes:

[0061] The intelligent data router, based on a deep learning-based data type recognition algorithm, automatically identifies the modal type of input data, including: text, images, audio, and video.

[0062] A multimodal data parser cluster, including a text data parser, an image data parser, an audio data parser, and a video data parser, processes text, image, audio, and video data respectively and outputs corresponding structured features;

[0063] A unified vector encoder is used to map the structured features of different modal data processed by a cluster of multimodal data parsers to a unified semantic space, and output the vector form of the multimodal data.

[0064] The semantic enhancement unit is used to construct knowledge graphs and perform semantic reasoning on the structured features of different modal data processed by the multimodal data parser cluster. It achieves deep semantic understanding through entity recognition, relation extraction and graph embedding, and outputs the graph structure form of multimodal data.

[0065] The multi-Agent collaborative generation module includes:

[0066] The retrieval agent is used to perform a three-way parallel retrieval based on the user's question, using a hierarchical storage database, a semantic vector library, and a knowledge graph library to obtain a set of candidate documents.

[0067] The reordering agent is used to reorder the candidate document set and obtain the final sorted document list;

[0068] Generate an Agent, which is used to generate answers to the user's questions based on the user's questions and the final document list;

[0069] The coordination and communication agent is used for data interaction between the data storage and management module and the multi-agent collaborative generation module, as well as for coordination and communication between the various agent units within the agent collaborative generation module.

[0070] Thirdly, the present invention provides a computer device, characterized in that it comprises:

[0071] Memory, used to store computer instructions;

[0072] A processor for executing the computer instructions to implement the steps of the multimodal RAG method described above.

[0073] Fourthly, the present invention provides a computer-readable storage medium having computer instructions stored thereon, characterized in that, when the computer instructions are executed by a processor, they implement the steps of the above-described multimodal RAG method.

[0074] Beneficial effects: The multimodal RAG method, apparatus, electronic device, and storage medium of this invention support retrieval and querying of text, image, voice, and video data. Through multi-agent collaboration, it improves the efficiency and accuracy of question-and-answer generation. The three-way parallel retrieval scheme ensures the diversity and accuracy of retrieval results. The multi-layer storage architecture enables refined storage of data. Real-time updates to the database through data migration ensure the real-time nature of retrieval data, improve retrieval speed, enable rapid retrieval of large amounts of data, enhance the ability to handle complex questions, and ensure the accuracy of the final answer. Attached Figure Description

[0075] Figure 1 is a flowchart of the multimodal RAG method of the present invention;

[0076] Figure 2 is a schematic diagram of the structure of the multimodal RAG device of the present invention. Detailed Implementation

[0077] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0078] Example 1

[0079] Referring to Figure 1, a multimodal RAG method includes the following steps:

[0080] The process involves acquiring raw multimodal data, identifying the modality type of the raw multimodal data, distributing each modality data to the corresponding modality data parser based on the identification results, extracting structured features from the received modality data by each modality data parser, and storing a copy of the raw multimodal data in a hierarchical storage database.

[0081] The structured features are processed by vector encoding and semantic enhancement respectively. The unified semantic vector obtained by vector encoding is stored in the semantic vector library, and the knowledge graph triple obtained by semantic enhancement is stored in the knowledge graph library.

[0082] When new multimodal data is input or after a fixed time period, the LSTM-Attention model is used to predict the access probability of data in the hierarchical storage database, semantic vector library and knowledge graph library, and the storage location of the data is updated according to the access probability, data timeliness and storage cost.

[0083] The system receives user questions and parses the search intent corresponding to those questions. The retrieval agent searches simultaneously in the updated hierarchical storage database, semantic vector database, and knowledge graph database based on the search intent. It then deduplicates and merges the three search results, outputting a set of candidate documents to the reordering agent. The reordering agent reorders the set of candidate documents to obtain a sorted document list. Finally, the generation agent generates the answer text based on the sorted document list and the user question.

[0084] In this embodiment, an intelligent router employing a deep learning-based data type recognition algorithm is used to identify the modality type of the original multimodal data. The identified modality types include text, image, audio, and video. The corresponding tuples (modality type label, original multimodal data) are output to the multimodal data parser cluster.

[0085] The multimodal data parser cluster integrates text data parser, image data parser, audio data parser and video data parser, and outputs corresponding tuples (structured features, original multimodal data copy).

[0086] Text data parser: An improved sliding time window block algorithm is used to block text data, calculate TF-IDF and perform semantic density analysis to obtain the structured features of the text data;

[0087] Image data parser: Multi-scale feature extraction and OCR fusion strategy, performs multi-scale feature extraction, OCR recognition and visual-text alignment on image data to obtain the structured features of image data;

[0088] Audio data parser: A multi-stage speech processing pipeline that performs noise reduction, speech transcription, and emotional prosody extraction on audio data to obtain the structured features of the audio data;

[0089] Video data parser: Spatiotemporal feature extraction and multimodal fusion, performing keyframe extraction, temporal modeling and 3D convolutional feature extraction on video data, realizing the transformation of raw video into high-dimensional semantic features, and obtaining the structured features of video data.

[0090] Before data is stored in the database, each modal parser performs format verification, quality checks, and anomaly detection on the raw data to ensure that the data stored in the hierarchical storage conforms to system standards. For example, image data needs to be verified for format, resolution, and color space; audio data needs to be checked for sampling rate, number of channels, and duration; and video data needs to be verified for encoding format, frame rate, and resolution. This avoids invalid data occupying storage space and improves the efficiency of subsequent retrieval and processing.

[0091] During processing, the modal parser extracts key metadata (such as file size, creation time, duration, resolution, etc.). This metadata can be used for fast retrieval and filtering, as well as for storage tier migration decisions (e.g., determining whether to store data in hot / warm / cold storage based on file size and access frequency). It also supports data version management and consistency maintenance. If the raw data is directly stored in the database, metadata needs to be extracted in real time during retrieval, which significantly reduces system performance.

[0092] Structured features are vector-encoded using a unified vector encoder, and cross-modal attention fusion, unified semantic space mapping, and contrastive learning optimization are employed to output a 512-dimensional unified semantic vector.

[0093] High-quality feature representations of structured features are extracted using dedicated pre-trained models for each modality, including text modality feature vectors, image modality feature vectors, audio modality feature vectors, and video modality feature vectors.

[0094]

[0095]

[0096]

[0097]

[0098] in: For the structured features of the input text data, For text modal feature vectors, RoBERTa(*) is an improved BERT model; For the structured features of the input image data, For image modal feature vectors, ViT-L / 14(*) is a large visual Transformer with an image block size of 14×14; For the structured features of the input audio data, For audio modal feature vectors, Wav2Vec2.0 (*) is a self-supervised audio representation learning model; For the structured features of the input video data, is the video modal feature vector, and VideoMAE(*) is the video mask autoencoder.

[0099] High-quality feature representations from multiple modalities are fused and mapped to a unified high-dimensional semantic space:

[0100]

[0101] in: The vector is a high-dimensional semantic feature vector. [;;;] represents the feature concatenation operation. The total dimension is 768+1024+768+768=3328. LayerNorm(*) is a normalization layer to stabilize the training process. MLP(*) is a multilayer perceptron to achieve nonlinear mapping.

[0102] By enhancing the semantic alignment capability between modalities through contrastive learning, high-dimensional semantic feature vectors are transformed into 512-dimensional unified semantic vectors.

[0103] The semantic enhancement module performs semantic enhancement processing on the structured vectors, and the structured features are used for named entity recognition based on the hybrid model of conditional random field and BERT. Bidirectional LSTM and attention mechanism are used to identify the semantic relationship between entities, and the RotatE model is used to construct the corresponding knowledge graph triples of entities and relations (head entity, relation, tail entity).

[0104] The hierarchical storage database, knowledge graph database, and semantic vector database all adopt a distributed multi-layer storage architecture to store data and use intelligent caching strategies to adjust the data in real time.

[0105] A temperature-sensing-based tiered storage strategy is adopted to schedule data between tiers based on access frequency, timeliness, and storage cost.

[0106] The tiered storage database includes: hot storage layer A1, warm storage layer A2, and cold storage layer A3.

[0107] The semantic vector library includes: hot storage layer B1, warm storage layer B2, and cold storage layer B3;

[0108] The knowledge graph database includes: hot storage layer C1, warm storage layer C2, and cold storage layer C3.

[0109] The system automatically selects the storage tier based on data timeliness and access frequency. Data accessed more than 100 times per hour within the past 7 days is read from the hot storage tier, with a response time of less than 1ms; data accessed between 10 and 100 times per hour within the past 7-30 days is read from the warm storage tier, with a response time of less than 50ms; and data accessed more than 30 days ago or less than 10 times per hour is read from the cold storage tier, with a response time of less than 200ms. If data in a certain storage tier does not meet the timeliness and access frequency conditions of the corresponding tier, a data migration decision is made.

[0110] Multi-level Bloom filters detect changes, and Merkle trees enable incremental synchronization.

[0111] The Extended Vector Clock (VCE) algorithm is used to manage version consistency of all stored data.

[0112]

[0113] in: For the event The extended vector clock contains clock information from k nodes. This is the unique identifier for the i-th node. Let be the logical clock value of the i-th node, which is monotonically increasing; is the weight of the i-th node; k is the total number of nodes participating in the distributed system.

[0114]

[0115] in, This indicates that event 1 occurred before event 2, suggesting a consistency issue with the versions of the stored data. This indicates that the clock logic value for event 1 is no greater than that for event 2 for all nodes. This indicates that there exists at least one node where the clock logic value of event 1 is strictly less than that of event 2.

[0116]

[0117] Where Conflict is the collision detection function. This means that event 1 did not occur before event 2. This indicates that event 2 did not occur before event 1.

[0118] The three-way merging strategy avoids data conflicts in a distributed environment. If the detection result indicates that a conflict exists, the conflict events are merged based on the conflict resolution mechanism of the common ancestor.

[0119] Based on the LSTM-Attention model, access patterns are predicted, and the future access probability of data is predicted. Taking into account access probability, timeliness, and storage cost, a migration score is calculated. Based on the migration score threshold, a hierarchical migration decision is executed, and the storage level of the data is automatically adjusted.

[0120]

[0121] in, H represents the probability of accessing the next time step predicted based on historical access sequences. t This represents a historical access sequence of length w. This is a binary access flag, where 1 indicates access and 0 indicates no access. LSTM is the hidden state vector computation function. W o Let b be the output layer weight matrix. o The output layer bias is set using Sigmoid as the activation function.

[0122]

[0123] in, For data d, a migration score is given. β represents the access probability weight, β represents the timeliness weight, and γ represents the storage cost weight. This represents the time-dependent decay function. This represents the reciprocal of the storage size; smaller data receives a higher score.

[0124] When the migration score is higher than the warm data storage threshold (0.1) and lower than the hot data storage threshold (0.3), the data is migrated from the hot data layer to the warm data layer, or from the cold data layer to the warm data layer.

[0125] When the migration score is higher than the hot data storage threshold (0.3), the data is migrated from the warm data layer to the hot data layer;

[0126] When the migration score is below the hot data storage threshold (0.1), the data is migrated from the warm data layer to the cold data layer.

[0127] In this embodiment, the retrieval process for the Agent includes:

[0128] Receive user queries and parse the search intent; perform a three-way parallel search based on the search intent (vector library: dense vector similarity-based search; graph library: structured query based on entities and relationships; hierarchical storage: BM25+ full-text search).

[0129] More specifically, dense vector similarity retrieval includes the following process:

[0130] Use a question-answering encoder (such as RoBERTa) to encode the user's question q into a 768-dimensional dense vector E. q Find the document vector E in the semantic vector library that is most similar to the query vector. d Calculate the similarity score sim:

[0131]

[0132] Returning the top-K documents with the highest similarity scores (usually 50≤K≤100) is a way to capture the semantic similarity between the query and the document. Even if the query terms and document terms do not match exactly, relevant documents can still be retrieved as long as they are semantically related.

[0133] More specifically, structured queries based on entities and relationships include the following process: Utilizing knowledge graph storage for query expansion and semantic reasoning, firstly, entities are extracted from the user's question using Named Entity Recognition (NER). Then, the identified entities are linked to corresponding nodes in the knowledge graph storage. Next, a two-hop depth graph traversal is performed in the knowledge graph to discover extended entities and relationships related to the user's question. Based on the extended entities, documents containing these entities or relationships are retrieved from the knowledge graph. Reasoning is then performed using the relationships in the knowledge graph to discover implicitly related documents. The structured information from the knowledge graph can enhance the semantic coverage of the query and improve the retrieval recall rate.

[0134] More specifically, the basic formula for full-text search based on the BM25+ algorithm is:

[0135]

[0136] Where q represents user questions and d represents documents (hierarchical database). Let be the document length, t be the query terms in the user question, IDF(*) be the inverse document probability of the query terms, f(*) be the probability of the query terms in the document, k1 be the term frequency saturation parameter, b be the document length normalization parameter, and avgdl be the average document length. This method can capture precise word matching between query terms in the user question and words in the document, making it suitable for scenarios requiring precise matching such as technical terms, names of people, and place names, and complementing the semantic matching of vector retrieval.

[0137] The three search results are deduplicated and merged based on document ID; a weighted fusion relevance score is calculated based on BM25+ score, similarity score, matching score and their corresponding preset weights; the search results are initially sorted based on the weighted fusion relevance score to obtain a candidate document set, and the candidate document set is output to the re-sorting Agent.

[0138] The weighted fusion relevance score is calculated by fully considering the similarity score of vector retrieval, the relevance score of knowledge graph retrieval (based on the matching degree of entities and relations), the BM25+ score of full-text retrieval (normalized to [0,1]), and the preset weights of the three paths. This allows for fair comparison and ranking of documents from different sources, providing a basis for subsequent re-ranking and avoiding expensive re-ranking calculations for low-relevance documents, thereby improving system efficiency.

[0139] In this embodiment, the reordering process of the reordering Agent includes:

[0140] Verify the completeness of the candidate document set, build a document index, and generate a list of candidate documents to be sorted;

[0141] Extract features from each document in the candidate document list, including: semantic similarity features, word matching features, structural matching features, timeliness features, and authority features; generate the corresponding multi-dimensional feature vector for the document.

[0142] Calculate the relevance scores between the document's multidimensional feature vector and the user's question to obtain a Top-K candidate set sorted by relevance scores;

[0143] Based on the multidimensional feature vectors of documents, the LambdaMART learning ranking algorithm is adopted, and the candidate set is finely ranked with NDCG as the optimization objective to obtain a finely ranked list of documents.

[0144] The MMR algorithm is applied to the finely sorted document list to increase the diversity of results while ensuring relevance, avoid content redundancy, and output the final sorted document list.

[0145] In this embodiment, the process of generating the Agent's response includes:

[0146] Parse the user's question, determine the generation goal and constraints (context length limit), and generate a structured task description;

[0147] Based on the task description, the document content in the sorted document list is concatenated into a long text. The TextRank algorithm is used to extract the most important sentences in the long text, determine the number of sentences to retain, and output a concise context.

[0148] The simplified context and corresponding knowledge graph information (including extended entity and relation information in the knowledge graph library) are dynamically fused through a knowledge gating mechanism. The RAG-Token model is used to select the fused content, and the most relevant knowledge fragments in the fusion process are used to generate a draft answer.

[0149] The initial draft of the answer is optimized and controlled in terms of attributes, including length, style, and professionalism, and the optimized answer text is output.

[0150] The factual statements are extracted from the response text. The NLI model is used to verify whether each statement is supported by the candidate document set, and the overall truthfulness score is calculated. If the truthfulness score is greater than a preset threshold, the response text is given to the user as the final response. If the truthfulness score is less than the preset threshold, the steps of generating the response draft and the response text are iterated. If the truthfulness score of the response text is greater than the preset threshold within the maximum number of iterations, the corresponding final response is output to the user. Otherwise, an error response is generated.

[0151] The candidate document set retains the complete document content and metadata information of the original search results.

[0152] Example 2

[0153] Referring to Figure 2, a multimodal RAG device includes: a multimodal data preprocessing module, a data storage and management module, a multi-agent collaborative generation module, and a secure access module.

[0154] The multimodal data preprocessing module is used to convert raw multimodal data into unified semantic vectors and knowledge graph triples;

[0155] The data storage and management module is used to independently and hierarchically store copies of the original multimodal data, unified semantic vectors, and knowledge graph triples, and to update the storage location of the data based on access probability, data timeliness, and storage cost.

[0156] The multi-agent collaborative generation module is used to search in the updated hierarchical storage database, semantic vector library and knowledge graph library according to the user's question, and generate the corresponding answer to the user's question after reordering the search results.

[0157] The secure access module is used to perform security checks on user identity and user questions to prevent malicious access, and to perform security controls on search results and answers to prevent illegal content from appearing.

[0158] In this embodiment, the multimodal data preprocessing module includes:

[0159] Intelligent Data Router: Based on a deep learning-based data type recognition algorithm, it automatically identifies the modal type of input data, including text, images, audio, and video;

[0160] Multimodal data parser cluster: processes text, image, audio, and video data;

[0161] Unified Vectorized Encoder: An improved multimodal contrastive learning framework that maps different modal data processed by a cluster of multimodal data parsers to a unified semantic space and outputs the vector form of the multimodal data;

[0162] Semantic Enhancement Unit: Knowledge graph construction and semantic reasoning capabilities. It achieves deep semantic understanding through entity recognition, relation extraction, and graph embedding, and outputs multimodal data in graph structure form.

[0163] The data storage and management module includes a hierarchical storage database, a semantic vector library, and a knowledge graph library. It adopts a temperature-aware hierarchical storage strategy and schedules data between layers based on access frequency, timeliness, and storage cost.

[0164] In this embodiment, the multi-agent collaborative generation module includes:

[0165] The retrieval agent is used to perform a three-way parallel retrieval based on the user's question, using a hierarchical storage database, a semantic vector library, and a knowledge graph library to obtain a set of candidate documents.

[0166] The reordering agent is used to reorder the candidate document set and obtain the final sorted document list;

[0167] Generate an Agent, which is used to generate answers to the user's questions based on the user's questions and the final document list;

[0168] The coordination and communication agent is used for data interaction between the data storage and management module and the multi-agent collaborative generation module, as well as for coordination and communication between the various agent units within the agent collaborative generation module.

[0169] Example 3

[0170] A computer device, comprising:

[0171] Memory, used to store computer instructions;

[0172] A processor for executing the computer instructions to implement the steps of the multimodal RAG method described in Embodiment 1.

[0173] Example 4

[0174] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the multimodal RAG method described in Embodiment 1.

[0175] In summary, the multimodal RAG method, apparatus, electronic device, and storage medium of this invention support retrieval and querying of text, image, voice, and video data. Through multi-agent collaboration, it improves the efficiency and accuracy of question-and-answer generation. The three-way parallel retrieval scheme ensures the diversity and accuracy of retrieval results. The multi-layered storage architecture enables refined data storage. Real-time database updates via data migration ensure the real-time nature of retrieved data, improve retrieval speed, enable rapid retrieval of large datasets, enhance the ability to handle complex problems, and guarantee the accuracy of the final answer.

[0176] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction system that implements the functions specified in one or more flowcharts and / or one or more block diagrams.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0180] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multimodal RAG method, characterized in that, The process includes the following steps: acquiring raw multimodal data; identifying the modality type of the raw multimodal data; distributing each modality data to its corresponding modality data parser based on the identification results; each modality data parser extracting structured features from the received modality data and storing a copy of the raw multimodal data in a hierarchical storage database; storing the unified semantic vector obtained by vector encoding the structured features in a semantic vector library, and storing the knowledge graph triples obtained by semantic enhancement in a knowledge graph library; and when new multimodal data is input or after a fixed time period, performing hierarchical processing based on the LSTM-Attention model. The system predicts access probabilities for data stored in the database, semantic vector library, and knowledge graph library, and updates the data storage location based on access probabilities, data timeliness, and storage costs. It receives user questions and parses the corresponding search intent. The retrieval agent simultaneously searches the updated hierarchical storage database, semantic vector library, and knowledge graph library based on the search intent. The three search results are deduplicated and merged, outputting a candidate document set to the re-ranking agent. The re-ranking agent re-ranks the candidate document set to obtain a ranked document list. The generation agent then generates an answer based on the ranked document list and the user question.

2. The multimodal RAG method according to claim 1, characterized in that, The method for identifying the modality type of the original multimodal data is as follows: an intelligent router using a data type identification algorithm based on deep learning is used to identify the modality type of the original multimodal data. The identified modality types include: text, image, audio and video.

3. The multimodal RAG method according to claim 1, characterized in that, The process by which each modal data parser extracts structured features from the received modal data includes: using a dedicated pre-trained model for each modality to extract high-quality feature representations of the structured features. in: For the structured features of the input text data, For text modal feature vectors, RoBERTa(*) is an improved BERT model; For the structured features of the input image data, For image modal feature vectors, ViT-L / 14(*) is a large visual Transformer with an image block size of 14×14; For the structured features of the input audio data, For audio modal feature vectors, Wav2Vec2.0 (*) is a self-supervised audio representation learning model; For the structured features of the input video data, is the video modal feature vector, and VideoMAE(*) is the video mask autoencoder.

4. The multimodal RAG method according to claim 1, characterized in that, The vector encoding process includes fusing high-quality feature representations of the structured features of multimodal data and mapping them to a unified high-dimensional semantic space. in: The high-dimensional semantic feature vector is represented by [;;;], which represents the feature concatenation operation. The total dimension is 768+1024+768+768=3328. LayerNorm(*) is the normalization layer. MLP(*) is the multilayer perceptron. By enhancing the semantic alignment ability between modalities through contrastive learning, the high-dimensional semantic feature vector is transformed into a 512-dimensional unified semantic vector.

5. The multimodal RAG method according to claim 1, characterized in that, The semantic enhancement process includes: performing named entity recognition on structured features based on a hybrid model of conditional random fields and BERT, identifying semantic relationships between entities using bidirectional LSTM and attention mechanisms, and constructing knowledge graph triples of corresponding entities and relationships using a rotation embedding model.

6. The multimodal RAG method according to claim 1, characterized in that, The hierarchical storage database, knowledge graph database, and semantic vector database all adopt a temperature-aware distributed multi-layer storage architecture for data storage. The hierarchical storage database includes: hot storage layer A1, warm storage layer A2, and cold storage layer A3; the semantic vector database includes: hot storage layer B1, warm storage layer B2, and cold storage layer B3; and the knowledge graph database includes: hot storage layer C1, warm storage layer C2, and cold storage layer C3. The hot storage layer is used to store data that has been accessed more than 100 times / hour within 7 days, the warm storage layer is used to store data that has been accessed between 10 and 100 times / hour within 7-30 days, and the cold data layer is used to store data that has been accessed more than 30 days or has an access frequency of less than 10 times / hour.

7. The multimodal RAG method according to claim 1, characterized in that, The process of updating the data storage location based on access probability, data timeliness, and storage cost includes: in, Give migration scores to data d. β represents the access probability weight, β represents the timeliness weight, and γ represents the storage cost weight. This represents the time-dependent decay function. The migration score is the reciprocal of the storage size, with smaller data receiving a higher score. When the migration score is higher than the warm data storage threshold but lower than the hot data storage threshold, the data is migrated from the hot data layer to the warm data layer, or from the cold data layer to the warm data layer. When the migration score is higher than the hot data storage threshold, the data is migrated from the warm data layer to the hot data layer. When the migration score is lower than the hot data storage threshold, the data is migrated from the warm data layer to the cold data layer.

8. The multimodal RAG method according to claim 1, characterized in that, The retrieval agent performs a retrieval process simultaneously in the updated hierarchical storage database, semantic vector database, and knowledge graph database based on the retrieval intent. This process includes: for the hierarchical storage database, performing full-text retrieval using the BM25+ algorithm to capture the exact word match between the query terms in the user's question and the document, and calculating the BM25+ score; for the semantic vector database, performing dense vector similarity retrieval, using a preset embedding model to match the user's question with a unified semantic vector for similarity, and calculating the similarity score; for the knowledge graph database, performing structured queries based on entities and relationships, extracting entities from the user's question through named entity recognition, linking the identified entities to corresponding nodes in the knowledge graph storage, and then performing a two-hop depth graph traversal in the knowledge graph to discover extended entities and relationships related to the user's question, and calculating the matching score between entities and relationships.

9. The multimodal RAG method according to claim 8, characterized in that, The process of deduplicating and merging the three search results includes: deduplicating and merging the three search results based on the document ID; calculating the weighted fusion relevance score of the three search results based on the BM25+ score, similarity score, matching score and their corresponding preset weights; and performing an initial sorting of the search results based on the weighted fusion relevance score to obtain a candidate document set.

10. The multimodal RAG method according to claim 1, characterized in that, The process by which the ranking agent re-ranks the candidate document set includes: verifying the completeness of the candidate document set, establishing a document index, and creating a list of candidate documents to be ranked; extracting features from each document in the candidate document list, calculating the relevance score between the document's multidimensional feature vector and the user's question, and obtaining a Top-K candidate set ranked by relevance score; based on the document's multidimensional feature vector, using the LambdaMART learning ranking algorithm with NDCG as the optimization objective to perform fine-grained ranking of the candidate set, resulting in a precisely ranked list of documents; applying the MMR algorithm to the fine-grained list of documents to increase the diversity of results while ensuring relevance, and outputting the final ranked list of documents.

11. The multimodal RAG method according to claim 1, characterized in that, The process of generating an agent based on a sorted document list and user questions to generate answer text includes: parsing the user question, determining the generation goal and constraints, and generating a structured task description; concatenating the document content from the sorted document list into a long text based on the task description, using the TextRank algorithm to extract the most important sentences from the long text, determining the number of sentences to retain, and outputting a simplified context; dynamically fusing the simplified context and corresponding knowledge graph information through a knowledge gating mechanism, using the RAG-Token model to select the fused content, and generating an answer draft based on the most relevant knowledge fragments selected during the fusion process; performing attribute optimization control on the initial answer draft, including length control, style control, and professionalism control, and outputting the optimized answer text; performing fact verification on the answer text, generating the final answer based on the verification results, and providing feedback to the user.

12. The multimodal RAG method according to claim 11, characterized in that, The process of fact-verifying the response text includes extracting factual statements from the response text, using an NLI model to verify whether each statement is supported by the candidate document set, and calculating an overall truthfulness score. If the truthfulness score is greater than a preset threshold, the response text is fed back to the user as the final response. If the truthfulness score is less than the preset threshold, the steps of generating the response draft and the response text are iterated. If the truthfulness score of the response text is greater than the preset threshold within the maximum number of iterations, the corresponding final response is output to the user; otherwise, an error feedback is generated.

13. A multimodal RAG device, characterized in that, include: The multimodal data preprocessing module is used to convert raw multimodal data into unified semantic vectors and knowledge graph triples; The data storage and management module is used to independently store copies of the original multimodal data, unified semantic vectors, and knowledge graph triples in a hierarchical manner, and to update the storage location of the data based on access probability, data timeliness, and storage cost. The multi-agent collaborative generation module is used to search the updated hierarchical storage database, semantic vector library, and knowledge graph library based on user questions, and to generate answers to the corresponding user questions after reordering the search results.

14. The multimodal RAG device according to claim 13, characterized in that, The multimodal data preprocessing module includes: an intelligent data router, based on a deep learning-based data type recognition algorithm, which automatically identifies the modality type of the input data, including text, image, audio, and video; a multimodal data parser cluster, including a text data parser, an image data parser, an audio data parser, and a video data parser, which process text, image, audio, and video data respectively and output corresponding structured features; a unified vector encoder, used to map the structured features of different modal data processed by the multimodal data parser cluster to a unified semantic space and output the vector form of the multimodal data; and a semantic enhancement unit, used to construct knowledge graphs and perform semantic reasoning on the structured features of different modal data processed by the multimodal data parser cluster, achieving deep semantic understanding through entity recognition, relation extraction, and graph embedding, and outputting the graph structure form of the multimodal data.

15. The multimodal RAG device according to claim 13, characterized in that, The multi-Agent collaborative generation module includes: a retrieval agent, used to perform parallel retrieval of a hierarchical storage database, a semantic vector database, and a knowledge graph database based on the user's question to obtain a candidate document set; a reordering agent, used to reorder the candidate document set to obtain a final sorted document list; a generation agent, used to generate an answer to the user's question based on the user's question and the final document list; and a coordination and communication agent, used for data interaction between the data storage and management module and the multi-Agent collaborative generation module, as well as for coordination and communication between the various agent units within the agent collaborative generation module.

16. A computer device, characterized in that, include: Memory, used to store computer instructions; A processor for executing the computer instructions to implement the steps of the multimodal RAG method according to any one of claims 1-12.

17. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the steps of the multimodal RAG method as described in any one of claims 1-12.

Images