Multimodal metadata retrieval-augmented generation method and system

Abstract

A multimodal metadata retrieval-augmented generation method and system, relating to the technical field of artificial intelligence. In the present invention, the method comprises: determining data modalities of query content, performing feature extraction on data of each modality on the basis of the determined data modalities, converting extracted features into feature vectors, and fusing the feature vectors of the modalities to generate a multimodal feature vector; extracting, from data of each modality in the query content, key metadata of the data of each modality, converting the key metadata into a key metadata vector and connecting same to the multimodal feature vector, and constructing an augmented vector; and performing retrieval by means of the augmented vector. The method helps improve knowledge coverage and information utilization efficiency, and enhances the accuracy, relevance, and credibility of generated content.

Description

A multimodal metadata retrieval enhancement generation method and system

[0001] The present application claims priority to the Chinese patent application No. 202411080245.X, filed on August 8, 2024, and entitled "A multimodal metadata retrieval enhancement generation method and system", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a multimodal metadata retrieval enhancement generation method and system. BACKGROUND

[0003] In the field of artificial intelligence and natural language processing (NLP), retrieval augmentation generation (RAG) systems have become a key technology to improve the performance of large language models (LLM). RAG systems combine the retrieval capabilities of external knowledge bases with the generation capabilities of LLMs to provide more accurate and reliable answers. This approach not only improves the knowledge coverage of the system, but also effectively alleviates the inherent hallucination problem of LLMs.

[0004] However, traditional RAG systems still face many challenges when dealing with complex multimodal and multi-hop queries. First, existing RAG systems often struggle to effectively integrate information from different modalities (such as text, images, audio, etc.), leading to poor performance when dealing with queries that require cross-modal reasoning. Second, for complex queries that require multi-step reasoning, traditional RAG systems often struggle to accurately capture and utilize intermediate reasoning results, affecting the quality of the final answer. In addition, existing RAG systems have obvious shortcomings in terms of factuality and explainability, making it difficult to ensure the accuracy of the generated content while lacking clear explanations of the reasoning process. SUMMARY

[0005] To solve the technical problems in the background art, the present application provides a multimodal metadata retrieval enhancement generation method and system, which helps to solve the limitations in complex multi-hop query processing and cross-modal reasoning scenarios, and improves the accuracy, relevance, and credibility of the generated content.

[0006] To achieve the above technical solutions, in a first aspect, the present application provides a multimodal metadata retrieval enhancement generation method, comprising:

[0007] Step one: input query content;

[0008] Step two: determine the data modality of the query content and process based on the determined data modality of the query content to generate a multimodal feature fusion vector;

[0009] Step three: extracting key metadata of each modality data in the query content using a large language model, and expanding the key metadata, and converting the expanded key metadata into a metadata vector;

[0010] Step four: connecting the metadata vector and the multi-modal feature fusion vector to generate an enhanced vector;

[0011] Step five: based on the enhanced vector, using the Faiss library to perform parallel retrieval on the GPU, and outputting the final retrieval result;

[0012] Step six: according to the output final retrieval result, using a large language model to generate an initial answer, and performing factual optimization on the generated initial answer to generate a final answer.

[0013] Further, the step two comprises:

[0014] analyzing the query content to determine the data modality of the query content, wherein the data modality of the query content includes text, image, audio and video;

[0015] preprocessing each modality data of the determined query content, and extracting each modality data feature of the query content from the preprocessed each modality data;

[0016] Converting the extracted each modality data feature of the query content into a fixed-dimension query data feature vector respectively;

[0017] Through the cross-modal attention mechanism and the modal importance dynamic weighting mechanism, the query data feature vector is fused to obtain a multi-modal feature fusion vector.

[0018] Further, the five comprises:

[0019] Based on the enhanced vector, using the Faiss library to perform parallel retrieval on the GPU to obtain a first retrieval result;

[0020] Relevance evaluation, quality evaluation and novelty calculation are performed on the first retrieval result;

[0021] According to the relevance evaluation result, the quality evaluation result and the novelty calculation result, using the maximum marginal relevance (MMR) algorithm, the first retrieval result is comprehensively sorted and analyzed and calculated;

[0022] According to the comprehensive sorting analysis calculation result, the first retrieval result is sorted;

[0023] Analyzing the first retrieval result to determine the complexity of the first retrieval result, and based on the complexity of the first retrieval result and the sorting result, the first final retrieval result is screened out;

[0024] The query content is analyzed to determine whether it is a multi-hop query. If not, the first final search result is output as the final search result. If yes, the key information of the first-hop search result is extracted, and the enhanced vector is updated according to the extracted key information to guide the search of the next hop. This process is repeated until the multi-hop query is completed, and the final search result is output.

[0025] Further, the step six comprises:

[0026] A GPT-3.5 large language model or a fine-tuned GPT-3.5 large language model is used to generate a preliminary answer based on the final search result and the input query content.

[0027] An NLI model is used to evaluate the consistency of the initial answer with the search result and the coverage of the key information to generate an evaluation result.

[0028] The evaluation result is used to guide the fine-tuning of the GPT-3.5 large language model to generate a final answer.

[0029] A rule-based method and a small BERT model are used to process the final answer, check possible errors, and standardize the terminology.

[0030] In another aspect, the present application provides a multi-modal metadata search enhancement generation system, comprising:

[0031] A human-computer interaction interface is used to input query content and display answers.

[0032] A multi-modal search and fusion module is used to process and fuse different modal data in the input content to generate a multi-modal feature fusion vector.

[0033] A metadata enhancement and filtering module is used to extract key metadata of each modal data from the query content and expand the key metadata, and convert the expanded metadata into a metadata vector.

[0034] A fusion module is used to combine the metadata vector and the multi-modal feature fusion vector to generate an enhanced vector.

[0035] A hierarchical search and reordering module is used to perform parallel search on GPU based on the enhanced vector using the Faiss library and output the final search result.

[0036] A fact-based optimization generation module is used to generate an initial answer using a large language model based on the output final search result, and to perform fact-based optimization on the generated initial answer to generate a final answer.

[0037] In yet another aspect, the present application provides a computer-readable storage medium comprising a stored program, wherein the program, when executed, controls a device in which the computer-readable storage medium is located to perform the above-mentioned multi-modal metadata retrieval enhancement generation method.

[0038] The present application has the following beneficial effects:

[0039] The present application determines the data modalities of the query content, extracts features from the data of each modality based on the determined data modalities, converts the extracted features into feature vectors, fuses the feature vectors of each modality to generate a multi-modal feature vector, which helps to improve the knowledge coverage range and information utilization efficiency. For example, when answering questions related to visual concepts, the system can utilize both textual descriptions and related image information to provide more comprehensive and accurate answers. The key metadata of each modality is extracted from the data of each modality, converted into a key metadata vector, connected with the multi-modal feature vector to construct an enhanced vector, and the enhanced vector is used for retrieval, which helps to improve the accuracy, relevance and credibility of the generated content. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, illustrate the preferred embodiments of the application, and assist in the explanation of the application. The drawings are not intended to limit the application in any way.

[0041] Fig. 1 is a flowchart of a multi-modal metadata retrieval enhancement generation method according to the present application.

[0042] Fig. 2 is a flowchart of generating a multi-modal feature fusion vector according to the present application.

[0043] Fig. 3 is a flowchart of generating a metadata vector according to the present application.

[0044] Fig. 4 is a flowchart of retrieving and outputting the final retrieval result based on the enhanced vector according to the present application. DETAILED DESCRIPTION

[0045] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, each technical and scientific term used in the present embodiment has the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0047] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, devices, components and / or combinations thereof, but do not preclude the presence or addition of one or more other features, steps, operations, devices, components and / or combinations thereof.

[0048] In the present application, the terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is a relationship word determined only for the purpose of describing the structural relationship of the components or elements of the present application, and cannot be understood as a limitation on the present application.

[0049] In the present application, the terms such as "fixedly connected", "connected", "connected" and the like should be understood broadly, which means that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant researchers or technicians in the art, the specific meaning of the above terms in the present application can be determined according to the specific circumstances, and cannot be understood as a limitation on the present application.

[0050] Multimodal data refers to data composed of two or more modalities, which can be different sources of information or different formats of the same information.

[0051] Embodiment 1:

[0052] As shown in FIG. 1, the present embodiment provides a multimodal metadata retrieval enhancement generation method, which comprises the following steps:

[0053] S1: input query content.

[0054] The query content is input through a human-computer interaction module, wherein the query content includes multimodal data.

[0055] For example, a doctor inputs text data and image data through a human-computer interaction module. For example:

[0056] Text description: analyze this chest X-ray film, combine the patient's symptom description and recent blood test results, evaluate the patient's likelihood of having a cold, and provide further examination recommendations.

[0057] Image data: data table for patient's chest X-ray image, symptom description text and blood test report.

[0058] S2: Determine the data modality of the query content, and process based on the determined data modality of the query content to generate a multi-modal feature fusion vector.

[0059] Specifically, as shown in FIG. 2:

[0060] S21: Analyze the query content to determine the data modality of the query content, wherein the data modality of the query content includes text, image, audio, and video.

[0061] S22: Preprocess the determined modality data of the query content, and extract the modality data features of the query content from the preprocessed modality data.

[0062] After processing such as word segmentation and stop word removal on the text data, the text features are extracted from the preprocessed text data by the RoBERTa model.

[0063] After processing such as normalization and enhancement on the image data, the image features are extracted from the preprocessed image data by the Vision Transformer (ViT) model.

[0064] After converting the audio data into a spectrum graph, the audio features are extracted from the spectrum graph by the Wav2Vec2.0 model.

[0065] S23: Convert the extracted modality query data features of the query content into fixed-dimension query data feature vectors, respectively.

[0066] Specifically, the extracted text features, image features, and audio features are projected into a unified dimensional space for normalization to convert into text feature vectors, image feature vectors, and audio feature vectors with the same dimension (such as 512 dimensions) as the vectors in the vector library.

[0067] S24: Fuse the query data feature vectors by a cross-modal attention mechanism and a modality importance dynamic weighting mechanism to obtain a multi-modal feature fusion vector.

[0068] The cross-modal attention mechanism is adopted by the Transformer structure to fuse the feature vectors of the fixed-dimension modality data. In the fusion process, the importance weight of each modality query data feature is dynamically calculated, and based on the calculated importance weight result, the weight of each modality query data feature vector is dynamically adjusted to obtain a query feature fusion vector.

[0069] The Transformer structure includes a self-attention layer and a cross-modal attention layer. The self-attention layer is used to capture the feature dependency of single-modal data, and the cross-modal attention layer is used to realize information interaction between different modal data.

[0070] S3: Extracting key metadata of each modality data from the query content by using a large language model, and expanding the key metadata, and converting the expanded key metadata into a metadata vector.

[0071] Specifically, as shown in FIG. 3:

[0072] S31: Extracting key metadata of each modality from the query content.

[0073] Using a fine-tuned T5 model to perform entity recognition, relation extraction, and attribute classification on each modality query data of the query content, so as to extract key metadata from each modality query data of the query content. The key metadata includes, but is not limited to, time range, information source, topic classification, and geographic location, etc.

[0074] S32: Reasoning out related factors of the key metadata according to the knowledge graph to expand the key metadata.

[0075] S33: Converting the expanded key metadata into a metadata vector.

[0076] S4: Connecting the metadata vector with the multi-modal feature fusion vector to generate an enhanced vector.

[0077] S5: Based on the enhanced vector, using the Faiss library to perform parallel retrieval on the GPU, and outputting the final retrieval result.

[0078] Wherein, the Faiss library is stored in the form of a file on the disk, and is loaded into the CPU or GPU for use when used.

[0079] Specifically, as shown in FIG. 4:

[0080] S51: Based on the enhanced vector, using the Faiss library to perform parallel retrieval on the GPU to obtain a first retrieval result.

[0081] For example, based on the enhanced vector, 100 relevant documents are retrieved.

[0082] S52: Performing relevance evaluation, quality evaluation, and novelty calculation on the first retrieval result.

[0083] Specifically, for example, 100 documents are retrieved, the content of the 100 retrieved relevant documents is evaluated, and the novelty and relevance of the content of the 100 retrieved relevant documents to the query content are calculated based on the query content, and the specific steps are as follows:

[0084] A) Quality evaluation:

[0085] The content of each document is scored using a pre-trained BERT model to obtain a language quality score, a coherence score and an informativeness score for each document.

[0086] Based on the obtained language quality score, coherence score and informativeness score of each document, a quality evaluation score for each document is calculated, and the specific calculation formula is as follows: Quality_Score = w1*Coherence + w2*Informativeness + w3*LanguageQuality

[0087] wherein w1, w2 and w3 are weight coefficients; Coherence is the coherence score; Informativeness is the informativeness score; LanguageQuality is the quality language score; Quality_Score is the quality evaluation score.

[0088] B) Relevance calculation:

[0089] The BM25 algorithm is used to calculate the relevance score of the query content and the content of each document, and the specific calculation formula is: Relevance_Score = IDF(qi)*((k+1)*tf(qi)) / (K+tf(qi)); wherein qi is the i-th word in the query term; IDF is the inverse document frequency; tf is the term frequency; k is the term frequency saturation parameter; K is the document length normalization factor; Relevance_Score is the relevance score.

[0090] C) Novelty calculation:

[0091] The Jaccard similarity is used to calculate the difference between the documents to obtain a novelty score, wherein the difference calculation formula is: Novelty_Score = 1-(|A∩B| / |A∪B|); wherein A and B are the keyword sets in the two documents.

[0092] Finally, the quality evaluation score, the relevance score and the novelty score are weighted and averaged to calculate the comprehensive score of each document, and the specific calculation formula is as follows:

[0093] Final_Score = α*Quality_Score + β*Relevance_Score + γ*Novelty_Score; wherein α, β and γ are adjustable weight parameters.

[0094] S53: According to the relevance evaluation result, the quality evaluation result and the novelty calculation result, the first search result is comprehensively sorted and analyzed using the Maximal Marginal Relevance (MMR) algorithm.

[0095] S54: According to the comprehensive ranking analysis calculation result, the first search result is ranked.

[0096] S55: The first search result is analyzed to determine the complexity of the first search result, and based on the complexity of the query content and the ranking result, the first final search result is screened out.

[0097] According to the length and type of the first final search result, the complexity of the search result is determined, for example, if video data and audio data are dominant in the search result, which consumes more system resources, then the number of final search results selected according to the ranking result should be less, for example, according to the ranking result, the first 10 search results can be selected as the final search result; if text data is dominant, the number of final search results selected can be appropriately expanded, for example, 20 first search results can be selected as the final search result.

[0098] S56: The query content is analyzed to determine whether it is a multi-hop query, if not, the screened first final search result is output as the final search result; if yes, the key information of the first-hop search result is extracted, and the enhanced vector is updated according to the extracted key information to guide the search of the next hop, and the process is repeated until the multi-hop query is completed, and the final search result is output.

[0099] Among them, the enhanced vector is updated according to the extracted key information to guide the search of the next hop, including:

[0100] Based on the updated enhanced vector, parallel search is performed on the GPU using the Faiss library to obtain the second search result;

[0101] The relevance, quality and novelty of the second search result are evaluated;

[0102] According to the relevance evaluation result, the quality evaluation result and the novelty calculation result, the maximum marginal relevance (MMR) algorithm is used to perform comprehensive ranking analysis calculation on the second search result;

[0103] According to the comprehensive ranking analysis calculation result, the second search result is ranked.

[0104] The second search result is analyzed to determine the complexity of the query content, and based on the complexity of the query content and the ranking result, the second final search result is screened out.

[0105] In turn, until the multi-hop is completed.

[0106] S6: According to the output final search result, an initial answer is generated using a large language model, and the generated initial answer is factually optimized to generate a final answer.

[0107] S61: Using a GPT-3.5 large language model or a fine-tuned GPT-3.5 large language model, generate a preliminary answer based on the final search results and the input query content.

[0108] S62: Use the NLI model to evaluate the consistency of the initial answer with the search results and the coverage of the key information to generate an evaluation result.

[0109] S63: Use the evaluation result to guide the fine-tuning of the GPT-3.5 large language model to generate the final answer.

[0110] Specifically, if the fine-tuning amplitude of the GPT-3.5 large language model is less than the preset threshold, the currently generated preliminary answer is the final answer, and if the fine-tuning amplitude of the GPT-3.5 large language model is not less than the preset threshold, return to step S61 until the fine-tuning amplitude of the GPT-3.5 large language model is less than the preset threshold.

[0111] S64: Use the rule-based method and small BERT model to process the final answer, check possible errors and standardize terms.

[0112] Embodiment 2:

[0113] The embodiment provides a multi-modal metadata retrieval enhancement generation system, comprising:

[0114] (I) A human-computer exchange interface for inputting query content and displaying answers.

[0115] (II) A multi-modal retrieval and fusion module responsible for processing and integrating data of different modalities in the query content, specifically including:

[0116] A modal data determination unit for determining the data modality of the query content;

[0117] A modal data processing unit for processing the determined data modality of the query content, such as using a RoBERTa model for encoding for text data, the RoBERTa model is pre-trained on large-scale text corpus and fine-tuned on specific domain data to extract text features and convert text features into text vectors. For image data, process through a pre-trained Vision Transformer (ViT) model to effectively capture local and global features of the image to extract image features and convert image features into image vectors. For audio data, use a Wav2Vec2.0 model to extract image features and convert image features into image vectors; the Wav2Vec2.0 model is trained on a self-supervised learning task and can effectively process various audio inputs.

[0118] A feature vector projection unit is configured to map the text vector, the image vector and the audio vector to a fixed, e.g., 512-dimensional vector space through a projection layer.

[0119] A fusion unit is configured to fuse the modality vectors by employing a Transformer structure, i.e., fuse the text vector, the audio vector and the image vector, to obtain a multi-feature fusion vector. The Transformer structure includes a modality-specific self-attention layer and a cross-modality attention layer. The self-attention layer captures feature dependencies within a single modality, while the cross-modality attention layer enables information interaction between different modalities.

[0120] A modality importance weighting unit is configured to adaptively adjust the weights of the modalities, i.e., the weights of the text vector, the audio vector and the image vector, according to the query content during the fusion of the modality vectors.

[0121] (ii) A metadata enhancement and filtering module extracts key metadata from the query content by using a fine-tuned T5 model, which specifically includes:

[0122] A key metadata extraction unit is configured to use a fine-tuned T5 model to train through a multi-task learning approach, while processing tasks such as entity recognition, relation extraction and attribute classification, to extract key metadata from the query content, including but not limited to time range, information source, topic classification and geographic location, etc.

[0123] A metadata expansion component is configured to use a knowledge graph to reason and expand the extracted key metadata, improve the recall rate of retrieval, and encode the expanded key metadata into a vector form to obtain a metadata vector.

[0124] (iv) An enhanced vector construction module is configured to connect the metadata vector and the multi-modal feature fusion vector to generate an enhanced vector.

[0125] (v) A hierarchical retrieval and re-ranking module implements an efficient multi-level retrieval strategy, which specifically includes:

[0126] A retrieval unit is configured to use the Faiss library to perform parallel retrieval on the GPU to obtain a first retrieval result.

[0127] A ranking unit is configured to use a BERT-based ranking model pre-trained on a large-scale ranking dataset and fine-tuned on a specific task to comprehensively rank and analyze the first retrieval result. The BERT-based ranking model considers multiple factors such as the relevance of the query content and the retrieval result, the quality of the retrieval result and the novelty of the retrieval result during the ranking process.

[0128] The dynamic depth controller analyzes the first search result, determines the complexity of the first search result, dynamically adjusts the search depth according to the complexity of the search result, balances the efficiency and recall rate, and screens the final search result based on the complexity of the search result and the sorting result.

[0129] The plurality of judgment units are configured to analyze the query content, determine whether it is a multi-hop query, if not, output the screened first final search result as the final search result; if yes, extract the key information of the first-hop search result, and update the enhanced vector according to the extracted key information to guide the search of the next hop, and the process is repeated until the multi-hop query is completed, and the final search result is output.

[0130] (Six) The fact-based optimization generation module adopts a PPO (Proximal Policy Optimization) algorithm framework, including:

[0131] The preliminary answer generation unit is configured to generate an initial answer based on the final search result using a pre-trained large language model (such as GPT-3.5).

[0132] The multi-round iteration unit is configured to perform multi-round quality evaluation on the generated initial answer to improve the factuality of the answer. In each round of iteration, an NLI (Natural Language Inference) model is used to evaluate the consistency of the initial answer with the final search result, and the coverage rate of the initial answer to the final search result is calculated. At the same time, the two indicators are combined into a comprehensive reward signal to guide the update of the NLI model parameters.

[0133] The post-processing component uses a rule-based method and a small-sized proofreading large language model to fine-tune to correct obvious factual errors and logical contradictions, and outputs the final answer.

[0134] Embodiment 3:

[0135] The embodiment provides a computer-readable storage medium including a stored program, wherein when the program is running, the computer-readable storage medium controls the device where the computer-readable storage medium is located to perform the multi-modal metadata search enhancement generation method of embodiment 1.

[0136] In the present specification, the same and similar parts among various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the description in the method embodiment.

[0137] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. For example, the division of the system embodiments described above is merely an example, and the units can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the display or discussion of a coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, and can be in electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0139] In addition, it should be noted that the flowchart in the accompanying drawings shows the method of the disclosed embodiment, and the operations or steps corresponding to the different blocks in the flowchart or block diagram in the accompanying drawings can also occur in an order different from that disclosed in the description. Sometimes, there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, which can depend on the functions involved. Each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0140] The above description is merely preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for multi-modal metadata retrieval augmentation generation, the method comprising: The method comprises the following steps: Step 1: input the query content; Step 2: determine the data modality of the query content and process based on the determined data modality of the query content to generate a multi-modal feature fusion vector; Step 3: use a large language model to extract key metadata of each modal data in the query content, and expand the key metadata, and convert the expanded key metadata into a metadata vector; Step 4: connect the metadata vector and the multi-modal feature fusion vector to generate an enhanced vector; Step 5: based on the enhanced vector, use the Faiss library to perform parallel retrieval on the GPU, and output the final retrieval result; Step 6: based on the output final retrieval result, use a large language model to generate an initial answer, and perform factual optimization on the generated initial answer to generate a final answer.

2. The multi-modal metadata retrieval augmentation generation method of claim 1, wherein, The step 2 comprises: analyze the query content to determine the data modality of the query content, wherein the data modality of the query content includes text, image, audio and video; preprocess each modal data of the determined query content, and extract the modal data features of the query content from the preprocessed modal data; convert the extracted modal data features of the query content into fixed-dimension query data feature vectors respectively; fuse the query data feature vectors through a cross-modal attention mechanism and a modal importance dynamic weighting mechanism to obtain a multi-modal feature fusion vector.

3. The multi-modal metadata retrieval augmentation generation method of claim 1, wherein, The step 5 comprises: based on the enhanced vector, use the Faiss library to perform parallel retrieval on the GPU to obtain a first retrieval result; perform relevance evaluation, quality evaluation and novelty calculation on the first retrieval result; based on the relevance evaluation result, the quality evaluation result and the novelty calculation result, use the maximum marginal relevance MMR algorithm to perform comprehensive sorting analysis calculation on the first retrieval result; sort the first retrieval result according to the comprehensive sorting analysis calculation result; analyze the first retrieval result to determine the complexity of the first retrieval result, and based on the complexity of the first retrieval result and the sorting result, filter out a first final retrieval result; analyze the query content to determine whether it is a multi-hop query, if not, output the filtered first final retrieval result as the final retrieval result; if yes, extract the key information of the first hop retrieval result, and update the enhanced vector based on the extracted key information to guide the retrieval of the next hop, and the same is repeated until the multi-hop query is completed, and the final retrieval result is output.

4. The multi-modal metadata retrieval augmentation generation method of claim 1, wherein, The step 6 comprises: use a GPT-3.5 large language model or a fine-tuned GPT-3.5 large language model to generate a preliminary answer based on the final retrieval result and the input query content; use an NLI model to evaluate the consistency of the initial answer with the retrieval result and the coverage of the key information to generate an evaluation result; use the evaluation result to guide the fine-tuning of the GPT-3.5 large language model to generate a final answer; use a rule-based method and a small BERT model to process the final answer, check possible errors and standardize terms.

5. A multi-modal metadata retrieval enhancement generation system, characterized by, The method comprises the following steps: a human-computer interaction interface for inputting query content and displaying answers; The multi-modal retrieval and fusion module is configured to process and fuse data of different modalities in the input content to generate a multi-modal feature fusion vector; The metadata enhancement and filtering module is configured to extract key metadata of the data of each modality from the query content, expand the key metadata, and convert the expanded metadata into a metadata vector; The fusion module is configured to fuse the metadata vector and the multi-modal feature fusion vector to generate an enhanced vector; The hierarchical retrieval and reordering module is configured to perform parallel retrieval on a GPU using a Faiss library based on the enhanced vector, and output a final retrieval result; The factuality optimization generation module is configured to use a large language model to generate an initial answer based on the output final retrieval result, and perform factuality optimization on the generated initial answer to generate a final answer.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to perform the multi-modal metadata retrieval enhancement generation method in any one of claims 1 to 4.

Images