A method, apparatus, device, medium, and product for enhancing search capabilities.

Abstract

This application discloses a retrieval enhancement generation method, apparatus, device, medium, and product, relating to the field of data processing. The method includes inputting a question into a trained multimodal embedding model to obtain a candidate text set, an image set, and a table set. The multimodal embedding model includes a multimodal encoding module for inputting text data, image data, and table data; inputting the question and candidate text set into a trained multigranularity retrieval model to obtain the optimal text combination; a text relationship modeling module based on a graph attention network in the multigranularity retrieval model for obtaining the importance score of each text data; and a dynamic programming module for obtaining the optimal text combination with the objective of maximizing the sum of importance scores. This application can improve the model's understanding, expressiveness, and retrieval capabilities when facing cross-modal scenarios, extremely long texts, and complex structured data.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, device, medium, and product for generating enhanced search results. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is a cutting-edge technique that incorporates external knowledge into generative models, attracting widespread attention in the field of natural language processing in recent years. The basic idea behind this method is to introduce an information retrieval module into the text generation process, searching for information relevant to the input question from an external knowledge base, thereby improving the accuracy and control of the generated text. This mechanism effectively alleviates the difficulties faced by models when dealing with long texts or insufficient domain-specific knowledge.

[0003] However, the data processed in reality is not limited to text; it also includes information in multiple modalities such as images and tables. Traditional RAG methods often face challenges such as semantic alignment difficulties, poor information fusion effects, and unstable quality of generated content when dealing with multimodal data. One reason is that different modalities of data differ significantly in their expression forms and structures, and direct splicing or fusion can easily lead to semantic bias or information loss. In addition, most current RAG methods use fixed-granularity retrieval strategies (such as fixed number of paragraphs or length), making it difficult for the system to flexibly adjust the retrieval scope when faced with diverse query requirements. This may introduce redundant information or omit key information, affecting the completeness and accuracy of the generated content.

[0004] To address these issues, academia and industry have proposed various strategies to improve multimodal RAG (Related Aspect-Oriented Data Collection), primarily including: unified feature space mapping methods, modality normalization processing strategies, and separate retrieval mechanisms. The unified vector space method attempts to map data from different modalities to the same semantic space, thereby achieving cross-modal relevance calculation and joint retrieval. While it has achieved some success in similarity calculation, the alignment accuracy between modalities still needs improvement. Modality unification methods improve processing efficiency by converting images, tables, etc., into text before processing, but this process easily leads to the loss of original information, especially with structured data. The separate retrieval mechanism establishes a separate retrieval system for each modality, extracting relevant content from each before merging them. While offering some flexibility, the fusion process still suffers from weak contextual coherence and insufficient integration of multimodal information.

[0005] Overall, existing multimodal RAG methods have not yet achieved a good balance between modality fusion capabilities and retrieval granularity control. This makes the model's understanding, expressiveness, and retrieval capabilities a core challenge that urgently needs to be overcome when facing cross-modal scenarios, ultra-long texts, and complex structured data. Summary of the Invention

[0006] The purpose of this application is to provide a retrieval enhancement generation method, apparatus, device, medium and product that can improve the understanding, expressiveness and retrieval capabilities of the model when facing cross-modal scenarios, ultra-long text and complex structured data.

[0007] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a retrieval enhancement generation method, comprising: inputting a question and an external knowledge base into a trained multimodal embedding model to obtain a candidate text set, an image set, and a table set; the multimodal embedding model includes: a multimodal encoding module for inputting text data, image data, and table data; the external knowledge base includes text data, image data, and table data.

[0008] The question and candidate text set are input into a trained multi-granularity retrieval model to obtain the optimal text combination. The multi-granularity retrieval model includes a text relationship modeling module based on graph attention network and a dynamic programming module connected in sequence. The text relationship modeling module based on graph attention network is used to obtain the importance score of each text data in the candidate text set. The dynamic programming module is used to solve the state transition equation with the objective of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination.

[0009] Input the question, the optimal combination of text, the set of images, and the set of tables into the generative model to obtain the answer to the question.

[0010] Secondly, this application provides a retrieval enhancement generation apparatus, comprising: a preliminary retrieval module for inputting a question and an external knowledge base into a candidate text set, an image set, and a table set obtained by a trained multimodal embedding model; the multimodal embedding model includes: a multimodal encoding module for inputting text data, image data, and table data; the external knowledge base includes text data, image data, and table data.

[0011] An optimal text combination generation module is used to input the question and candidate text set into a trained multi-granularity retrieval model to obtain the optimal text combination. The multi-granularity retrieval model includes a text relationship modeling module based on a graph attention network and a dynamic programming module connected in sequence. The text relationship modeling module based on the graph attention network is used to obtain the importance score of each text data in the candidate text set. The dynamic programming module is used to solve the state transition equation with the goal of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination.

[0012] The answer generation module is used to input the question, the optimal text combination, the image set, and the table set into the generation model to obtain the answer to the question.

[0013] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described retrieval enhancement generation method.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described retrieval enhancement generation method.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described search enhancement generation method.

[0016] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a retrieval enhancement generation method, apparatus, device, medium, and product. Most multimodal embedding models only consider text and image modalities. The structure of the multimodal embedding model in this application considers three modalities: text, image, and table. Furthermore, in the multi-granularity retrieval model, the image attention network calculates the importance score of each text through a multi-head attention mechanism, ensuring that relevant texts can be identified regardless of their granularity. It dynamically captures the relationships between texts of different granularities, avoiding the limitations of fixed granularity. It can achieve a good balance between modal fusion capability and retrieval granularity control, thereby improving the model's understanding, expressiveness, and retrieval capabilities when facing cross-modal scenarios, ultra-long texts, and complex structured data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a search enhancement generation method provided in an embodiment of this application.

[0019] Figure 2 This is an overall framework diagram of a search enhancement generation method provided in an embodiment of this application.

[0020] Figure 3 This is a flowchart of multimodal embedding model data processing provided in an embodiment of this application.

[0021] Figure 4 This is a diagram of a multimodal embedding model architecture provided in an embodiment of this application.

[0022] Figure 5 This is a flowchart of the image-text pair mask training task.

[0023] Figure 6 This is a flowchart for a text-image comparison learning task.

[0024] Figure 7 This is a flowchart for the image-text matching task.

[0025] Figure 8 Flowchart for text comparison training.

[0026] Figure 9 This is a schematic diagram of the GAMGR algorithm process.

[0027] Figure 10 A schematic diagram of the multi-head attention mechanism in graph attention networks.

[0028] Figure 11 This is a schematic diagram of the functional modules of a search enhancement generation device provided in an embodiment of this application.

[0029] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] This application provides a retrieval enhancement generation method. In one exemplary embodiment, such as... Figure 1 As shown, the retrieval enhancement generation method includes the following steps 101 to 103.

[0033] Step 101: Input the question and the external knowledge base into the candidate text set, image set and table set of the trained multimodal embedding model; the multimodal embedding model includes: a multimodal encoding module for inputting text data, image data and table data; the external knowledge base includes text data, image data and table data.

[0034] Step 102: Input the question and candidate text set into the trained multi-granularity retrieval model to obtain the optimal text combination; the multi-granularity retrieval model includes: a text relationship modeling module based on graph attention network and a dynamic programming module connected in sequence; the text relationship modeling module based on graph attention network is used to obtain the importance score of each text data in the candidate text set; the dynamic programming module is used to solve the state transition equation with the goal of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination.

[0035] Step 103: Input the question, the optimal text combination, the image set, and the table set into the generative model to obtain the answer to the question.

[0036] The multi-granularity retrieval model provided in this application is a GAT-based multi-granularity retrieval model (GAMGR), which can solve the problem of fixed granularity in traditional retrieval. By inputting the retrieved candidate text and question into this model, it can output an optimal text combination as the final text retrieval answer. Finally, by inputting the retrieved text, image, table, and question together into the generative model, the final answer can be obtained.

[0037] like Figure 2 As shown, the multimodal embedding model provided in this application is a Mixture-of-Experts (MoE) based Multimodal Embedding Model (MMEM). For multimodal data such as text, images, and tables, this model can convert them into feature vectors in the same space. Through similarity matching, multimodal retrieval can be performed using the input question, outputting text, images, and tables related to the question. In practical applications, the multimodal embedding model further includes: a multimodal fusion module connected to the multimodal encoding module; the multimodal encoding module includes: a table encoder, an image encoder, and a text encoder, all connected to the multimodal fusion module; the multimodal fusion module includes multiple Transformer modules connected in sequence; the fully connected feedforward layer in the Transformer module includes a parallel fusion modality expert feedforward network, an image modality expert feedforward network, and a text modality expert feedforward network.

[0038] In practical applications, text, image, and tabular data are encoded separately using dedicated encoders. The image encoder extracts visual features for the ViT model. The text encoder extracts semantic information for the BERT model. The tabular encoder obtains structured information for the TAPAS model. This multimodal encoding transforms different data into feature vectors, laying the foundation for subsequent multimodal fusion.

[0039] In another exemplary embodiment of this application, such as Figure 4 As shown, the Transformer module further includes: a first normalization layer, a multi-head self-attention mechanism, a first concatenation operation, a second normalization layer, and a second concatenation operation; the output of the first normalization layer is connected to the input of the multi-head self-attention mechanism, and the input of the first normalization layer is connected to the output of the multi-head self-attention mechanism through the first concatenation operation; the output of the first concatenation operation is connected to the input of the second normalization layer; the output of the second normalization layer is connected to the input of the fusion modality expert feedforward network, the input of the image modality expert feedforward network, and the input of the text modality expert feedforward network, respectively; the outputs of the fusion modality expert feedforward network, the image modality expert feedforward network, and the text modality expert feedforward network are all connected to the output of the first concatenation operation through the second concatenation operation.

[0040] The core architecture of MMEM includes a multimodal coding module, a multi-head self-attention mechanism, and a modality expert feedforward network (a fusion modality expert feedforward network, an image modality expert feedforward network, and a text modality expert feedforward network). A shared multi-head self-attention mechanism is used to align visual and linguistic content, while a modality expert feedforward network is assigned to each modality to capture modality-specific information. For example... Figure 3 As shown, the multimodal coding module first encodes the input multimodal data independently, generating their respective feature vectors. Then, it uses the MoE structure to dynamically select experts suitable for the current task for information fusion, performing unified feature encoding on the multimodal data. This not only effectively reduces the computational burden but also improves retrieval accuracy.

[0041] The overall architecture of MMEM is as follows: Figure 4As shown, image, text, and table data are first processed by independent encoding modules (image encoder, text encoder, and table encoder) to extract features, generating visual features, text features, and table features, respectively. These features then interact through a first normalization layer and a multi-head self-attention mechanism to achieve cross-modal information fusion. To better preserve the unique information of each modality and enhance intermodal collaborative understanding, the model introduces three paths in the feedforward neural network: a vision-specific image modality expert feedforward network (V-FFN), a text-specific text modality expert feedforward network (T-FFN), and a fusion modality expert feedforward network (Mix-FFN) that integrates modal features. Each path processes features for different modalities and then aggregates them through residual connections, improving the model's ability to model and represent multimodal semantics.

[0042] In practical applications, the model comprises 12 Transformer modules. Each Transformer module has 768 hidden layers and 12 attention heads. The design of the Transformer modules ensures the model's powerful representational capabilities, enabling it to capture long-range dependencies. At the top two layers of the Transformer modules, the model introduces hybrid modality expert modules. These expert modules are specifically designed to handle interaction information from different modalities, further enhancing the model's ability to process complex cross-modal information.

[0043] In another exemplary embodiment of this application, the training process of the multimodal embedding model specifically includes: pre-training the multimodal embedding model by sequentially using a text-image pair mask training task, a text-image comparison learning task, and a text-image matching task to obtain a pre-trained multimodal embedding model.

[0044] The pre-trained multimodal embedding model was fine-tuned using image-text retrieval tasks and table-text retrieval tasks in sequence to obtain a trained multimodal embedding model.

[0045] In practical applications, during the pre-training process of MMEM, three key training tasks are employed sequentially to learn the cross-modal association between images and text: Masked Multimodal Learning (MLM), Image-Text Contrastive Learning (ITC), and Image-Text Matching (ITM). These tasks work together to enable the model to effectively jointly represent images and text, providing stronger semantic understanding capabilities for downstream multimodal tasks.

[0046] In multimodal embedding models (MLMs), image-text pair masking training is a masked language modeling method that combines textual and visual information. This method follows BERT's masking strategy, masking 15% of the tokens in the input text. The model not only uses the remaining unmasked text tokens for prediction but also incorporates visual features from the image to provide more contextual information. The final output vector of the masked tokens is fed into a classifier whose output covers the entire text vocabulary and is trained using cross-entropy loss. The goal of MLM is to enable the model to learn to infer missing information from the remaining text and visual content, even when some textual information is missing, thereby enhancing multimodal representation learning capabilities and improving the model's performance in downstream tasks such as image-text matching and visual question answering.

[0047] The core of image-text pair mask training lies in dynamically fusing text and image features, such as Figure 5 As shown, it is divided into the following four steps.

[0048] (1) Input preparation: The model receives image-text pairs (images and their associated text descriptions). The text is converted into a token sequence by the BERT encoder, and the image is extracted into a visual feature vector by the ViT encoder.

[0049] (2) Text masking: Following BERT's masking strategy, 15% of the tokens in the text are randomly selected for masking (replaced with [MASK]), and the remaining 85% of unmasked tokens are retained as context clues.

[0050] (3) Multimodal feature fusion: The model obtains the missing part by sharing the unmasked token with the image visual feature input through a multi-head self-attention mechanism.

[0051] (4) Prediction and optimization: Calculate the cross-entropy loss function value based on the missing part and the mask part, obtain the difference between the predicted value and the original token, and pre-train the model based on the loss function value.

[0052] During training, image features and text tokens interact at the attention layer, and the prediction of mask positions (such as "[MASK]") directly benefits from visual input. This method overcomes the limitation of traditional mask language modeling, which relies solely on text, by explicitly combining visual features. In pre-training, it enhances the model's multimodal alignment capability, enabling generative question answering systems to more accurately complete information in visually rich scenes.

[0053] Image-text comparison learning is a multimodal representation learning method that improves the discriminative ability in retrieval tasks, making matched image-text pairs closer together and unmatched pairs further apart, thereby improving the overall retrieval performance. The image-text comparison learning process is as follows: Figure 6 As shown.

[0054] During the image-text comparison training process, given a batch containing... Given a pair of image-text pairs, if [I_CLS] and [T_CLS] are used as the aggregated representation of the image and text, and unified encoding is performed using MMEM, the final image vector set can be obtained. and text vector set . The vector representing the i-th image. Let represent the vector of the i-th text. From this, we can calculate the similarity between text and image, and then use softmax normalization to calculate the matching probability between image and text, and between text and image.

[0055] .

[0056] .

[0057] in, Indicates the similarity between images and text. Indicates the similarity between text and images. This represents the probability of an image matching the text. This represents the probability of a text-to-image match. It's a temperature over-parameter. Indicates calculation transpose, Indicates calculation The transpose of exp() denotes an exponential function with the natural constant e as its base.

[0058] During training, cross-entropy loss is used. To optimize the matching probability of image-to-text and text-to-image: .

[0059] Its core principle is to maximize the similarity of matched samples while minimizing the similarity of unmatched samples, thereby improving the model's ability to identify cross-modal associations.

[0060] Image-text matching tasks aim to teach models the matching relationships between images and text, thereby accurately determining whether input image-text pairs match. Training with ITM combined with ITC allows the model to further learn finer-grained matching relationships, optimizing its semantic reasoning ability and enabling it to understand more complex contextual information through matching tasks, thus improving its reasoning capabilities.

[0061] like Figure 7 As shown, the image-text matching task can be viewed as a binary classification task. The input is an image-text pair, and the model needs to determine whether they match. The training objective of the model is to maximize the probability of matching image-text pairs while minimizing the probability of not matching them. Specifically, the cross-entropy loss function is used. To optimize the model.

[0062] .

[0063] in, This represents the total number of training samples, which are image-text pairs. Indicates the first The labels of each training sample (a value of 1 indicates a match, and a value of 0 indicates a mismatch). Indicates the first The predicted matching probability of each training sample.

[0064] In practical applications, the main goal of the fine-tuning phase is to improve the model's performance in cross-modal retrieval tasks. Specifically, two downstream fine-tuning tasks were set up: image-to-text retrieval and table-to-text retrieval. In this phase, image-to-text and table-to-text comparison training were employed on domain datasets to optimize model performance. These two fine-tuning strategies can improve the model's performance in cross-modal retrieval tasks, making it more adaptable and usable in practical applications.

[0065] Image-text retrieval tasks aim to retrieve the most relevant images based on given text or the most relevant text based on given images. To improve the model's performance on image-text retrieval tasks in a specific domain, image-text comparison training is performed on a domain-specific dataset during the model fine-tuning phase. The core process is as follows.

[0066] (1) Retrieve the most relevant text based on the given image: Input the sample image into MMEM and load the text library simultaneously. The model calculates the similarity between the image features and all text features, and outputs the top-ranked sample candidate text set. Based on the sample candidate text set and the ground truth label (the matching text corresponding to the image), the image-text comparison loss is calculated using the loss function in the image-text comparison training, and the model parameters are updated through backpropagation.

[0067] (2) Retrieve the most relevant image based on the given text: Symmetrically, input the sample text into MMEM and load the image library. After the model retrieves the candidate images, calculate the loss value using the loss function in the image-text comparison training based on the candidate images and the real labels (the matching images corresponding to the text), and further optimize the model.

[0068] This process significantly enhances the model's ability to model the relationship between text and images through two-way supervised fine-tuning.

[0069] Table-text retrieval tasks refer to retrieving the most relevant tables based on text queries or retrieving the most relevant text based on table content. To improve the model's performance in table-text retrieval tasks within a specific domain, table-text contrastive training was introduced during the fine-tuning phase. Through contrastive learning, the representations of tables and text are mapped to the same semantic space, thereby achieving cross-modal alignment. For example... Figure 8 As shown, the idea behind table-text contrastive training is to bring the representations of matching table-text pairs closer together, while simultaneously pushing the representations of mismatched table-text pairs further apart. MMEM was trained on the table-text pairs in the test dataset for contrastive training. The contrastive loss function is the InfoNCE loss. The formula is: .

[0070] in, It is a table and text similarity, It is a positive sample. It is a negative sample. It is a temperature parameter.

[0071] In practical applications, multimodal embedding models can use a shared encoder architecture or a modal independent encoder plus a post-fusion mechanism to replace the expert model based on MoE.

[0072] The shared encoder architecture specifically employs a multimodal encoding module, which includes three parallel feature vector transformation modules for converting text, image, and tabular data into feature vectors.

[0073] The Transformer encoder module is used to input feature vectors of text, image, and tabular data into the Transformer encoder, and achieve cross-modal feature fusion through a multi-head self-attention mechanism.

[0074] The modal independent encoder plus post-fusion mechanism specifically means that the multimodal fusion module is a fully connected layer or a weighted average operation.

[0075] The text relationship modeling module based on graph attention networks first constructs a relationship graph between the retrieved candidate text sets using semantic similarity. Then, it calculates the importance scores of text nodes using the graph attention network and learns the correlations between texts using an attention mechanism. This module weights the influence of different nodes based on similarity and contextual information, enabling the model to capture the logical relationships between texts at the semantic level. Therefore, in another exemplary embodiment of this application, the text relationship modeling module based on graph attention networks includes: a graph construction module and a graph attention network connected sequentially. The graph construction module is used to construct a text relationship graph based on the cosine similarity between nodes; the nodes are the question and each text data in the candidate text set; the graph attention network is used to determine the importance score of each text data in the candidate text set based on the text relationship graph.

[0076] In another exemplary embodiment of this application, the graph construction module specifically includes: a vector representation unit, used to obtain a vector representation of the problem and a vector representation of each text data in the candidate text set.

[0077] The similarity calculation unit is used to calculate the cosine similarity between each text data in the candidate text set and the cosine similarity between the question and each text data in the candidate text set, based on the vector representation of the question and the vector representation of each text data in the candidate text set.

[0078] The text association graph construction unit is used to construct edges between nodes based on the cosine similarity between the text data in the candidate text set and the cosine similarity between the question and the text data in the candidate text set, thereby obtaining a text association graph.

[0079] In practical applications, GAMGR's input is a given problem. And using a multimodal embedding model to retrieve the top rankings A candidate text set consisting of 10 texts The goal is to Select the optimal text combination As the answer. A formula can be used. It indicates. Among them, This represents the relevance score of the answer obtained from the question and the set of candidate texts. This means finding the solution using dynamic programming that satisfies the evaluation function. Maximize S, ) is used to calculate the relevance score of the optimal text combination S given a question q and a candidate text set T.

[0080] To represent multi-granular text combinations Sentences need to be considered ,paragraph and chapters The hierarchical structure between sentences. If multiple sentences in a paragraph are selected, the entire paragraph is returned. If multiple paragraphs within a chapter are selected, the entire chapter will be returned. Otherwise, return individual sentences. The final returned text combination Represented as: .

[0081] in, From the candidate text set The optimal sentence combination selected from , This represents a paragraph, and each paragraph consists of multiple sentences. , This represents the m-th sentence in the j-th paragraph. This indicates a chapter, and each chapter consists of multiple paragraphs. , This represents the p-th paragraph in the k-th chapter. It should be a collection of paragraphs returned at the paragraph level. It should be a collection of chapters returned at the chapter level. This represents the set of all chapters, paragraphs, and independent sentences that do not form paragraphs that meet the criteria.

[0082] Defined as: .

[0083] Where | represents taking the absolute value, Indicates intersection, To represent the empty set, This is a paragraph-level threshold; if more than a certain number of characters in a paragraph are selected, the entire paragraph is returned. .

[0084] Defined as: .

[0085] in, This is a chapter-level threshold; if more than a certain number of paragraphs in a chapter are selected, the entire chapter is returned. .

[0086] Based on this, given a problem and candidate text set First, the vector representation unit needs to use a multimodal embedding model to convert the text into a vector representation, and the specific formula is as follows.

[0087] .

[0088] , .

[0089] in, ( ) represents a multimodal embedding model. The vector representation of problem q. Indicates candidate text The vector representation of , The set of real numbers is represented to ensure the mathematical continuity and computational feasibility of the embedded values. d represents the dimension, which determines the complexity and expressive power of the embedded representation.

[0090] Next, we construct a containing problem. and candidate text set The picture , where nodes The edges between nodes include two types: one is the problem. With candidate text The edge between them, another is candidate text. and candidate text The edges between them. First, a similarity calculation unit is used to calculate the similarity as the weight of the edge, and the specific formula is as follows.

[0091] .

[0092] .

[0093] in, Indicate the problem With candidate text Cosine similarity between them, as a problem With candidate text The weight of the edges between them. Indicates candidate text and candidate text Cosine similarity between them, as candidate texts and candidate text The weight of the edges between them. ( ) represents the cosine similarity function. When constructing the graph, it must satisfy... , This is the similarity threshold parameter.

[0094] Then, a text association graph is constructed using text association graph construction units.

[0095] picture After construction, a graph attention network is used to update the node representations, and importance scores are calculated based on the updated node representations. The process of updating node representations is as follows: Figure 10 As shown, the calculation formula is: .

[0096] in, This represents the updated node representation. Represents a node The neighborhood group, ( ) is a non-linear activation function. It is a trainable weight matrix. It is a node and nodes The attention weights between them are calculated using the following formula: .

[0097] in, It is an attention vector. Indicates calculation transpose, This represents vector concatenation. It is an activation function A variant of .

[0098] Finally, the importance score for each text is calculated using the updated node representation: .

[0099] in, Representing text Importance score The L2 norm of a vector is used to calculate the vector's L2 norm. The Euclidean length is used to map high-dimensional features to importance values. The larger the value, the more critical the text is. This formula evaluates importance based on the magnitude of the node representation vector, ensuring that the score directly reflects the weight of the text in the semantic graph.

[0100] In another exemplary embodiment of this application, the goal of the dynamic programming module is to extract from the graph constructed above. In this process, a set of candidate texts is selected that are complementary in content and rich in information. The formula is as follows: .in, This represents the optimal text combination. It is text The score reflects the relevance between the text and the question. It is text and text The edge weights between them reflect the correlation or redundancy information between them. It is an adjustment parameter used to adjust the strength of the redundancy penalty, thereby controlling the impact of duplicate information on the overall score.

[0101] To solve this optimization problem, a dynamic programming algorithm is used. The state transition equation is defined as follows: .in, Represents the i-th text in the candidate text set. The set of selected texts at the i-th iteration number The j-th text cosine similarity, Indicates the adjustment parameter. Represents the i-th text in the candidate text set. Importance score This represents the set with the best score among each first target text set; the first target text set is the top-scoring set among the candidate text sets. A set consisting of k randomly selected texts from a given set; This represents the set with the best score among each second target text set; the second target text set is the top-scoring set among the candidate text sets. A set consisting of k-1 randomly selected texts from a set of texts. This represents the set with the best scores among the third target text sets; the third target text set is the top-scoring set among the candidate text sets. A set consisting of k randomly selected texts from a given set of texts. Representing text With the selected text set Redundancy penalty term between them.

[0102] When considering whether to select text If not selected The state is If selected The score will increase. However, the text needs to be deducted. With the currently selected text set Redundancy penalty between Finally, through the stepwise recursion of dynamic programming, both the high relevance of individual texts and the information complementarity of the overall text combination can be considered, thus selecting the optimal text set. .

[0103] In this way, although the problem As nodes in the graph, they have already scored the candidate texts in the GAT stage. While this has an impact, the dynamic programming module only needs to focus on the complementarity and redundancy penalties among candidate texts, ensuring that the final selected text set can fully and without redundancy answer the question. Maximizing the total importance score of the selected texts while penalizing redundant similarity achieves dynamic adjustment of granularity, ensuring that the output combination is both complete and without redundancy.

[0104] In another exemplary embodiment of this application, the training of the GAMGR model mainly involves the graph attention network and the dynamic programming module. The training process optimizes the model parameters through backpropagation to improve the quality of the final text combination selection. The specific training process includes the following steps.

[0105] First, define a joint loss function. The function, comprising the loss from the graph attention network and the loss from the dynamic programming part, is as follows: .in, It is the binary cross-entropy loss, used to measure whether the question and candidate text are relevant; It is a graph structure loss, which encourages the graph attention network to learn the correct node representations; It is the optimization loss of dynamic programming, which aims to optimize the text selection process.

[0106] The goal of training is to minimize the joint loss function. During training, graph structure loss optimizes the node representations in the graph structure, making connections between similar text pairs closer and connections between unrelated text pairs more distant. Text selection loss optimizes text combination selection through dynamic programming, maximizing the score of correct text combinations while penalizing irrelevant text selections.

[0107] For graph structure loss, a contrastive loss is defined to measure the effectiveness of edges in the graph. For each text pair... If their true relationship is positively correlated, then we want their embeddings to be similar; conversely, if their relationship is negatively correlated, we want their embeddings to be differentiated. Therefore, we define the graph structure loss function as: .in, It is a set of sample pairs, consisting of related texts. It is a set of negative sample pairs, consisting of sample pairs composed of unrelated text. () is the Sigmoid function.

[0108] The goal of the dynamic programming module is to maximize the score of the final text combination by optimally selecting and combining texts. Text selection is calculated based on text importance scores and text similarity. The final selected text combination must maximize the score while avoiding redundant selections (i.e., repeatedly selecting multiple texts on the same topic). Therefore, the loss of dynamic programming can be defined as: .in, Representing text and the problem The correlation; and It refers to adjusting parameters.

[0109] Secondly, construct training data and labels.

[0110] The training data includes: sample query text: sample natural language questions input by the user.

[0111] Sample candidate text set: Text fragments corresponding to sample natural language questions obtained through MMEM retrieval, covering multiple granularities such as sentences, paragraphs, and chapters.

[0112] Sample text relationship graph: A graph structure constructed using texts in the sample candidate text set as nodes and semantic similarity as edge weights.

[0113] Tags include: text relevance tags; supervised binary cross-entropy loss. Label whether the sample query text is related to the sample candidate text set (label: 0 / 1).

[0114] Textual relationship label: supervised graph structural loss Label whether candidate sample text pairs belong to the same semantic topic (positive / negative sample pairs).

[0115] Optimal combination label: Supervision signal of dynamic programming module (implied in) (In the middle), the optimal combination of texts selected by manual screening is marked to maximize information coverage and minimize redundancy.

[0116] Finally, the model is trained, and the joint loss function L is optimized through end-to-end training. Specifically, labeled data is used to drive GAT to learn node representations and text importance scores.

[0117] The dynamic programming module learns the optimal combination strategy based on labels and adjusts parameters using a redundancy penalty term. During training, the sample query text and the sample candidate text set are input into the GAMGR model. The graph attention network learns node representations and text importance scores based on the sample text relationship graph (nodes include query and candidate texts, and edge weights are defined by similarity). The dynamic programming module then optimizes the strategy based on these scores and text similarity, combining the optimal combination labels (labeling the optimal text combination) and the redundancy penalty term. Specifically, during training, text relevance labels are input into the model to calculate the binary cross-entropy loss LBCE (measuring the relevance between the query and the text), and text relationship labels are used to calculate the graph structure loss L. graph (Optimize graph edge weights using positive and negative samples), calculate dynamic programming loss L using optimal combination labels. dp (Combining relevance score and redundancy penalty). The three are added together to obtain the joint loss L', and the GAT parameters and dynamic programming penalty coefficient λ are optimized through backpropagation to improve model performance.

[0118] In multimodal information retrieval and question-answering generation tasks, MMEM improves the modeling capability of multimodal tasks by jointly learning different modal data through expert networks. However, this model can only index and match text data of fixed granularity during the retrieval stage, which leads to the following problems in the retrieval results: 1. Redundant information: Since the model can only process text of fixed granularity (such as sentences or paragraphs), it may return multiple similar or repetitive text fragments during the retrieval process. This redundant information not only increases the computational burden but may also lead to unnecessary repetition in the generated answers, affecting the conciseness and readability of the answers. 2. Information loss: In some cases, queries need to integrate information from multiple texts of different granularities. Since the model can only process text of the same granularity during retrieval, it cannot flexibly capture more fine-grained or larger-grained correlations, leading to the omission of some key details and affecting the completeness and quality of the final answer. To solve the above problems, this application proposes GAMGR, which, for text data retrieved by MMEM, constructs a multi-granularity retrieval mechanism to dynamically adjust the granularity of the retrieval results, thereby improving the accuracy of text retrieval and the quality of question-answering generation. The algorithm data flow is as follows: Figure 9 As shown, the pseudocode for the algorithm is as follows.

[0119]

[0120] This application also has the following technical advantages: The MMEM provided in this application uses a hybrid expert architecture to weightedly fuse the outputs of the graph linear layer and the table linear layer, which can dynamically adapt to visual question answering tasks oriented towards data graphs and data tables, thereby enhancing the model's ability to dynamically adapt to cross-domain question answering tasks.

[0121] Existing RAG methods primarily focus on text-based modalities, failing to fully leverage the value of non-textual information such as images and tables, resulting in incomplete information representation in multimodal question-answering scenarios. To address this issue, this application introduces MMEM, which designs dedicated expert networks for different modalities (text, image, and table) and employs a dynamic routing mechanism to activate experts on demand. This ensures that information from each modality receives optimal representation and is mapped to a unified semantic space. The model encodes and effectively fuses multimodal information more accurately, significantly improving retrieval accuracy and enhancing the system's adaptability and generalization capabilities in complex multimodal retrieval and question-answering scenarios.

[0122] Fixed-granularity retrieval strategies cannot flexibly adapt to different query needs, easily leading to redundant information interference or omission of key information, thus affecting the accuracy and readability of the generated results. To address this issue, this application proposes GAMGR, which, based on preliminary retrieval results, models the correlation between texts using graph attention networks and then optimizes the combination path using dynamic programming algorithms. This enables dynamic filtering and combination of information at different granularities (such as sentences, paragraphs, and chapters), significantly improving the completeness, relevance, and semantic consistency of the generated answers, thereby enhancing the overall performance of the question-answering system.

[0123] Data from different modalities exhibit significant differences in their representation and structure. Existing methods, such as shared vector spaces or unified modality transformation, while achieving cross-modal processing to some extent, are prone to losing key information during feature alignment or modality transformation, especially when processing structured data such as tables, ultimately affecting the integrity and accuracy of the generated content. This application improves the representation and alignment capabilities of multimodal data (such as text, images, tables, etc.) at the feature level through MMEM, reducing information loss during transformation or fusion.

[0124] Most current RAG systems employ fixed-granularity retrieval strategies (such as fixed-length paragraphs or sentence counts), lacking the ability to flexibly adjust based on query semantics. This can lead to redundant returned information, increased generation burden, omission of key content, reduced answer quality, and impact on the system's responsiveness to complex questions. This application constructs a flexible and adjustable retrieval mechanism using GAMGR, supporting the dynamic selection of text fragments with appropriate granularity based on query semantics, thereby improving retrieval accuracy.

[0125] The MMEM proposed in this application aims to improve the accuracy of the retrieval system in the RAG architecture, thereby enhancing the question-answering performance of the entire system. This model is based on an improved Transformer architecture and utilizes the MoE concept to introduce a modality-aware multi-path expert feedforward network, enhancing the model's understanding and fusion capabilities of multimodal data (images, text, tables).

[0126] The GAMGR proposed in this application achieves more refined result filtering and organization based on the initially retrieved candidate texts. The model introduces GAT to model the relationships between candidate texts and uses a dynamic programming algorithm for text selection. By combining reordering and candidate selection, the performance of the final generation module is improved.

[0127] Based on the same inventive concept, this application also provides a retrieval enhancement generation apparatus for implementing the retrieval enhancement generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more retrieval enhancement generation apparatus embodiments provided below can be found in the limitations of the retrieval enhancement generation method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 11 As shown, a retrieval enhancement generation apparatus is provided, comprising:

[0129] The preliminary retrieval module is used to input the question and an external knowledge base into the candidate text set, image set, and table set of the trained multimodal embedding model; the multimodal embedding model includes a multimodal encoding module for inputting text data, image data, and table data; the external knowledge base includes text data, image data, and table data.

[0130] An optimal text combination generation module is used to input the question and candidate text set into a trained multi-granularity retrieval model to obtain the optimal text combination. The multi-granularity retrieval model includes a text relationship modeling module based on a graph attention network and a dynamic programming module connected in sequence. The text relationship modeling module based on the graph attention network is used to obtain the importance score of each text data in the candidate text set. The dynamic programming module is used to solve the state transition equation with the goal of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination.

[0131] The answer generation module is used to input the question, the optimal text combination, the image set, and the table set into the generation model to obtain the answer to the question.

[0132] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 12As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores retrieval-enhanced generation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a retrieval-enhanced generation method.

[0133] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0135] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0136] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0138] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0139] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A retrieval enhancement generation method, characterized in that, The retrieval enhancement generation method includes: The problem and an external knowledge base are input into a trained multimodal embedding model to obtain a candidate text set, an image set, and a table set; the multimodal embedding model includes a multimodal encoding module for inputting text data, image data, and table data; the external knowledge base includes text data, image data, and table data; The question and candidate text set are input into a trained multi-granularity retrieval model to obtain the optimal text combination. The multi-granularity retrieval model includes a text relationship modeling module based on a graph attention network and a dynamic programming module connected in sequence. The text relationship modeling module based on the graph attention network is used to calculate the importance score of each text data using the multi-head attention mechanism in the graph attention network to obtain the importance score of each text data in the candidate text set. The dynamic programming module is used to solve the state transition equation with the objective of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination. Input the question, the optimal combination of text, the set of images, and the set of tables into the generative model to obtain the answer to the question.

2. The retrieval enhancement generation method according to claim 1, characterized in that, The multimodal embedding model further includes a multimodal fusion module connected to the multimodal coding module; the multimodal coding module includes a table encoder, an image encoder, and a text encoder, all connected to the multimodal fusion module; the multimodal fusion module includes multiple Transformer modules connected in sequence; the fully connected feedforward layer in the Transformer module includes a parallel fusion modality expert feedforward network, an image modality expert feedforward network, and a text modality expert feedforward network.

3. The retrieval enhancement generation method according to claim 2, characterized in that, The Transformer module also includes: The system comprises a first normalization layer, a multi-head self-attention mechanism, a first concatenation operation, a second normalization layer, and a second concatenation operation. The output of the first normalization layer is connected to the input of the multi-head self-attention mechanism. The input of the first normalization layer is connected to the output of the multi-head self-attention mechanism via the first concatenation operation. The output of the first concatenation operation is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the inputs of the fusion modality expert feedforward network, the image modality expert feedforward network, and the text modality expert feedforward network, respectively. The outputs of the fusion modality expert feedforward network, the image modality expert feedforward network, and the text modality expert feedforward network are all connected to the output of the first concatenation operation via the second concatenation operation.

4. The retrieval enhancement generation method according to claim 1, characterized in that, The text relationship modeling module based on graph attention network includes: a graph construction module and a graph attention network connected in sequence; The graph construction module is used to construct a text association graph based on the cosine similarity between nodes; the nodes are the questions and each text data in the candidate text set; the graph attention network is used to determine the importance score of each text data in the candidate text set according to the text association graph.

5. The retrieval enhancement generation method according to claim 4, characterized in that, The graph construction module specifically includes: A vector representation unit is used to obtain a vector representation of the problem and a vector representation of each text data in the candidate text set; The similarity calculation unit is used to calculate the cosine similarity between each text data in the candidate text set and the cosine similarity between the question and each text data in the candidate text set, based on the vector representation of the question and the vector representation of each text data in the candidate text set. The text association graph construction unit is used to construct edges between nodes based on the cosine similarity between the text data in the candidate text set and the cosine similarity between the question and the text data in the candidate text set, thereby obtaining a text association graph.

6. The retrieval enhancement generation method according to claim 1, characterized in that, The training process of a multimodal embedding model specifically includes: The multimodal embedding model was pre-trained by sequentially employing image-text masking training tasks, image-text comparison learning tasks, and image-text matching tasks to obtain the pre-trained multimodal embedding model. The pre-trained multimodal embedding model was fine-tuned using image-text retrieval tasks and table-text retrieval tasks in sequence to obtain a trained multimodal embedding model.

7. A retrieval enhancement generation device, characterized in that, The retrieval enhancement generation device includes: The preliminary retrieval module is used to input the question and an external knowledge base into the candidate text set, image set, and table set of the trained multimodal embedding model; the multimodal embedding model includes a multimodal encoding module for inputting text data, image data, and table data; the external knowledge base includes text data, image data, and table data. An optimal text combination generation module is used to input the question and candidate text set into a trained multi-granularity retrieval model to obtain the optimal text combination. The multi-granularity retrieval model includes a text relationship modeling module based on a graph attention network and a dynamic programming module connected in sequence. The text relationship modeling module based on the graph attention network is used to calculate the importance score of each text data using the multi-head attention mechanism in the graph attention network to obtain the importance score of each text data in the candidate text set. The dynamic programming module is used to solve the state transition equation with the objective of maximizing the sum of the importance scores of each text data in the optimal text combination to obtain the optimal text combination. The answer generation module is used to input the question, the optimal text combination, the image set, and the table set into the generation model to obtain the answer to the question.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the retrieval enhancement generation method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the retrieval enhancement generation method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the retrieval enhancement generation method as described in any one of claims 1-6.

Images