Retrieval enhancement generation method and system based on two-stage multi-dimensional fusion

Abstract

The invention discloses a two-stage multi-dimensional fusion retrieval enhancement generation method and system, and belongs to the field of large model language processing. Comprising the following steps: dual-stage mixed retrieval: performing BM25 retrieval and vector retrieval in parallel and performing dynamic weighted fusion to realize file-level rough selection and content-level fine selection so as to give consideration to literal matching and semantic understanding; performing intelligent content integrity judgment: performing three-dimensional evaluation on a retrieval result by utilizing a lightweight language model, and triggering supplementary retrieval when information is insufficient; and multi-source information fusion generation: fusing multi-source information by adopting a model based on an attention mechanism, and driving a large language model to generate accurate and traceable answers. The method is also matched with a continuous training optimization method of a small model and an efficient system management architecture based on an asynchronous queue. According to the method, the accuracy and the integrity of the answers are remarkably improved, the retrieval efficiency and the expandability of the system are improved, and the calculation cost is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of large model language processing technology, and in particular to a two-stage, multi-dimensional fusion retrieval enhancement generation method and system. Background Technology

[0002] Retrieval Augmentation (RAG) is a technique that combines large language models (LLMs) with external knowledge bases. It aims to improve the accuracy and timeliness of large language model responses by providing precise, fact-based context, thereby effectively mitigating the inherent "illusion" problem.

[0003] Existing RAG technologies face multiple challenges in implementation and application. The inherent limitations of retrieval methods are manifested in the fact that while sparse retrieval, represented by BM25, performs well for precise term matching, it cannot understand deep semantics, resulting in limited recall. Conversely, dense retrieval, represented by vector retrieval, can handle semantic matching but may ignore key specific words, and the model cost is high. Using either method alone is unlikely to achieve optimal results when handling complex queries. System efficiency and scalability issues are also prominent. When the knowledge base is huge, global dense retrieval will bring huge computational overhead and unacceptable query latency. The construction and maintenance of indexes are costly, while some optimization methods, such as tree-based retrieval, are limited by their complex structure and the difficulty of implementation and maintenance.

[0004] Poor quality of contextual information is also a challenge. If the retrieved text fragments have low information density, contain noise, or lack key metadata, it will directly reduce the quality of the context provided to the LLM and affect the accuracy of the final generated content. Summary of the Invention

[0005] Therefore, the purpose of this invention is to provide a two-stage, multi-dimensional fusion retrieval enhancement generation method, which aims to solve how to improve the accuracy and completeness of responses from large language models through intelligent retrieval strategies, while effectively reducing computational costs and improving system scalability.

[0006] To achieve the above objectives, the present invention provides a two-stage, multi-dimensional fusion-based retrieval enhancement generation method, comprising the following steps: S1. Two-stage hybrid retrieval: For user queries, firstly, a file-level coarse selection is performed. Candidate document sets based on literal matching and semantic matching are obtained through parallel execution of BM25 retrieval and vector retrieval, respectively. The two retrieval results are dynamically weighted and fused to filter out the initial selected documents. Then, a content-level fine selection is performed. The initial selected documents are further refined to extract the text paragraphs most relevant to the query and form the initial retrieval context. S2. Intelligent content integrity judgment: Using a trained small language model, the initial retrieval context is evaluated in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered. S3. Multi-source information fusion generation: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generating an enhanced context representation. Based on the context representation, a large language model is driven to generate the final answer with information source annotations.

[0007] Furthermore, in S1, the content-level selection specifically involves: using a sliding window to divide the initially selected document into text blocks, using a Transformer-based paragraph reordering model to calculate the relevance score between each text block and the query, and thereby selecting a predetermined number of text paragraphs.

[0008] Furthermore, when obtaining the candidate document set based on literal matching and semantic matching, and dynamically weighting and fusing the two search results, BM25 and vector retrieval run in two independent thread pools. The dynamic weighting and fusing adopts the specificity-aware dynamic weighting and fusing method shown in the following formula: ; in, It's about query specificity. A monotonically increasing function. For BM25 search results, This is the result of a vector search.

[0009] Furthermore, in S2, the small language model adopts the DistilBERT architecture, which is a 6-layer Transformer architecture. Model compression employs knowledge distillation technology. Distillation loss function L = α* L_task + β* L_distill + γ* L_hidden Where L_task is the task loss, L_distill is the distillation loss, L_hidden is the hidden layer loss, α=0.5, β=0.3, γ=0.2.

[0010] Furthermore, in S2, the triggering supplementary retrieval mechanism includes: expanding the retrieval scope, adjusting the similarity threshold, or calling a tree-structured hierarchical retrieval algorithm to obtain more comprehensive information.

[0011] Furthermore, the tree-structured hierarchical retrieval algorithm is executed based on a tree-structured hierarchical index, which is constructed in the following way: starting from the bottom semantic unit of the document, a large language model is recursively called to summarize the semantics of adjacent nodes and generate upper-level nodes, until a root node representing the core semantics of the entire document is formed, thereby constructing a bottom-up tree structure.

[0012] Furthermore, the training data construction for S4 and the small model is automated. Based on the tree-structured retrieval results, positive and negative samples are automatically labeled through comparative analysis. Positive samples contain all the information needed to answer the question, while negative samples have information gaps. Based on the positive and negative samples, a continuous optimization strategy including online learning and adversarial training is adopted to improve the model's judgment accuracy and robustness.

[0013] Furthermore, it also includes: S5, asynchronous queue processing and system management: adopting an asynchronous processing architecture based on distributed message queues to schedule document processing, retrieval and generation tasks; the architecture supports multi-priority task management, a caching mechanism based on LRU policy, and a multi-level fault recovery strategy including retry, circuit breaking and degradation to ensure the system's high concurrency processing capability and high availability.

[0014] This invention also provides a two-stage, multi-dimensional fusion-based retrieval enhancement generation system, comprising: The dual-stage hybrid retrieval engine first performs a document-level coarse selection for user queries. It obtains candidate document sets based on literal matching and semantic matching through parallel execution of BM25 retrieval and vector retrieval, respectively, and dynamically weights and merges the two retrieval results to filter out the initial selected documents. Then, it performs a content-level fine selection to refine the initial selected documents and extract the most relevant text paragraphs to form the initial retrieval context. Intelligent content integrity judgment module: Utilizes a trained small language model to evaluate the initial retrieval context in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered; Multi-source information query fusion module: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generate an enhanced context representation, and drive a large language model based on the context representation to generate the final answer with information source annotations.

[0015] The two-stage, multi-dimensional fusion-based retrieval enhancement generation method and system disclosed in this application have at least the following advantages compared with the prior art: By employing a two-stage hybrid retrieval mechanism, the precise term matching capability of BM25 and the deep semantic understanding capability of vector retrieval are integrated in parallel during the initial document-level coarse selection, achieving dual coverage of "literal + semantic". In the content-level fine selection, a Transformer-based paragraph reordering method is used to accurately extract the most relevant text fragments from the initially selected documents. This design ensures retrieval speed in large-scale knowledge bases while significantly improving recall and precision, effectively addressing the limitations of single retrieval methods under complex queries. It innovatively introduces a lightweight intelligent judgment model to evaluate the relevance, completeness, and sufficiency of search results in real time across three dimensions. This model can proactively trigger supplementary searches (such as calling tree-structured searches) when information is insufficient, thus avoiding the problem of large language models generating "illusions" or incorrect answers due to search defects, and significantly improving the reliability of generated answers and user credibility. By constructing a tree-like hierarchical index and implementing a top-down retrieval strategy, multi-granular and hierarchical semantic mining can be performed on long documents and complex structural knowledge. Combined with a multi-source information fusion model based on an attention mechanism, heterogeneous information from different retrieval paths and text fragments can be dynamically weighted and integrated to generate enhanced context with high information density and strong coherence, thereby supporting the large language model to produce more comprehensive and accurate answers. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the two-stage, multi-dimensional fusion-based retrieval enhancement generation method proposed in this invention.

[0017] Figure 2 This is a flowchart of a two-stage hybrid retrieval process.

[0018] Figure 3 This is a schematic diagram of a tree-like hierarchical index structure.

[0019] Figure 4 This is a schematic diagram of a tree-structured hierarchical retrieval process. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] like Figure 1 As shown, one embodiment of the present invention provides a two-stage, multi-dimensional fusion-based retrieval enhancement generation method, comprising the following steps: S1. Two-stage hybrid retrieval: For user queries, firstly, a file-level coarse selection is performed. Candidate document sets based on literal matching and semantic matching are obtained through parallel execution of BM25 retrieval and vector retrieval, respectively. The two retrieval results are dynamically weighted and fused to filter out the initial selected documents. Then, a content-level fine selection is performed. The initial selected documents are further refined to extract the text paragraphs most relevant to the query and form the initial retrieval context. The two-stage hybrid retrieval mechanism divides the retrieval process into two levels: document-level coarse selection and content-level fine selection. This design ensures both retrieval efficiency and result quality.

[0022] During the document-level coarse selection phase, the system runs two algorithms in parallel: BM25 and vector retrieval. The BM25 algorithm focuses on literal matching, accurately identifying key information such as technical terms, product names, and specifications within documents. Vector retrieval, on the other hand, understands the query intent at the semantic level, capturing synonyms, related concepts, and implicit meanings. The two algorithms run independently without interference, and the results are finally integrated through a weighted fusion. The fusion weights are dynamically adjusted based on the actual application scenario; BM25 has a higher weight in technical document retrieval, while vector retrieval has a higher weight in conceptual queries.

[0023] The content-level selection phase involves in-depth analysis of the initially selected files. Instead of searching the entire document library, the system focuses on the candidate files themselves, employing more sophisticated algorithms to extract relevant paragraphs. Each file is segmented into appropriately sized text blocks, and the system calculates the relevance score of each text block to the query. The score calculation comprehensively considers multiple factors such as keyword matching, semantic similarity, and text position, ultimately selecting the most relevant text fragments to form the context.

[0024] The file-level coarse selection stage employs a parallel computing architecture, with BM25 and vector retrieval running in two independent thread pools. The BM25 algorithm is based on an inverted index, using block compression technology during index construction. Each document block contains 1000 documents, with variable byte encoding used to compress the document ID list within each block, and a skip list structure used between blocks to accelerate navigation. During query processing, the system first performs word segmentation and stemming on the query, using a statistical word segmentation algorithm combined with a domain dictionary for specialized terminology recognition. The BM25 scoring formula uses a standard implementation: score(q,d) = Σ IDF(qi) * (tf(qi,d) *(k1 + 1)) / (tf(qi,d) + k1 * (1 - b + b * |d| / avgdl)), where k1=1.2 and b=0.75.

[0025] Vector retrieval employs an approximate nearest neighbor search algorithm based on the HNSW (Hierarchical Navigative Small World) graph. During graph construction, each node maintains a maximum number of connections M=16, and the hierarchical structure uses an exponentially decaying probability distribution, with a probability of 1 at the bottom layer and the probability of each upper layer multiplied by 1 / ln(M). During querying, a multi-path search strategy is used, starting from the entry node and performing greedy searches at multiple levels simultaneously. The number of search steps at each level is limited to ef=200. Finally, the results from each level are merged and deduplicated. Vector similarity calculation uses cosine similarity, optimized using the FAISS library and supported for acceleration via the SIMD instruction set.

[0026] The fusion algorithm employs a dynamic weighted fusion method based on specificity awareness: .in It's about query specificity. The function is monotonically increasing to ensure that high-specificity queries focus on lexical matching, while low-specificity queries focus on semantic reasoning. Finally, the top 10 files with the highest scores are selected for the next stage.

[0027] Content-level selection employs a BERT-based paragraph ranking model. The model structure is a 12-layer Transformer with 768 hidden layers and 12 attention heads. Training data uses a manually labeled query-paragraph relevance dataset containing 100,000 samples. The model input is in the format "[CLS]query[SEP]paragraph[SEP]", and the output is the relevance score. During inference, paragraphs in each file are batch-processed using a sliding window strategy with a window size of 512 and a stride of 256 to ensure full content coverage. Finally, the top 5 paragraphs with the highest scores from each file are selected as context.

[0028] S2. Intelligent content integrity judgment: Using a trained small language model, the initial retrieval context is evaluated in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered. Traditional RAG systems often blindly trust search results, failing to determine whether the obtained information is sufficient to answer the question. This invention introduces a small language model for intelligent evaluation, effectively solving this problem.

[0029] This small model has far fewer parameters than the generative model, but possesses professional judgment capabilities. It evaluates retrieval quality from three dimensions: relevance measures the degree of match between the search content and the query; completeness determines whether the information covers all aspects of the question; and sufficiency predicts whether a satisfactory answer can be generated based on the current information.

[0030] The training data for the small model comes from real-world application scenarios. The system collects a large number of query-retrieval-answer triples, using the tree-structured retrieval results as the standard answer. The union of the retrieval results for this project and the standard answer is taken as the positive sample, and the difference is taken as the negative sample. In this way, the model learns to identify information gaps and acquires professional judgment capabilities.

[0031] In practical applications, the small model quickly analyzes the search results. If insufficient information is found, a supplementary search mechanism is immediately triggered. This supplementary search employs a more aggressive strategy, expanding the search scope and invoking advanced algorithms such as tree-structured retrieval to ensure comprehensive information is obtained. This intelligent judgment mechanism significantly improves the quality of answers and avoids incorrect responses due to insufficient information.

[0032] The small language model adopts the DistilBERT architecture, with 6 Transformer layers, 768 hidden layer dimensions, and 66M parameters. Model compression employs knowledge distillation. The teacher model is BERT-based, and the student model is trained by mimicking the output distribution and attention matrix of the teacher model. The distillation loss function is L = α * L_task + β * L_distill + γ * L_hidden, where L_task is the task loss, L_distill is the distillation loss, and L_hidden is the hidden layer loss, with α=0.5, β=0.3, and γ=0.2.

[0033] The training data was constructed using an automated annotation process. The system collected 1 million real queries, performing both tree-structured search and item-specific search on each query. The tree-structured search employed a recursive search algorithm based on document hierarchy, refining the search scope layer by layer from the document level to the paragraph level and then to the sentence level. The annotation rule was: if the item-specific search results contained all the key information from the tree-structured search results, they were marked as positive samples; if information was missing, they were marked as negative samples. In this way, 800,000 training samples were automatically generated, with a positive-to-negative sample ratio of 1:1.

[0034] The model training employs a progressive learning strategy. The initial stage uses a large-scale general corpus for pre-training to learn fundamental language knowledge; the second stage uses domain-specific corpora for fine-tuning to adapt to the characteristics of the professional domain; and the third stage uses task-specific data for fine-tuning to optimize judgment accuracy. During training, techniques such as dynamic masking and data augmentation are used to enhance the model's generalization ability.

[0035] The inference phase employs batch processing optimization, handling up to 32 queries per session with an average response time of 50ms. Judgment results are categorized into three levels: complete (confidence > 0.8), partially complete (0.5 < confidence ≤ 0.8), and incomplete (confidence ≤ 0.5). For incomplete cases, the system triggers supplementary retrieval, expanding the search scope to the entire knowledge base and employing a more lenient similarity threshold (reduced to 0.6) to ensure comprehensive information is obtained.

[0036] S3. Multi-source information fusion generation: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generating an enhanced context representation. Based on the context representation, a large language model is driven to generate the final answer with information source annotations.

[0037] High-quality information fusion is key to generating accurate answers. This invention employs a multi-level fusion strategy to organically integrate information from different sources.

[0038] The preprocessing stage involves cleaning and standardizing the search results. The system removes duplicate content, standardizes formatting, and eliminates inconsistencies in terminology. For different expressions of the same concept, the system establishes a standardized mapping to ensure information consistency.

[0039] The tree-structured hierarchical index employs a bottom-up construction strategy based on an embedding model. The construction process begins with the smallest semantic unit of the document and recursively summarizes it to form a hierarchical tree structure.

[0040] The specific process of index construction is as follows: First, the original document is semantically sliced ​​using an embedding model to form the base layer n0. The slicing process adopts a sliding window strategy with a window size of 512 tokens and a step size of 256 tokens to ensure semantic integrity. Each slice is converted into a 768-dimensional vector representation through the embedding model, forming the leaf nodes of a tree structure.

[0041] Then, a large language model is invoked to semantically summarize the adjacent nodes of layer n0, forming the upper-level node n1. The summarization process employs a clustering-based merging strategy, grouping semantically similar leaf nodes into groups of 4-8 nodes each. The LLM summary hint template requires the model to extract key information, generate a concise summary, and retain important details. The vector representation of the nodes in layer n1 is obtained through average pooling: v_n1 = mean(v_n0_1, v_n0_2, ..., v_n0_k), where k is the number of child nodes.

[0042] The recursive summarization process continues, with level n2 generated from the summarization of level n1 nodes, level n3 from the summarization of level n2 nodes, and so on, until the top-level node n is generated. The number of nodes at each level decreases exponentially, forming a typical tree structure. The top-level node n contains the core semantics of the entire document and is the root node of the entire tree. The entire construction process is optimized using parallel computing, allowing different branches to be processed simultaneously, significantly improving construction efficiency.

[0043] The tree-structured search employs a top-down hierarchical search strategy. At the start of the search, the system first converts the user query into a vector representation, then calculates its similarity with the top-level node n. A flexible pruning strategy based on dynamic confidence is used. The pruning threshold is no longer a fixed value (0.7), but rather depends on the specificity of the current query. The current layer depth is dynamically adjusted. When the query specificity is low, the threshold is automatically lowered to recall more 'semantic sibling nodes'; at the same time, when calculating node similarity, the residual connection weight of the node's parent node is introduced to prevent erroneous pruning due to the loss of upper-layer summary information.

[0044] At layer n-1, the system performs fine-grained matching on the selected nodes, calculating the similarity between the query and each child node. The similarity calculation uses weighted cosine similarity: sim(q, n_i) = cos(v_q, v_n_i) * w_layer, where w_layer is the hierarchical weight coefficient, with deeper nodes having higher weights. The top-k nodes (k=3) with the highest similarity are selected to proceed to the next layer of retrieval.

[0045] This process continues until the leaf node layer n0 is reached. At the leaf node layer, the system collects all selected original text blocks to form a candidate context set. These text blocks are then finely ranked using a re-ranking model, selecting the most relevant segments to form the final context. The re-ranking uses a BERT-based cross-encoder model, with the input being the query and the concatenation of text blocks, and the output being a relevance score.

[0046] The entire tree-structured search process optimizes performance through a caching mechanism. Search results from intermediate nodes are cached to avoid redundant calculations. The cache employs an LRU (Least Recently Used) strategy, automatically evicting infrequently used results to balance memory usage and search efficiency.

[0047] The fusion phase employs an attention-based algorithm. The system assigns dynamic weights to each text segment, reflecting the importance, reliability, and relevance of the information. Important information receives higher weights, while redundant information is appropriately downweighted. The fusion process is not a simple stitching together, but a deep integration of information, generating new and more comprehensive information representations.

[0048] The generation phase leverages the powerful capabilities of large language models. Carefully designed prompt templates guide the model to generate answers based on fused information, requiring the model to remain faithful to the source material and avoid subjective assumptions. Simultaneously, the model needs to annotate the sources of information to enhance the traceability of the answers. The generated answers undergo post-processing steps such as grammar checking and fact-checking to ensure quality standards are met.

[0049] Multi-source information fusion employs a deep fusion algorithm based on an attention mechanism. The fusion model uses an Encoder-Decoder architecture, where the Encoder part uses a bidirectional LSTM to process the input sequence, and the Decoder part uses a unidirectional LSTM to generate the fused representation. The attention mechanism uses Bahdanau Attention to calculate the importance weight of each position in the input sequence. The fusion process consists of three stages: feature extraction, weight calculation, and information integration.

[0050] In the feature extraction stage, the system extracts semantic features, positional features, and confidence features from each text segment. Semantic features are represented as 768-dimensional vectors using the BERT model; positional features include paragraph position, sentence position, and word position; confidence features are calculated based on the retrieval score and the model's predicted probability. The weight calculation stage employs a bilinear attention mechanism: e_ij = v^T tanh(W_1 h_i + W_2 s_j), where h_i is the query representation, s_j is the text segment representation, W_1 and W_2 are learnable parameters, and v is the attention vector. The information integration stage uses a weighted summation method: c_i = Σ α_ij * s_j, where α_ij is the normalized attention weight.

[0051] Answer generation employs a prompt engineering strategy. The prompt template comprises four parts: task description, contextual information, constraints, and output format. The task description explicitly states that the model must answer the question based on given information; the contextual information includes the fused text content; the constraints require the model not to fabricate information and to base it on the given content; the output format requires the inclusion of the answer and source annotations. The generation process uses a beam search algorithm with a beam width of 4 and a length penalty coefficient of 0.6 to ensure the fluency and accuracy of the generated answers.

[0052] High-quality information fusion is key to generating accurate answers. This invention employs a multi-level fusion strategy to organically integrate information from different sources. The preprocessing stage cleans and standardizes the retrieval results. The system removes duplicate content, standardizes formatting, and eliminates inconsistencies in terminology. For different expressions of the same concept, the system establishes a standardized mapping to ensure information consistency. The fusion stage uses an attention-based algorithm. The system assigns dynamic weights to each text segment, reflecting the importance, reliability, and relevance of the information. Important information receives higher weights, while redundant information is appropriately downweighted. The fusion process is not a simple splicing but a deep integration of information, generating new and more comprehensive information representations. The generation stage leverages the powerful capabilities of a large language model. Carefully designed prompt templates guide the model to generate answers based on the fused information, requiring the model to be faithful to the source material and avoid subjective assumptions. Simultaneously, the model needs to annotate the information sources to enhance the traceability of the answers. The generated answers undergo post-processing steps such as grammar checking and fact verification to ensure quality standards are met.

[0053] The training data construction for S4 and smaller models is automated, using tree-structured search results as a benchmark and automatically generating labels through comparative analysis. Positive samples contain all the information needed to answer the question, while negative samples have information gaps. This data construction method avoids the cost of manual annotation while ensuring data quality. An online learning mechanism allows for continuous model improvement. The system collects user feedback, analyzes the discrepancies between the model's judgments and actual needs, and adds new training data to the dataset. Regular retraining adapts the model to new application scenarios, maintaining accuracy. Adversarial training enhances model robustness. The system generates difficult samples containing noise and misleading information, training the model to maintain accurate judgments in complex environments. This training method significantly improves the model's practicality, enabling it to handle various complex queries.

[0054] The small-model online learning approach employs a policy gradient method based on experience replay. The system maintains an experience pool, storing recent judgments and user feedback. Each experience sample contains information such as query and retrieval results, model judgments, user feedback, and the quality of the actual answer. When the experience pool accumulates to a certain size (1000 samples), the model update process is triggered.

[0055] The update algorithm employs the Proximal Policy Optimization (PPO) algorithm, which optimizes model parameters by maximizing the reward function. The reward function is designed as: R = R_accuracy + R_coverage + R_user, where R_accuracy is based on answer accuracy, R_coverage is based on information coverage, and R_user is based on user satisfaction. The PPO algorithm uses a clipped surrogate objective: L^CLIP(θ) = E[min(r_t(θ)A_t, clip(r_t(θ), 1-ε, 1+ε)A_t)], where ε=0.2 to ensure the stability of model updates.

[0056] Adversarial training employs a GAN-based framework. The generator produces noisy, difficult samples, while the discriminator judges the authenticity of the samples. The generator's goal is to minimize the discriminator's accuracy, while the discriminator's goal is to maximize classification accuracy. The training process uses minimax optimization: min_G max_D V(D,G) = E_x[log D(x)] + E_z[log(1-D(G(z)))]. Through adversarial training, the model learns to maintain accurate judgments in noisy environments, significantly improving its robustness.

[0057] The model evaluation employs a multi-dimensional indicator system. Accuracy indicators include precision, recall, and F1 score; robustness indicators include noise tolerance and adversarial example accuracy; efficiency indicators include inference time and memory usage. The system undergoes comprehensive evaluation periodically, and when performance degradation exceeds a threshold (5%), model retraining is triggered to ensure continuous performance optimization.

[0058] It also includes: S5, asynchronous queue processing and system management: adopting an asynchronous processing architecture based on distributed message queues to schedule document processing, retrieval and generation tasks; the architecture supports multi-priority task management, a caching mechanism based on LRU policy, and a multi-level fault recovery strategy including retry, circuit breaking and degradation to ensure the system's high concurrency processing capability and high availability.

[0059] Efficient system management ensures the practicality of the entire solution. This invention adopts an asynchronous architecture, supporting high-concurrency processing and intelligent resource scheduling. Document processing tasks are distributed through a message queue. The system supports batch uploading and parallel processing, significantly improving processing efficiency. A priority mechanism ensures that important tasks are executed first, and timeout control prevents resource waste.

[0060] Status monitoring provides a real-time view of system operation. Administrators can view key metrics such as queue length, processing speed, and error rate to promptly identify and resolve issues. A comprehensive logging system records detailed operational information, facilitating troubleshooting and performance optimization.

[0061] A fault recovery mechanism ensures system stability. Failed tasks are automatically retried, and the retry strategy is configurable. The system supports graceful degradation, allowing it to continue providing services even when some components fail. This design ensures high system availability and meets the needs of production environments.

[0062] The asynchronous queue system adopts a distributed message queue architecture based on Redis. The queue design supports features such as multiple priorities, multiple consumer groups, and message persistence. The message structure includes fields such as message ID, message type, priority, creation time, retries, and business data. Queue operations use atomic commands to ensure that messages are not lost or duplicated.

[0063] Task scheduling employs a priority-based multi-level feedback queue algorithm. The system maintains multiple priority queues, with higher-priority tasks processed first. Tasks are dynamically adjusted among queues; low-priority tasks that have been waiting for a long time are gradually promoted in priority to avoid starvation. The scheduler uses a work-stealing mechanism, allowing idle consumers to steal tasks from other consumers' queues, thus improving system throughput.

[0064] Fault recovery employs a multi-layered strategy. The network layer uses a retry mechanism, automatically retrying failed operations three times with exponential backoff intervals of 1 second, 2 seconds, and 4 seconds. The application layer uses a circuit breaker pattern; when the failure rate exceeds a threshold (50%), the circuit breaker enters an open state, suspending service calls, and then enters a half-open state for tentative calls. The data layer uses a transaction mechanism to ensure data consistency, automatically rolling back upon failure.

[0065] The monitoring and alerting system uses a Prometheus-based metrics framework. Key metrics include queue length, processing latency, failure rate, retries, and system load. Alert rules employ multiple threshold settings, with different notification methods for different severity levels. Emergency alerts are sent via telephone, important alerts via SMS, and general alerts via email. The system supports an alert escalation mechanism; unprocessed alerts will be escalated for processing after a certain period.

[0066] The logging system employs a structured log design, with each log entry containing fields such as timestamp, log level, service name, trace ID, error code, and detailed information. Logs are collected uniformly using Fluentd, stored in Elasticsearch, and visualized and analyzed using Kibana. The system supports log tracing, connecting the entire request processing flow through a unique trace ID, facilitating problem localization and analysis.

[0067] This invention also provides a two-stage, multi-dimensional fusion-based retrieval enhancement generation system, comprising: The dual-stage hybrid retrieval engine first performs a document-level coarse selection for user queries. It obtains candidate document sets based on literal matching and semantic matching through parallel execution of BM25 retrieval and vector retrieval, respectively, and dynamically weights and merges the two retrieval results to filter out the initial selected documents. Then, it performs a content-level fine selection to refine the initial selected documents and extract the most relevant text paragraphs to form the initial retrieval context. Intelligent content integrity judgment module: Utilizes a trained small language model to evaluate the initial retrieval context in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered; Multi-source information query fusion module: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generate an enhanced context representation, and drive a large language model based on the context representation to generate the final answer with information source annotations.

[0068] The overall system architecture of this invention comprises five core modules. The user query input module receives and processes user-input query requests, preprocesses, segments, and vectorizes the queries, and uses a pre-trained embedding model to convert the queries into 768-dimensional vector representations. The dual-stage hybrid retrieval engine consists of a document-level coarse selection stage and a content-level fine selection stage. In the document-level coarse selection stage, BM25 retrieval and vector retrieval algorithms run in parallel. BM25 retrieval accurately identifies technical terms based on an inverted index, while vector retrieval uses the HNSW graph algorithm to understand the query intent at the semantic level. The two retrieval results are integrated through a weighted fusion algorithm. In the content-level fine selection stage, a sliding window strategy is used to segment documents into 512-token text blocks. A BERT-based paragraph ranking model is used to calculate relevance scores, and the top 5 paragraphs with the highest scores are selected as the final context. The intelligent content integrity judgment module uses a small language model to evaluate the retrieval results in three dimensions, triggering a supplementary retrieval mechanism when the judgment information is insufficient. The multi-source information fusion generation module uses a deep fusion algorithm based on an attention mechanism, assigning dynamic weights to each text segment and using a large language model to generate answers based on the fused information. The asynchronous queue processing module is implemented based on Redis, supports multi-priority scheduling and fault recovery, and uses a multi-level feedback queue algorithm to ensure that high-priority tasks are processed first.

[0069] In practice, the user query input module uses the FastAPI framework to build a RESTful API interface, supporting HTTP / HTTPS protocol communication. The query preprocessing submodule uses regular expressions for text cleaning, jieba for Chinese word segmentation, and snowballstemmer for stemming. Vectorization processing uses the Qwen3-Embedding-8B model, generated through an OpenAI-like API, producing a 768-dimensional dense vector representation. The system is configured with connection pool management, supporting a maximum of 64 concurrent connections with a timeout of 30 seconds.

[0070] The two-stage hybrid search engine is implemented using the Python asynchronous programming framework asyncio, employing a thread pool Executor to manage concurrent tasks. The BM25 search submodule uses the rank-bm25 library, storing document metadata in a PostgreSQL database and extending the storage vector representation using pgvector when building the inverted index. The vector search submodule implements the HNSW graph algorithm based on the FAISS library, configured with M=16 maximum connections, efConstruction=200 construction parameters, and efSearch=200 search parameters. The fusion algorithm uses a linear weighting approach, setting α=0.7 for technical document queries and α=0.4 for concept understanding queries, dynamically adjusting the weight parameters through a configuration file.

[0071] The content-level selection stage employs a BERT-based paragraph re-ranking model. The model architecture consists of a 12-layer Transformer encoder with 768 hidden layers, 12 attention heads, and approximately 110M parameters. Paragraph segmentation uses a sliding window strategy with a window size of 512 tokens, a stride of 256 tokens, and a 50% overlap rate to ensure semantic coherence. The re-ranking model uses a cross-encoder architecture, taking the concatenated sequence of query and paragraph as input and outputting a relevance score, normalized to the 0-1 range using a sigmoid function. The inference stage utilizes batch processing optimization with a batch size of 32, employing GPU acceleration to achieve an average response time of less than 100ms.

[0072] Two-stage hybrid retrieval process implementation The two-stage hybrid retrieval process introduces a generative query expansion mechanism (HyDE). First, a structured pseudo-document suggestion template is built to guide a large language model to generate queries based on user queries. Hypothetical documents are generated. The user query and the hypothetical documents are vectorized separately. The arithmetic mean of the hypothetical document vectors is calculated and weighted proportionally with the original query vector to generate a final 768-dimensional enhanced semantic vector representation. The system simultaneously launches BM25 retrieval and vector retrieval threads. BM25 retrieval uses literal matching based on an inverted index, calculates term frequency statistics using TF-IDF, and calculates relevance scores using the standard BM25 scoring formula. Vector retrieval uses an HNSW graph for approximate nearest neighbor search, maintaining a maximum of M=16 connections per node, with a search step limit of ef=200, and calculates semantic relevance using cosine similarity. The two retrieval results are sorted and normalized, and a weighted fusion algorithm is used to select the Top 10 documents for the next stage. Candidate documents are segmented into paragraphs. After initially selecting the Top-K high-scoring document paragraphs (center blocks), a neighborhood expansion algorithm is executed. The system automatically obtains the preceding and following neighbor chunks of each center block in the original document's physical location, using the distance decay formula... Calculate neighborhood scores. The central block and its neighboring segments are used together as a candidate set, and a deep relevance calculation is performed using the BERT cross-encoder to maintain the semantic coherence of the context.

[0073] In practice, query preprocessing employs a multi-language support strategy: the jieba word segmenter for Chinese and the NLTK toolkit for English, supporting user-defined dictionary expansion. The word segmentation algorithm uses a statistically based Hidden Markov Model, combined with a domain dictionary for specialized terminology recognition; the dictionary contains over 100,000 specialized words. Stemming uses the Porter Stemmer algorithm, supporting word form reconstruction in multiple languages. Vectorization processing uses batch processing with a batch size of 64, supporting dynamic batch optimization that automatically adjusts the batch size based on query length.

[0074] The BM25 retrieval thread is implemented using an independent process pool, with the number of processes set to twice the number of CPU cores, supporting a maximum of 16 concurrent retrieval tasks. The inverted index construction employs block compression technology, with each document block containing 1000 documents. Within each block, the document ID list is compressed using Variable Byte encoding, and a Skip List structure is used between blocks to accelerate navigation. TF-IDF calculation uses an incremental update strategy, supporting real-time index updates. Term frequency statistics are stored using 32-bit unsigned integers, and document frequencies are stored using 16-bit unsigned integers. The BM25 scoring formula parameters are set to k1=1.2 and b=0.75, conforming to standard configuration and supporting parameter tuning interfaces.

[0075] The vector retrieval thread is GPU-accelerated using the CUDA parallel computing framework, supporting multi-GPU parallel processing. The HNSW graph construction employs a hierarchical navigation strategy, with randomized selection of the entry node and an exponentially decreasing number of nodes at each level. Similarity calculation uses cosine similarity, optimized through vectorization and supported for acceleration via the SIMD instruction set. Search results are sorted using a multi-way merge sort, with a priority queue size of 100, supporting dynamic adjustment of the TopK results.

[0076] The result fusion stage employs a ranking-based weighted fusion algorithm, normalizing the BM25 and vector retrieval results to the 0-1 interval. Weight allocation utilizes a dynamic adaptive strategy based on query specificity. The system first calculates the specificity value S(q) of the query q, calculated as the sum of TF×IDF of all words in the query, where TF is the term frequency and IDF is the inverse document frequency based on the corpus. The dynamic fusion coefficient α is calculated in real-time based on the value of S(q): when S(q) is high (containing rare terms), the BM25 weight α is automatically increased via a Sigmoid variant function; when S(q) is low (fuzzy query), the vector retrieval weight (1−α) is automatically increased, replacing the fixed manually preset weights. The fusion results are processed using a heap sort algorithm with a time complexity of O(nlogk) and a space complexity of O(k), supporting Top 10 document selection.

[0077] Tree-like hierarchical index structure implementation The hierarchical tree index employs a bottom-up construction strategy. An embedding model is used to semantically slice the original document, with a window size of 512 tokens and a stride of 256 tokens, generating a 768-dimensional vector representation, forming leaf nodes n0. A large language model is then invoked to semantically summarize adjacent nodes, using a clustering and merging strategy, grouping 4-8 nodes into each group to generate upper-level summary nodes. The upper-level node vectors are obtained through average pooling, and the above process is repeated until the top-level root node is generated, forming a complete tree structure.

[0078] In practice, document semantic slicing employs a sentence boundary detection-based algorithm, using regular expressions to identify delimiters such as Chinese periods and English punctuation marks, ensuring the semantic integrity of the slices. The slice size is set to 512 tokens, with an overlapping area of ​​256 tokens, and a 50% overlap rate to guarantee contextual coherence. The embedding model uses Qwen3-Embedding-8B, accessible via an API interface, supporting batch processing with a batch size of 32, and an average response time controlled within 200ms.

[0079] Node clustering employs the density-based DBSCAN algorithm, with the epsilon parameter set to 0.3 and the min_samples parameter set to 4 to automatically determine the number of clusters. Clustering features use vector cosine similarity as the distance metric, and a KD-tree is used to accelerate nearest neighbor search. Semantic summarization utilizes the large-scale language model Qwen3-32B, accessed via an API interface. The prompt template includes a task description, contextual information, and output format requirements, with the generated length limited to 200 tokens.

[0080] The hierarchical construction employs a recursive merging strategy, reducing the number of nodes at each level by a ratio of 4:1, forming a typical tree structure. The root node contains the core semantics of the entire document, generated through recursive summarization. Vector representation updates utilize average pooling, v_n1 = mean(v_n0_1, v_n0_2, ..., v_n0_k), ensuring complete transmission of semantic information. The construction process is optimized for parallel computation, allowing different branches to run concurrently, and improving construction efficiency through multithreading techniques.

[0081] Tree-structured hierarchical retrieval process implementation The tree-structured search employs a top-down hierarchical strategy. User queries are converted into 768-dimensional vector representations, and the similarity between the query and the root node is calculated, followed by branch selection and pruning. The child nodes of the selected branches are traversed, and weighted cosine similarity is calculated. Hierarchical weight coefficients ensure that deeper nodes receive higher weights, and the results are cached using an LRU strategy. All selected original text blocks are collected, reordered using a BERT cross-encoder, and the Top K most relevant segments are selected to form the final context.

[0082] In practice, query vectorization uses the same embedding model as the index construction to ensure vector space consistency. Similarity calculation uses cosine similarity, optimized through vectorization, and supports SIMD instruction set acceleration. Branch selection employs a Top 3 strategy, choosing the three child nodes with the highest similarity to proceed to the next level of retrieval, balancing search accuracy and efficiency. The pruning threshold is set to 0.7; nodes with similarity below this threshold are pruned to avoid unnecessary computation.

[0083] The hierarchical weight coefficients adopt a linear increasing strategy, w_layer = 1 + 0.1 * layer, ensuring that deeper nodes receive higher weights. The caching mechanism employs an LRU strategy, with a cache size of 10,000 nodes, maintaining a cache hit rate above 85%, significantly improving retrieval efficiency. Cache updates utilize a write-back strategy, asynchronously updating persistent storage to reduce real-time computation overhead.

[0084] The re-ranking model employs a BERT-based cross-encoder architecture. The input is a concatenated sequence of queries and text blocks, with a maximum length of 512 tokens. The output is a relevance score. Model training uses a manually labeled dataset containing 100,000 query-text block pairs, optimized with a binary classification loss function. Batch processing is used for inference, with a batch size of 16, and GPU acceleration is employed, resulting in an average response time of less than 150ms.

[0085] System performance optimization implementation This invention achieves performance optimization at multiple levels. BM25 and vector retrieval are executed in parallel using independent thread pools, significantly improving retrieval efficiency. Intermediate layer retrieval results are cached using LRU to avoid redundant calculations, balancing memory usage and retrieval efficiency. Paragraph sorting and re-sorting processes are batch-processed, handling up to 32 queries at a time with an average response time of 50ms. Document processing tasks are executed asynchronously via message queues, supporting high concurrency. Real-time monitoring of system operation status allows for timely detection and resolution of problems, ensuring system stability.

[0086] In practice, parallel computing is implemented using a thread pool (Executor). The core thread count is set to the number of CPU cores, the maximum thread count is set to twice the number of cores, and the queue capacity is set to 1000, supporting dynamic thread pool expansion. The thread pool employs a pre-creation strategy to reduce thread creation overhead and improve response speed. Task scheduling uses a work-stealing algorithm, where idle threads actively steal tasks from other threads' task queues, improving CPU utilization.

[0087] The caching mechanism employs a multi-level caching architecture: a first-level cache uses memory, a second-level cache uses a Redis distributed cache, and a third-level cache uses a local disk cache. The memory cache uses an LRU strategy, with a cache size of 1GB and a maximum of 100,000 cache items. The Redis cache is deployed in a cluster, supporting master-slave replication and sentinel mode to ensure high availability. The disk cache uses SSD storage, with a cache size of 10GB, and supports cache preheating and batch loading.

[0088] Batch processing optimization employs a dynamic batching strategy, automatically adjusting the batch size based on query length and system load, with a minimum batch size of 8 and a maximum batch size of 64. Batch processing utilizes a pipelined architecture, supporting parallel execution of query preprocessing, model inference, and result post-processing, thereby improving overall throughput. Memory management employs object pooling technology, reusing frequently created objects and reducing garbage collection overhead.

[0089] Asynchronous processing is implemented using a Redis-based message queue, supporting publish-subscribe and point-to-point modes. The message queue is deployed in a cluster, supporting master-slave replication and partitioned storage to ensure high availability and scalability. Message persistence uses a combination of AOF logs and RDB snapshots to ensure no message loss. Consumers are deployed in a cluster, supporting load balancing and failover; the failure of a single consumer does not affect the overall service.

[0090] The monitoring system adopts a Prometheus-based monitoring architecture, supporting multi-dimensional metric collection and storage. Key metrics include query latency, throughput, error rate, and resource utilization. Monitoring data is stored in a time-series database, supporting long-term storage and historical data analysis. The alarm system supports multi-level alarm configuration, including emergency, important, and general levels, using different notification methods. The system supports automatic fault recovery; when service anomalies are detected, it automatically restarts the service or switches to a backup node to ensure service continuity.

[0091] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A two-stage, multi-dimensional fusion-based retrieval enhancement generation method, characterized in that, Includes the following steps: S1. Two-stage hybrid retrieval: For user queries, firstly, a file-level coarse selection is performed. Candidate document sets based on literal matching and semantic matching are obtained through parallel execution of BM25 retrieval and vector retrieval, respectively. The two retrieval results are dynamically weighted and fused to filter out the initial selected documents. Then, a content-level fine selection is performed. The initial selected documents are further refined to extract the text paragraphs most relevant to the query and form the initial retrieval context. S2. Intelligent content integrity judgment: Using a trained small language model, the initial retrieval context is evaluated in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered. S3. Multi-source information fusion generation: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generating an enhanced context representation. Based on the context representation, a large language model is driven to generate the final answer with information source annotations.

2. The two-stage, multi-dimensional fusion-based retrieval enhancement generation method according to claim 2, characterized in that, In S1, the content-level selection specifically involves: using a sliding window to divide the initially selected document into text blocks, using a Transformer-based paragraph reordering model to calculate the relevance score between each text block and the query, and thereby selecting a predetermined number of text paragraphs.

3. The dual-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 2, characterized in that, When obtaining the candidate document set based on literal and semantic matching and dynamically weighting and fusing the two search results, BM25 and vector retrieval run in two independent thread pools. The dynamic weighting and fusing adopts the specificity-aware dynamic weighting and fusing method shown in the following formula: ； in, It's about query specificity. A monotonically increasing function. For BM25 search results, This is the result of a vector search.

4. The two-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 2, characterized in that, In S2, the small language model adopts the DistilBERT architecture, which is a 6-layer Transformer architecture. Model compression employs knowledge distillation technology. Distillation loss function L = α* L_task + β* L_distill + γ* L_hidden Where L_task is the task loss, L_distill is the distillation loss, L_hidden is the hidden layer loss, α=0.5, β=0.3, γ=0.

2.

5. The two-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 2, characterized in that, In S2, the triggering supplementary retrieval mechanism includes: expanding the retrieval scope, adjusting the similarity threshold, or calling a tree-structured hierarchical retrieval algorithm to obtain more comprehensive information.

6. The two-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 5, characterized in that, The tree-structured hierarchical retrieval algorithm is executed based on a tree-structured hierarchical index, which is constructed in the following way: starting from the bottom semantic unit of the document, a large language model is recursively called to summarize the semantics of adjacent nodes and generate upper-level nodes, until a root node representing the core semantics of the entire document is formed, thereby constructing a bottom-up tree structure.

7. The two-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 5, characterized in that, It also includes the S4 model. The training data construction is automated. Based on the tree-structured retrieval results, positive and negative samples are automatically labeled through comparative analysis. Positive samples contain all the information needed to answer the question, while negative samples have information gaps. Based on the positive and negative samples, a continuous optimization strategy including online learning and adversarial training is adopted to improve the model's judgment accuracy and robustness.

8. The two-stage, multi-dimensional fusion retrieval enhancement generation method according to claim 5, characterized in that, It also includes: Step 5, Asynchronous queue processing and system management: An asynchronous processing architecture based on distributed message queues is adopted to schedule document processing, retrieval and generation tasks; the architecture supports multi-priority task management, a caching mechanism based on LRU policy, and a multi-level fault recovery strategy including retry, circuit breaking and degradation to ensure the system's high concurrency processing capability and high availability.

9. A two-stage, multi-dimensional fusion-based retrieval enhancement generation system, characterized in that, include: The dual-stage hybrid retrieval engine first performs a document-level coarse selection for user queries. It obtains candidate document sets based on literal matching and semantic matching through parallel execution of BM25 retrieval and vector retrieval, respectively, and dynamically weights and merges the two retrieval results to filter out the initial selected documents. Then, it performs a content-level fine selection to refine the initial selected documents and extract the most relevant text paragraphs to form the initial retrieval context. Intelligent content integrity judgment module: Utilizes a trained small language model to evaluate the initial retrieval context in three dimensions, including relevance to the query, completeness of the answer, and sufficiency of the generated answer; when the evaluation result is lower than a preset threshold, a supplementary retrieval mechanism is triggered; Multi-source information query fusion module: The initial retrieval context or the context expanded by the supplementary retrieval mechanism is used as multi-source information input. An attention-based fusion model is used to integrate and remove redundancy, generate an enhanced context representation, and drive a large language model based on the context representation to generate the final answer with information source annotations.

Images