Retrieval system and method based on retrieval enhancement generation

Abstract

The invention discloses a retrieval system and method based on retrieval enhancement generation. The method comprises the following steps: analyzing and determining a query type according to a received query request; distributing a corresponding retrieval path according to the query type; under the condition that the retrieval path is an accurate matching retrieval path, extracting a key part from the query request, constructing a filtering query condition aiming at a metadata field in the vector database, and obtaining a retrieval result according to the filtering query condition; under the condition that the retrieval path is a mixed retrieval path, according to whether the query request needs to be subjected to query preprocessing or not, a corresponding query process is distributed, and a retrieval result is obtained. According to the method, the technical effects of system architecture simplification, retrieval efficiency and precision improvement, result quality optimization and observability of the processing process can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology applications, and in particular to a retrieval system and method based on retrieval enhancement generation. Background Technology

[0002] With the rapid development of information technology, various industries have accumulated massive amounts of unstructured document data, such as technical standards, operation and maintenance manuals, research reports, and regulations. How to quickly and accurately extract knowledge from these massive, heterogeneous documents has become a key challenge for improving enterprise operational efficiency and decision-making quality.

[0003] Traditional knowledge retrieval techniques primarily rely on keyword matching. This method achieves document retrieval by building a keyword index, but its drawbacks are also quite obvious: First, it cannot deeply understand the semantics of natural language and has difficulty handling synonyms, near-synonyms, or abbreviations of domain terms. Second, keyword matching is only sorted based on word frequency, which cannot determine the true relevance of the content and often returns a large number of irrelevant results, requiring users to spend a lot of time manually filtering them.

[0004] In existing engineering practices, mainstream RAG systems generally suffer from the following unresolved technical defects during implementation: First, the system architecture is generally complex and inefficient. Current RAG systems, when implementing hybrid retrieval (i.e., combining keyword exact matching and semantic similarity retrieval), typically employ a separate architecture where a full-text search engine (such as Elasticsearch) and a vector database coexist. This architecture introduces significant engineering complexity: firstly, it imposes a heavy operational burden, requiring the deployment, monitoring, and maintenance of two independent, heterogeneous data storage systems; secondly, data consistency is difficult to guarantee, as the same knowledge document needs to be processed and written to both systems, easily leading to data synchronization delays, failures, or inconsistencies; and thirdly, performance bottlenecks exist, as the application layer needs to send requests to both systems separately during retrieval, and then manually merge and sort the two sets of heterogeneous results, increasing network overhead and computational complexity, thus limiting overall retrieval efficiency.

[0005] Second, the retrieval strategy is rigid and lacks adaptability. Existing RAG systems often use a fixed retrieval process to handle all types of user queries. This model cannot dynamically adjust based on the content and characteristics of the query itself. For example, when a user enters a well-formatted standard number or device model, this is a highly deterministic query that should quickly return a unique result through exact matching. However, the system still executes a time-consuming and potentially unstable semantic retrieval process. Conversely, when a user enters a fuzzy query with insufficient information, the system lacks an effective query enhancement mechanism to improve recall, leading to retrieval failures.

[0006] Third, the information quality of the search results is low. Even if the retrieval engine recalls a batch of candidate documents, the context provided to the large language model often suffers from information redundancy and spurious relevance. On the one hand, multiple candidate segments may differ literally, but their core content is highly repetitive, occupying valuable context window length and reducing "information density." On the other hand, some segments, although semantically similar to the query, are not direct answers to the question and belong to "spurious relevance" information. This not only interferes with the comprehensive judgment of the large language model but may even induce it to produce incorrect "illusionary" answers, thus affecting the accuracy of the final answer.

[0007] There are currently no effective solutions to the problems that RAG systems generally suffer from, such as complex system architecture, poor data consistency, rigid retrieval strategies, and low quality of returned results. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention aims to provide a retrieval system and method based on retrieval enhancement generation, thereby at least resolving the common issues in current RAG systems, such as complex system architecture, poor data consistency, rigid retrieval strategies, and low quality of returned results.

[0009] The technical solution of this invention is implemented as follows: This invention provides a retrieval system based on retrieval enhancement, comprising: a client, a vector database, a log database, an adaptive query processing module, a unified hybrid retrieval module, a multi-stage result optimization module, and a log recording module. The client is used to send query requests to the adaptive query processing module; the adaptive query processing module is used to receive query requests, analyze them, determine the query type, and allocate corresponding retrieval paths based on the query type; the unified hybrid retrieval module is connected to the vector database and is used to send approximate nearest neighbor search requests and keyword search requests to the vector database when the retrieval path is a hybrid retrieval path; the vector database is used to perform search based on the approximate nearest neighbor search requests and keyword search requests. The keyword search request returns a list of semantic search results for the corresponding nearest neighbor search request and a list of keyword search results for the corresponding keyword search request to the unified hybrid retrieval module. The unified hybrid retrieval module also generates a candidate result set based on the semantic search result list and the keyword search result list. The multi-stage result optimization module optimizes the candidate result set through an optimization pipeline to obtain the optimized result set. The log recording module, connected to the log database, adaptive query processing module, unified hybrid retrieval module, and multi-stage result optimization module, records the processing records of these modules in the current business flow and stores these records in the log database.

[0010] Optionally, the adaptive query processing module is also used to analyze the structure and content of the query request based on pattern matching technology, determine whether the query request belongs to a predefined specific category, and determine the retrieval path corresponding to the query request based on the judgment result.

[0011] Furthermore, optionally, the adaptive query processing module is also used to call the large language model to perform preprocessing operations such as query enhancement or query decomposition when the content of the query request is ambiguous or complex, through the preset large language model API interface.

[0012] Optionally, the schema of the vector database's dataset defines both dense vector fields for storing text semantic information and sparse vector fields for storing text keyword information.

[0013] Furthermore, optionally, the unified hybrid retrieval module is also used to, when the retrieval path is a hybrid retrieval path, convert the query text of the query request into a dense vector according to the embedding model, and then generate an approximate nearest neighbor search request based on the dense vector field; generate a keyword search request based on a specific algorithm using the query text of the query request through the sparse vector field; the unified hybrid retrieval module is also used to receive the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request returned by the vector database; and generate a candidate result set based on the semantic retrieval result list and the keyword retrieval result list.

[0014] Optionally, the multi-stage result optimization module is further used when the optimization pipeline includes a preprocessing filtering stage, a precise re-ranking stage, and a post-processing filtering stage. In the preprocessing filtering stage, duplicate document fragments with identical content in the candidate result set are removed, and redundant fragments with similarity greater than a preset threshold are eliminated by calculating the vector cosine similarity between each fragment, resulting in preprocessed candidate documents. In the precise re-ranking stage, the preprocessed candidate documents are paired with the original user query in the query request to form multiple data pairs. These multiple data pairs are input into a cross-encoder re-ranking model, which analyzes the inherent correlation between the query and each document and outputs a corresponding relevance score. The preprocessed candidate documents are then re-ranked based on the relevance score, resulting in re-ranked candidate documents. In the post-processing filtering stage, the re-ranked candidate documents are filtered based on their metadata to obtain an optimized result set.

[0015] Optionally, the logging module is also used to persistently write processing records to the log database in an asynchronous, non-blocking manner.

[0016] This invention provides a retrieval method based on retrieval enhancement, applied to a retrieval system based on retrieval enhancement, comprising: analyzing a received query request to determine the query type; allocating a corresponding retrieval path according to the query type; wherein, when the query type is a high-determinism exact query, the allocated retrieval path is an exact match retrieval path; when the query type is a general semantic query, the allocated retrieval path is a mixed retrieval path; when the retrieval path is an exact match retrieval path, constructing filtering query conditions for metadata fields in a vector database from the key parts extracted from the query request, and obtaining retrieval results based on the filtering query conditions; when the retrieval path is a mixed retrieval path, allocating a corresponding query process according to whether the query request requires query preprocessing, and obtaining retrieval results; wherein, query preprocessing includes: query enhancement and query decomposition.

[0017] Optionally, the key parts extracted from the query request are used to construct filtering query conditions for the metadata fields in the vector database. The search results obtained based on the filtering query conditions include: calling the keyword search function in the unified hybrid search module and applying the filtering query conditions to locate the target document; and stabilizing the target as the search result.

[0018] Optionally, depending on whether query preprocessing is required, a corresponding query process is allocated to obtain the retrieval results, including: if query preprocessing is not required, sending an approximate nearest neighbor search request and a keyword search request to the vector database; receiving the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request; generating a candidate result set based on the semantic retrieval result list and the keyword retrieval result list; optimizing the candidate result set through an optimization pipeline to obtain an optimized result set; if query preprocessing is required, enhancing the query request through a large language model to obtain... Upon receiving the enhanced query request, if the enhanced query request is detected to contain multiple parallel or comparative intents, the large language model is invoked to decompose the enhanced query request into multiple independent, atomic sub-problems; based on the sub-problems, approximate nearest neighbor search requests and keyword search requests are generated; the approximate nearest neighbor search requests and keyword search requests are sent to the vector database, and the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request are received; a candidate result set is generated based on the semantic retrieval result list and the keyword retrieval result list; the candidate result set is optimized through the optimization pipeline to obtain the optimized result set.

[0019] This invention provides a retrieval system and method based on retrieval enhancement. It analyzes received query requests to determine the query type and allocates corresponding retrieval paths accordingly. Specifically, for high-determinism, precise queries, an exact-match retrieval path is allocated; for general semantic queries, a hybrid retrieval path is allocated. When the retrieval path is an exact-match path, key parts extracted from the query request are used to construct filtering query conditions for metadata fields in a vector database, and retrieval results are obtained based on these conditions. When the retrieval path is a hybrid retrieval path, a corresponding query process is allocated based on whether query preprocessing is required, and retrieval results are obtained. Query preprocessing includes query enhancement and query decomposition, thereby achieving technical effects such as simplified system architecture, improved retrieval efficiency and accuracy, optimized result quality, and observable processing. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of a retrieval system based on retrieval enhancement generation provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of another retrieval system based on retrieval enhancement provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of an adaptive query processing module processing a query request in another retrieval system based on retrieval enhancement provided in an embodiment of the present invention. Figure 4 This is a flowchart illustrating how a unified hybrid retrieval module processes retrieval paths in a retrieval system based on retrieval enhancement generation, as provided in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the data processing flow of a log recording module in a retrieval system based on retrieval enhancement generation, as provided in an embodiment of the present invention. Figure 6 This is a flowchart illustrating a retrieval method based on retrieval enhancement provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish different objects, rather than to limit a specific order.

[0023] It should also be noted that the various embodiments of the present invention described below can be executed individually or in combination with each other, and the embodiments of the present invention do not impose specific limitations in this regard.

[0024] This invention provides a retrieval system based on retrieval enhancement generation. Figure 1 This is a schematic diagram of a retrieval system based on retrieval enhancement generation provided by an embodiment of the present invention; as shown below. Figure 1 As shown, the retrieval system based on retrieval enhancement provided in this application includes: Client 11, Vector Database 12, Log Database 13, Adaptive Query Processing Module 14, Unified Hybrid Retrieval Module 15, Multi-stage Result Optimization Module 16, and Log Recording Module 17, among which, Client 11 is used to send query requests to adaptive query processing module 14; adaptive query processing module 14 is used to receive query requests, analyze query requests, determine query types, and allocate corresponding retrieval paths based on query types; unified hybrid retrieval module 15 is connected to vector database 12 and is used to send approximate nearest neighbor search requests and keyword search requests to vector database 12 when the retrieval path is a hybrid retrieval path; vector database 12 is used to return a list of semantic search results corresponding to the approximate nearest neighbor search requests and corresponding keywords to unified hybrid retrieval module 15 based on the approximate nearest neighbor search requests and keyword search requests. The search request includes a keyword search result list; a unified hybrid search module 15, which generates a candidate result set based on the semantic search result list and the keyword search result list; a multi-stage result optimization module 16, which optimizes the candidate result set through an optimization pipeline to obtain an optimized result set; and a log recording module 17, which is connected to the log database 13, the adaptive query processing module 14, the unified hybrid search module 15, and the multi-stage result optimization module 16, and records the processing records of the adaptive query processing module 14, the unified hybrid search module 15, and the multi-stage result optimization module 16 in the current business flow, and stores the processing records in the log database 13.

[0025] Optionally, the adaptive query processing module 14 is also used to analyze the structure and content of the query request based on pattern matching technology, determine whether the query request belongs to a predefined specific category, and determine the retrieval path corresponding to the query request based on the judgment result.

[0026] Furthermore, optionally, the adaptive query processing module 14 is also used to call the large language model to perform preprocessing operations such as query enhancement or query decomposition through the preset large language model API interface when the content of the query request is ambiguous or complex.

[0027] Optionally, the schema of the data set of the vector database 12 defines both dense vector fields for storing text semantic information and sparse vector fields for storing text keyword information.

[0028] Furthermore, optionally, the unified hybrid retrieval module 15 is also used to, when the retrieval path is a hybrid retrieval path, convert the query text of the query request into a dense vector according to the embedding model, and then generate an approximate nearest neighbor search request based on the dense vector field; generate a keyword search request based on a specific algorithm based on the query text of the query request through the sparse vector field; the unified hybrid retrieval module 15 is also used to receive the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request returned by the vector database 12; and generate a candidate result set based on the semantic retrieval result list and the keyword retrieval result list.

[0029] Optionally, the multi-stage result optimization module 16 is further configured to, in the case that the optimization pipeline includes: a preprocessing filtering stage, a precise re-ranking stage, and a post-processing filtering stage, remove duplicate document fragments with identical content in the candidate result set during the preprocessing filtering stage, and eliminate redundant fragments with similarity greater than a preset threshold by calculating the vector cosine similarity between each fragment, thereby obtaining preprocessed candidate documents; during the precise re-ranking stage, pair the preprocessed candidate documents with the original user query in the query request to form multiple data pairs; input the multiple data pairs into the cross-encoder re-ranking model, analyze the inherent correlation between the query and each document through the cross-encoder re-ranking model, and output the corresponding relevance score; re-rank the preprocessed candidate documents according to the relevance score to obtain re-ranked candidate documents; and during the post-processing filtering stage, filter the re-ranked candidate documents according to their metadata to obtain the optimized result set.

[0030] Optionally, the logging module 17 is also used to persistently write processing records to the log database 13 in an asynchronous, non-blocking manner.

[0031] Specifically, Figure 2This is a schematic diagram of another retrieval system based on retrieval enhancement provided in an embodiment of the present invention; as shown below. Figure 2 As shown in the embodiments of this application, the retrieval system based on retrieval enhancement generation is specifically as follows: In this embodiment, the adaptive query processing module 14, the unified hybrid retrieval module 15, the multi-stage result optimization module 16, and the log recording module 17 work collaboratively and interact with the vector database 12, which serves as a unified backend. These modules are deployed on computing devices such as servers, aiming to efficiently and intelligently process user query requests and return the most relevant document information from the knowledge base. In this embodiment, the vector database 12 can be a single vector database, and the adaptive query processing module 14, the unified hybrid retrieval module 15, the multi-stage result optimization module 16, and the log recording module 17 can be deployed in the application layer.

[0032] In this embodiment, the adaptive query processing module 14 is configured to receive the user's original query, determine the type of the query through built-in pattern recognition and analysis logic, and determine the subsequent processing path based on the determination result. For specific types of queries, the adaptive query processing module 14 can also call the large language model interface to perform preprocessing operations such as query enhancement or query decomposition; In this application embodiment, the analysis logic can be implemented through regular expression matching. The specific type in this application embodiment can be a fuzzy query or a complex query.

[0033] In a preferred example, the adaptive query processing module 14 serves as the system's entry point, responsible for receiving and initially parsing the user's original query request. The core function of the adaptive query processing module 14 lies in its built-in intelligent analysis and decision-making logic. Specifically, the adaptive query processing module 14 uses pattern matching techniques such as regular expressions to analyze the structure and content of the query text in the query request to determine whether the structure and content of the query text belong to a predefined specific category, such as whether it is a "standard number" query containing a specific combination of letters and numbers. Based on the determination result, the adaptive query processing module 14 will decide which retrieval strategy to adopt and perform corresponding routing for the query.

[0034] In addition, for queries with relatively vague or complex content, the adaptive query processing module 14 can also be configured to call the API interface of an external large language model to perform preprocessing operations such as query enhancement or query decomposition, thereby improving the quality of subsequent retrieval from the source.

[0035] In this embodiment, the large language model can be an LLM model; query enhancement can be to generate a hypothetical, more detailed answer text to assist retrieval, i.e., HyDE technology; query decomposition can be to break down a complex question into multiple independently searchable sub-questions.

[0036] The core feature of the unified hybrid retrieval module 15 in this embodiment is its connection to a single vector database 12 backend that simultaneously supports both dense and sparse vector indexes. The unified hybrid retrieval module 15 executes corresponding retrieval tasks according to the instructions of the adaptive query processing module 14. During hybrid retrieval, the unified hybrid retrieval module 15 initiates semantic retrieval requests and keyword retrieval requests to the vector database 12 in parallel, receives two independent raw recall results, and merges and sorts the results at the application layer using a preset fusion algorithm to generate a unified candidate result set; wherein, in this embodiment, the preset fusion algorithm can be a reciprocal sorting fusion RRF.

[0037] In a preferred example, unlike existing solutions that require simultaneous connections to two different databases, the unified hybrid retrieval module 15 connects only to the vector database 12. The vector database 12 is specially configured such that its collection schema defines both dense vector fields for storing text semantic information and sparse vector fields for storing text keyword information.

[0038] When performing a hybrid retrieval task, the unified hybrid retrieval module 15 will construct two types of retrieval requests: The first type of retrieval request is an approximate nearest neighbor search request initiated on the dense vector field after the query text is converted into a dense vector using an embedding model. The second type of retrieval request is a keyword search request based on the BM25 algorithm initiated directly using the original query text and targeting the sparse vector field.

[0039] Nearest neighbor search requests and keyword search requests are sent to vector database 12 in parallel. After receiving the two independent recall result lists returned by vector database 12, each with its own score, the unified hybrid retrieval module 15 executes a preset fusion algorithm, such as a reverse sorting fusion algorithm, at the application layer (i.e., in the system's own logic), to merge the two results and perform a unified sorting, ultimately generating a candidate result set that takes into account both semantic relevance and keyword accuracy.

[0040] In this embodiment, the multi-stage result optimization module 16 is used to refine and optimize the candidate result set output by the unified hybrid retrieval module 15. The processing flow includes at least the following: The preprocessing and filtering stage is used to remove redundant information that is completely duplicated or highly similar in meaning; In the precise re-ranking stage, the cross-encoder model is used to perform in-depth interactive analysis of the query and candidate documents and re-score and rank them. The post-processing screening stage is used to limit the diversity of the final results based on factors such as the source.

[0041] In a preferred example, the multi-stage result optimization module 16 performs in-depth optimization and refinement on the candidate result set generated by the unified hybrid retrieval module 15 to ensure the high quality of the final output. Internally, the multi-stage result optimization module 16 is implemented as an ordered optimization pipeline containing multiple stages.

[0042] The optimized pipeline in this embodiment includes at least: Preprocessing and filtering stage: Before the formal rearrangement, the candidate set is cleaned. This stage removes duplicate document fragments with identical content and eliminates redundant fragments with highly similar content (e.g., similarity higher than a preset threshold of 0.95) by calculating the vector cosine similarity between fragments. In this embodiment, the elimination of highly similar redundant fragments can be based on a similarity higher than the preset threshold of 0.95.

[0043] Precise re-ranking stage: Preprocessed candidate documents are paired with the original user query, forming multiple [Query, Document] pairs. These data pairs are fed into a cross-encoder re-ranking model. This model can deeply analyze the inherent correlation between the query and each document and output a high-precision relevance score. The module re-ranks the candidate set based on this score.

[0044] Post-processing filtering stage: After re-sorting, to further improve the diversity of results, this stage filters based on metadata such as the document's source. For example, it limits the number of segments ultimately selected from the same original document to no more than a preset value, avoiding the final results being dominated by documents from a single source. In this embodiment, the preset value can be two.

[0045] In this embodiment, the log recording module 17 generates a globally unique identifier (request_id) for each retrieval request (i.e., the query request in this embodiment) entering the system. This module is responsible for capturing and recording key status data, intermediate results, and performance time consumption of the request as it flows through all the above modules, associating this data with the unique identifier, and finally persistently storing the complete tracing log through an asynchronous processing mechanism for subsequent system debugging and performance optimization.

[0046] In a preferred example, the logging module 17 provides the system with strong observability and diagnostics. When a query request enters the system, the logging module 17 assigns it a globally unique request identifier (e.g., UUID). Throughout the request processing lifecycle, as data flows through the adaptive query processing module 14, the unified hybrid retrieval module 15, and the multi-stage result optimization module 16, these modules send key intermediate state data, input / output content, and performance time information for each stage to the logging module 17. The logging module 17 associates all information with the unique request identifier to form a structured and complete runtime log. While the main processing thread returns the final result to the user, the logging module 17 starts an independent background thread to asynchronously and non-blockingly persist this complete log data to the log database 13. The retrieval system based on retrieval enhancement provided in this embodiment ensures problem tracing capabilities without affecting the system's real-time response performance.

[0047] Figure 3 This is a schematic diagram illustrating the process of an adaptive query processing module handling query requests in another retrieval system based on retrieval enhancement generation, as provided in an embodiment of the present invention; for example... Figure 3 As shown, the adaptive query processing module 14 in the retrieval system based on retrieval enhancement provided in this application aims to improve the overall efficiency and effectiveness of retrieval by intelligently analyzing user queries and dynamically matching the optimal retrieval strategy for the user. The process of the unified hybrid retrieval method in this application specifically includes the following steps: Step 1. Receive user query. The system receives the original query request from the user (i.e., the query request in this embodiment of the application). The original query request is usually a piece of natural language text, such as a question, keywords or technical terms, wherein the number of questions, keywords or technical terms can be a single one.

[0048] Step 2. Perform feature analysis and type determination on the query text in the original query request. After receiving the original query request, the system does not immediately perform a search, but first analyzes the inherent features of the query text in the original query request to determine the query type to which the original query request belongs.

[0049] In a preferred embodiment, this application uses a pattern matching method, for example, using a predefined regular expression library to detect whether a string with a specific format exists in the query text; wherein, a regular expression used to match national or industry standard numbers can be used to identify queries in the form of "DL / T 5190.5-2012". If the query text successfully matches a preset pattern, the query text is judged as a "high-determinism exact query" type; otherwise, if no pattern is matched, it is judged as a "general semantic query" type.

[0050] Step 3. Adaptively select the retrieval path based on the query type as a decision-making step. Based on the judgment result of Step 2, determine the subsequent execution path: If the result is "high certainty exact query", the process proceeds to Step 4, entering the exact match retrieval path.

[0051] If the result is "general semantic query", the process will proceed to Step 5, entering the hybrid retrieval path that requires deep semantic understanding.

[0052] Step 4. Execute the exact match search path. When the query is identified as highly certain information such as a standard number, the system will adopt a more direct and efficient search method. Specifically, in this embodiment of the application, the key parts extracted from the query, which in this embodiment of the application can be various fields of the standard number, are used to construct filtering query conditions for the metadata fields in the vector database 12; where the metadata fields in this embodiment of the application can be the source or title fields.

[0053] The retrieval system based on retrieval enhancement provided in this application embodiment only calls the keyword retrieval function in the unified hybrid retrieval module 15. The keyword retrieval function in the unified hybrid retrieval module 15 can be the BM25 retrieval function, and the filtering query conditions are applied.

[0054] By bypassing the time-consuming vector generation and semantic similarity calculation, the above method can directly locate the target document with extremely high speed and 100% accuracy; after execution, the search results are output.

[0055] Step 5. Determine whether the query needs to be preprocessed. For general semantic queries, the retrieval system based on retrieval enhancement provided in this application embodiment will further determine whether it needs to be optimized before performing the retrieval.

[0056] In a preferred example, this determination can be based on parameter flags carried in the API request (e.g., whether hyde_mode or query_decomposition_mode is true), or through more complex semantic analysis to determine whether the query itself lacks sufficient information or is overly complex. Specifically, in this embodiment, the determination can be based on parameter flags carried in the API request, specifically whether hyde_mode or query_decomposition_mode is true.

[0057] If it is determined that preprocessing is required, the process proceeds to Step 6.

[0058] If it is determined that no preprocessing is required, the process proceeds directly to Step 7.

[0059] Step 6. Perform query preprocessing. As needed, the system will perform one or more query optimization operations; for example: Query enhancement: When a query is detected to be too short, the retrieval system based on retrieval enhancement provided in this application embodiment calls the large language model to generate a "hypothetical answer" containing rich context based on the original query, and uses this more detailed text to replace the original query for subsequent retrieval.

[0060] Query decomposition: When a query is detected to contain multiple parallel or comparative intents, the retrieval system based on retrieval enhancement provided in this application embodiment calls the large language model to decompose the original query into multiple independent, atomic sub-questions.

[0061] Step 7. Execute the hybrid retrieval path. The retrieval system based on retrieval enhancement provided in this application embodiment uses the original query (if it has not been preprocessed) or the query text optimized by Step 6 to start the complete hybrid retrieval process of the unified hybrid retrieval module 15.

[0062] As mentioned earlier, this will involve parallel semantic retrieval and keyword retrieval, as well as subsequent RRF fusion at the application layer. The detailed process of Step 7 will be discussed in the reference... Figure 4 The description further elaborates on this. After execution, the fused candidate result set is output for processing by the subsequent multi-stage result optimization module 16.

[0063] Through the above steps, the adaptive query processing method of the retrieval system based on retrieval enhancement provided in this application embodiment can effectively distinguish different types of queries and match them with the most suitable "green channel" or "refined process", thereby greatly improving the efficiency of simple queries while ensuring the effect of complex queries, and realizing the intelligent and optimal allocation of system resources.

[0064] Figure 4 This is a schematic diagram illustrating the process of a unified hybrid retrieval module processing retrieval paths in a retrieval system based on retrieval enhancement generation, as provided in an embodiment of the present invention; for example... Figure 4 As shown in the embodiments of this application, the internal working principle of the unified hybrid retrieval module 15 in the retrieval system based on retrieval enhancement generation when performing hybrid retrieval tasks is clearly demonstrated to show how to efficiently collaborate between the application layer and a single database backend to achieve deep integration of semantic and keyword retrieval. Figure 4 The execution environment is divided into two layers: the application layer and the vector database 12.

[0065] In this embodiment, the application layer represents the logic executed within the retrieval system itself, which is based on retrieval enhancement provided in this embodiment, while the vector database 12 represents the operations executed in a separate vector database service. The process specifically includes the following steps: Step 1. In the application layer, prepare to initiate retrieval requests in parallel. When the process enters the hybrid retrieval path, the application layer first prepares two different forms of query data for a single retrieval: First, the user's original query text or preprocessed query text, used for keyword retrieval; Second, by calling the embedded model API interface, the query text is transformed into a high-dimensional dense vector for semantic retrieval.

[0066] After preparing the two types of query data, the application layer sends two retrieval requests to the vector database 12 in parallel: The first type of retrieval request is the keyword retrieval request. The keyword retrieval request takes the original query text (i.e., the query request in this embodiment) as input data and specifies that a retrieval based on BM25 or a similar algorithm is performed on the predefined sparse vector field in the vector database 12.

[0067] The second type of retrieval request is a semantic retrieval request. The semantic retrieval request takes the dense vector of the query text (i.e., the query request in this embodiment) as input data and specifies that an approximate nearest neighbor search is performed on the predefined dense vector field in the vector database 12.

[0068] Both requests point to the same collection within the same database instance.

[0069] Step 2. Parallel retrieval is performed in vector database 12. After receiving two requests from the application layer, vector database 12 utilizes the internal capabilities of the application layer to process these two tasks in parallel: The first sub-step involves performing a keyword search. The search engine within Vector Database 12 calculates and returns a batch of document IDs and their relevance scores (BM25 Score) that best match the keywords based on the query text in the first search request, using the sparse vector index.

[0070] The second sub-step involves performing semantic retrieval. The retrieval engine within the vector database 12 calculates and returns a batch of document IDs that are most similar at the semantic level, along with their distance or similarity scores, based on the query vector in the second retrieval request, using the dense vector index.

[0071] After the retrieval is completed, the vector database 12 returns the two independent result lists, each with its original sorting and scores, to the application layer.

[0072] Step 3: The application layer receives and merges the multiple recall results. The application layer receives two result lists returned from the vector database 12. At this point, the retrieval system based on retrieval enhancement provided in this embodiment does not directly use either result, but instead initiates a fusion process to combine the advantages of the two results in a better way.

[0073] In a preferred embodiment, the retrieval system based on retrieval enhancement provided in this application uses a reciprocal ranking fusion algorithm. The specific process of the reciprocal ranking fusion algorithm in this application embodiment is as follows: Initialize an empty dictionary (rrf_scores) with document ID as the key and fusion score as the value.

[0074] Iterate through the list of results for the keyword search. For each document in the list, add a score of 1 / (rank + K) based on its rank in the list, where K is a smoothing constant (e.g., 60). Add this score to the value of the corresponding document ID in the rrf_scores dictionary.

[0075] Similarly, the semantic search result list is traversed, and for each document in the list, a score of 1 / (rank + K) is added to it according to its ranking, and accumulated in the rrf_scores dictionary.

[0076] In this way, a document that ranks highly in both recalls will have a significantly higher final fusion score.

[0077] Step 4: Generate a unified candidate set. After calculating the RRF scores for all recalled documents, the application layer sorts all unique document IDs in descending order based on the fusion scores in the rrf_scores dictionary. The sorted list of document IDs constitutes a high-quality unified candidate set that balances keyword accuracy and semantic relevance.

[0078] The unified candidate set is then passed to the multi-stage result optimization module 16 for further ranking and optimization. This embodiment of the application achieves efficient, concise, and highly scalable hybrid retrieval by implementing a flexible and observable fusion strategy at the application layer and fully leveraging the ability of a single database backend to simultaneously support two types of indexes.

[0079] Figure 5 This invention provides a schematic diagram illustrating the data processing flow of a log recording module in a retrieval system based on retrieval enhancement generation, as shown in the embodiments of the present invention. Figure 5 As shown in the embodiment of this application, the function coordinated by the log recording module 17 in the retrieval system based on retrieval enhancement generation aims to establish a complete, transparent, and traceable log for each retrieval request, while ensuring that the log recording process does not affect the main service performance of the system. This process specifically includes the following steps: Step 1. Generate a unique identifier (ID) for the retrieval request. When a new user query request arrives at the system, before any substantive processing, the retrieval system based on retrieval enhancement provided in this embodiment first generates a globally unique request identifier (Request ID), such as a UUID (Universally Unique Identifier). This unique identifier will serve as the "identity card" for the request throughout its entire processing lifecycle, used to associate all subsequently generated log information.

[0080] Step 2. Create and associate the tracking data structure. The retrieval system based on retrieval enhancement provided in this embodiment creates a structured data object in memory for this request, such as a dictionary or JSON object, called the tracking log data structure. This data structure is bound to the unique identifier generated in Step 1, and predefined fields are used to store information for subsequent processing stages.

[0081] Step 3. Record intermediate data at each key node of the main retrieval process. As the request flows through the various adaptive query processing modules 14, unified hybrid retrieval module 15, and multi-stage result optimization module 16 in the system's main processing thread, after each key processing node completes its operation, information such as the node's input, output, key parameters, intermediate results, and performance time consumption is sent and recorded in the tracing log data structure associated with the current request ID. For example: After query processing, record the final query text used for retrieval and the selected retrieval path.

[0082] After the mixed search, record the original document ID list and score returned by the keyword search and semantic search respectively.

[0083] After the results are fused, the candidate set sorted by the RRF algorithm is recorded.

[0084] After optimizing the results, record the list of documents sent to the reordering model and the final results returned by the model.

[0085] In this way, the tracking log data structure becomes like a constantly being filled archive, gradually accumulating the complete evolution of this request from the original question to the final answer.

[0086] Step 4. The main processing thread returns the final result, when the main retrieval process (i.e., ...) is completed. Figures 2 to 4 After the process shown is completed and the final list of document results is generated, the main processing thread immediately returns this result to the user who initiated the request or the upper-layer application. This is the core task of the main thread, and its execution is not affected by subsequent log writing operations.

[0087] Step 5. Asynchronously persist the tracing logs. In this embodiment, the log persistence operation is decoupled from the main processing flow. Simultaneously with or immediately after Step 4, the retrieval system based on enhanced retrieval provided in this embodiment will start an independent background thread (or place the task in an asynchronous task queue) and pass the tracing log data structure containing complete information to this background thread.

[0088] Step 6. The background thread performs the database write operation. The independent thread running in the background is responsible for communicating with the log database, serializing the received structured log data (for example, converting a list or dictionary into a JSON string), and then performing the database INSERT operation to safely write this complete log, associated with a unique ID, to persistent storage.

[0089] Because database write operations are performed in a background thread that does not affect user response, the main system's retrieval service can maintain high availability and low latency even if the log system experiences slow writes or temporary failures. Through the above process, the retrieval system based on retrieval enhancement provided in this application embodiment achieves a high-performance, non-blocking, and information-complete end-to-end tracing mechanism, providing a solid data foundation for system fault diagnosis, performance analysis, and future model iteration and optimization.

[0090] Compared with existing technologies, the retrieval system based on retrieval enhancement provided in this application has the following significant advantages: The architecture of the retrieval system based on retrieval enhancement provided in this application is simplified. By adopting a single vector database backend to achieve hybrid retrieval, it completely replaces the complex heterogeneous architecture composed of a full-text search engine and a vector database in existing technologies. This not only greatly simplifies the deployment, monitoring, and maintenance processes of the system but also fundamentally eliminates the need for cross-system data synchronization, ensuring high data consistency and real-time performance while reducing hardware resource consumption. It achieves intelligent retrieval. The introduced adaptive query processing module can match the most efficient processing path for different types of queries, enabling simple precise queries to obtain accurate responses, while complex fuzzy queries achieve higher recall rates through intelligent enhancement, significantly improving user experience and retrieval success rates in various scenarios. It achieves result refinement. The unique multi-stage result optimization module, by adding preprocessing and post-processing stages before and after reordering, can effectively remove redundant information and pseudo-relevant content from the candidate set. This not only improves the "information density" and diversity of the final results but also provides a cleaner, higher-quality context for subsequent large language models, thereby reducing the probability of the model generating "illusions" and ensuring the accuracy of the final generated answer. Furthermore, it achieves process transparency by fully preserving detailed information for each retrieval through end-to-end log tracking and asynchronous recording mechanisms. This provides a solid data foundation for accurate system problem localization, performance bottleneck analysis, and BadCase repair, and constructs an observable and diagnosable closed-loop optimization system, ensuring the long-term stability and continuous evolution capability of the system.

[0091] This invention provides a retrieval system based on enhanced retrieval generation. A client sends query requests to an adaptive query processing module. The adaptive query processing module receives the query requests, analyzes them, determines the query type, and allocates a corresponding retrieval path based on the query type. A unified hybrid retrieval module, connected to a vector database, sends approximate nearest neighbor search requests and keyword search requests to the vector database when the retrieval path is a hybrid retrieval path. The vector database returns a list of semantic search results for the approximate nearest neighbor search request and a list of results for the keyword search request to the unified hybrid retrieval module based on the approximate nearest neighbor search request and the keyword search request. The system includes a keyword search results list; a unified hybrid search module, which generates a candidate result set based on the semantic search results list and the keyword search results list; a multi-stage result optimization module, which optimizes the candidate result set through an optimization pipeline to obtain the optimized result set; and a log recording module, which connects to the log database, the adaptive query processing module, the unified hybrid search module, and the multi-stage result optimization module, to record the processing records of these modules in the current business flow and store these records in the log database. This achieves the technical effects of simplified system architecture, improved search efficiency and accuracy, optimized result quality, and observable processing.

[0092] This invention provides a retrieval method based on retrieval enhancement generation. Figure 6 This is a flowchart illustrating a retrieval method based on retrieval enhancement generation, provided as an embodiment of the present invention; for example... Figure 6 As shown, applied to Figures 1 to 5 The retrieval system based on retrieval enhancement shown in this application embodiment includes the following retrieval method based on retrieval enhancement: Step S602: Analyze the received query request to determine the query type; Specifically, step S602 in this embodiment corresponds to Figures 1 to 3 The adaptive query processing module 14 in the middle processes the query request, especially determining the query type of the query request.

[0093] Step S604: Assign a corresponding retrieval path based on the query type; wherein, when the query type is a high-determinism exact query, the assigned retrieval path is an exact match retrieval path; when the query type is a general semantic query, the assigned retrieval path is a mixed retrieval path. Specifically, step S604 in this embodiment corresponds to Figures 1 to 3 The adaptive query processing module 14 in the middle handles the retrieval path.

[0094] Step S606: When the search path is an exact match search path, the key parts extracted from the query request are used to construct filtering query conditions for the metadata fields in the vector database, and the search results are obtained based on the filtering query conditions. Optionally, in step S606, the key parts extracted from the query request are used to construct filtering query conditions for the metadata fields in the vector database. The search results obtained based on the filtering query conditions include: calling the keyword search function in the unified hybrid search module and applying the filtering query conditions to locate the target document; and stabilizing the target as the search result.

[0095] Specifically, step S606 in this embodiment corresponds to Figure 2 The process of sparse retrieval in sparse search.

[0096] Step S608: When the retrieval path is a mixed retrieval path, allocate the corresponding query process according to whether the query request needs to be preprocessed, and obtain the retrieval results; wherein, the query preprocessing includes: query enhancement and query decomposition.

[0097] Optionally, in step S608, depending on whether the query request requires query preprocessing, the corresponding query process is allocated to obtain the retrieval results, including: if the query request does not require query preprocessing, sending an approximate nearest neighbor search request and a keyword search request to the vector database; receiving the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request; generating a candidate result set based on the semantic retrieval result list and the keyword retrieval result list; optimizing the candidate result set through an optimization pipeline to obtain an optimized result set; if the query request requires query preprocessing, processing the query request through a large language model... The process involves enhancing the query request, obtaining an enhanced query request, and if the enhanced query request is found to contain multiple parallel or comparative intents, then invoking the large language model to decompose the enhanced query request into multiple independent, atomic sub-problems. Based on the sub-problems, approximate nearest neighbor search requests and keyword search requests are generated. These requests are then sent to the vector database, and the semantic retrieval result list corresponding to the approximate nearest neighbor search request and the keyword retrieval result list corresponding to the keyword search request are received. A candidate result set is generated based on the semantic retrieval result list and the keyword retrieval result list. Finally, the candidate result set is optimized through an optimization pipeline to obtain the optimized result set.

[0098] Specifically, step S608 in this embodiment corresponds to Figure 1 and Figure 4 The processing flow of the unified hybrid retrieval module 15.

[0099] This invention provides a retrieval method based on retrieval enhancement. It analyzes received query requests to determine the query type and allocates corresponding retrieval paths accordingly. Specifically, for high-determinism, precise queries, an exact-match retrieval path is allocated; for general semantic queries, a mixed retrieval path is allocated. When the retrieval path is an exact-match path, key parts extracted from the query request are used to construct filtering query conditions for metadata fields in a vector database, and retrieval results are obtained based on these conditions. When the retrieval path is a mixed retrieval path, a corresponding query process is allocated based on whether query preprocessing is required, and retrieval results are obtained. Query preprocessing includes query enhancement and query decomposition, thereby achieving technical effects such as simplified system architecture, improved retrieval efficiency and accuracy, optimized result quality, and observable processing.

[0100] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A retrieval system based on retrieval enhancement generation, characterized in that, include: The system comprises a client, a vector database, a log database, an adaptive query processing module, a unified hybrid retrieval module, a multi-stage result optimization module, and a logging module. The client is used to send a query request to the adaptive query processing module; The adaptive query processing module is used to receive the query request, analyze the query request, determine the query type, and allocate the corresponding retrieval path according to the query type. The unified hybrid retrieval module is connected to the vector database and is used to send an approximate nearest neighbor search request and a keyword search request to the vector database when the retrieval path is a hybrid retrieval path. The vector database is used to return a list of semantic search results corresponding to the approximate nearest neighbor search request and a list of keyword search results corresponding to the keyword search request to the unified hybrid retrieval module based on the approximate nearest neighbor search request and the keyword search request. The unified hybrid retrieval module is also used to generate a candidate result set based on the semantic retrieval result list and the keyword retrieval result list; The multi-stage result optimization module is used to optimize the candidate result set through the optimization pipeline to obtain the optimized result set; The log recording module is connected to the log database, the adaptive query processing module, the unified hybrid retrieval module, and the multi-stage result optimization module, and is used to record the processing records of the adaptive query processing module, the unified hybrid retrieval module, and the multi-stage result optimization module in the current business flow; and store the processing records in the log database.

2. The retrieval system based on retrieval enhancement generation according to claim 1, characterized in that, The adaptive query processing module is further configured to analyze the structure and content of the query request based on pattern matching technology, determine whether the query request belongs to a predefined specific category, and determine the retrieval path corresponding to the query request based on the determination result.

3. The retrieval system based on retrieval enhancement generation according to claim 2, characterized in that, The adaptive query processing module is also used to, when the content of the query request is ambiguous or complex, call the large language model through the preset large language model API interface to perform preprocessing operations such as query enhancement or query decomposition.

4. The retrieval system based on retrieval enhancement generation according to claim 1, characterized in that, The schema of the vector database's data set simultaneously defines dense vector fields for storing text semantic information and sparse vector fields for storing text keyword information.

5. The retrieval system based on retrieval enhancement generation according to claim 4, characterized in that, The unified hybrid retrieval module is further configured to, when the retrieval path is a hybrid retrieval path, convert the query text of the query request into a dense vector according to the embedding model, and then generate the approximate nearest neighbor search request based on the dense vector field; and generate the keyword search request based on a specific algorithm using the query text of the query request through the sparse vector field. The unified hybrid retrieval module is further configured to receive a list of semantic retrieval results corresponding to the approximate nearest neighbor search request and a list of keyword retrieval results corresponding to the keyword search request returned by the vector database; and generate the candidate result set based on the list of semantic retrieval results and the list of keyword retrieval results.

6. The retrieval system based on retrieval enhancement generation according to claim 1, characterized in that, The multi-stage result optimization module is also used when the optimization pipeline includes: a preprocessing filtering stage, a precise re-ranking stage, and a post-processing screening stage. In the preprocessing filtering stage, duplicate document segments with the same content are removed from the candidate result set, and redundant segments with similarity greater than a preset threshold are eliminated by calculating the vector cosine similarity between each segment, thus obtaining preprocessed candidate documents; In the precise reordering stage, the preprocessed candidate documents are paired with the original user query in the query request to form multiple data pairs; the multiple data pairs are input into the cross encoder reordering model, the cross encoder reordering model analyzes the inherent correlation between the query and each document, and outputs the corresponding relevance score; the preprocessed candidate documents are reordered according to the relevance score to obtain the reordered candidate documents. In the post-processing filtering stage, the candidate documents are filtered based on their metadata after being reordered to obtain the optimized result set.

7. The retrieval system based on retrieval enhancement generation according to claim 1, characterized in that, The logging module is also used to persistently write the processing records to the log database in an asynchronous, non-blocking manner.

8. A retrieval method based on retrieval enhancement generation, characterized in that, Applications to retrieval systems based on retrieval enhancement include: Analyze the received query requests to determine the query type; The corresponding retrieval path is assigned according to the query type; wherein, when the query type is a high deterministic exact query, the retrieval path is assigned as an exact match retrieval path; when the query type is a general semantic query, the retrieval path is assigned as a hybrid retrieval path. When the search path is the exact matching search path, the key parts extracted from the query request are used to construct filtering query conditions for the metadata fields in the vector database, and the search results are obtained based on the filtering query conditions. When the search path is the hybrid search path, the corresponding query process is allocated according to whether the query request needs to be preprocessed, and the search results are obtained; wherein, the query preprocessing includes: query enhancement and query decomposition.

9. The retrieval method based on retrieval enhancement generation according to claim 8, characterized in that, The key parts extracted from the query request are used to construct filtering query conditions for metadata fields in the vector database. The retrieval results obtained based on these filtering query conditions include: Invoke the keyword search function in the unified hybrid search module and apply the filtering query conditions to locate the target document; The target is stably determined as the search result.

10. The retrieval method based on retrieval enhancement generation according to claim 8, characterized in that, The step of allocating a corresponding query process based on whether the query request requires query preprocessing, and obtaining the retrieval results, includes: If the query request does not require query preprocessing, send an approximate nearest neighbor search request and a keyword search request to the vector database; receive a semantic search result list corresponding to the approximate nearest neighbor search request and a keyword search result list corresponding to the keyword search request; generate a candidate result set based on the semantic search result list and the keyword search result list; optimize the candidate result set through an optimization pipeline to obtain an optimized result set; If the query request requires query preprocessing, the query request is enhanced using a large language model to obtain an enhanced query request. If the enhanced query request is detected to contain multiple parallel or comparative intents, the large language model is invoked to decompose the enhanced query request into multiple independent, atomic sub-problems. Approximate nearest neighbor search requests and keyword search requests are generated based on the sub-problems. The approximate nearest neighbor search requests and keyword search requests are sent to the vector database, and a semantic retrieval result list corresponding to the approximate nearest neighbor search requests and a keyword retrieval result list corresponding to the keyword search requests are received. A candidate result set is generated based on the semantic retrieval result list and the keyword retrieval result list. The candidate result set is optimized through an optimization pipeline to obtain an optimized result set.

Images