Bid document-oriented dual-channel hybrid retrieval knowledge base query method and system

Abstract

The application relates to the technical field of natural language processing, and discloses a double-channel mixed retrieval knowledge base query method and system for bidding documents. The method comprises the following steps: S1, obtaining a score basis text and preprocessing the text to obtain a text block, and establishing dense vector indexes and sparse retrieval indexes for a text channel and a metadata channel; S2, obtaining bidding document information and a structured review sub-element to generate a batch query request; S3, reading a search term of the batch query request, reading corresponding query vectors from a preloaded vector cache, and obtaining a query vector set; S4, concurrently executing each query request in the batch query request; outputting a retrieval result; and S5, aggregating the retrieval results of the batch query request to complete batch knowledge base query. The application can improve the retrieval capability of metadata such as regulation names, chapter titles, keywords and structured numbers through double-channel independent retrieval, so that the retrieval accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically a dual-channel hybrid retrieval knowledge base query method and system for bidding documents. Background Technology

[0002] When using large language models to evaluate tender documents, relevant industry laws and regulations are required as a scoring basis. However, the knowledge derived from large models originates from general corpora and struggles to cover industry-specific regulations and dynamic policies (such as those in the construction industry). Relying solely on internalized knowledge often leads to "illusions," generating incorrect clauses and compromising the accuracy and authority of the review conclusions. However, by retrieving relevant text fragments from external knowledge bases and injecting them into the large model's contextual hints, the problems of knowledge lag and "illusions" inherent in the model itself can be significantly alleviated.

[0003] Traditional knowledge base retrieval methods mainly rely on a single retrieval channel, such as keyword-based full-text retrieval or vector-based semantic retrieval. Simple keyword retrieval is difficult to handle queries that are synonymous but have different expressions, such as "temperature control of large-volume concrete" and "temperature control measures for concrete pouring," leading to missed detections; while pure vector retrieval can understand semantics, it cannot accurately match structured numbers such as "GB 50411-2019" and "Article 120," resulting in false positives.

[0004] To balance accuracy and semantic understanding, existing technologies employ a hybrid retrieval scheme, followed by a second round of fine-tuning using a re-ranking model. However, these existing schemes still have the following shortcomings: most schemes only index the original text blocks of documents, failing to utilize extended descriptions such as document titles and keywords as independent retrieval channels. This results in a large amount of structured, high-value information not being recalled, impacting retrieval accuracy.

[0005] For example, CN119961433A uses a large model to calculate the perplexity of the recalled fragments and then re-ranks and filters the recalled content. However, this method is still limited to the traditional single retrieval channel and lacks an architecture design that allows for multi-channel independent storage and parallel retrieval. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides a dual-channel hybrid retrieval knowledge base query method and system for bidding documents. It can improve the retrieval capability of metadata such as law names, chapter titles, keywords and structured numbers through independent dual-channel retrieval, thereby improving the retrieval accuracy.

[0007] To achieve the above objectives, the present invention provides the following technical solution: This invention discloses a dual-channel hybrid retrieval knowledge base query method for bidding documents, including: S1. Obtain the scoring basis text and preprocess it to obtain text blocks. Save the text blocks containing the main text content in the main text channel and save the text blocks containing the metadata extended description content in the metadata channel. Establish dense vector indexes and sparse search indexes for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel. S2. Obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms; S3. Determine whether the batch query request requires vector retrieval. If not, perform keyword retrieval on the batch query request in the dual-channel knowledge base using the sparse retrieval index, output the retrieval results, and proceed to S5. If yes, read the search terms of the batch query request, read the corresponding query vectors from the pre-loaded vector cache, and obtain the query vector set. S4. Execute each query request in the batch query request concurrently; Based on the query vector set, the query vector is used to perform vector retrieval of the query request in the dual-channel knowledge base through dense vector index; and / or, the query request is used to perform keyword retrieval in the dual-channel knowledge base through sparse retrieval index; Output the search results; S5. Aggregate the retrieval results of each query request in the batch query requests to complete the batch knowledge base query.

[0008] Preferably, the step of obtaining the scoring basis text and preprocessing it to obtain text blocks specifically includes: S101. Obtain the text of the scoring basis; S102. After parsing and cleaning the scoring data, the data is divided into blocks to obtain text blocks.

[0009] Preferably, S2 specifically includes: S201. Obtain bidding document information and structured review sub-elements, and input query terms to generate search terms using the model; S202. After cleaning and deduplicating the search terms, divide the search terms into at least one batch query group according to sub-elements; determine the search configuration for each search term, including the number of search results returned, the scope of the knowledge base, the search mode, the reordering switch, and the search configuration of the context parameters; assemble the search terms and search configuration of the batch query group into a batch query request in JSON format.

[0010] Preferably, before S3, the method further includes: Extract all search terms from the batch query request and remove duplicate search terms.

[0011] Preferably, S3 further includes: Preload vector cache: Iterate through the batch query requests, read the search terms for each query request, and deduplicate the search terms to form a set of search terms that need to be vectorized; Check the local vector cache to determine if there is a query vector corresponding to the search term. If there is, call the query vector in the vector cache; otherwise, add the search term to the list to be vectorized and generate a query vector for quantization. Write the query vector into the vector cache with the search term as the key and the query vector as the value, thus completing the vector cache preloading.

[0012] Preferably, the step of vectorizing the search terms specifically includes: The search terms are added to the vectorization list, which is then grouped according to a preset batch size and sent to the bge-m3 semantic vector model to generate query vectors.

[0013] Preferably, before step S5, the search results are further semantically reordered; Specifically, the following steps are included: Step a: For each batch of query requests, the search results obtained in the text channel and the metadata channel are used as the candidate result set for the text channel and the candidate result set for the metadata channel, respectively. Then, the search results in the candidate result sets of the text channel and the metadata channel are scored and ranked to obtain the channel recall results. The recall method is determined according to the search mode: When the search mode is keyword search, the search results in the candidate result set are sorted in descending order according to the BM25 score of the keyword and the search results in the candidate result set. The top-ranked search results are selected to obtain the channel recall results under the keyword search mode. When the retrieval mode is vector retrieval, the retrieval results in the candidate result set are sorted in descending order based on the cosine similarity between the retrieval results in the candidate result set and the query vector. A predetermined number of the top-ranked retrieval results are then selected to obtain the channel recall results under the vector retrieval mode. The cosine similarity is calculated as follows: ; Where q represents the query vector and d represents the search result; It is a norm 2; The cosine similarity between the query vector q and the search result d; When the search mode is hybrid search, the RRF method is used to fuse the rankings of search results in keyword search mode and vector search mode to obtain the RRF value of the search results. The search results in the candidate result set are then sorted in descending order based on the RRF value, and a predetermined number of the top-ranked search results are selected to obtain the channel recall results under the hybrid search mode. The RRF value is calculated as follows: ; in, Indicates search results The ranking of the i-th search pattern; m is the number of search patterns; k is the smoothing parameter; For search results RRF value; Step b: Input the recall results of each channel into the re-ranking model, re-rank them according to the semantic matching degree between the search terms and the search results, and assign scores to the search results to obtain the score of each search result in each channel: ; in, The normalized score for search result d; The semantic matching score output by the reordering model for the retrieval result d; The highest semantic matching score among all search results; The lowest semantic match score among all search results; Step c: retrieve the search results The fusion score is used as the final score; among which, the search results Fusion score The calculation method is as follows: ; in, This represents the text channel score of search result d. Weight of the main text channel; This represents the metadata channel score of search result d. Metadata channel weight; Step d: Sort all the search results obtained from the query request in the text channel and the metadata channel in descending order according to the final score, and extract a specified number of search results as the final search results for the query request.

[0014] Preferably, after S4, the method further includes: Expand the context of the search results: determine the document and chapter position to which the search result belongs, query the adjacent segments before and after it, and combine the context with the search result in the order of the article.

[0015] This invention also proposes a dual-channel hybrid retrieval knowledge base query system for bidding documents, applicable to any of the aforementioned dual-channel hybrid retrieval knowledge base query methods for bidding documents. The system includes: The knowledge base construction module is used to obtain the scoring basis text and preprocess it to obtain text blocks. The text blocks containing the main text content are stored in the main text channel, and the text blocks containing the metadata extended description content are stored in the metadata channel. Dense vector indexes and sparse search indexes are built for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel. The query request generation module is used to obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms; The query vector generation module reads the search terms from the batch query requests, preloads the vector cache, and checks whether there is a query vector corresponding to the search terms in the vector cache. If there is, it calls the query vector in the vector cache; otherwise, it vectorizes the search terms to generate query vectors and writes the query vectors into the vector cache with the search terms as keys and the query vectors as values. The retrieval module is used to concurrently execute each query request in the batch query requests. If vector retrieval is not required, the query request is searched for keywords in the dual-channel knowledge base using a sparse retrieval index, and the retrieval results are output. If vector retrieval is required, the query request is searched for vectors in the dual-channel knowledge base using a dense vector index based on the query vector set obtained by the query vector generation module, and / or the query request is searched for keywords in the dual-channel knowledge base using a sparse retrieval index, and the retrieval results are output. The results generation module is used to aggregate the retrieval results of each query request in a batch query request to complete the batch knowledge base query.

[0016] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a dual-channel hybrid retrieval knowledge base query method for any tender document.

[0017] Compared with the prior art, the beneficial effects of the present invention are: (1) The dual-channel hybrid retrieval knowledge base query method and system for bidding documents proposed in this invention improves the retrieval capability of information such as law names, chapter titles, keywords and structured numbers by constructing a text channel and a metadata channel.

[0018] (2) The dual-channel hybrid retrieval knowledge base query method and system for bidding documents proposed in this invention avoids repeated calls to the embedded model by configuring vector caching and using vector preloading and caching mechanisms. Combined with high-concurrency scheduling, it significantly reduces the total time consumption in batch scenarios and provides efficient support for multi-factor review.

[0019] (3) The dual-channel hybrid retrieval knowledge base query method and system for bidding documents proposed in this invention automatically attaches adjacent text blocks through context expansion of document hierarchical structure, so that the large model can obtain a complete context, avoid deviation caused by isolated fragments, and improve the RAG effect.

[0020] (4) The dual-channel hybrid retrieval knowledge base query method and system for bidding documents proposed in this invention supports three modes: BM25, vector, and hybrid, as well as multiple filtering conditions to adapt to different scenarios; at the same time, it returns the full version and the deduplicated simplified version of the results, which is convenient for downstream users to choose flexibly.

[0021] (5) The dual-channel hybrid retrieval knowledge base query method and system for bidding documents proposed in this invention can reduce the problems of keyword mismatch and weak semantic differentiation by performing semantic reordering, thereby improving the accuracy of retrieval results. Attached Figure Description

[0022] Figure 1 This is a flowchart of the dual-channel hybrid retrieval knowledge base query method for bidding documents proposed in Embodiment 1 of the present invention.

[0023] Figure 2 This is a structural block diagram of the dual-channel hybrid retrieval knowledge base query system for bidding documents proposed in Embodiment 1 of the present invention.

[0024] Figure 3 This is a diagram showing the search results for a housing construction project related to "Traditional Chinese Medicine in a certain region". Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1 Please see Figure 1 This embodiment provides a dual-channel hybrid retrieval knowledge base query method for bidding documents, including the following steps; S1. Obtain the scoring basis text and preprocess it to obtain text blocks. Save the text blocks containing the main text content in the main text channel and save the text blocks containing the metadata extended description content in the metadata channel. Establish dense vector indexes and sparse search indexes for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel.

[0027] S101. Obtain the text of the scoring basis; S102. After parsing and cleaning the scoring data, the data is divided into blocks to obtain text blocks.

[0028] In this embodiment, taking the bidding process in the construction industry as an example, the texts of scoring criteria, such as construction industry laws and regulations, construction acceptance standards, and engineering quality specifications, are pre-imported into a knowledge base. For each text of scoring criteria, it is first parsed, cleaned, and segmented. Segmentation can be achieved in various ways, such as using fixed character size cutting, recursive character text segmentation, and semantic segmentation. In this embodiment, a sliding window strategy with overlap ratio can be used to perform semantic-level slicing on the cleaned plain text. The step size of the sliding window is dynamically adjusted according to the paragraph length of the construction industry business rules. Finally, a high-purity text block with complete contextual logic is output, and metadata information such as document name, professional category, topic, subtopic, keywords, and article paragraphs is recorded.

[0029] When building the knowledge base, two retrieval channels are established for the text blocks: the main text channel stores the text blocks containing the main content of regulations or standards, and the metadata channel stores the text blocks containing extended descriptions of metadata, such as document name, topic, subtopics, and keywords. Subsequently, dense vector indexes and sparse retrieval indexes are built for both the main text channel and the metadata channel. This allows for retrieval of relevant content not only through the main text but also by standard name, chapter topic, and keywords.

[0030] The sparse retrieval index can be implemented using BM25, sparse vector, or inverted index.

[0031] For example, for the clauses in the "Standard for Construction and Acceptance of Automatic Fire Alarm Systems", not only is the main text about the acceptance requirements for detectors, alarm buttons, and fire control rooms saved, but also extended descriptions such as "automatic fire alarm system", "fire protection acceptance", "construction and acceptance", "detector", and "alarm button" are saved to improve the hit rate of subsequent searches.

[0032] S2. Obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms.

[0033] S201. Obtain bidding document information and structured review sub-elements, and input query terms to generate search terms using the model; S202. After cleaning and deduplicating the search terms, divide the search terms into at least one batch query group according to sub-elements; determine the search configuration for each search term, including the number of search results, knowledge base scope, search mode, reordering switch and context parameter requirements, and assemble the search terms and search configuration of the batch query group into a batch query request in JSON format.

[0034] Specifically, by reading the project bidding documents and structured review sub-elements, and inputting the sub-element titles, superior titles, document descriptions, existing regulations names, and core judgment points into the query term generation model, 3 to 5 search terms for searching regulations and technical specifications can be generated.

[0035] In this embodiment, the query term generation model can adopt the Query Expansion model to meet the offline high-concurrency query requirements.

[0036] In detail, the query term generation model generates multiple search terms based on bidding document information and structured review sub-elements. These generated search terms are then divided into batch query groups according to sub-elements such as "acceptance and handover," and the number of search results output for each search term is determined, such as top-k=2. Next, search configurations are bound to the search terms, which can include the knowledge base scope, search mode, reordering switch, and context parameter requirements. Finally, the batch query group numbers, search terms, and search configurations are assembled into a batch query request in JSON format.

[0037] The knowledge base scope is used to limit the knowledge base for retrieval; the retrieval modes include vector retrieval, keyword retrieval, and hybrid retrieval (vector retrieval + keyword retrieval); the reordering switch is used to determine whether semantic reordering is performed, which can be turned on or off according to the user's choice; the context parameter requirements are used to determine whether the retrieval results are expanded in context, which can also be turned on or off according to the user's choice.

[0038] Taking the "Traditional Chinese Medicine in a Certain Region" project as an example, for the "Acceptance and Handover" sub-element of the "Traditional Chinese Medicine in a Certain Region" project, the model actually generated four search terms: "Construction Project Acceptance and Handover Specifications," "Construction Engineering Data Management Acceptance and Handover," "Construction Temporary Facility Demolition Specifications," and "Engineering Acceptance and Handover Procedure Document Archiving." The search terms were cleaned, deduplicated, and automatically assembled into batch query groups. Each search term was configured with top-k=2, meaning the top two most relevant search results for each term were output. These, along with the selected knowledge base scope, mixed search mode, re-ranking, and context parameters, were then sent to the batch search interface. This example ultimately retained eight search results.

[0039] Extract all search terms from the batch query request and remove duplicate search terms.

[0040] After generating batch query requests, in order to reduce the pressure on subsequent retrieval, the search terms can be deduplicated to reduce retrieval time.

[0041] For example, this tender evaluation includes multiple themes such as fire safety inspection, building construction quality inspection, and steel structure construction inspection, with several query questions under each theme. The system first collects all search terms, removes duplicates, and generates vectors in batches, which are then reused in subsequent search tasks. This approach reduces repetitive vectorization calculations and network calls, improves the inference throughput of the vector model, and shortens the overall time for batch retrieval.

[0042] S3. Determine whether the batch query request requires vector retrieval. If not, perform keyword retrieval on the batch query request in the dual-channel knowledge base using the sparse retrieval index, output the retrieval results, and proceed to S5. If yes, read the search terms of the batch query request, read the corresponding query vectors from the preloaded vector cache, and obtain the query vector set.

[0043] In this embodiment, for batch query requests, it is first determined whether vector retrieval is required. Specifically, the retrieval configuration for batch query requests includes the knowledge base scope, retrieval mode, reordering switch, and context expansion requirements. The retrieval modes include vector retrieval, keyword retrieval, and mixed retrieval, with mixed retrieval combining vector retrieval and keyword retrieval. If the retrieval mode is keyword retrieval, it is determined that the batch query request does not include vector retrieval; if the retrieval mode is vector retrieval or mixed retrieval, it is determined that the batch query request includes vector retrieval.

[0044] Specifically, vectorizing search terms includes: The search terms are added to the vectorization list, which is then grouped according to a preset batch size and sent to the bge-m3 semantic vector model to generate query vectors.

[0045] This embodiment uses the bge-m3 semantic vector model to generate query vectors. This model supports relatively long text inputs and can cover texts of varying lengths, including legal provisions, chapter titles, and review questions. It is suitable for vectorizing long articles, long titles, and complex query questions in construction industry standards. Furthermore, the model's training objectives cover dense retrieval, sparse retrieval, and multi-vector retrieval capabilities. The generated semantic representation not only expresses overall semantic similarity but also helps retain retrieval clues such as standard numbers, technical terms, and chapter names. The model also demonstrates good adaptability to the mixed expressions of Chinese and English abbreviations, specification codes, and engineering terms commonly found in bidding documents.

[0046] In this step, vector cache can also be preloaded, as implemented below: Retrieve the vector cache and determine if there is a query vector corresponding to the search term in the vector cache. If there is, call the query vector in the vector cache; otherwise, vectorize the search term to generate a query vector, use the search term as the key and the query vector as the value, and write the query vector into the vector cache to complete the vector cache preloading.

[0047] In detail, batch query requests can include multiple batch query groups, each containing multiple query requests. First, all query groups can be iterated through, the search terms for each query request read, and the query content converted into string format. After filtering out empty queries, duplicate search terms are removed to form the set of search terms that need to be vectorized for the current batch.

[0048] The local vector cache is then checked. If a query vector already exists for a particular search term, the cached result is reused. If the term does not exist in the cache, it is added to the vectorization list. The vectorization list is grouped according to a preset batch size and sent to the bge-m3 semantic vector model to generate dense query vectors. After the model returns the vectors, they are written to the vector cache with the search term as the key and the query vector as the value.

[0049] S4. Execute each query request in the batch query request concurrently; Based on the query vector set, the query vector is used to perform vector retrieval of the query request in the dual-channel knowledge base through dense vector index; and / or, the query request is used to perform keyword retrieval in the dual-channel knowledge base through sparse retrieval index; Output the search results.

[0050] For each query request, parallel searches are performed in both the text search channel and the metadata search channel. Search methods can include keyword search, vector search, or a hybrid search combining both. A hybrid search mode can be used to improve search accuracy.

[0051] In the hybrid retrieval method, on the one hand, vector retrieval results are obtained based on regulatory fragments with similar semantic meanings to the query request; on the other hand, keyword retrieval results are obtained based on precisely matched regulatory fragments based on keywords and standard names. The vector retrieval results and keyword retrieval results are then initially fused. The initial fusion can be achieved using existing technologies such as merging and deduplication, and weighted fusion.

[0052] For example, regarding the question "What should be checked during the acceptance inspection after the construction of an automatic fire alarm system is completed?", the main text channel can retrieve clauses that directly describe the acceptance content of fire alarm controllers, linkage equipment, detectors, manual alarm buttons, fire communication, emergency broadcasts, etc.; the metadata channel can retrieve relevant standards whose names, titles, or keywords contain information such as "automatic fire alarm system", "construction and acceptance", and "fire control room".

[0053] After completing the query, output the search results, and then determine whether to request semantic reordering. If so, reorder the search results using semantic reordering.

[0054] Specifically, semantic reordering is optional. When using this method for retrieval, you can choose whether to enable semantic reordering to refine the retrieval results according to your actual needs.

[0055] In this embodiment, before step S5, S3 can output the search results of keyword retrieval, and S4 can output the search results obtained by keyword retrieval, vector retrieval, or mixed retrieval. If semantic reordering is not requested, the search results output by S3 and S4 are directly aggregated through S5. If semantic reordering is requested, the search results output by S3 and S4 will be semantically reordered. The specific process is as follows: After executing the batch query requests in steps S3 and S4, one or more search results will be output for each query request. Taking the output of 4 search results per query request as an example, for a single query request, the 4 search results are used as the candidate result set in the text channel and the candidate result set in the metadata channel. Then, candidate results are selected from the candidate result sets in the text channel and the metadata channel, respectively. The selection method is determined according to the search mode, as follows: Step a: For each batch of query requests, the search results obtained in the text channel and the metadata channel are used as the candidate result set for the text channel and the candidate result set for the metadata channel, respectively. Then, the search results in the candidate result sets of the text channel and the metadata channel are scored and ranked to obtain the channel recall results (i.e., candidate results). The recall method is determined according to the search mode: When the search mode is keyword search, the search results in the candidate result set are sorted in descending order according to the BM25 score of the keyword and the search results in the candidate result set. The top-ranked search results are selected to obtain the channel recall results under the keyword search mode. When the retrieval mode is vector retrieval, the retrieval results in the candidate result set are sorted in descending order based on the cosine similarity between the retrieval results in the candidate result set and the query vector. A predetermined number of the top-ranked retrieval results are then selected to obtain the channel recall results under the vector retrieval mode. The cosine similarity is calculated as follows: ; Where q represents the query vector and d represents the search result; It is a norm 2; To find the cosine similarity between the query vector q and the search result d; when calculating the cosine similarity, d is actually input as the vector of the search result (text fragment).

[0056] When the search mode is hybrid search, the RRF method is used to fuse the rankings of search results in keyword search mode and vector search mode to obtain the RRF value of the search results. The search results in the candidate result set are then sorted in descending order based on the RRF value, and a predetermined number of the top-ranked search results are selected to obtain the channel recall results under the hybrid search mode. The RRF value is calculated as follows: ; in, This represents the ranking of search result d under the i-th search pattern; m is the number of search patterns; and k is the smoothing parameter. The RRF value of the search result d; Step b: Input the recall results of each channel into the re-ranking model, re-rank them according to the semantic matching degree between the search terms and the search results, and assign scores to the search results to obtain the score of each search result in each channel: ; in, The normalized score for search result d; The semantic matching score output by the reordering model for the retrieval result d; The highest semantic matching score among all search results; The lowest semantic match score among all search results; Step c: retrieve the search results The fusion score is used as the final score; among which, the search results Fusion score The calculation method is as follows: ; in, The score represents the main text channel score, primarily reflecting the degree of direct matching between the legal text and the query question. Weight of the main text channel; The metadata channel score primarily reflects the degree of matching between structured information such as document name, topic, subtopic, and keywords and the query question. This is the weight of the metadata channel.

[0057] Step d: Sort all the search results obtained from the query request in the text channel and the metadata channel in descending order according to the final score, and extract a specified number of search results as the final search results for the query request.

[0058] In this embodiment, based on actual retrieval needs, It can be set to 0.7. It can be set to 0.3, and the fusion score is calculated as follows: ; In this application, the re-ranking model can be the bge-reranker-v2-m3 model. The bge-reranker-v2-m3 model performs relevance ranking, inputting the "query question" and "candidate results" as pairs into the model. The model determines the semantic relevance between the two and assigns scores to the candidate results according to their ranking. Relevance is sorted in descending order, with higher-ranked results receiving higher scores. The specific scores can be determined based on the actual situation. When the same segment (search result) appears in both the main text channel and the metadata channel, it can receive scores from both channels simultaneously. Therefore, search results that are relevant in both the main text channel and the metadata channel will receive a higher ranking after fusion. Of course, if the search result only appears in the main text channel, its score in the metadata channel will be 0 when calculating the score.

[0059] To further illustrate the scoring method of semantic reordering, let's consider the retrieved fragments "A", "B", and "C" from the main text channel, and the retrieved fragments "C", "D", and "E" from the metadata channel. The scores for "A", "B", and "C" in the main text channel are 0.9, 0.8, and 0.7, respectively; while the scores for "C", "D", and "E" in the metadata channel are 0.9, 0.8, and 0.7, respectively. Therefore, the final score for A is 0.7. 0.9; B Final score: 0.7 0.8; C Final score: 0.7 0.7 + 0.3 0.9; D Final score: 0.3 0.8; Final score: 0.3 0.7.

[0060] By performing semantic reordering, we can reduce the problems of keyword mismatch and weak semantic differentiation, and improve the accuracy of search results.

[0061] After completing the query, the search results are output. It can also determine whether the search results need to be expanded with context. If so, the search results are expanded with context: determine the document and chapter position to which the search result belongs, query the adjacent segments before and after it, and concatenate the context with the search result according to the article order.

[0062] Similarly, context expansion can also be an option. When using this method for retrieval, you can choose whether to expand the context of the retrieval results according to your actual needs.

[0063] For example, a search result fragment might contain a clause about the acceptance inspection of an automatic fire alarm system, which may be preceded and followed by clauses on the scope of application, system functional requirements, equipment installation requirements, or concealed works acceptance requirements. The system supplements these adjacent contents into the returned results, enabling the subsequent large language model to not only see a single isolated clause but also understand the complete context of the chapter in which the clause is located.

[0064] Specifically, for each search result, obtain its document ID and subtopic. Within this range, query several adjacent segments, the number of which is determined by the request parameters. Then, concatenate the context with the target text, i.e., the search result, according to the article order.

[0065] S5. Aggregate the retrieval results of each query request in the batch query requests to complete the batch knowledge base query.

[0066] The search results for each query request in all query groups are aggregated according to their original structure, generating both a full version and a deduplicated simplified version. Finally, the results are returned to the caller as JSON, completing the batch knowledge base query. The full version can include: file title, score, keywords, context, etc.

[0067] For each query request in the aggregation, the large language model can subsequently generate review conclusions, problem descriptions, and modification suggestions based on the search results, which will not be elaborated upon in this article.

[0068] Through the above implementation methods, the present invention can perform batch retrieval of multiple review topics and multiple query questions in the bidding evaluation scenario of the construction industry, and obtain accurate, traceable and contextual legal basis, thereby improving the efficiency, accuracy and interpretability of bidding evaluation.

[0069] Example 2 like Figure 2 As shown in the figure, this invention also proposes a dual-channel hybrid retrieval knowledge base query system for bidding documents, including: Knowledge base construction module 1 is used to obtain the scoring basis text and preprocess it to obtain text blocks. The text blocks containing the main text content are stored in the main text channel, and the text blocks containing the metadata extended description content are stored in the metadata channel. Dense vector indexes and sparse retrieval indexes are built for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel. The query request generation module 2 is used to obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms; The query vector generation module 3 reads the search terms from the batch query requests, reads the corresponding query vectors from the preloaded vector cache, and obtains a set of query vectors. The query vector generation module reads the search terms from the batch query requests, preloads the vector cache, and determines whether there are query vectors corresponding to the search terms in the vector cache. If there are, it calls the query vectors in the vector cache; otherwise, it vectorizes the search terms to generate query vectors, and writes the query vectors into the vector cache with the search terms as keys and the query vectors as values. The retrieval module 4 is used to concurrently execute each query request in the batch query requests. If vector retrieval is not required, the query request is searched for keywords in the dual-channel knowledge base using a sparse retrieval index, and the retrieval results are output. If vector retrieval is required, the query request is searched for in the dual-channel knowledge base using a dense vector index based on the query vector set obtained by the query vector generation module, and / or the query request is searched for keywords in the dual-channel knowledge base using a sparse retrieval index, and the retrieval results are output. Result generation module 5 is used to aggregate the retrieval results of each query request in the batch query requests to complete the batch knowledge base query.

[0070] Example 3 This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the dual-channel hybrid retrieval knowledge base query method for bidding documents as described in Embodiment 1.

[0071] Example 4 like Figure 3 As shown, this invention conducts a search experiment using the "Traditional Chinese Medicine in a certain region" housing construction project.

[0072] This search experiment focuses on the "Traditional Chinese Medicine Inheritance and Innovation Development Center Project in a Certain Region". The project's bidding documents and review criteria were read, and search terms were automatically generated for each sub-element. Relevant standards were also retrieved from the engineering construction standards knowledge base. The following explanation uses the "acceptance and handover" sub-element as an example.

[0073] This project's search scope includes the "Basic Database," "Building Construction," "Municipal and Highway Engineering," and "Water Conservancy" standards databases. The search employs a hybrid approach combining semantic vector retrieval and BM25 keyword retrieval, with results reordered and supplemented with context. Returned results include standard name, search terms, relevance score, relevant knowledge base, keywords, and excerpts from the main text.

[0074] In the actual review workflow, users only need to select the project, expert model, and model version, without needing to input search terms one by one. The project bidding information, along with the title, superior title, grade description, existing regulation name, and core judgment points of each review sub-element, are input as search terms to generate the model. The model then generates 3 to 5 phrase-style search terms and corresponding search intents for each sub-element. The generation rules require that search terms revolve around professional objects, management actions, engineering stages, and regulatory directions, and do not generate scoring, deduction, or downgrading content, nor fabricate specific clause numbers.

[0075] The “Search Terms” field in the image shows the automatically generated and actually submitted query, and the “Knowledge Base” field shows the standard database to which the result belongs; in this example, the “GBT50430-2017 Engineering Construction Enterprise Quality Management Standard” that was matched comes from the “Basic Database”.

[0076] Taking the "Acceptance and Handover" sub-element as an example, a sample query group request is as follows: { "sub_id": 1, "title": "Acceptance and Handover", "queries": [ "Construction Project Acceptance and Handover Specifications" "Management, Acceptance and Handover of Construction Project Data" "Specifications for the Dismantling of Temporary Facilities in Building Construction" "Archiving of Project Acceptance and Handover Procedure Documents" ], "topk": [ 2, 2, 2, 2 ] } This example ultimately retains 8 search results.

[0077] Taking the "Acceptance and Handover" sub-element as an example, after completing the batch retrieval, the response sample for a single query group is as follows. For ease of display, only one representative result is retained in each of the retrieved_source and retrieved_cleaned fields, and the content field is excerpted: { "sub_id": 1, "title": "Acceptance and Handover", "queries": [ "Construction Project Acceptance and Handover Specifications" "Management, Acceptance and Handover of Construction Project Data" "Specifications for the Dismantling of Temporary Facilities in Building Construction" "Archiving of Project Acceptance and Handover Procedure Documents" ], "topk": [ 2, 2, 2, 2 ], "retrieved_source": [ { "query": "Construction Project Acceptance and Handover Specifications", "topk": 2, "standard_name": "GBT50430-2017 Quality Management Standard for Construction Enterprises", "content": "Before the project handover, quality inspection and data organization should be completed, and the handover should be carried out in accordance with regulations... (excerpt)" "id": 50430, "page_idx": 0, "score": 0.91, "type": "Basic Library", "keywords": [ "Acceptance and handover" "Engineering Data" "Quality Management" ] } ], "retrieved_cleaned": [ { "standard_name": "GBT50430-2017 Quality Management Standard for Construction Enterprises", "content": "Before the project is handed over, quality inspections and data organization should be completed, and the handover should be carried out in accordance with regulations... (excerpt)" } ] } In the above example, the system actually performed searches for each of the four search terms using top-k=2, obtaining a total of eight complete search results. The retrieved_source field retains traceability information such as standard name, hit text, result number, page number, relevance score, knowledge base type, and keywords according to the query; the retrieved_cleaned field removes duplicates and cleans the results under the same sub-element, forming a set of legal bases that can be directly used for subsequent review. Taking the query "Construction Project Acceptance and Handover Specification" as an example, the sample returns relevant excerpts from GB / T 50430-2017 "Quality Management Specification for Construction Enterprises".

[0078] The large language model can then compare the acceptance and handover requirements reflected in this segment with the corresponding chapters of the tender documents to generate review conclusions, problem descriptions, and modification suggestions.

[0079] Through the above implementation methods, the present invention can perform batch retrieval of multiple review topics and multiple query questions in the bidding evaluation scenario of the construction industry, and obtain accurate, traceable and contextual legal basis, thereby improving the efficiency, accuracy and interpretability of bidding evaluation.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memory, special components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0081] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0082] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A dual-channel hybrid retrieval knowledge base query method for bidding documents, characterized in that, include: S1. Obtain the scoring basis text and preprocess it to obtain text blocks. Save the text blocks containing the main text content in the main text channel and save the text blocks containing the metadata extended description content in the metadata channel. Establish dense vector indexes and sparse search indexes for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel. S2. Obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms; S3. Determine whether the batch query request requires vector retrieval. If not, perform keyword retrieval on the batch query request in the dual-channel knowledge base using the sparse retrieval index, output the retrieval results, and proceed to S5. If yes, read the search terms of the batch query request, read the corresponding query vectors from the pre-loaded vector cache, and obtain the query vector set. S4. Execute each query request in the batch query request concurrently; Based on the query vector set, the query vector is used to perform vector retrieval of the query request in the dual-channel knowledge base through dense vector index; and / or, the query request is used to perform keyword retrieval in the dual-channel knowledge base through sparse retrieval index; Output the search results; Semantic reordering of search results; Specifically, the following steps are included: Step a: For each batch of query requests, the search results obtained in the text channel and the metadata channel are used as the candidate result set for the text channel and the candidate result set for the metadata channel, respectively. Then, the search results in the candidate result sets of the text channel and the metadata channel are scored and ranked to obtain the channel recall results. The recall method is determined according to the search mode: When the search mode is keyword search, the search results in the candidate result set are sorted in descending order according to the BM25 score of the keyword and the search results in the candidate result set. The top-ranked search results are selected to obtain the channel recall results under the keyword search mode. When the retrieval mode is vector retrieval, the retrieval results in the candidate result set are sorted in descending order based on the cosine similarity between the retrieval results in the candidate result set and the query vector. A predetermined number of the top-ranked retrieval results are then selected to obtain the channel recall results under the vector retrieval mode. The cosine similarity is calculated as follows: Where q represents the query vector and d represents the search result; It is a 2-norm; The cosine similarity between the query vector q and the search result d; When the search mode is hybrid search, the RRF method is used to fuse the rankings of search results in keyword search mode and vector search mode to obtain the RRF value of the search results. The search results in the candidate result set are then sorted in descending order based on the RRF value, and a predetermined number of the top-ranked search results are selected to obtain the channel recall results under the hybrid search mode. The RRF value is calculated as follows: in, Indicates search results The ranking of the i-th search pattern; m is the number of search patterns; k is the smoothing parameter; For search results RRF value; Step b: Input the recall results of each channel into the re-ranking model, re-rank them according to the semantic matching degree between the search terms and the search results, and assign scores to the search results to obtain the score of each search result in each channel: in, The normalized score for the search result d; The semantic matching score output by the reordering model for the retrieval result d; The highest semantic matching score among all search results; The lowest semantic match score among all search results; Step c: retrieve the search results The fusion score is used as the final score; among which, the search results Fusion score The calculation method is as follows: in, This represents the text channel score of search result d. Weight of the main text channel; This represents the metadata channel score of search result d. Metadata channel weight; Step d: Sort all the search results obtained from the query request in the text channel and the metadata channel in descending order according to the final score, and extract a specified number of search results as the final search results for the query request; S5. Aggregate the retrieval results of each query request in the batch query requests to complete the batch knowledge base query.

2. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 1, characterized in that, The process of obtaining a score based on text and preprocessing it to obtain text blocks specifically includes: S101. Obtain the text of the scoring basis; S102. After parsing and cleaning the scoring data, the data is divided into blocks to obtain text blocks.

3. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 1, characterized in that, S2 specifically includes: S201. Obtain bidding document information and structured review sub-elements, and input query terms to generate search terms using the model; S202. After cleaning and deduplicating the search terms, divide the search terms into at least one batch query group according to sub-elements; determine the search configuration for each search term, including the number of search results returned, the scope of the knowledge base, the search mode, the reordering switch, and the search configuration of the context parameters; assemble the search terms and search configuration of the batch query group into a batch query request in JSON format.

4. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 1, characterized in that, Before S3, it also includes: Extract all search terms from the batch query request and remove duplicate search terms.

5. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 1, characterized in that, S3 further includes: Preload vector cache: Iterate through the batch query requests, read the search terms for each query request, and deduplicate the search terms to form a set of search terms that need to be vectorized; Check the local vector cache to determine if there is a query vector corresponding to the search term. If there is, call the query vector in the vector cache; otherwise, add the search term to the list to be vectorized and generate a query vector for quantization. Write the query vector into the vector cache with the search term as the key and the query vector as the value, thus completing the vector cache preloading.

6. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 5, characterized in that, The specific steps of vectorizing the search terms include: The search terms are added to the vectorization list, which is then grouped according to a preset batch size and sent to the bge-m3 semantic vector model to generate query vectors.

7. The dual-channel hybrid retrieval knowledge base query method for bidding documents as described in claim 1, characterized in that, Following S4, it also includes: Expand the context of the search results: determine the document and chapter position to which the search result belongs, query the adjacent segments before and after it, and combine the context with the search result in the order of the article.

8. A dual-channel hybrid retrieval knowledge base query system for bidding documents, characterized in that, The system, applicable to the dual-channel hybrid retrieval knowledge base query method for bidding documents as described in any one of claims 1-7, comprises: The knowledge base construction module is used to obtain the scoring basis text and preprocess it to obtain text blocks. The text blocks containing the main text content are stored in the main text channel, and the text blocks containing the metadata extended description content are stored in the metadata channel. Dense vector indexes and sparse search indexes are built for both the main text channel and the metadata channel to form a dual-channel knowledge base containing the main text channel and the metadata channel. The query request generation module is used to obtain bidding document information and structured review sub-elements, input query terms to generate search terms, and generate batch query requests based on the search terms; The query vector generation module reads the search terms from the batch query requests, preloads the vector cache, and checks whether there is a query vector corresponding to the search terms in the vector cache. If there is, it calls the query vector in the vector cache; otherwise, it vectorizes the search terms to generate query vectors and writes the query vectors into the vector cache with the search terms as keys and the query vectors as values. The retrieval module is used to concurrently execute each query request in a batch query request. If vector retrieval is not required, the module performs keyword retrieval on the query request in the dual-channel knowledge base using a sparse retrieval index and outputs the retrieval results. If vector retrieval is required, the module performs vector retrieval on the query request in the dual-channel knowledge base using a dense vector index based on the query vector set obtained by the query vector generation module, and / or performs keyword retrieval on the query request in the dual-channel knowledge base using a sparse retrieval index, and outputs the retrieval results. The results generation module is used to aggregate the retrieval results of each query request in a batch query request to complete the batch knowledge base query.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the dual-channel hybrid retrieval knowledge base query method for bidding documents as described in any one of claims 1 to 7.

Images