Retrieval system for bidding law document large language model

Abstract

The invention relates to the field of text processing, in particular to a retrieval system for a bidding law document large language model. Comprising a document preprocessing module, a semantic dicing module, an entity and relation recognition module, a knowledge graph construction module, a multi-strategy mixed retrieval module and a retrieval result preferential module. During working, the bidding law document is preprocessed, based on document word number judgment, a chapter-level slicing strategy or a clause-level slicing strategy is adopted for cutting, a LateChunking algorithm is used for determining cutting points, semantic units are extracted, entity elements are recognized, a knowledge graph is constructed, and an optimal retrieval result is obtained through multi-strategy mixed retrieval and preferential processing. According to the method, the limitation of a traditional single retrieval mode in bidding legal chief document processing is overcome, and the provision retrieval accuracy and context coherence are improved, so that the illusion risk caused by information missing or misunderstanding of a large language model is greatly reduced, and the credibility and practicability of a legal intelligent question and answer result are enhanced.

Description

Technical Field

[0001] This application relates to the field of text processing, and more particularly to a retrieval system for a large language model of bidding and tendering legal documents. Background Technology

[0002] Large language models commonly exhibit the illusion phenomenon in text generation and question answering tasks, meaning that the model may output seemingly reasonable content that is actually inconsistent with the facts or input information. This problem is particularly prominent in the highly specialized legal field. Due to the high rigor, standardization, and logical dependencies of legal texts, if the model fails to fully understand the semantic relationships and hierarchical validity of the provisions, it is highly susceptible to generating erroneous or misleading responses. Tender and bidding legal documents, as a specific branch of legal texts, not only possess the general characteristics of legal language but also exhibit significant domain-specific features. These documents typically have a rigorous hierarchical structure, including chapter divisions, clause numbering, nested sub-items, and numerous clause references. Furthermore, their content covers multiple business dimensions such as tendering procedures, bidding requirements, and legal responsibilities, resulting in long texts, dense terminology, and complex logical connections.

[0003] Traditional RAG (Retrieval Augmentation) systems often employ a single retrieval channel (such as vector recall), making it difficult to cover different types of knowledge. In complex scenarios, dynamic retrieval routing has become a research hotspot. Recent research has proposed introducing intelligent routers into the RAG framework, dynamically switching retrieval tools or data sources based on the query; this is known as the Agentic RAG Router. For example, the SAGE framework uses intelligent routing units to adaptively switch between search engine retrieval and internal knowledge base retrieval, achieving dynamic adjustments to early, mid, and late-stage routing. This multi-routing strategy can select different retrieval strategies based on query intent, significantly improving the coverage and accuracy of recall. On the other hand, how to evaluate the reliability of retrieval results is a weak point in existing systems. Currently, it usually relies on vector matching scores or cross-coding re-ranking scores, but lacks intuitive confidence output. In 2024, scholars proposed using Bayesian inference to verify the quality of RAG retrieval fragments: estimating the relevance likelihood of a paragraph to the question using LLM, and combining prior probabilities (such as the paragraph's position in the document) to calculate posterior confidence to decide whether to include it in the context. This method demonstrates the value of the Bayesian probabilistic framework in improving the quality of retrieval content. Currently, no adjustable Bayesian dynamic routing scheme has been found in publicly available information—that is, dynamically assigning weights or paths to different retrieval methods (such as keyword retrieval, vector retrieval, knowledge graph query, etc.) and outputting statistically significant confidence intervals. This not only limits the retrieval effect, but also further exacerbates the risk of large language models generating illusions due to missing or misunderstanding information. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a retrieval system for a large language model of bidding and tendering legal documents, comprising: The document preprocessing module is used to clean and extract structural identifiers from bidding and tendering legal documents; The semantic segmentation module is used to segment the preprocessed document based on the number of words in the document and adopt chapter-level segmentation or clause-level segmentation strategy. During segmentation, the LateChunking algorithm is used to determine the segmentation point and output the semantic units obtained from the segmentation. The entity and relationship identification module is used to identify entity elements from the semantic unit, and to label the entity elements with validity level tags and business domain tags, and output entity element and tag data, wherein the entity elements include the regulatory name; The knowledge graph construction module is used to construct a knowledge graph in the bidding and tendering field based on the name of the regulation as the root node and the reference, causal and supplementary relationships between entities. The multi-strategy hybrid retrieval module is used to receive user queries, extract legal entity keywords from the queries, and simultaneously perform semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword matching retrieval, and output multi-path recall results. The search result selection module is used to perform semantic similarity calculation and business tag weighted sorting on the multi-path recall results, and select the optimal candidate answer set for generating results.

[0005] Specifically, when the document preprocessing module cleans and extracts structural identifiers from the bidding legal documents, it removes irrelevant characters from the headers and footers, and marks the starting position and hierarchical relationship of the content in the title, chapter number, clause number, sub-clause, item and / or preset important clause chapter library.

[0006] Specifically, when the semantic segmentation module segments the preprocessed document using a chapter-level segmentation or clause-level segmentation strategy based on the document's word count, if the document's word count is less than 500 characters, it dynamically determines the first segmentation point based on the title and the chapter number. When the document has a word count of not less than 500 characters, a second cutting point is dynamically determined based on the clause number, and when the word count of the clause exceeds 300 characters, it is further subdivided and cut according to the sub-clauses or the sub-items.

[0007] Specifically, when the semantic segmentation module uses the LateChunking algorithm to determine the segmentation point, it combines sentence boundary punctuation marks and semantic coherence analysis, and limits the segmentation point to a period, semicolon or clause number by recognizing the clause number sequence, title level and patterns in the preset important clause chapter library.

[0008] Specifically, the entity elements extracted by the entity and relationship identification module also include the issuing organization, the date of issuance and effectiveness, and the scope of application; When the entity and relation recognition module identifies entity elements from the semantic unit, it uses an embedding model to map the text into a vector representation, and then inputs the vector into a large language model for entity and relation extraction. The large language model is fine-tuned through a reinforcement learning strategy. During fine-tuning, the reward function is constructed based on entity recognition accuracy, relation extraction accuracy, or the degree of matching with existing entities in the knowledge graph.

[0009] Specifically, the validity level labels include national-level bidding and tendering laws, departmental rules, and local regulations, and the business area labels include comprehensive, engineering, finance, and administrative management categories; When the entity and relationship identification module labels the entity elements with validity level tags and business domain tags, it automatically classifies them based on the issuing organization and content theme of the bidding legal documents.

[0010] Specifically, when the knowledge graph construction module constructs a knowledge graph in the bidding and tendering field, it uses the name of the regulation as the root node and constructs graph edges by identifying the citation relationship, causal relationship and supplementary explanation relationship between clauses.

[0011] Specifically, the multi-strategy hybrid retrieval module also includes a routing decision unit and a result reliability evaluation module. The routing decision unit is used to receive the vector representation of the user query and dynamically select at least one retrieval path based on a pre-trained Bayesian decision network. The retrieval paths include: vector similarity retrieval, keyword retrieval, and knowledge graph query. The Bayesian decision network analyzes the relationship between query semantic features and historical performance data of each retrieval path to output a path selection probability distribution. The result reliability assessment module is used to integrate the path selection probability distribution output by the Bayesian decision network and the quality score of the retrieval results returned by the path, calculate the overall confidence of the final answer through Bayesian inference, and output a quantified confidence interval value or confidence level to characterize the reliability of the final answer.

[0012] Specifically, when the search result selection module selects the optimal candidate answer set, it calculates the posterior probability of the search results that meet the confidence interval requirements or confidence level requirements. When calculating the posterior probability, the prior probability is determined based on the validity level and publication time of the clause. The validity level includes national bidding laws, departmental regulations, and local regulations. The prior probability weight of national laws is higher than that of local regulations. The likelihood function is constructed based on the semantic similarity score, and the top N results with the highest posterior probability are selected as the candidate answer set, where N is a preset constant.

[0013] Specifically, the retrieval system also includes an illusion discrimination module, which is used to evaluate the credibility of the generated results based on a pre-trained factual discrimination model. The discrimination model outputs a confidence score by comparing the semantic consistency and factual conformity of the generated text with the real terms in the knowledge base. Results below the threshold are filtered or marked as pending review.

[0014] This application has the following technical advantages: By employing a multi-strategy hybrid retrieval method and structured semantic processing, the limitations of traditional single retrieval methods in processing long legal documents related to bidding and tendering are effectively overcome. This significantly improves the accuracy and contextual coherence of text retrieval, thereby greatly reducing the risk of illusions caused by missing or misunderstandings in large language models and enhancing the credibility and practicality of legal intelligent question answering results. Attached Figure Description

[0015] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts.

[0016] Figure 1 This is a schematic diagram of the architecture of a retrieval system for a large language model of bidding legal documents in an embodiment of this application. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0018] This invention aims to provide a systematic solution to key problems in the intelligent retrieval and processing of legal documents related to bidding and tendering. Existing retrieval systems based on large language models often suffer from the "illusion" phenomenon, where the content generated by the model deviates from the actual legal provisions. This is mainly due to the insufficient consideration of the unique characteristics of bidding and tendering legal texts. Such documents are characterized by high structure, frequent citations between clauses, rigorous terminology, and clear hierarchical validity. Traditional retrieval methods often rely on single strategies, such as vector similarity or keyword matching, which struggle to capture deep semantic relationships and logical structures. More importantly, long documents are often mechanically segmented, disrupting the continuity of the text's context and preventing the model from fully understanding the legislative intent and relationships between clauses. This leads to misinterpretations or missing information, exacerbating the risk of the "illusion." This deficiency is particularly prominent in the bidding and tendering field, where documents often involve numerous cross-references, supplementary relationships, and differences in validity levels, making a single retrieval strategy clearly insufficient to handle complex query needs.

[0019] Based on the aforementioned shortcomings, the core motivation of this invention lies in constructing a system that can deeply integrate the structural characteristics of bidding and tendering legal documents, support multi-dimensional retrieval and result optimization, and improve retrieval accuracy and suppress illusions through collaborative multi-module processing. The following embodiment illustrates a retrieval system for a large language model of bidding and tendering legal documents proposed in this application, such as... Figure 1 As shown, it includes: The document preprocessing module is used to clean and extract structural identifiers from bidding and tendering legal documents; The semantic segmentation module is used to segment the preprocessed document based on the number of words in the document and adopt chapter-level segmentation or clause-level segmentation strategy. During segmentation, the LateChunking algorithm is used to determine the segmentation point and output the semantic units obtained from the segmentation. The entity and relationship identification module is used to identify entity elements from semantic units, and to label entity elements with validity level tags and business domain tags, and output entity element and tag data. Entity elements include the name of the regulation. The knowledge graph construction module is used to build a knowledge graph for the bidding and tendering field based on the law name as the root node and the reference, causal and supplementary relationships between entities. The multi-strategy hybrid retrieval module is used to receive user queries, extract legal entity keywords from the queries, and simultaneously perform semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword matching retrieval, and output multi-path recall results. The search result selection module is used to calculate semantic similarity and sort business tags for multiple recall results, and select the optimal candidate answer set to generate results.

[0020] When the retrieval system described in this application is in operation, it first cleans and extracts structural identifiers from the original bidding legal documents through a document preprocessing module. Original documents often contain irrelevant characters such as headers and footers, as well as complex formatting elements, which need to be removed to purify the text. Simultaneously, the module identifies and marks the starting position and hierarchical relationship of content in a pre-defined important clause and chapter library, including titles, chapter numbers, clause numbers, sub-clauses, sections, and other relevant clauses. This process not only improves text quality but also provides crucial metadata for subsequent processing through structural extraction.

[0021] Subsequently, the semantic segmentation module dynamically selects a segmentation strategy based on the word count of the preprocessed document. When the document has a small word count, such as less than 500 characters, a chapter-level segmentation strategy is used, dynamically determining the cutting points based on the extracted titles and chapter numbers to ensure content integrity. For longer documents, such as those exceeding 500 characters, the strategy switches to clause-level segmentation, dividing the document according to clause numbers. When a single clause exceeds 300 characters, it is further subdivided based on sub-clauses or items to avoid information overload. During the segmentation process, the system uses the LateChunking algorithm to determine the cutting points. This algorithm combines sentence boundary punctuation marks such as periods and semicolons with semantic coherence analysis, while also recognizing patterns in clause number sequences, title levels, and a pre-set important clause chapter library to ensure that the cutting points are always located at semantic boundaries, thereby maintaining the contextual coherence of the clauses.

[0022] Next, the entity and relation recognition module extracts key entity elements from the semantic units obtained from the segmentation, including the law name, issuing agency, issuance and effective dates, and scope of application. Unlike traditional static methods based on BERT fine-tuning, this system employs a joint optimization framework integrating embedding representations, Large Language Models (LLM), and reinforcement learning for entity recognition and relation extraction. Specifically, firstly, a domain-adaptive embedding model is used to map textual semantic units into high-dimensional vector representations; then, these are input into the LLM for deep semantic parsing and structured extraction, identifying entities and their relationships. To further improve extraction quality and domain adaptability, a reinforcement learning mechanism is introduced to fine-tune the LLM. By designing reward functions, such as those based on entity recognition accuracy, relation extraction accuracy, and consistency with existing knowledge graphs, the model's optimization strategy is guided, effectively reducing noise annotation interference and improving the understanding of the complex semantics and logical structure of bidding and tendering legal texts. After identification, the module automatically labels entity elements with validity level tags, such as national laws, departmental rules, and local regulations, and business domain tags, such as comprehensive, engineering, finance, and administrative management. The tag classification is automatically determined based on the document publishing institution, content theme, and contextual semantics, providing structured support for subsequent knowledge graph construction and multi-strategy retrieval.

[0023] The knowledge graph construction module uses the name of the regulation as the root node and dynamically constructs a knowledge graph for the bidding and tendering domain based on the entity references, causal relationships, and supplementary information extracted from semantic units. By parsing explicit references between clauses (such as "see Article X") and implicit logical connections (such as causal relationships and supplementary explanations), edge relationships are constructed in the graph, forming a structured knowledge network with semantic connectivity. For example, the reference relationship between the "Bidding and Tendering Law" and the "Bidding and Tendering Implementation Regulations" will be extracted and represented as an edge in the graph, thus intuitively showing the hierarchical structure and connection paths between legal provisions. Building upon this, the system further introduces a Bayesian dynamic routing mechanism to achieve adaptive fusion and weight allocation of multi-source retrieval strategies (including vector retrieval, knowledge graph association retrieval, and keyword matching). This mechanism dynamically adjusts the contribution weight of each retrieval channel based on historical retrieval accuracy and recall, and calculates the confidence interval of the retrieval results based on a Bayesian inference framework. This achieves reliable fusion and ranking optimization of multi-path recall results, improving the system's coverage and answer credibility under complex queries, and providing reliable structured knowledge support for subsequent multi-strategy retrieval.

[0024] The multi-strategy hybrid retrieval module receives user queries and extracts legal entity keywords from them. The module includes a routing decision unit and a result reliability assessment module. The routing decision unit dynamically selects retrieval paths based on a pre-trained Bayesian decision network. This network analyzes the relationship between query semantic features and historical performance data of each retrieval path, outputting a path selection probability distribution to determine at least one retrieval strategy, such as semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, or keyword-based matching retrieval. The module synchronously executes the selected retrieval paths: semantic retrieval calculates similarity by mapping queries and document blocks to a vector space; association retrieval expands the retrieval scope by exploring relationships between entities using a knowledge graph; and keyword retrieval ensures exact matching. The result reliability assessment module integrates the path selection probability distribution and the quality score of the retrieval results, calculates the overall confidence of the final answer through Bayesian inference, and outputs a confidence interval or confidence level to characterize reliability.

[0025] The search result optimization module integrates and ranks the multiple recall results. The module calculates the semantic similarity score for each result and performs a weighted ranking based on business tags and effectiveness levels. Specifically, a Bayesian inference framework is used to calculate the posterior probability, where the prior probability is determined based on the clause's effectiveness level and publication time, and the likelihood function is constructed based on the semantic similarity score. This ranking strategy ensures that the results are not only relevant but also efficient and timely. Finally, the module selects the top N results with the highest posterior probabilities as the candidate answer set to generate the final output.

[0026] In addition, the system includes a hallucination detection module to assess the credibility of the generated results. This module, based on a pre-trained factual discrimination model, compares the semantic consistency and factual accuracy of the generated text with real terms in the knowledge base, outputting a confidence score. Results below a certain threshold are filtered or marked for review, further reducing the risk of hallucinations.

[0027] Obviously, the embodiments described above are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be understood that when the terms "first," "second," etc., are used in the claims, description, and drawings of this application, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the description and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

Claims

1. A retrieval system for a large language model of legal documents for bidding and tendering, characterized in that, include: The document preprocessing module is used to clean and extract structural identifiers from bidding and tendering legal documents; The semantic segmentation module is used to segment the preprocessed document based on the number of words in the document, using either chapter-level or clause-level segmentation strategies. During segmentation, the LateChunking algorithm is used to determine the segmentation points, and the resulting semantic units are output. The entity and relationship identification module is used to identify entity elements from the semantic unit, and to label the entity elements with validity level tags and business domain tags, and output entity element and tag data, wherein the entity elements include the regulatory name; The knowledge graph construction module is used to construct a knowledge graph in the bidding and tendering field based on the name of the regulation as the root node and the reference, causal and supplementary relationships between entities. The multi-strategy hybrid retrieval module is used to receive user queries, extract legal entity keywords from the queries, and simultaneously perform semantic retrieval based on vector similarity, association retrieval based on knowledge graph relationships, and keyword matching retrieval, and output multi-path recall results. The search result selection module is used to perform semantic similarity calculation and business tag weighted sorting on the multi-path recall results, and select the optimal candidate answer set for generating results.

2. The system as described in claim 1, characterized in that, When the document preprocessing module cleans and extracts structural identifiers from bidding legal documents, it removes irrelevant characters from headers and footers and marks the starting position and hierarchical relationship of titles, chapter numbers, clause numbers, sub-clauses, items and / or content in the preset important clause chapter library.

3. The system as described in claim 2, characterized in that, When the semantic segmentation module segments the preprocessed document using a chapter-level or clause-level segmentation strategy based on the document's word count, if the document's word count is less than 500 characters, it dynamically determines the first segmentation point based on the title and the chapter number. When the document has a word count of not less than 500 characters, a second cutting point is dynamically determined based on the clause number, and when the word count of the clause exceeds 300 characters, it is further subdivided and cut according to the sub-clauses or the sub-items.

4. The system as described in claim 1, characterized in that, When the semantic segmentation module uses the LateChunking algorithm to determine the segmentation point, it combines sentence boundary punctuation marks and semantic coherence analysis. By identifying the clause number sequence, title level, and patterns in the preset important clause chapter library, it limits the segmentation point to a period, semicolon, or clause number.

5. The system as described in claim 1, characterized in that, The entity elements extracted by the entity and relationship identification module also include the issuing organization, the date of issuance and effectiveness, and the scope of application; When the entity and relation recognition module identifies entity elements from the semantic unit, it uses an embedding model to map the text into a vector representation, and then inputs the vector into a large language model for entity and relation extraction. The large language model is fine-tuned through a reinforcement learning strategy. During fine-tuning, the reward function is constructed based on entity recognition accuracy, relation extraction accuracy, or the degree of matching with existing entities in the knowledge graph.

6. The system as described in claim 1, characterized in that, The validity level labels include national-level bidding and tendering laws, departmental rules, and local regulations; the business area labels include comprehensive, engineering, finance, and administrative management categories. When the entity and relationship identification module labels the entity elements with validity level tags and business domain tags, it automatically classifies them based on the issuing organization and content theme of the bidding legal documents.

7. The system as described in claim 1, characterized in that, When constructing a knowledge graph in the bidding and tendering field, the knowledge graph construction module uses the name of the regulation as the root node and constructs graph edges by identifying the citation relationship, causal relationship and supplementary explanation relationship between clauses.

8. The system as described in claim 1, characterized in that, The multi-strategy hybrid retrieval module also includes a routing decision unit and a result reliability evaluation module. The routing decision unit is used to receive the vector representation of the user query and dynamically select at least one retrieval path based on a pre-trained Bayesian decision network. The retrieval paths include: vector similarity retrieval, keyword retrieval, and knowledge graph query. The Bayesian decision network analyzes the relationship between query semantic features and historical performance data of each retrieval path to output a path selection probability distribution. The result reliability assessment module is used to integrate the path selection probability distribution output by the Bayesian decision network and the quality score of the retrieval results returned by the path, calculate the overall confidence of the final answer through Bayesian inference, and output a quantified confidence interval value or confidence level to characterize the reliability of the final answer.

9. The system as described in claim 8, characterized in that, When the search result selection module selects the optimal candidate answer set, it calculates the posterior probability of the search results that meet the confidence interval requirements or confidence level requirements. When calculating the posterior probability, the prior probability is determined based on the validity level and publication time of the clause. The validity level includes national bidding laws, departmental regulations, and local regulations. The prior probability weight of national laws is higher than that of local regulations. The likelihood function is constructed based on the semantic similarity score, and the top N results with the highest posterior probability are selected as the candidate answer set, where N is a preset constant.

10. The system as claimed in claim 1, characterized in that, The retrieval system also includes an illusion discrimination module, which is used to evaluate the credibility of the generated results based on a pre-trained factual discrimination model. The discrimination model outputs a confidence score by comparing the semantic consistency and factual conformity of the generated text with the real terms in the knowledge base. Results below the threshold are filtered or marked as pending review.

Images